Building a platform from open source at Yahoo!

Building a platform from open source at Yahoo!

Dustin Whittle – Yahoo! Developer Network

Building a platform from open source at Yahoo! | Dus$n Whi*le 

Building a platform from open source at Yahoo! | Dus$n Whi*le 

Overview •  •  •  •  •  •  •  • 

Why symfony? symfony vs ysymfony What does scaling really mean? Social Search: Delicious and Answers Yahoo! Open Strategy –  – 

What is the Yahoo! Open Stack? Application Platform + Developer Tool

– 

YUI, Design Patterns, Tutorials

–  – 

YQL, Geo, Profiles, Connections, Updates, … YOS SDK for PHP

– 

OAuth / YQL / OpenSocial

Developer Tools

Data (YQL) & Social APIs

Building an open application with symfony and YOS

Building a platform from open source at Yahoo! | Dus$n Whi*le 

Who am I? •  Working with symfony since open source •  symfony Core Team Member •  Responsible for the development and support of symfony at Yahoo! –  Worked with Y! Answers, Delicious, Y! Widgets, Y! Bookmarks, Yahoo! Application Platform

•  Consultant –  Commercial symfony support + training (USA)

Building a platform from open source at Yahoo! | Dus$n Whi*le 

EXAMPLES | TUTORIALS | CODE SAMPLES

DEVELOPER.YAHOO.COM

Building a platform from open source at Yahoo! | Dus$n Whi*le 

FREEBSD | LINUX | APACHE | PHP | MYSQL | BUGZILLA | HADOOP | SYMFONY

YAHOO! IS POWERED BY OPEN SOURCE TECHNOLOGIES Building a platform from open source at Yahoo! | Dus$n Whi*le 

W3C | MICROFORMATS | OAUTH | OPENID | OPENSOCIAL

YAHOO! EMBRACES OPEN STANDARDS

Building a platform from open source at Yahoo! | Dus$n Whi*le 

RASMUS LERDORF | DOUG CROCKFORD | DOUG CUTTING | CHRISTIAN HEILMANN

YAHOO! HIRES OPEN SOURCE DEVELOPERS

Building a platform from open source at Yahoo! | Dus$n Whi*le 

YUI | BROWSER PLUS | DESIGN PATTERNS | R3 | YSLOW + PERFORMANCE RULES

YAHOO! GIVES BACK TO OPEN SOURCE

Building a platform from open source at Yahoo! | Dus$n Whi*le 

YQL | PIPES | BOSS | CONTACTS | UPDATES | MAIL | DELICIOUS | FLICKR | UPCOMING | HOTJOBS | MAPS | FIREEAGLE | GEOLOCATION | LOCAL | TRAFFIC | WEATHER | MUSIC | ANSWERS | SHOPPING | FINANCE | TRAVEL

YAHOO! SHARES ITS DATA THROUGH OPEN APIS AND WEB SERVICES Building a platform from open source at Yahoo! | Dus$n Whi*le 

Conferences | Hack Days | HackU | Tech Talks | YDN Theater

YAHOO! ENGAGES COMMUNITIES WITH OPEN HACK EVENTS AROUND THE WORLD Building a platform from open source at Yahoo! | Dus$n Whi*le 

Building a platform from open source at Yahoo! | Dus$n Whi*le 

Users  Load Balancers  Frontend  Apache 

Custom Modules 

PHP 

APC, PEAR, PECL, Custom Extensions 

ysymfony / YUI 

Backend  MySQL/Oracle 

Web Services 

Ad API 

FreeBSD 4.x/6.x, Linux 2.6.x  Building a platform from open source at Yahoo! | Dus$n Whi*le 

User API 

Y! needs from a frontend platfrom •  Fit existing environment (RHEL/PHP5/Apache) •  Development Cycle – How easy to develop, test, and deploy? •  Clean separation between data, logic, and display (MVC) •  Independent model layer to fit service oriented architecture •  Extensible and pluggable •  Internationalization and localization support •  Detailed documentation and active community of support •  Open source and ability to contribute back Building a platform from open source at Yahoo! | Dus$n Whi*le 

Why a frontend presentation platform? •  Rasmus says “frameworks are not well suited for Y!” –  Build applications to requirements •  Do exactly what you need: no more, no less •  Understand that frameworks add a lot of overhead •  Choosing functional components is a better fit

•  Despite choosing open source or building your own –  Everyone uses a framework –  If you use open source, use only the pieces you need

Building a platform from open source at Yahoo! | Dus$n Whi*le 

Why a framework at all? •  Another software layer (ysymfony, yphp, yapache) •  Factors out common patterns –  Code Layout –  Configuration –  URL Routing –  Authentication / Security –  Form Validation / Repopulation –  Internationalization / Localization

•  Encourages good design •  Abstraction > Consistency > Maintainability Building a platform from open source at Yahoo! | Dus$n Whi*le 

The choice to adopt symfony? •  Philosophy –  Full-stack framework for building complex web applications –  Adopt best ideas from anywhere, using existing code if available (Mojavi, Prado, Rails, Django)

•  Design –  Clean separation between Model, View, and Controller –  Controller using modules and actions –  Views using templates in straight PHP with helpers –  Easy to reuse view modules to compose a page •  Layouts, Components, Partials, Slots Building a platform from open source at Yahoo! | Dus$n Whi*le 

The choice to use symfony •  •  •  •  •  • 

Configurability / Flexibility Features we do not want are easily disabled Use of factories for easy customization Documentation / Support Community The Definitive Guide to symfony (free online) Excellent tutorials and example applications –  Askeet & Jobeet

•  Active community with wiki, mailing lists, forums, irc channel Building a platform from open source at Yahoo! | Dus$n Whi*le 

Why ysymfony for Yahoo! teams? •  Eliminate common patterns by adding a layer on PHP –  Code layout/structure (MVC) –  Configuration –  Internationalization

•  ysymfony is just a toolkit –  Learn one set of tools •  Shift between multiple projects •  Consistency

–  Long term maintainability through platform

Building a platform from open source at Yahoo! | Dus$n Whi*le 

Building a platform from open source at Yahoo! | Dus$n Whi*le 

How Yahoo! uses symfony

•  Full Stack –  Yahoo! Bookmarks –  Yahoo! Answers –  Delicious –  Yahoo! Widgets –  Yahoo! Application Platform Developer Tool –  Corporate Reporting Tools

•  Components –  Forms –  Caching –  Routing Building a platform from open source at Yahoo! | Dus$n Whi*le 

symfony MVC framework

Building a platform from open source at Yahoo! | Dus$n Whi*le 

Building a platform from open source at Yahoo! | Dus$n Whi*le 

Building a platform from open source at Yahoo! | Dus$n Whi*le 

A look at Yahoo! Answers •  http://answers.yahoo.com •  Yahoo! Answers is the largest collection of human knowledge on the Web with more than 135 million users and 515 million answers worldwide (Yahoo! Internal Data, March 2008). •  Yahoo! Answers is the 2nd ranked education & reference site on the web (comScore) •  Available in 26 markets and 12 languages

Building a platform from open source at Yahoo! | Dus$n Whi*le 

Yahoo! Answers at the beginning •  Started as a small development team on PHP4 from a fork of Yahoo! Taiwan Knowledge+ •  Launched December 2005 by December 2006 there were 60 million users and 65 million answers •  The code base eventually became difficult to maintain and iterate new features •  Large distributed development teams (US / UK)

Building a platform from open source at Yahoo! | Dus$n Whi*le 

Building a platform from open source at Yahoo! | Dus$n Whi*le 

Building a platform from open source at Yahoo! | Dus$n Whi*le 

Building a platform from open source at Yahoo! | Dus$n Whi*le 

Building a platform from open source at Yahoo! | Dus$n Whi*le 

Building a platform from open source at Yahoo! | Dus$n Whi*le 

Building a platform from open source at Yahoo! | Dus$n Whi*le 

Building a platform from open source at Yahoo! | Dus$n Whi*le 

The big picture •  A complete platform for building web applications from frameworks –  PHP Framework –  JavaScript Framework –  CSS Framework –  UI Design Patterns + Best Practices –  Development Tools (logger, profiler, debugger, docs) –  Unit + Functional Testing Frameworks (LIME / YUI Test) –  Deployment Tools (rsync deployment system)

Building a platform from open source at Yahoo! | Dus$n Whi*le 

What does Yahoo! change? •  Minor changes to fit our environment (bsd/php/apache) –  Most of our changes are easily implemented via factories

•  Added dimensions to configurations (ysfDimensionsPlugin) •  Integrate R3 translation/template management (ysfR3Plugin) –  R3 - http://developer.yahoo.com/r3/

•  Dropped the ORM and pushed down the stack (SOA) –  Added a parallel API Dispatcher (ysfAPIClientPlugin)

•  Created a build and deployment solution (ysfBuildPlugin) •  Integrate support for Y! User Interface libraries (ysfYUIPlugin) Building a platform from open source at Yahoo! | Dus$n Whi*le 

Propel or Doctrine or ??? •  No ORM for large projects •  Propel or Doctrine for large projects –  Doctrine for internal projects (best supported)

•  Service Oriented Architecture –  Platforms as services (reusable to all) –  No heavy lifting, push down the stack –  Thin Controller/Fat Model (where model == services)

•  Java/C++/Erlang + JSON/XML

Building a platform from open source at Yahoo! | Dus$n Whi*le 

Localizing the experience with dimensions

•  Cascading Configuration based on YAML –  Framework -> Project -> Application -> Module

•  Extending the cascade to be based on dimensions –  Dimensions can be anything (and can be chained together) •  •  •  • 

Data Center + Environment for customizing configurations Culture for localizing user interface + data Theme for customizing look and feel User info (is user on corporate intranet?)

•  Caching

Building a platform from open source at Yahoo! | Dus$n Whi*le 

symfony I18n + L10n •  Translation + Template Management –  Integrated with symfony i18n + view layers •  hello vs __(‘hello’) •  Dictionary Management (YALA)

–  SQLite/XLIFF Backend

•  Added tasks –  Importing/Exporting XLIFF dictionary –  Generating translations for projects –  Find missing strings

Building a platform from open source at Yahoo! | Dus$n Whi*le 

A build and deployment system •  •  •  •  •  •  •  • 

Aggregate and minify stylesheets and javascripts Rewrite templates, css, js for CDN (Akamai, S3, …) Generate translations for configurations + templates Generate configuration cache Aggregate core classes + remove debug statements Run lint, unit, functional tests Package applications as .tgz Deployment via rsync

Building a platform from open source at Yahoo! | Dus$n Whi*le 

What does it mean to scale? •  A system whose performance improves after adding hardware, proportionally to the capacity added, is said to be a scalable system. •  High Availability + Scalability + Performance •  Bigger dataset, more traffic, maintainable •  Not about performance –  PHP is slow, but it is not your bottleneck

•  Languages do not scale, architectures do. •  Planning to grow and planning to fail –  Capacity Planning –  Business Continuity Planning Building a platform from open source at Yahoo! | Dus$n Whi*le 

Scaling – Planning •  Planning hardware purchases and hosting options to have as much as you need without breaking your wallet •  Partitioning and distributing databases to support large datasets and simultaneous transactions •  Monitoring your applications to find and clear bottlenecks •  Providing services APIs and using services from other providers to increase your site's reach and capabilities •  Think Minimal, Plan to grow, Plan to fail. Building a platform from open source at Yahoo! | Dus$n Whi*le 

Scaling – The basics in PHP •  PHP is rarely the bottleneck •  “Most performance comes not from the language, but from application design” - Rasmus •  Share Nothing Architecture –  Independent, self-sufficient, no single point of contention –  No local storage = No PHP Sessions •  Use a database (works for distributed) •  Use a small signed cookie (ideal)

–  Important data in database –  Individual expiration on session objects –  Small data items

–  Use a distributed cache •  Memcache

•  Forget about small efficiencies -- Premature optimization is the root of all evil. Building a platform from open source at Yahoo! | Dus$n Whi*le 

Scaling Databases – The basics •  Master/Slave Replication –  First steps –  Helps with reads, writes are still bottleneck

•  Partitioning –  Segmenting data

•  Sharding (horizontal partitioning) –  Segmenting data onto different physical machines –  Make problems smaller, easier to grow

Building a platform from open source at Yahoo! | Dus$n Whi*le 

Improving latency with Caching •  Always use PHP opcode cache (APC, Xcache, etc) –  Use for routing and i18n cache

•  Memcache (distributed cache) –  Use for view cache •  Distributed invalidation can be a pain •  sfViewCacheManager makes this easy! •  Be intelligent about cache_keys (uri, user, state)

•  There is a fine line to caching –  At what point do you spend more time managing the cache, than reading from it? Building a platform from open source at Yahoo! | Dus$n Whi*le 

Tweaking Performance •  Don’t use features you do not need –  settings.yml / factories.yml

•  •  •  •  •  •  •  • 

Use core_compile (aggregate classes) Remove debug statements (sfOptimizerPlugin) Do not use .htaccess (move to real apache config) Set a minimal include path Increase realpath_cache_size + realpath_cache_ttl Use apc.stat=0 Use @routeName Do not use components in loop Building a platform from open source at Yahoo! | Dus$n Whi*le 

Do it yourself for cheap •  Open source software = Free –  Apache –  PHP –  MySQL –  Memcache / Perlbal / MogileFS / Squid / Gearman –  symfony / Doctrine / Propel / Swift –  Nagios

•  Amazon Shared Infrastructure = Cheap –  EC2 Cloud Computing –  S3 Distributed Storage –  SimpleDB Building a platform from open source at Yahoo! | Dus$n Whi*le 

Yahoo! Open Strategy

Building a platform from open source at Yahoo! | Dus$n Whi*le 

OPENID | XRDS | OAUTH | PORTABLE CONTACTS | OPEN SOCIAL

OPEN PLATFORMS + COLLABORATION

Building a platform from open source at Yahoo! | Dus$n Whi*le 

The Open Web

Building a platform from open source at Yahoo! | Dus$n Whi*le 

Y! OS – The Open Stack

Building a platform from open source at Yahoo! | Dus$n Whi*le 

What does Yahoo! offer developers? •  Yahoo! Developer Network  •  Developer Tools (YUI, etc)  •  Social APIs  •  Profiles  •  ConnecWons  •  Updates  •  Data APIs  •  OAuth  •  Yahoo! Query Language  •  Yahoo! ApplicaWon PlaYorm  •  OpenSocial  Building a platform from open source at Yahoo! | Dus$n Whi*le 

Building a platform from open source at Yahoo! | Dus$n Whi*le 

developer.yahoo.com

/

0

1

2

3

4

5

6

7

8

9

:

;





!

!

"

"

3D3@GB677a ZSb RSdSZ]^S`a bO^ W\b] GOV]]¸a e]`ZR

2SaWU\PgAbS^VO\2]c`Wa:cQOa>SbbW\ObW

QZOaaRObOO\RW\T`Oab`cQbc`Sb]VSZ^ ^]eS`bVSW`PcaW\SaaSaO\RW\bSU`ObS RObO a]c`QSa W\ \Se eOga [OYW\U $

bVS eSP O []`S caSTcZ O\R W\bS`Sab

$

W\U^ZOQST]`SdS`g]\S

/

0

/>7aA3@D713A8C:G /22@3AA0==9



1



 

 

4"

>`]dWRScaS`US\S`ObSRORdWQS]\O\gb]^WQ]\g]c`]e\ eSPaWbS

/AB@/

7



: ;





0cWZR^S`a]\OZWhSRSf^S`WS\QSaeWbVOQQSaab]caS` a^SQWTWQRObOaSS00OcbVO\R=/cbV

00/CB6



; !



/ZZ]eGOV]]caS`ab]UWdSg]c^S`[WaaW]\b]OQQSaa bVSW`^S`a]\OZRObOPgcaW\UbVSGOV]]0`]eaS`0OaSR /cbVS\bWQObW]\^`]QSaaaSS/cbV

0=AA



7"

0cWZRg]c`=e\ASO`QVAS`dWQS0=AAWaGOV]]¸a]^S\ aSO`QVeSPaS`dWQSa^ZObT]`[bVObOZZ]eag]cb]PcWZR\Sfb US\aSO`QV[OaVc^a0=AAUWdSag]cTW\SU`OW\SRQ]\b`]Z ]dS`bVSRWa^ZOgO\R`O\YW\U]TcaS`aSO`QV`SacZba

1=7a QZWS\baWRScaW\UbVS8OdOAQ`W^b/>7O\RaS`dS`aWRS caW\U@3ABTcZRObO/>7aT]`PcWZRW\Ua]QWOZO^^ZWQObW]\a

:$

A3/@16;=