This slideshow could not be started. Try refreshing the page or viewing it in another browser.
Deploys at WordPress.com VIP
About me: Mohammad (Mo) Jangda
mo@automattic.com | batmoo@gmail.com | @mjangda
- Code Wrangler at Automattic
- Ice Cream Fan
What is WordPress.com?
Hosted platform for running blogs, sites, etc. using WordPress.
Big multisite network.
What is VIP?
WordPress hosting and services for big companies (CNN, TIME, ESPN, CBS, TechCrunch, Williams, etc.)
A VIP site on WordPress.com is the same as a free site, except with some custom code a bit of magic sauce.
The VIP team works with external developers providing code review, developer support, etc.
Fancy Numbers
54 million sites
62 million users
30 million posts (per month)
13 billion pageviews / 400 uniques (per month)
500 million MySQL tables
2500 servers
3+ DCs
How Many Deploys?
Yesterday: 259
VIP: 165
WP.com: 75
Other: 19
How is code deployed on WordPress.com?
Production servers run trunk
– Commit changes to trunk
– Run deploy script
– Deploy script syncs svn mirrors across DCSs and runs `svn up` on all servers
$ deploy wpcom
Going to update from 93786 to 93787 for /public_html/
Syncing wpcom SVN Mirrors
DFW (1s)...
IAD (2s)...
SAT (2s)...
Deploying wpcom revision 93787
Deploying to static webs
SAT (1s)...
IAD (1s)...
DFW (1s)...
Deploying to dynamic webs
DFW (5s)...
IAD (7s)...
SAT (9s)...
What about VIPs?
Similar to WordPress.com except:
– Code comes from external developers
– Deploy done internally
– We only push the particular folder (code changes are limited to the “theme”)
VIP Numbers
5 Software Engineers; 1 Happiness Engineer; other biz people
7m lines of code (on top of WP.com codebase)
1000s of sites
~100 active developers
Over 140K commits (currently r140321)
70K deploys all time
Avg deploy time (commit => review => deploy): 130 minutes
Challenges
– External developers writing PHP, HTML, and JS so anything is fair game
– Issues on one site can spill over to another (also helps with scaling)
Code Review is Essential
Make sure they’re not doing not-so-good things
Guidelines: http://vip.wordpress.com/documentation/code-review-what-we-look-for/
Challenge: Code Review is hard and time-consuming! Blocks business needs!
Deploy Page
– custom built tool to optimize review process
– Used to track pending commits, review changes, and deploy
– Can send feedback on issues, alert rest of team when issues spotted
– Handy revert commands!
– “real-time”
Pre-Deploy Tests
Run when deploy button is pressed.
Loads up the site against a sandbox server in production with live database (crazy!) and latest code.
Static Analysis
Using custom scanning tools + PHP Code Sniffer to catch things like restricted functions, bad coding patterns/practices, etc.
Coming soon: post-commit tests
Using Jenkins to run tests immediately after commit, instead of on-demand using Sandbox server in production.
Other Tools
– Reference Page with handy revert commands
– IRC Channel piping PHP and MySQL errors from 2 servers
– IRC alerts for internal commit and deploy notifications
– Email and webhooks for external commit and deploy notifications
Future: Scaling
– More developers and automation
– Externally initiated deploys and post-deploy reviews
Hiring, Hiring, Hiring!
- VIP Wranglers, VIP Wranglers, VIP Wranglers!
- Code Wranglers, Code Wranglers, Code Wranglers!
- Designers, Designers, Designers!
- Happiness Engineers, Happiness Engineers, Happiness Engineers!
