Field Notes Inside an Integrated Communications Agency

Every popular website eventually runs into the same problem: how to scale. In general, you hope for as much traffic as possible when you are developing a new application, but what would happen if you actually got it? Could you keep up with the increase in users, data, bandwidth, servers, and everything else that makes your application go? This panel aims to expose some of the key issues that you will face as your traffic grows, with insight from people with proven track records working on complex, high volume systems. Topics such as load balancing, caching technologies, and database use will be discussed. The end goal is that we would like the audience to leave with a better understanding of the issues that they will face when their applications start to get material amounts of traffic, with the hope that they will be able to better plan for growth.

Kevin Rose - Founder - Diggnation /Digg Inc
Cal Henderson - Badass MC - Flickr
Joe Stump - Lead Architect - Digg.com Inc
Chris Lea - Media Temple
Garett Camp - StumbleUpon
Matt Mullenweg - Founding Dev - Automattic /WordPress

Should I buy or rent space? Media Temple (and other hosting companies) have more hardware available than any of their customers. When you get customers that develop something bad (on accident) and you get a lot of traffic, no amount of hardware is going to protect that site or the others on the box from crashing.

If you need something specialized to your application, and it will affect others, the vendor is not going to do it. Scalability requires specification and customability. Architectures are different because each company is different. This allows for customized problems, digg and flickr have totally different problems.

Some of the panelists think buying boxes is not scalable and would rather keep renting space.

YOu may be having database scalabilty issues and going from 10 MySQL instances to Oracle. The easiest ways to solve problems is to through money at it. The more MySQL databases you ahve the more likely you are going to have one down. You have to make sure you partition the data as you build.

My SQL Proxy project - imbed lua language. You talk to the proxy and the proxy knows what to do.

Scalability of teams must be considered. You generally have to add more structure to the team when you have several developers. There is always one dissenting opnion when you have 7 developers. The perfect number of developers working on a single project seems to be between 3-6. You can almost always get consensus with that number.

Documenting code is essential. It takes a lot of time for everyone to explain and get up to speed when you don't.

Recommend coding in a wiki so that each developer can easily work on the code. You must complete cross-training, you have to have two people learn something at the same time.

You can't do check-ins on a ticket unless you actually own it (Panelist recommends using Trac - which we use at Capstrat).

They mentioned that most developers are anti-agile or anti-scrum processes (Thoughts on this?? Our developers love team development with everyone working on one project for a period of time.)

Every company is different, at WordPress, the employees are 100% remote. There is talk of having some location redundancy and requiring those int he same area get together once a week to discuss issues and company ideas.

When you are trying to be scalable, the bottleneck will never be your language. It is always your database or your file storage or access. (As Cal Henderson from Flickr said:Flickrs first issue was they ran out of disk space (hmm), but there are ways you can figure this out with a tiny bit of foresight.)