How Facebook Sees How Thousands of Devices React to Code Changes

Why is there a wall featuring thousands of mobile devices, older and newer, in Facebook's data center in Prineville, Ore.? It's not there for decoration.

Facebook production engineer Antoine Reversat described how Facebook examines the impact of code changes on thousands of devices in a blog post.

Reversat wrote:

Initially, engineers tested code by running CT-Scan on a single device that they had at their desks. This didn't scale-we needed to be able to run tests on more than 2,000 mobile devices to account for all the combinations of device hardware, operating systems and network connections that people use to connect on Facebook. Today, in our Prineville data center, we have a mobile device lab-outfitted with a custom-built rack-that allows us to run tests on thousands of phones. The process of building a lab in our data center wasn't a direct path, and we learned a lot along the way as we worked to scale out the promise of CT-Scan.

However, before the mobile device lab came the sled, the gondola and the “slatwall.” Reversat explained:

Our first setup was “the sled.” On the sled, phones were placed on a metal rack, which slid into a metal case. When you slide a bunch of metal sleds into a metal rack, you end up with too much metal, and we lost all Wi-Fi.

Our second version was named “the gondola” and allowed us to deploy 100 devices for testing. The gondola was a plastic rack-which didn't interfere with the Wi-Fi-but the short length of the USB cables caused a lot of issues. The gondola was a tangled mess.

Next up, we built “the slatwall.” The slatwall took up an entire room, and we were able to deploy 240 devices. To accommodate 2,000 phones (our target number based on several factors, including the number of commits per week, the number of iterations needed per test to get statistically significant results, etc.), we'd need to scale to nine of these rooms in our Menlo Park, Calif., headquarters within a year. This wouldn't work, so we decided to move our mobile device lab into a data center.

For much more on the challenges faced by the team working on the mobile device lab in Prineville, as well as future plans, please see Reversat's blog post.