Showing posts from March 14, 2010

Scalable Tools: Murder: a bittorent based, file transfer tool for faster deployments

Deploying to one server can be done with a single script doing a bunch of ssh/scp commands. If you have a few more servers, you can run it in a loop sequentially or fork the processes to do them in parallel. At some point though, it will get unmanageable especially if you have the challenge of updating multiple datacenters at the same time. So how does a company like twitter do their releases ? Murder is an interesting P2P/Bittorent based tool which twitter uses to distribute files during its own software updates. Here are some more details. Murder is a method of using Bittorrent to distribute files to a large amount of servers within a production environment. This allows for scalable and fast deploys in environments of hundreds to tens of thousands of servers where centralized distribution systems wouldn't otherwise function. A "Murder" is normally used to refer to a flock of crows, which in this case applies to a bunch of servers doing something. In order to do a