royans

Kosmix , a search startup has released source to C++ implementation of something which looks like a clustered file system. This looks very similar to Hadoop/HDFS , but the C++ factor will be a big performance boost. From Skrenta blog Incremental scalability - New chunkserver nodes can be added as storage needs increase; the system automatically adapts to the new nodes. Availability - Replication is used to provide availability due to chunk server failures. Re-balancing - Periodically, the meta-server may rebalance the chunks amongst chunkservers. This is done to help with balancing disk space utilization amongst nodes. Data integrity - To handle disk corruptions to data blocks, data blocks are checksummed. Checksum verification is done on each read; whenever there is a checksum mismatch, re-replication is used to recover the corrupted chunk. Client side fail-over - During reads, if the client library determines that the chunkserver it is communicating with is unreacha

When asked what they mean by scalability, a lot of people talk about improving performance, about implementing HA , or even talk about a particular technology or protocol. Unfortunately, scalability is none of that. Don't get me wrong. You still need to know all about speed, performance, HA technology, application platform, network, etc. But that is not the definition of scalability. Scalability, simply, is about doing what you do in a bigger way. Scaling a web application is all about allowing more people to use your application. If you can't figure out how to improve performance while scaling out, its okay. And as long as you can scale to handle larger number of users its ok to have multiple single points of failures as well. There are two key primary ways of scaling web applications which is in practice today. " Vertical Scalability " - Adding resource within the same logical unit to increase capacity. An example of this would be to add CPUs to an existing serve

Smugmug.com , a 5 year old company with just 23 employees has 315000 paying customers and 195 million photographs. CEO & "Chief Geek" Don MacAskill has a nice set of slides where he talks about its 5 year journey during which it went from small startup to a profitable business. The talk was given during Amazon's "Startup project" so it talks mostly about how it uses AWS (Amazon Web services). Other than wonderful fun loving employees who are also "super heroes" in his eyes, he talks about the how they are doubling the storage requirements on an yearly basis. They already have about 300 TB in use, and as of today all of that is on Amazon's S3. Don estimates that based on his estimates, for the storage they are using, they are saving about 500K per year which is pretty big for a small operation like theirs. The Smugmug architecture has evolved over time. Internally it can serve images in 3 different ways. It could either "proxy" th

Zoppr is a Custom Search engine which allows you to create custom Google search engine on the fly, by appending your bookmark page, wikipage, or any other kind of page with lots of interesting bookmarks/links on it. Once setup, google will search only across your bookmarks/links. For example this URL will help you search across an OPML file published somewhere on the internet http://www.zoppr.com/cse/http://share.opml.org/

If you haven't noticed already there is a second blog which I maintain which is currently more busy than this particular blog. " Scalable web architectures " is a collection of posts about how web architectures which scale and technologies which make it happen. Here are some of the posts on that blog Scaling Powerset using Amazon’s EC2 and S3 Powerset could have gone the way most dot-com companies have gone, but instead they decided to try out Amazon’s EC2 (Elastic Cloud Computing) and S3(Simple Storage Service) to augment their computational needs. P2P network scalability Youtube is said to be pushing about 25 petabytes per month which is about 77 Gbps sustained data rate on an average. The bandwidth usage at the peaks would be even higher. Thanks to Limelight networks , Youtube doesn’t really need to scale or provision for that kind of bandwidth and based on the some reports from 2006 it had cost them close to 4 million a month back then. Youtube

There have been a lot of interesting stories from last week for me to share. If you have interesting links you want to add to this post please forward them to me or post a comment to this post. Sun is planning to acquire majority stake of " Cluster File Systems, Inc ". [ Talk on Lustre File System ] Sun intends to add support for Solaris Operating System (Solaris OS) on Lustre and plans to continue enhancing Lustre on Linux and Solaris OS across multi vendor hardware platforms. As previously announced in July 2007, Sun also plans to deliver Lustre servers on top of Sun's industry-leading open source Solaris ZFS solution, which is one of the fastest growing storage virtualization technology in the marketplace. Making Facebook Apps scale on cheap : An interesting writeup By Surj Patel about Scalability issues Facebook itself and the 3rd Party apps on it have. Also discusses EC2 and S3 as an alternative solution to scale in a cost effective way. Welcome to Amazon

The first thing most doc-com companies do before going public is setup an infrastructure to provide the service. And though it might sound straight forward to most of you, it can be a very expensive affair. To come up with the right kind of infrastructure for any new service a few key architectural decisions have to be made. Design the infrastructure and architecture to handle the traffic peaks (not average) Design with long term scaling in mind Design power and cooling infrastructure to support the servers Hire Hardware+Systems+network support staff ( more if 24/7 operations are required) Add buffers to support failures and short term grow requirements Take into account lead times of ordering and procuring hardware (which can be weeks if not months).. And a few others.. which I won't bore you with here. The point is that the initial capital investment can be in millions even before the first customer starts using the service. And once the capital investment is made,

Youtube is said to be pushing about 25 petabytes per month which is about 77 Gbps sustained data rate on an average. The bandwidth usage at the peaks would be even higher. Thanks to Limelight networks , Youtube doesn't really need to scale or provision for that kind of bandwidth and based on the some reports from 2006 it had cost them close to 4 million a month back then. Youtube and services like that have to invest a lot in their infrastructure before they can really launch their service and though using shared Content delivery networks is not ideal, its probably not a bad deal. In Youtube's case, it helped them survive until Google bought it out. Newer Internet television service providers, however need not build their services around the traditional CDN model. Joost Network architecture presentation from Colm MacCarthaigh is an interesting example to discuss to prove my point. Joost was founded by the same guys who founded Kazaa and Skype . Kazaa was one of notorious

Ive been hearing this word "sharding" more and more often, and its spreading like fire. Theo Schlossnagle, the author of " Scalable internet architecutres " argues that federation is form of partitioning, and that sharding is nothing but a form of partitioning and federation. Infact, according to him, Sharding has already been in use use for a long time . I'm not a dba, and I don't pretend to be one in my free time either, so to understand the differences I did some research and found some interesting posts. The first time I heard about "Sharding" was on Been Admininig's blog about Unorthodox approach to database design ( Part I and Part II ). Here is the exact reference... Splitting up the user data so that User A exists on one serverwhile User B exists on another server, each server now holds a s hard ofthe data in this federated model . A couple of months ago Highscalability.com picked it up and made it sound (probably unintentionally)

Patrick Lenz , founder and lead developer of freshmeat.net was also responsible for the relaunch of another website eins.de which recently moved from php to ruby. Eins.de site serves about 1.2 million dynamic pages a day. He wrote a series of articles describing how they redesigned the site to scale for growth. I found these articles very informative with a extreemly mature discussion of the colorful world of scalability. The adventures of scaling, Stage 1 Questions and answers for Stage 1 The adventures of scaling, Stage 2 The adventures of scaling, Stage 3 The adventures of scaling, Stage 4 Here is the final summary of what they ended up after 4 months of optimizations Systems optimization Use Linux 2.6 instead of 2.4 Use self-compiled Ruby 1.8.4 instead of all else Use MySQL-supplied binaries Use lighttpd 1.4.11 instead of all else Use memcache-client instead of Ruby-MemCache Use a smaller amount of dispatchers Watch your dispatchers Code optimizatio

A 55 minute talk by ' Stewart Smith' from MySQL AB, about Mysql Clusters. He talks about NDB storage engine and synchronous replication between storage nodes. Also talks about new features in 5.1 including cluster to cluster replication, disk based data and a bunch of other things. And another Mysql talk on Google about Performance Tuning Best practices for Mysql. An interesting talk with Leah Culver about how Pownce was created . They use LAMP (Python) stack with Perlbal , Memcached , Django , AIR with Amazon S3 as the backend storage. Discussion about  High-Availability Mongrel Packs using Seesaw A blog about " Future of Data Center Computing " mentions Terracotta Sessions. If you read my previous post about "Session, state and scalabililty" and understood the problem, do look at this as a solution as well EC2 and S3 are being used more than before. Unfortunately because storage on EC2 doesn't persist across reboots creative ways of k

In my other life I work with a medium scale web application which has had many different kinds of growing problems over time. One of the most painful one is the issue about "statelessness". If I could only give one recommendation to anyone building a brand new web application, I'd say " go stateless ". But going stateless is not the same as going session-less. One could implement a perfectly stateless web architecture which still uses sessions to authenticate, authorize and track user activity. And to complicate matters further, when I say stateless, I really mean that the server should be stateless, not the client. Basic authentication Most interactive web applications today which allow you to manipulate data in some form require authentication and authorization mechanisms to control access. In the good old days "Basic Authentication " was the most commonly used authentication mechanism. Once authenticated, the browser would send the credentials to