September 15, 2007
Zoppr is a Custom Search engine which allows you to create custom Google search engine on the fly, by appending your bookmark page, wikipage, or any other kind of page with lots of interesting bookmarks/links on it. Once setup, google will search only across your bookmarks/links. For example this URL will help you search across an OPML file published somewhere on the internet http://www.zoppr.com/cse/http://share.opml.org/
Posted by Royans at 2:05 PM
If you haven't noticed already there is a second blog which I maintain which is currently more busy than this particular blog. "Scalable web architectures" is a collection of posts about how web architectures which scale and technologies which make it happen.
Here are some of the posts on that blog
Here are some of the posts on that blog
Powerset could have gone the way most dot-com companies have gone, but instead they decided to try out AmazonÃ¢â‚¬â„¢s EC2 (Elastic Cloud Computing) and S3(Simple Storage Service) to augment their computational needs.
Youtube is said to be pushing about 25 petabytes per month which is about 77 Gbps sustained data rate on an average. The bandwidth usage at the peaks would be even higher. Thanks to Limelight networks, Youtube doesnÃ¢â‚¬â„¢t really need to scale or provision for that kind of bandwidth and based on the some reports from 2006 it had cost them close to 4 million a month back then. Youtube and services like that have to invest a lot in their infrastructure before they can really launch their service and though using shared Content delivery networks is not ideal, its probably not a bad deal. In YoutubeÃ¢â‚¬â„¢s case, it helped them survive until Google bought it out.
Theo Schlossnagle, the author of Ã¢â‚¬Å“Scalable internet architecutresÃ¢â‚¬Â argues that federation is form of partitioning, and that sharding is nothing but a form of partitioning and federation. Infact, according to him, Sharding has already been in use use for a long time.
Eins.de site serves about 1.2 million dynamic pages a day. He wrote a series of articles describing how they redesigned the site to scale for growth. I found these articles very informative with a extreemly mature discussion of the colorful world of scalability.
If I could only give one recommendation to anyone building a brand new web application, IÃ¢â‚¬â„¢d say Ã¢â‚¬Å“go statelessÃ¢â‚¬Å“. But going stateless is not the same as going session-less. One could implement a perfectly stateless web architecture which still uses sessions to authenticate, authorize and track user activity. And to complicate matters further, when I say stateless, I really mean that the server should be stateless, not the client.
Loadbalancers, by definition, are supposed to solve performance bottlenecks by distributing or balancing load between different components its managing. Though you would normally find loadbalancers in front of a webserver, a lot of different individuals have found other interesting ways of using it.
Posted by Royans at 7:12 AM
There have been a lot of interesting stories from last week for me to share. If you have interesting links you want to add to this post please forward them to me or post a comment to this post.
- Sun is planning to acquire majority stake of "Cluster File Systems, Inc". [ Talk on Lustre File System ]
- Sun intends to add support for Solaris Operating System (Solaris OS) on Lustre and plans to continue enhancing Lustre on Linux and Solaris OS across multi vendor hardware platforms. As previously announced in July 2007, Sun also plans to deliver Lustre servers on top of Sun's industry-leading open source Solaris ZFS solution, which is one of the fastest growing storage virtualization technology in the marketplace.
- Making Facebook Apps scale on cheap : An interesting writeup By Surj Patel about Scalability issues Facebook itself and the 3rd Party apps on it have. Also discusses EC2 and S3 as an alternative solution to scale in a cost effective way.
- Welcome to Amazon and S3 and EC2 Ã¢â‚¬â€ processing power (EC2) and storage (S3) on demand. These services let you access computational power and storage only when you need it and, better yet, pay only for what you use. The last time I checked, it was 10 cents an hour for the server, 10 cents for every gigabyte of data written and 18 cents per gigabyte read out Ã¢â‚¬â€œ all for a virtual box with 1.7Ghz x86 processor/1.75Gbytes of RAM/250Mbs of bandwidth. Nor are you limited to one usage; use as many as you need or want and can afford.
- Interesting blog post on Google Reader Numbers. They have made significant progress lately and thanks to the scalable architecture they now store 10 terabytes of raw feed data from 8 million feeds in their index.
- Todd Hoff has an interesting writeup on Scaling Twitter: Making Twitter 10000 Percent Faster. And an interview with Biz Stone (Co-Founder of Twitter) here.
- If you use Mysql and your app is not yet designed to handle federated database architecture, you should take a look at a new product in development called "Mysql Proxy"
- The most powerful feature is Read/Write Splitting which allows you to scale a application which is unaware of replication automatically cross several slaves without changes to your application. Instance Scale Out we say. The Proxy also became a 1st class citizen in the MySQL world with full docs, win32 support and easy to install.