Sharding: Different from Partitioning and Federation ?

Ive been hearing this word "sharding" more and more often, and its spreading like fire. Theo Schlossnagle, the author of "Scalable internet architecutres" argues that federation is form of partitioning, and that sharding is nothing but a form of partitioning and federation. Infact, according to him, Sharding has already been in use use for a long time.

I'm not a dba, and I don't pretend to be one in my free time either, so to understand the differences I did some research and found some interesting posts.

The first time I heard about "Sharding" was on Been Admininig's blog about Unorthodox approach to database design (Part I and Part II). Here is the exact reference...
Splitting up the user data so that User A exists on one serverwhile User B exists on another server, each server now holds a shard ofthe data in this federated model.

A couple of months ago Highscalability.com picked it up and made it sound (probably unintentionally) that sharding is actually different from Federation and Partitioning. Todd's post also points at Flickr using sharding.The search for Flickr architecture lead me to Colin Charles' post about Federation at Flickr: A tour of the Flickr architecture where he does mention shards as a component of Federation key. Again no mention of Sharding being anything new.
Federation Key Components:

  • Shards: My data gets stored on my shard, but the record ofperforming action on your comment, is on your shard. When making acomment on someone elses’ blog

  • Global Ring: Its like DNS, you need to know where to go and whocontrols where you go. Every page view, calculate where your data is,at that moment of time.

  • PHP logic to connect to the shards and keep the data consistent (10 lines of code with comments!)



Based on the discussions on these and other blogs, "Shards" sounds more like a terminology used to describe fragments of data which is federated across multiple databases instead of an architecture by itself. I think Theo Schlossnagle has a valid argument. If any of you disagree I'm interested to hear what you have to say. A clearer definition between sharding and federation would be very helpful as well.

Here are more references to Shard/Sharding.



      Comments

      Post a Comment

      Popular posts from this blog

      Chrome Frame - How to add command line parameters

      Image
      Chrome frame intentionally does stuff without getting in the way of the user. This sometimes makes things harder to debug. For example how can one debug an issue if chrome frame doesn't even launch ? Apparently there is a flag for that. But you have to know how to enable it. Here are the steps. Make sure chrome frame is installed. We can enable startup flags for dumping debug logs using a policy called AdditionalLaunchParameters If this is just for one desktop, I recommend doing a registry edit (it can be pushed via GPO as well) Add a REG_SZ property " AdditionalLaunchParameters " to " SoftwarePoliciesGoogleChromeAdditionalLaunchParameters " with the value "--enable-logging --v=1" (also documented here  and mentioned here )  [ Attachment 1 ] Next kill the IE browser and make sure chrome is also dead by checking taskmgr Restart IE and go to " gcf:about:version " and confirm that the parameters you added show up next to "Command Line:"
      Read more

      Creating your first chrome app on a Chromebook

      Image
      Doing development on Chromebook usually means developing online. There are a lot of sites around for that. But with the APIs getting more mature, its just a matter of time before someone builds a kick-ass IDE which runs natively on Chromebooks without network connectivity. One such tool which I've been exploring for a last few weeks is from Google and called " Chrome Dev Editor " If you have never built and run a chrome app or extension, I'll show you how to do this in 3 easy steps.     Step 1 Launch the app and click on the "+" button to add a new project. Pick a project name and select "JavaScript Chrome App" in the drop down menu.       Step 2 Notice how it automatically adds the minimum code required for it to run. This will be the template you are going to work with. You are almost done, but lets review the files in there to understand why they are there.   manifest.json - This is the most important file. Modify the app name, version number
      Read more

      Brewers CAP Theorem on distributed systems

      Image
      Large distributed systems run into a problem which smaller systems don’t usually have to worry about. “ Brewers CAP Theorem ” [ Ref 1 ] [ Ref 2 ] [ Ref 3 ]  defines this problem in a very simple way. It states, that though its desirable to have Consistency, High-Availability and Partition-tolerance in every system, unfortunately no system can achieve all three at the same time . C onsistent : A fully Consistent system is one where the system can guarantee that once you store a state (lets say “x=y”) in the system, it will report the same state in every subsequent operation until the state is explicitly changed by something outside the system. [ Example 1 ] A single MySQL database instance is automatically fully consistent since there is only one node keeping the state. [ Example 2 ] If two MySQL servers are involved, and if the system is designed in such a way that all keys starting “a” to “m” is kept on server 1, and keys “n” to “z” are kept on server
      Read more