Google Storage : What it really is…

Yesterday Google formally announced Google Storage to a few (5000?) of us at Google I/O. Here is the gist of this as I see it from the various discussions/talks I attended.

To begin with, I have to point out that there is almost nothing new in what Google has proposed to provide. Amazon has been doing this for years with its S3.  The key difference is that if you are a google customer you won’t have to look elsewhere for storage services like this one.

Lets get the technical details out

• Its tries to implement a Strong consistency model (CA of the CAP: Consistent and Available). Which means data you store is automatically replicated in a consistent way across multiple datacenter
  • Currently it replicates to multiple locations within US. In future it does plan to replicate across continents.
  • Currently there are no controls to control how replication happens or to where. They plan to learn from usage in beta period and develop controls over time.
• There are two basic building blocks for objects
  • Buckets – Containers
    All objects are stored in flat container. However, the tools understand “/” and “*” (wild cards) and does the right thing when used correctly
    
  • Objects – objects/files inside those containers
• Implements RESTful APIs (GET/PUT/POST/DELETE/HEAD/etc)
  • All resources are identified by a URI
• No theoretical size limit of Buckets or containers. However a 100GB limit per account would be imposed during beta phase.
• Its of course built on Google very well tested, scalable, highly available infrastructure
• It provides multiple, flexible authentication and sharing models
  • Does support standard public/private key based auth
  • Will also have integration with some kind of groups which will allow object to be shared with  or controlled by with multiple identities.
  • ACLs can be applied to both Buckets and Objects
    • Buckets
      • Control who can list objects
      • Who can create/delete objects
      • Who can read/write into the bucket
    • Objects
      • Who can read
      • Who can read/write
• Tools
  • There were two tools mentioned during the talk
    • GS manager looks like a web application which allows an admin to manage this service
    • GS util is more like the shell tools AWS provides for S3.
      • As I mentioned before GS util accepts wild card
        • So something like this is possible
          • gsutil cp gs://gs2010/*  /home/rkt/gs2010
• The service was created with “data liberation” as one of the goals. As shown by the previous command it takes just one line of code to transfer all of your data out.
• Resume feature (if connection breaks during a big upload) is not available yet, but thats on the roadmap.
• Groups feature was discussed a lot, but its not ready in the current release
• Versioning feature is not available. Wasn’t clear if its on the roadmap or how long before its implemented.

Few other notes.

• Its not clear how this plays with the “storage service” Google currently provides for Gmail/Docs storage. From what I heard this is not related to that storage service at all and there are no plans to integrate it.
• The service is free in beta period to all developers who get access to it, but when its released it will follow a pricing model similar others in the industry. The pricing model is already published on their website
• The speakers and the product managers didn’t comment on whether storage access from google apps engine would be charged (or at what rate)
• They do provide MD5 signatures as a way of verifying if an object on the client is same as the object on the server, but its not used storing files itself. (So MD5 collisions issue shouldn’t be a problem)
• US Navy is already using this service with about 80TB of data on Google Storage, and from what I heard they looked pretty happy talking about it.

I suspect this product will be in beta for a while before they release it out in the open.