Versioning data in S3 on AWS

One of the problem with Amazon’s S3 was the inability to take a “snapshot” of the state of S3 at anyAmazon Web Services given moment. This is one of the most important DR (disaster recovery) steps of any major upgrade which could potentially corrupt data during a release. Until now the applications using S3 would have had to manage versioning of data, but it seems Amazon has launched a versioning feature built into S3 itself to do this particular task. In addition to that, they have made it a requirement that delete operations on versioned data can only be done using MFA (Multi factor authentication).

Versioning allows you to preserve, retrieve, and restore every version of every object in an Amazon S3 bucket. Once you enable Versioning for a bucket, Amazon S3 preserves existing objects any time you perform a PUT, POST, COPY, or DELETE operation on them. By default, GET requests will retrieve the most recently written version. Older versions of an overwritten or deleted object can be retrieved by specifying a version in the request.

The way AWS Blog describes the feature, it looks like a version would be created every time an object is modified and each object in S3 could have different number of copies depending on the number of times it was modified.

This kind of reminds me of SVN/CVS like versioning control system and I wonder how long it will take for someone to build a source code versioning system on S3.

BTW, data requests to a versioned object is priced the same way as regular data, which basically means you are getting this feature for free.

References

Comments

Adrian said…
Actually you're not getting it for free as you have to pay for all the versions being stored on S3. Here's what the AWS blog is saying:
"Normal S3 pricing applies to each version of an object. You can store any number of versions of the same object, so you may want to implement some expiration and deletion logic if you plan to make use of this feature."


So depending on how many versions you need to store and supposing all your object versions will have about the same size, it's strictly proportional:
2 versions - pay twice the original amount for storage
...
N versions - pay N times the amount for storage
12:53 AM
Royans said…
@Adrian: Thats true but if I understand that correctly its only for that particular object, not for your entire S3 store. Since they also talk about being able to configure how to set expiry, I'm hoping users could tune that to their satisfaction. Personally, I think for some providers the cost of storage is insignificant compared to risk of data corruption.

But, I agree its not entirely free :)
1:08 AM
Ronak said…
I am one of developer team member. Anyone can try graphical user interface (GUI) like Bucket Explorer to get working with all features of Amazon S3 and CloudFront.

Prevent Amazon S3 Data Loss by Version and Trash Feature of Bucket Explorer
Download Bucket Explorer
3:54 PM
Post a Comment

Popular posts from this blog

Chrome Frame - How to add command line parameters

Image
Chrome frame intentionally does stuff without getting in the way of the user. This sometimes makes things harder to debug. For example how can one debug an issue if chrome frame doesn't even launch ? Apparently there is a flag for that. But you have to know how to enable it. Here are the steps. Make sure chrome frame is installed. We can enable startup flags for dumping debug logs using a policy called AdditionalLaunchParameters If this is just for one desktop, I recommend doing a registry edit (it can be pushed via GPO as well) Add a REG_SZ property " AdditionalLaunchParameters " to " SoftwarePoliciesGoogleChromeAdditionalLaunchParameters " with the value "--enable-logging --v=1" (also documented here  and mentioned here )  [ Attachment 1 ] Next kill the IE browser and make sure chrome is also dead by checking taskmgr Restart IE and go to " gcf:about:version " and confirm that the parameters you added show up next to "Command Line:"
Read more

Creating your first chrome app on a Chromebook

Image
Doing development on Chromebook usually means developing online. There are a lot of sites around for that. But with the APIs getting more mature, its just a matter of time before someone builds a kick-ass IDE which runs natively on Chromebooks without network connectivity. One such tool which I've been exploring for a last few weeks is from Google and called " Chrome Dev Editor " If you have never built and run a chrome app or extension, I'll show you how to do this in 3 easy steps.     Step 1 Launch the app and click on the "+" button to add a new project. Pick a project name and select "JavaScript Chrome App" in the drop down menu.       Step 2 Notice how it automatically adds the minimum code required for it to run. This will be the template you are going to work with. You are almost done, but lets review the files in there to understand why they are there.   manifest.json - This is the most important file. Modify the app name, version number
Read more

Brewers CAP Theorem on distributed systems

Image
Large distributed systems run into a problem which smaller systems don’t usually have to worry about. “ Brewers CAP Theorem ” [ Ref 1 ] [ Ref 2 ] [ Ref 3 ]  defines this problem in a very simple way. It states, that though its desirable to have Consistency, High-Availability and Partition-tolerance in every system, unfortunately no system can achieve all three at the same time . C onsistent : A fully Consistent system is one where the system can guarantee that once you store a state (lets say “x=y”) in the system, it will report the same state in every subsequent operation until the state is explicitly changed by something outside the system. [ Example 1 ] A single MySQL database instance is automatically fully consistent since there is only one node keeping the state. [ Example 2 ] If two MySQL servers are involved, and if the system is designed in such a way that all keys starting “a” to “m” is kept on server 1, and keys “n” to “z” are kept on server
Read more