Tech Blog

Rosetta and Amazon Storage

Rosetta and Amazon Storage. Chris Erickson. February 2015.
In the search for more file storage, as well as more affordable file storage, we tried Amazon Simple Storage Service (Amazon S3). The plan was to connect the Rosetta Digital Preservation System to the Amazon cloud storage, and evaluate it as a possible storage solution. There is a free trial. The Free Tier includes 5GB storage, 20,000 Get Requests, and 2,000 Put Request.

I tried various configurations, but decided on a single bucket for the files. I setup buckets for the IEs and metadata, but after trying it, decided to only keep the files on Amazon. That would keep the metadata local. I had tried nested folders, but couldn’t figure out how to designate that in the storage rules and definition. So I create the folders by time period.

In the Rosetta Admin interface I create a File storage group, using the S3 storage plugin, and then entered the Bucket name, Secret Access Key, Access Key ID, and left the Maximum waiting time at the default. For the test, I set up a retention code for Amazon, and the storage rule used that code to determine what went to the Amazon storage. In a real storage instance, it would be better to use something that would not change, like the producer, etc.

It took a few tests to get everything in sync. The result was that Rosetta stored the content in Amazon just fine. I also tried adding content with a one day retention period, and the content was removed from Amazon after the day. A fixity check task was also able to work without a problem.

This gives us another storage option, though we decided to not use it at present.

Pricing at the time of this comparison, was:

1 TB50 TBs
Digital Storage CostsAnnual  Cost20 Year ProjectedYearly  Charge10 Year Projected20 Year Projected50 Year Projected
Cloud Storage
Amazon S3 – Regular$360$7,200$17,706$177,060$354,120$885,300
Amazon S3 – Copy / Glacier$480$9,600$23,706$237,060$474,120$1,185,300
Amazon S3 – Reduced & Glacier$288$5,760$14,165$141,648$283,296$708,240
DuraSpace – Preservation$1,800$36,000$36,100$361,000$722,000$1,805,000
DuraSpace – Dark copy/Glacier$1,925$38,500$42,350$423,500$847,000$2,117,500
DuraSpace – Enterprise Plus$5,625$112,500$64,425$644,250$1,288,500$3,221,250

More storage options will be considered in the future.

Digital Preservation Matters.

Leave a Reply