Second Tier Storage
Object storage platform for new access modes and inactive data parking
MSI first added a Ceph object storage system in November 2014 as a second tier storage option. The system currently has around 10 PB of usable storage installed.
MSI's second tier storage is designed to address the growing need for resources that support data-intensive research. It is tightly integrated with other MSI storage and computing resources in order to support a wide variety of research data life cycles and data analysis workflows. In addition, this object storage platform offers new access modes, such as Amazon’s S3 (Simple Storage Service) interface, so that researchers can better manage their data and more seamlessly share data with other researchers whether or not the other researcher has an MSI account or is at the University of Minnesota.
These characteristics make it a good option for exchanging data with outside colleagues, porting cloud-based workflows to MSI systems, or storing inactive data separately from our high performance storage systems. Usage quotas will be set by the allocations committee and reviewed on an annual basis.
Files in Second Tier are not backed up by MSI and cannot be recovered if deleted or lost.
Currently, MSI groups are limited to a 120 TB second tier storage quota. If you would like to know how much Second Tier Storage your group is using, please email help@msi.umn.edu
Access to MSI's Second Tier Storage system is currently available via the S3 interface and Globus. Click here for more information on how to use Globus to access MSI's Second Tier Storage. The S3 interface was developed by Amazon to make accessing its storage resources easy and robust, and it is now widely used.
It is possible to access MSI's Second Tier Storage system via graphical clients, a command-line interface (s3cmd), and also directly via the S3 API. The s3cmd command-line interface on MSI machines is preconfigured to access your Second Tier Storage account.
- For examples of using s3cmd at MSI, see: How do I use Second Tier Storage from the command line?
You can also transfer files to and from Second Tier Storage using a Globus interface:
- For documentation on using Globus to transfer files at MSI, see: How do I use Globus to transfer data to Second Tier Storage at MSI?
Other access methods require the use of a personalized access key and secret key associated to your MSI account.
-
For examples of graphical clients, see: What are some user-friendly ways to use Second Tier Storage via S3?
-
To access S3 directly from programs, see: How can my programs interface with Second Tier Storage?
If you are logged in to the MSI website, your current usage, access and secret keys are listed below:
Your current data usage in Second Tier Storage is [current-user:cephS3_usage].
Your S3 access key is: [current-user:cephS3_access_key]
Your S3 secret key is: [current-user:cephS3_secret_key]Your S3 sharing address is: [current-user:cephS3_sharing_address]
Log in to view your S3 access and secret keys.
If you are concerned that your access keys have been compromised or wish to reset your S3 access credentials for any reason, you can do so on our S3 Credentials page.
For more information, see these slides from a Birds of a Feather session led by MSI at SC15 -- Ceph in HPC Environments at SC15.