CEDA document repository

Semantic storage of climate data on object stores

Massey, Neil R. (2018) Semantic storage of climate data on object stores. In: NCAS Staff Conference 2018. (Unpublished)

[img] Text

Download (3MB)


New methods for storing large volumes of data, such as cloud systems and object storage present opportunities: • Improved access and user management over POSIX • Parallel reads / writes • Read / write from anywhere (no need to mount disks) 
and challenges: • Interface and access methods for users are changed • Exploit parallel and remote access nature of stores with S3 HTTP API • While maintaining a familiar and consistent API for the user 
 • S3-netCDF-python: • Library to read/write netCDF 3 and netCDF 4 via a S3 HTTP API • Files can be streamed directly to memory or cached to disk • Files can be split into smaller objects • Multiple objects can be streamed in parallel

Item Type: Conference or Workshop Item (Poster)
Subjects: Atmospheric Sciences
Computer Science
Data and Information
Depositing User: Dr Neil Massey
Date Deposited: 28 Feb 2018 11:56
Last Modified: 28 Feb 2018 11:56
URI: http://cedadocs.ceda.ac.uk/id/eprint/1362

Actions (login required)

View Item View Item