CEDA document repository

Semantic Storage of climate data on object stores

Massey, Neil R. (2018) Semantic Storage of climate data on object stores. Other. STFC.

[img] Text
nrmassey_ncas2018_poster.pdf

Download (3MB)
[img] Slideshow
nrmassey_ncas2018_poster.pptx

Download (2MB)

Abstract

New methods for storing large volumes of data, such as cloud systems and object storage present opportunities: Improved access and user management over POSIX Parallel reads / writes Read / write from anywhere (no need to mount disks) and challenges: Interface and access methods for users are changed Exploit parallel and remote access nature of stores with S3 HTTP API While maintaining a familiar and consistent API for the user S3-netCDF-python: Library to read/write netCDF 3 and netCDF 4 via a S3 HTTP API Files can be streamed directly to memory or cached to disk Files can be split into smaller objects Multiple objects can be streamed in parallel

Item Type: Monograph (Other)
Subjects: Earth Sciences
Ecology and Environment
Meteorology and Climatology
Depositing User: Miss Hayley Gray
Date Deposited: 21 Oct 2019 13:32
Last Modified: 21 Oct 2019 13:32
URI: http://cedadocs.ceda.ac.uk/id/eprint/1443

Actions (login required)

View Item View Item