Speaking at the 2020 Earth Sciences Information Partners* online summer meeting, Aleksandar Jelenak from the HDF5 Group described HDF5 as ‘one of the most widely used file formats for Earth Science data’. The HDF Group is building on this popularity to enable a ‘seamless transition’ to cloud computing for all applications that use the HDF5 API. An HDF Town Hall covered several use cases of HDF Group’s own and third-party solutions on how to access and analyze HDF5 data in cloud object stores.
Michael Lee Rilee presented NASA’s STARE-PODS spatio-temporal data store that leverages the HDF virtual object layer and a STARE location service (SLS), aka a ‘domain name service’ for geolocated data in the cloud.
Joe Lee (HDF Group) compared three cloud access approaches (Apache Drill, Unidata TDS a.k.a. Thredds and HDF5 VFD) for accessing earth observation data in the Amazon S3 cloud. Unidata THREDDS 5.0 came out as a ‘clear winner’ in the benchmark results.
John Readey (HDF Group) presented the new HDF Server features for the cloud. The idea is to keep the existing HDF5 API and data model but migrate to a new cloud friendly storage format. Enter the HSDS, the highly scalable data service.
Aleksandar Jelenak (HDF Group) presented on bi-directional exchange between HDF5 and the Zarr format as used by the USGS. Zarr is a Python package that implements chunked, compressed, N-dimensional arrays.
Other ESIP presentations covered a proposal for a ‘consistent, operational-grade, digital knowledge commons for the earth and environmental science community’ to ‘bring together data and information siloed behind disciplinary boundaries’. The digital knowledge commons will leverage semantic technologies and knowledge representation initiatives such as the OBO Foundry merging such with the Semantic Web for Earth and Environmental Terminology (SWEET).
More from the ESIP Summer Meeting website.
* ESIP is a collaboration of partner organizations that promotes the collection, stewardship and use of Earth science data, information and knowledge. Its activities are sponsored by NASA, NOAA and the USGS.
© Oil IT Journal - all rights reserved.