Multi-Repository Metadata Assessment

The metadata of energy-related datasets from the following repositories is used in this multi-repository assessment of conformance of metadata with OEMetadata v2:

• Energy Data eXchange: https://edx.netl.doe.gov/
• EU Open Data Portal: https://data.europa.eu/data/datasets
• Eurostat: https://ec.europa.eu/eurostat
• FfE Open Data Portal: https://opendata.ffe.de/
• Inspire-HEP: https://inspirehep.net
• Open Energy Data Initiative: https://data.openei.org/
• Open Energy Platform: https://openenergyplatform.org/database/
• OpenAIRE: https://explore.openaire.eu/
• Open Power System Data (OPSD): https://open-power-system-data.org/
• Open Access Power-Grid Frequency Database: https://osf.io/m43tg/

Unfortunately, we cannot share the scripts that we used for downloading the metadata from these repositories. However, we are sharing our preprocessing scripts for the repositories which needed preprocessing before mapping them to OEMetadata. These scripts would be executed directly after retrieving the metadata from the repository.

In order to map the raw metadata (see metadata_raw) to the OEMetadata, execute

bash pip install -r requirements.txt python parser.py

The transformed metadata is then stored in a directory called metadata_mapped. To fix some reporting issues with nested fields, execute the cleanup script for the OEP:

bash python cleanup_oep.py

Now you are ready to reproduce our report by running

bash python report.py

The reports for each of the repositories is then stored in the directory report.