Combined team effort and a cloud-based approach were behind a hugely successful campaign, resulting in the first full reprocessing of all Sentinel-5P datasets since the beginning of routine operations.
As part of the Copernicus programme the Copernicus Sentinel-5 Precursor (Sentinel-5P) mission continues to make a real difference in monitoring air quality and supporting the European Union’s contribution to combatting global air pollution.
Launched on 13 October 2017, this atmosphere-monitoring mission from the EU’s Copernicus programme delivers highly accurate and detailed data, thanks to its state-of-the-art multispectral imaging spectrometer, Tropomi.
Atmosphere monitoring mission
Copyright: Contains modified Copernicus Sentinel data (2022), processed by ESA
The recent complete reprocessing of all datasets ensures even more impact for Sentinel-5P. Multiple teams worked tirelessly to manage the reprocessing of already huge repositories of data, since the start of routine operations in April 2018.
The processing was prepared, implemented and executed by the German Aerospace Center (DLR), while the Mission Performance Centre (MPC) - led by KMNI, but including many industrial teams - helped attest the quality of the datasets through the development of tailor-made algorithms.
The reprocessed datasets are now available on the Copernicus Space Data Ecosystem (CSDE) identifiable by the file class “RPRO”.
Whilst Sentinel-5P has been in operation for some time, the reprocessing could only be executed when the algorithms were fully developed, which took time to complete, and when a reprocessing system capable of processing this extensive data was ready.
The reprocessing team used a novel approach by processing on the cloud. Previously reprocessing had been undertaken on spare resources within the nominal processing system, meaning it was executed alongside the day-to-day processing that generates Sentinel-5p products from fresh satellite data.
“Traditionally, in a ground-processing system there’s usually enough hardware resources to process the data plus a small extra capacity. For reprocessing activities, if the reprocessing is only for a small data set, then using hardware already existing in the nominal processing system works fine,” explains Nigel Houghton, ESA’s Sentinels Ground Segment Processing and Archiving Centre (PAC) Manager.
“Since our goal was a complete mission reprocessing, using the nominal system could have taken years. So, we decided to use cloud-based processing - giving the DLR team autonomy to choose the cloud provider, whilst ESA set the requirements.”
The paradigm shift to cloud-based processing meant the speed of the reprocessing is governed by available cloud resources. This aim was largely achieved in Sentinel-5P reprocessing but was impacted somewhat since some of the processing is not linear, meaning some products needed to be processed in advance of others.
As a first priority, ESA released Sentinel-5P full mission reprocessed datasets for four products, which served the needs of the Copernicus Atmosphere Monitoring Service (CAMS), in their work to provide quality controlled information related to air pollution and health, solar energy and greenhouse gases. The second release of full mission reprocessed datasets included other Level-1B products and Level-2 products, until finally, the third (and final) full mission reprocessed datasets - covering the whole mission – were released.
Sentinel 5-P detecting methane plumes
One challenge of the reprocessing campaign was the huge volume of products generated from the reprocessing, which needed to be carefully checked post-processing.
The reprocessed data are welcomed by the data community, especially since the processing is in near real-time. For the entire operation window of the mission there are now consistent datasets - generated with the same algorithm – for each data product, serving applications dependent on longer term trends.
About the Copernicus Sentinels
The Copernicus Sentinels are a fleet of dedicated EU-owned satellites, designed to deliver the wealth of data and imagery that are central to the European Union's Copernicus environmental programme.
The European Commission leads and coordinates this programme, to improve the management of the environment, safeguarding lives every day. ESA is in charge of the space component, responsible for developing the family of Copernicus Sentinel satellites on behalf of the European Union and ensuring the flow of data for the Copernicus services, while the operations of the Copernicus Sentinels have been entrusted to ESA and EUMETSAT.
Did you know that?
Earth observation data from the Copernicus Sentinel satellites are fed into the Copernicus Services. First launched in 2012 with the Land Monitoring and Emergency Management services, these services provide free and open support, in six different thematic areas.
The Copernicus Atmosphere Monitoring Service (CAMS) provides continuous data and information on atmospheric composition, supporting many applications in a variety of domains including health, environmental monitoring, renewable energies, meteorology and climatology.