Online Dissemination Help Page

Introduction

Welcome to the ESA Aeolus Online Dissemination System. This is not an operational system. All functionalities are available for evaluation and testing only.

Supported Browsers

The following Web Browsers are supported by this system: Internet Explorer, Firefox, Google Chrome, Safari. It is recommended that a recent version is used.

Available Data

The available data is organised into different collections. All products are provided in native Earth Explorer (EE) Ground Segment File Format Standard, as files with .TGZ or .EEF extension.

Users can openly search for and browse the available data products, but download of files is only possible after login.

Aeolus Collections

The following online collections contain the operational free and open Aeolus Science data, accessible to all users:

The following collections contain restricted data, only accessible to CalVal users:

  • Level 0 Products (ALD_U_N_0_): raw instrument data, cleaned, time-ordered (Format Specification)
  • Level 1A Products (ALD_U_N_1A): geo-located, un-processed observational data (Format Specification)
  • Auxiliary Data
  • Calibration Data
  • Telemetry Data

1. EO-SSO Account

In order to access the restricted functionalities provided by the ESA Aeolus Online Dissemination System (e.g. products download) you need an EO-SSO account.
The ESA Earth Observation Users Single Sign On service (EO-SSO) allows end users to access all the EO Services by using a unique account.

EO-SSO TEST accounts, configured with different access privileges, have been created at this stage for selected test users.

1.1 Login

You can login into the system following these steps:

  • Navigate to http://aeolus-ds.eo.esa.int/
  • Click on "Login" at the top right of the page
  • Enter your EO-SSO TEST account
  • Click on the "Login" button

2. Search for Data

The Aeolus Online Dissemination System provides two different mechanisms for searching data, one based on HTTP via a dedicated web portal and another one based on FTP. Below the details on how to use the two methods.

2.1 Search via the Web Portal

To start searching for data, navigate to http://aeolus-ds.eo.esa.int/, click on the "Collections" button and choose a collection. Three product search mechanisms are provided and listed.

2.1.1 Search by Tree View Navigation

This function allows navigating a dynamic structure (as shown below) via the Web Browser, where data is organised in collections. Inside these the user can navigate the available data by:

  • Baseline: processing baseline (Science and Calibration data only), e.g. "1B01" (where the first two chars indicate the processing level and the last two a sequential counter).
    This is a qualifier summarizing all quality characteristics of a specific product, depending on the applied algorithm and on the input used to generate it.
  • File Type: type of file (among those available in the specific collection)
  • Year-Month: year&month of sensing or validity
  • Day: day of sensing or validity

2.1.2 Search by File Name

This function allows searching a product by its file name. You can use the whole name of the file of interest, or part of it (using “*” and “?” wildcards) in order to search for multiple products.

A few examples on the usage of this specific function are reported hereafter:

  1. AE_OPER_ALD_U_N_2B_20071030T155029_20071030T172353_0001.TGZ
    only the ALD_U_N_2B product with that specific file name will match
  2. *OPER_ALD_U_N_2A_20071030*
    all files of type ALD_U_N_2A and "sensing-start-time" 2007-10-30 will match
  3. *OPER_ALD_U_N_2B_*_20071124*
    all files of type ALD_U_N_2B and "sensing-stop-time" 2007-11-24 will match
  4. *OPER_ALD_U_N_2C*
    all files of type ALD_U_N_2C, irrespective of sensing time, will match

To display the results in a list of products, tick the "Show search result as product list". To display the results in a short URL list, tick "Show search result as URL list". Note that in case of more than 500 results the view defaults to a "URL list".

2.1.3 Catalogue Search

This function allows searching for data using different parameters and obtaining different results.

2.1.3.1 Search Form

  • Tick the "draw bbox" checkbox and draw a rectangular area to define the bounding box of interest on the world map
  • When a bounding box is drawn on the world map, the "Geographical Coverage" information gets filled by the coordinates of the displayed rectangular area. This information can be manually adjusted by changing latitude and longitude values. The area will be adjusted as well on the map
  • In order to refine your search, if needed, you can specify additional metadata parameters:
    • "Format": "HTML search response", "Bulk download lists", "Plain text metadata (index)"
    • "Items per page": it only refers to the "HTML search response"
    • "Geographical Coverage": it allows indicating the coordinates, as min and max latitude and min and max longitude values, of the area of interest
    • "Temporal Coverage": it defines the "sensing-start-time" and the (optional) "sensing-stop-time" of interest. Search goes from 00:00:00 of the start date to 23:59:59 of the stop date, or from 00:00:00 to 23:59:59 of the single date. The date can be entered either manually (as YYYY-MM-DD UTC, time specification is not allowed) or by means of the provided Date Selection widget
    • "Product/FileType": it allows selecting the product/file of interest by chosing one of the possible values from the drop-down menu
    • "Baseline": it allows selecting the processing baseline among those available. It is used to identify group of products presenting similar characteristics from a user point of view, i.e. allowing listing a temporal series of products presenting physical consistency and interoperability. All files characterised by the same baseline can be used together without losing consistency
  • Click on the "Search" button: the relevant satellite ground tracks will be displayed on the map

The search results can be adjusted as described in Section 2.1.3.2.

2.1.3.2 Search Results Format

The Simple Online Catalogue allows selecting among five different response formats, by clicking on the "Format" blank field and selecting one of the options from the drop-down menu:

  • "HTML search response": it returns an HTML page with the list of all requested products (ref. to Section 2.1.3.2.1 for further details)
  • "Bulk download lists": it returns a list of download URLs in different formats suitable for easy bulk download. The list can be saved in "plain text" format (containing one download URL per line) or as a "metalink" XML file, or simply shown in an HTML page ("View HTML")
  • "Plain text metadata (index)": it returns a plain text page containing the full metadata records for the matched products, encoded in a tab separated value (TSV) format

It is also possible to set the pagination parameter ("Items per page") to define the maximum number of records per page of the "HTML search response".

2.1.3.2.1 HTML Search Response

The "HTML search response" (see Section 2.1.3.2) contains a list of data items matching the search parameters, organised into different pages each with as many items as indicated in "Items per page".

The displayed list of results contains the data item file name with links to download the Product (it requires authentication), the Quality Report and the Metadata file, and to display the relevant Browse Image if available.

In order to identify a given item on the map, by clicking on its file name, the item will be highlighted in the map, zoomed to its extent. On the other hand, by clicking on a given satellite ground track displayed on the map, the corresponding search record in the list will be highlighted as shown below: .

Please note that, in case of multiple overlapping tracks at the same location, a clickable list of product identifiers will appear in a tooltip.

2.2 Searching via an FTP Client

This function allows navigating a dynamic structure via a standard FTP Client (FileZilla, CyberDuck, etc.). Data is organised into different collections the user can navigate by Baseline (Science and Calibration data only), File Type, Year-Month and Day (of the sensing or validity date).

You can refer to Section 3.2 for additional details on the FTP access.

3. Download Products

There are different mechanisms to access the data and download the products of interest.

3.1 Download via HTTP

The Aeolus data can be accessed and downloaded via HTTP as follows:

  • Login with EO-SSO TEST credentials
  • For interactive or individual downloads, select the data of interest by using a search mechanism as from Section 2
  • Download the files by using the product download links

3.2 Download via FTP

The same data can be accessed and downloaded via FTP as follows:

  • Open an FTP Client (such as FileZilla, CyberDuck, Rumba FTP, etc.) or the command prompt
  • Choose "explicit" FTP over SSL/TLS settings (FTPS)
  • Set to Passive (PASV) connect mode
  • Enter "aeolus-ds.eo.esa.int" as Host
  • Enter your EO-SSO TEST credentials (username and password)
  • The Aeolus directory tree will be displayed

In case of problems with FTPS (FTP encrypted with SSL/TLS), please try with plain FTP.

It is also recommended to use the LFTP command-line client for UNIX and UNIX-like systems.

3.3 Download in Bulk

It is possibile to download in bulk a set of data contained in a web page. In order to download multiple products all at once, it is suggested to use Firefox with the DownThemAll! extension able to automatically parse a list of URLs inputs:

  • Execute a search as described in either File name search or Catalogue search , respectively by selecting "Show search result as URL list" or "Bulk download lists" in the "Format" options, to get a list of download URLs
  • In the result page, select "Tools" > "DownThemAll! Tools" > "DownThemAll!" from the Browser's Toolbar
  • In the "DownThemAll!" window use the "Filters" or "Fast Filtering" options in order to select ".TGZ" or ".EEF" files only
  • Select "Start" for immediate download, or "Add paused" and activate the download at wish from the "DownThemAll! Manager" window
Recommended "Preferences/Network" settings for DownThemAll!:
  • Concurrent downloads: 8
  • Downloads per server: 4

3.4 Advanced Query and Download via Command Line

The catalogue can be queried via a command-line tool like cURL, saving the output as a URL list file. A tool supporting SSO authentication schemas (such e.g. https://sourceforge.net/projects/secp/) must then be used to perform URL downloads to a target directory.

This is an example of the sequence of commands to run on UNIX-like systems:

  1. To fetch a plain text url list from the catalogue and store it in a temp_file.
    In the following example the user is interested in 1 year of L2 data, processed within a specific baseline and intersecting a given area of interest (rectangular box). The query on the curl command can be adjusted as needed by setting any of the following parameters:
    • query.footprint.minlat - (latitude of bottom-left corner of a rectangular box)
    • query.footprint.minlon - (longitude of bottom-left corner of a rectangular box)
    • query.footprint.maxlat - (latitude of upper-right corner of a rectangular box)
    • query.footprint.maxlon - (longitude of upper-right corner of a rectangular box)
    • query.beginAcquisition.start - (start time as "YYYY-MM-DD")
    • query.beginAcquisition.stop - (stop time as "YYYY-MM-DD")
    • query.productVersion - (processing baseline)
    • query.productType - (product/file of interest)

    curl --data "service=SimpleOnlineCatalogue&version=1.0&request=search&format=text%2Fplain&pageCount=10&query.footprint.minlat=-19.336&query.footprint.minlon=15.345&query.footprint.maxlat=38.320&query.footprint.maxlon=48.340&query.beginAcquisition.start=2007-05-01&query.beginAcquisition.stop=2008-05-01&query.productType=ALD_U_N_2B&query.productVersion=2B02" http://aeolus-ds.eo.esa.int/socat/L2B_Products/search > temp_file

  2. To clean-up the temp_file

    dos2unix temp_file

  3. To download all products, without compression, using the SSO credentials (requested the first time and then stored locally for further accesses) one at a time to the local directory.
    Another directory can be specified with -o /some/local/dir

    secp -U -F temp_file

4. Troubleshooting

In case of problems with the authentication process (inconsistent session), you can try to solve the issue by cleaning the cookie cache (Browser memory) or by closing and reopening again your Web Browser.