CDDIS Archive Access

The Crustal Dynamics Data Information System (CDDIS) supports data archiving and distribution activities for a global user community. Since its inception, users have relied on anonymous ftp for accessing and downloading files from the system. Although this protocol allows users to easily automate file downloads, many organizations, data systems, and users have already migrated from ftp or are actively pursuing a move away from the protocol due to problems from a system and security standpoint. U.S. Government agencies have become increasingly concerned about this legacy protocol and ensuring data integrity for the user community and have recently begun to disallow the use of the ftp protocol. The CDDIS, located at NASA GSFC, must therefore address these concerns and provide alternative methods for access to its archive for continued easy and automated download of its contents.

Therefore, the CDDIS will discontinue anonymous ftp access to its archive in October 2020. Users must now begin their transition to utilize more secure access protocols such as https or ftp-ssl.

In order to access the CDDIS archive, users will need to use a client that supports Transport Security Layer (TLS). When using the https protocol (for example, cURL, Wget, or Python commands/scripts) users will need to have an Earthdata Login account. In addition, when utilizing the cURL and Wget, users will need to create a .netrc file; instructions for creating this file are also available.

Examples of accessing the archive via TLS using various methods can be found below. Click on the name of a method to view the instructions; click on a link within a method to see expected results of that command. In the examples, <email address> means type in your actual email address without the < and > symbols.

cURL

cURL is a command line tool for accessing URLs

For all of these examples:

  • [file] = file to hold cookies
  • -n = use .netrc file
  • -L = follow redirects
  • -O = write to output file
  • -c = file to save cookies to
  1. Download Single File
    -- curl -c [file] -n -L -O "https://cddis.nasa.gov/archive/doris/data/cs2/2017/cs2rx17001.001.Z"
  2. Get a list of all files in a directory
    -- curl -c [file] -n -L "https://cddis.nasa.gov/archive/gps/data/daily/2019/045/19o/*?list"
  3. Get a list of files in a directory matching a pattern: (*_2018*.log)
    -- curl -c .urs_cookies -n -L "https://cddis.nasa.gov/archive/slr/slrlog/*_2018*.log?list"
  4. Download a group of files from a range:
    -- curl -c .urs_cookies -n -L -O "https://cddis.nasa.gov/archive/doris/data/cs2/2017/cs2rx1700[1-3].001.Z"
  5. Download a specific group of files from a list of files:
    -- curl -c .urs_cookies -n -L -O "https://cddis.nasa.gov/archive/vlbi/ivsdata/ngs/2019/19JAN{12,19,20}XK_N003.gz"
  6. Download a set of files matching a pattern and output those to a single .tar file.
    -- curl -c .urs_cookies -n -L -o files.tar "https://cddis.nasa.gov/archive/doris/data/cs2/2017/cs2rx1700*.001.Z"

Wget

GNU Wget is a free software package for retrieving files using HTTP, HTTPS, FTP, and FTPS, the most widely used Internet protocols.

  1. Download Single File
    -- wget --auth-no-challenge "https://cddis.nasa.gov/archive/doris/data/cs2/2017/cs2rx17001.001.Z"
  2. Get a list of all files in a directory
    -- wget --auth-no-challenge -qO- "https://cddis.nasa.gov/archive/doris/data/cs2/2017/*?list"
  3. Get a list of files in a directory matching a pattern: (cs2rx1700*.001.Z)
    -- wget --auth-no-challenge -qO- "https://cddis.nasa.gov/archive/doris/data/cs2/2017/cs2rx1700*.001.Z*?list"
  4. Download Single File via ftp-ssl (requires Wget version 1.18 or higher)
    -- wget --ftp-user anonymous --ftp-password <email address>
    ftps://gdc.cddis.eosdis.nasa.gov/doris/data/cs2/2017/cs2rx17001.001.Z
  5. Download all files in a directory via ftp-ssl (requires Wget version 1.18 or higher)
    -- wget --ftp-user anonymous --ftp-password <email address> -r ftps://gdc.cddis.eosdis.nasa.gov/vlbi/ivsdata/ngs/2019/

ftp-ssl

ftp-ssl is an extension to the commonly used File Transfer Protocol (FTP) that adds support for the Transport Layer Security (TLS).

NOTE: In this technique, the path to the archive is gdc.cddis.eosdis.nasa.gov. This is different than for the other techniques.

  1. Download Single File
    -- lftp -d -u anonymous,<email address> -e 'set ftp:ssl-force true' -e 'get doris/data/cs2/2017/cs2rx17001.001.Z;exit' gdc.cddis.eosdis.nasa.gov
  2. Download an entire directory using lftp:
    -- lftp -d -u anonymous,<email address> -e 'set ftp:ssl-force true' -e 'mget vlbi/ivscontrol/*;exit' gdc.cddis.eosdis.nasa.gov
  3. Log into the archive using ftp commands
    -- lftp -d -u anonymous,<email address> -e 'set ftp:ssl-force true' gdc.cddis.eosdis.nasa.gov
  4. Download Single File via ftp-ssl (requires Wget version 1.18 or higher)
    -- wget --ftp-user anonymous --ftp-password <email address>
    ftps://gdc.cddis.eosdis.nasa.gov/doris/data/cs2/2017/cs2rx17001.001.Z
  5. Download all files in a directory via ftp-ssl (requires Wget version 1.18 or higher)
    -- wget --ftp-user anonymous --ftp-password <email address> -r ftps://gdc.cddis.eosdis.nasa.gov/vlbi/ivsdata/ngs/2019/

Python

Software written in the Python language can access the archive using libraries such as lib-curl.

  1. Download a single file
    python [code_filename] https://cddis.nasa.gov/archive/doris/data/cs2/2017/ cs2rx17001.001.Z