CDDIS now has a new https-based capability for users to download files from its archive. For now, users can continue to use anonymous ftp to access the CDDIS archive and retrieve files. Utilizing the https-based method will require that users make a few adjustments in how files are downloaded from the CDDIS.
Outline of https access:
Access to CDDIS data via https uses a NASA single sign-on system (https://urs.earthdata.nasa.gov/). Users will need to create an account with that system before they can download CDDIS data via https.
Files can be downloaded using a standard web browser (https://cddis.nasa.gov/archive/) or with a standard command line tool like cURL (http://curl.haxx.se/) that is available for OS X, Windows, and Linux operating systems.
Steps to using the new system:
Initial use of the system:
- Create an Earthdata login account if you do not already have one. Go to https://urs.earthdata.nasa.gov/ and follow the instructions to create an account.
- Important: the very first time you try to use the system you need to use the web interface: https://cddis.nasa.gov/archive/, this page will redirect you to the Earthdata login page to input your username and password. After that, the system will ask you to authorize the application's access to your Earthdata profile. You must do this the first time you use the system; this step will not be required in future download sessions.
Downloading files:
Option 1 - Using the web interface
Go to https://cddis.nasa.gov/archive/. When redirected to the Earthdata log in page, input your username and password. When you are returned to CDDIS, simply navigate to the files you want.
Option 2 - Using the command line interface through cURL
This process has two steps like the web interface, however, unlike that process, you will not see the redirect to Earthdata login happening, and you will need to set up a .netrc file on your system with your username and password for cURL to use when the redirect occurs behind the scenes.
** If your operating system does not have cURL installed by default, you will need to install it before you can use this process. **
-
To create a .netrc file, you will need to create a text file with the name .netrc; this file needs to have read permissions set to only yourself, so that no one can read your file and get your username and password. The format of the file is:
machine urs.earthdata.nasa.gov login <username> password <password>
where <username> and <password> are the values you set when you created your Earthdata login account.
Below are examples on how to use cURL to obtain a list of files as well as download files via a script.
For these examples below:
<-n>
look for a text file in my homespace under the name .netrc with read only permission, see above for instructions on creating this.
- Get a list of all files in a directory:
curl -c .urs_cookies -b .urs_cookies -n -L "https://cddis.nasa.gov/archive/doris/data/cs2/2017/*?list"
returns:
cs2rx17183.001.Z 1661933
cs2rx17184.001.Z 1704621
cs2rx17185.001.Z 1680949
..
..
cs2rx17365.001.Z 1669185
# Total number of files = 364
# Total file size = 609023913
- Get a list of files in a directory matching a pattern (cs2rx1700*.001.Z)
curl -c .urs_cookies -b .urs_cookies -n -L "https://cddis.nasa.gov/archive/doris/data/cs2/2017/cs2rx1700*.001.Z?list"
returns:
cs2rx17001.001.Z 1597475 cs2rx17002.001.Z 1710715 cs2rx17003.001.Z 1649195 cs2rx17004.001.Z 1716190 cs2rx17005.001.Z 1640610 cs2rx17006.001.Z 1726173 cs2rx17007.001.Z 1645541 cs2rx17008.001.Z 1717177 cs2rx17009.001.Z 1665687
# Total number of files = 9
# Total file size = 15068763
- Download a group of files within a range
curl -c .urs_cookies -b .urs_cookies -n -L "https://cddis.nasa.gov/archive/doris/data/cs2/2017/cs2rx1700[1-9].001.Z" -O
downloads:
- cs2rx17001.001.Z
- cs2rx17002.001.Z
- cs2rx17003.001.Z
- cs2rx17004.001.Z
- cs2rx17005.001.Z
- cs2rx17006.001.Z
- cs2rx17007.001.Z
- cs2rx17008.001.Z
- cs2rx17009.001.Z
- Download a group of files from a list of specific files
curl -c .urs_cookies -b .urs_cookies -n -L "https://cddis.nasa.gov/archive/doris/data/cs2/2017/cs2rx1700{1,3,7}.001.Z" -O
downloads:
- cs2rx17001.001.Z
- cs2rx17003.001.Z
- cs2rx17007.001.Z
- Download a set of files matching a pattern and output those to a single .tar file
curl -c .urs_cookies -b .urs_cookies -n -L "https://cddis.nasa.gov/archive/doris/data/cs2/2017/cs2rx1700*.001.Z" -o files.tar
- Download a single file
curl -c .urs_cookies -b .urs_cookies -n -L "https://cddis.nasa.gov/archive/doris/data/cs2/2017/cs2rx17001.001.Z" -O