Table of Contents

Module: globalDownload madrigalWeb/globalDownload.py

This script runs a global search through Madrigal data from a given URL, and downloads all matching files.

This script is a stand-alone application, and can be run from anywhere with a connection to the internet. It runs on either unix or windows. It requires only the MadrigalWeb python module to be installed.

Usage:

globalDownload.py --url=<Madrigal url> --outputDir=<output directory> \ --user_fullname=<user fullname> --user_email=<user email> \ --user_affiliation=<user affiliation> --format=<ascii,hdf5> [options]

where:

--url=<Madrigal url> - url to homepage of site to be searched (ie, http://madrigal.haystack.mit.edu/madrigal/) This is required.

--outputDir=<output directory> - the output directory to store all files in. Default is to store all files in the same directory, and a number is added to the filename if a file might be overwritten. Set --tree flag to store all files in the same directory structure they appear in Madrigal. This allows all files to keep their original names.

--user_fullname=<user fullname> - the full user name (probably in quotes unless your name is Sting or Madonna)

--user_email=<user email>

--user_affiliation=<user affiliation> - user affiliation. Use quotes if it contains spaces.

--format=<ascii or hdf5>

and options are:

--startDate=<MM/DD/YYYY> - start date to filter experiments before. Defaults to allow all experiments.

--endDate=<MM/DD/YYYY> - end date to filter experiments after. Defaults to allow all experiments.

--inst=<instrument list> - comma separated list of instrument codes or names. See Madrigal documentation for this list. Defaults to allow all instruments. If names are given, the argument must be enclosed in double quotes. An asterick will perform matching as in glob. For example:

           --inst=10,30

           --inst=&quot;Jicamarca IS Radar,Arecibo*&quot;

--expName - filter experiments by the experiment name. Give all or part of the experiment name. Matching is case insensitive. Default is no filtering by experiment name.

--kindat=<kind of data list> - comma separated list of kind of data codes. See Madrigal documentation for this list. Defaults to allow all kinds of data. If names are given, the argument must be enclosed in double quotes. An asterick will perform matching as in glob. For example:

            --kindat=3001,13201

            --kindat=&quot;INSCAL Basic Derived Parameters,*efwind*,2001&quot;

--seasonalStartDate=<MM/DD> - seasonal start date to filter experiments before. Use this to select only part of the year to collect data. Defaults to Jan 1. Example:

--seasonalStartDate=07/01 would only allow experiments after July 1st from each year.

--seasonalEndDate=<MM/DD> - seasonal end date to filter experiments after. Use this to select only part of the year to collect data. Defaults to Dec 31. Example:

--seasonalEndDate=10/31 would only allow experiments before Oct 31 of each year.

--tree - add if you want to store the downloaded files in the same hierarchy as in Madrigal: <YYYY/<instCode>/<experimentDir>. Without --tree, stores all downloaded files in one directory.

--includeNonDefault - if given, include all files, including history. Default is to search only default files.

--verbose - if given, print each file processed info to stdout. Default is to run silently.

Example:

globalDownload.py --url=http://madrigal.haystack.mit.edu/madrigal --outputDir=/tmp --user_fullname="Bill Rideout" --user_email=brideout@haystack.mit.edu --user_affiliation=MIT --startDate=01/01/1998 --endDate=-01/30/1998 --inst=30

$Id: globalDownload.py 4087 2012-11-08 18:24:49Z brideout $

Functions   
filterExperimentFilesUsingFileDesc
filterExperimentFilesUsingKindat
filterExperimentFilesUsingStatus
filterExperimentsUsingExpName
filterExperimentsUsingSeason
getExperimentFileList
getInstrumentList
getSubdirectoryFromFullFile
getTimesOfExperiment
  filterExperimentFilesUsingFileDesc 
filterExperimentFilesUsingFileDesc ( expFileList,  fileDesc )

filterExperimentFilesUsingFileDesc returns a subset of the experiment files in expFileList with filtered using fileDesc string and case-insensitive fnmatch.

Input:

expFileList - a list of MadrigalExperimentFile objects to be filtered.

Returns:

a subset of expFileList with default status

  filterExperimentFilesUsingKindat 
filterExperimentFilesUsingKindat ( expFileList,  kindat )

filterExperimentFilesUsingKindat returns a subset of the experiment files in expFileList whose kindat is found in kindat argument.

Input:

expFileList - a list of MadrigalExperimentFile objects to be filtered

kindat - the kindat argument passed in by the user - comma separated list of kind of data codes. If names are given, the argument must be enclosed in double quotes. An asterick will perform matching as in glob.

Returns:

a subset of expFileList whose kindat values are accepted

  filterExperimentFilesUsingStatus 
filterExperimentFilesUsingStatus ( expFileList )

filterExperimentFilesUsingStatus returns a subset of the experiment files in expFileList with default status.

Input:

expFileList - a list of MadrigalExperimentFile objects to be filtered.

Returns:

a subset of expFileList with default status

  filterExperimentsUsingExpName 
filterExperimentsUsingExpName ( expList,  expName )

filterExperimentsUsingExpName returns a subset of the experiments in expList whose name matches.

Input:

expList - a list of MadrigalExperiment objects to be filtered

expName - filter experiments by the experiment name. Can be all or part of the experiment name. Matching is case insensitive.

Returns:

a subset of expList whose names are accepted

  filterExperimentsUsingSeason 
filterExperimentsUsingSeason (
        expList,
        seasonalStartDate,
        seasonalEndDate,
        )

filterExperimentsUsingSeason returns a subset of the experiments in expList whose date is within the given season.

Input:

expList - a list of MadrigalExperiment objects to be filtered

seasonalStartDate - in form MM/DD - seasonal start date to filter experiments before

seasonalEndDate - in form MM/DD - seasonal end date to filter experiments after

Returns:

a subset of expList whose times are accepted

Exceptions   
ValueError, 'seasonalEndDate must be in form MM/DD: ' + str( seasonalEndDate )
ValueError, 'seasonalStartDate must be in form MM/DD: ' + str( seasonalStartDate )
  getExperimentFileList 
getExperimentFileList (
        server,
        expList,
        includeNonDefault,
        )

getExperimentFileList returns a list of MadrigalExperimentFile objects given an experiment list.

Inputs:

        server - the active MadrigalData object to get information from

        expList - the list of desired MadrigalExperiment objects

        includeNonDefault - 1 if should include non-default files, 0 otherwise

Returns:

a list of MadrigalExperimentFile objects

  getInstrumentList 
getInstrumentList ( inst,  server )

getInstrumentList takes the user argument inst and coverts it into a list of instrument codes.

Inputs:

inst - a string containing a comma separated list of instrument codes or names. If names are given, the argument must be enclosed in double quotes. An asterick will perform matching as in glob. Both names and codes may be mixed together.

server - the active MadrigalData object to get information from

Returns:

a list of instrument codes (int). Instrument code 0 means all instruments

  getSubdirectoryFromFullFile 
getSubdirectoryFromFullFile ( fullFilename )

get the subdirectory to store this file in from the full file name

Return everything after experiments* and before filename

Exceptions   
ValueError, 'Illegal filename %s' %( fullFilename )
  getTimesOfExperiment 
getTimesOfExperiment ( expList,  expId )

getTimesOfExperiment returns a list of the start and end time of the experiment given expId.

Input:

expList - the list of MadrigalExperiment objects

expId - the experiment id

Returns:

a list of: (startyear, startmonth, startday, starthour, startmin, startsec, endyear, endmonth, endday, endhour, endmin, endsec)


Table of Contents

This document was automatically generated on Thu Nov 8 14:17:24 2012 by HappyDoc version r1_5