-->
Previous: Remote access using Python   Up: Remote access programming tutorial toc   Next: Remote access reference toc

Remote access using IDL

 

The following are the methods for accessing Madrigal data remotely:

A good way to learn how to use this IDL API is to run this example.

   ***********************************************************
;+
; NAME: 
;    madgetallinstruments(madurl)
;
; PURPOSE: 
;    Get information on all instruments with data at the Madrigal site specified
;    by madUrl
;
; INPUTS: 
;      madurl - scalar string giving a fully qualified url to the Madrigal site
;
; OUTPUT: an array of structures with the following fields:
;		1. name - (string) Example: 'Millstone Hill Incoherent Scatter Radar'
;		2. code - (int) Example: 30
;		3. mnemonic - (3 char string) Example: 'mlh'
;		4. latitude - (double) Example: 45.0
;		5. longitude (double) Example: 110.0
;		6. altitude (double)  Example: 0.015 (km)
; EXAMPLE: 
;     result = madGetAllInstruments('http://madrigal.haystack.mit.edu/madrigal')
;
; $Id: madidl.html 3539 2011-08-26 18:00:05Z brideout $
;
FUNCTION madgetallinstruments, madurl
   ***********************************************************
;+
; NAME: 
;    madgetexperiments(madurl, instcode, startyear, startmonth, startday, starthour, startmin, startsec,
;                                      endyear, endmonth, endday, endhour, endmin, endsec, local)
;
; PURPOSE: 
;    Get information on experiments for a given instrument and date range
;
; INPUTS: 
;      madurl - scalar string giving a fully qualified url to the Madrigal site
;      instcode = int or array of ints representing instrument code(s). Special value of 0 selects all instruments.
;      startyear, startmonth, startday, starthour, startmin, startsec - 6 ints representing start time
;      endyear, endmonth, endday, endhour, endmin, endsec - 6 ints representing end time
;      local - 0 if all sites desired, 1 (default) if only local experiments desired
;
; OUTPUT: an array of structures representing experiments between the dates with the following fields:
;   1. strid (string) Example: "10000111".  Uniquely identifies experiment. Can be converted to a long long.
;         Not stored as a long long because that just doesn't seem to work, so user needs to convert.
;   2. url (string) Example: 'http://www.haystack.mit.edu/cgi-bin/madtoc/1997/mlh/03dec97'
;   3. name (string) Example: 'Wide Latitude Substorm Study'
;   4. siteid (int) Example: 1
;   5. sitename (string) Example: 'Millstone Hill Observatory'
;   6. instcode (int) Code of instrument. Example: 30
;   7. instname (string) Instrument name. Example: 'Millstone Hill Incoherent Scatter Radar'
;   8. startyear, startmonth, startday, starthour, startmin, startsec - 6 ints representing start time of experiments
;   9. endyear, endmonth, endday, endhour, endmin, endsec - 6 ints representing end time of experiments
;   10. isLocal - 1 if a local experiment, 0 if not
;   11. madrigalUrl - url of Madrigal site.  Used if not a local experiment.
;   If no experiements found, returns a Null Pointer (IDL does not allow empty arrays)
;
;    Note that if the returned experiment is not local, the experiment id will be -1.  This means that you
;    will need to rerun madgetexperiments with the url of the 
;    non-local experiment (experiment.madrigalUrl).
;    This is because 
;    while Madrigal sites share metadata about experiments, the real experiment ids are only
;    known by the individual Madrigal sites.  See example_madidl.pro for an example of this
;           for an example of this.
; EXAMPLE: 
;     result = madGetExperiments('http://madrigal.haystack.mit.edu/madrigal', 30, 1998,1,1,0,0,0, 1998,1,31,23,59,59,1)
;
; $Id: madidl.html 3539 2011-08-26 18:00:05Z brideout $
;
FUNCTION madGetExperiments, madurl,  instcode, startyear, startmonth, startday, starthour, startmin, startsec, $
                                      endyear, endmonth, endday, endhour, endmin, endsec, local
   ***********************************************************
;+
; NAME: 
;    madgetexperimentfiles(madurl, expid, getnondefault)
;
; PURPOSE: 
;    returns a list of all  Madrigal Experiment Files for a given experiment id
;
; INPUTS: 
;      madurl - scalar string giving a fully qualified url to the Madrigal site
;      expid - experiment id as returned by madgetexperiments
;      getnondefault - an optional argument.  If 1, get all files, including non-default.  If 0 (the default)
;             get only default files
;
; OUTPUT: an array of structures representing experiment files for that experiment with the following fields:
;     1. name (string) Example '/opt/madrigal/blah/mlh980120g.001'
;     2. kindat (int) Kindat code.  Example: 3001
;     3. kindatdesc (string) Kindat description: Example 'Basic Derived Parameters'
;     4. category (int) (1=default, 2=variant, 3=history, 4=real-time)
;     5. status (string)('preliminary', 'final', or any other description)
;     6. permission (int)  0 for public, 1 for private
;     7. expId - experiment id (int) of the experiment this MadrigalExperimentFile belongs in
;   If no experiement files found, returns a Null Pointer (IDL does not allow empty arrays)
; EXAMPLE: 
;     result = madGetExperimentFiles('http://madrigal.haystack.mit.edu/madrigal', 300041, 0)
;
; $Id: madidl.html 3539 2011-08-26 18:00:05Z brideout $
;
FUNCTION madGetExperimentFiles, madurl,  expId, getNonDefault
   ***********************************************************
;+
; NAME: 
;    madgetexperimentfileparameters(madurl, fullFilename)
;
; PURPOSE: 
;    returns a list of all measured and derivable parameters in file
;
; INPUTS: 
;      madurl - scalar string giving a fully qualified url to the Madrigal site
;      fullFilename - full path to experiment file as returned by madGetExperimentFiles.
;
; OUTPUT: an array of structures representing parameters for that experiment file with the following fields:
;     1. mnemonic (string) Example 'dti'
;     2. description (string) Example: "F10.7 Multiday average observed (Ott)"
;     3. isError (int) 1 if error parameter, 0 if not
;     4. units (string) Example "W/m2/Hz"
;     5. isMeasured (int) 1 if measured, 0 if derivable
;     6. category (string) Example: "Time Related Parameter"
;     7. isSure (int)  1 if parameter can be found for every record, 0 if can only be found for some
;     8. isAddIncrement - 1 if additional increment, 0 if normal, -1 if unknown (this capability
;                        only added with Madrigal 2.5)
; EXAMPLE: 
;     result = madGetExperimentFileParameters('http://madrigal.haystack.mit.edu/madrigal', '/opt/madrigal/blah/mlh980120g.001')
;
; $Id: madidl.html 3539 2011-08-26 18:00:05Z brideout $
;
FUNCTION madGetExperimentFileParameters, madurl,  fullFilename
   ***********************************************************
;+
; NAME: 
;    madsimpleprint(madurl, fullFilename, user_fullname, user_email, user_affiliation)
;
; PURPOSE: 
;    madSimplePrint prints the data in the given file is a simple ascii format.
;
;        madSimplePrint prints only the parameters in the file, without filters or derived
;        parameters.  To choose which parameters to print, to print derived parameters, or
;        to filter the data, use isprint instead.
;
; INPUTS: 
;      madurl - scalar string giving a fully qualified url to the Madrigal site
;      fullFilename - full path to experiment file as returned by madGetExperimentFiles.
;      user_fullname - full name of user making request
;      user_email - email address of user making request
;      user_affiliation - affiliation of user making request
;
; OUTPUT: a structure with following fields:
;     1. parameters - an array of strings giving the mnemonics of the parameters.  Equal to the number of columns in data
;     2. data - a two dimensional array of double.  Number of columns = number of parameters above.  Number of rows
;         is the number of measurments.  Note that Madrigal often contains a number a measurments at the same time
;         (such as when a radar makes different range measurements at the same time), so two or more rows may have the same time.
; EXAMPLE: 
;     result = madSimplePrint('http://madrigal.haystack.mit.edu/madrigal', '/opt/madrigal/blah/mlh980120g.001', $
;                                              'Bill Rideout', 'brideout@haystack.mit.edu', 'MIT')
;
; $Id: madidl.html 3539 2011-08-26 18:00:05Z brideout $
;
FUNCTION madSimplePrint, madurl,  fullFilename, user_fullname, user_email, user_affiliation
   ***********************************************************
;+
; NAME: 
;    madprint(madurl, fullFilename, parms, filters, user_fullname, user_email, user_affiliation, ignore_no_data)
;
; PURPOSE: 
;    madPrint returns a two-dimensional array of doubles based on user-specified parmeters and filters.
;
;       See madSimplePrint to print a file with all data and just those parameters in the file itself.
;
;        madPrint allows you to choose which parameters to print, to print derived parameters, or
;        to filter the data.
;
; INPUTS: 
;      madurl - scalar string giving a fully qualified url to the Madrigal site
;      fullFilename - full path to experiment file as returned by madGetExperimentFiles.
;      parms - Comma delimited string listing requested parameters (no spaces allowed).
;      filters - Space delimited string listing filters desired, as in isprint command (see
;                   http://madrigal.haystack.mit.edu/madrigal/ug_commandLine.html#isprint for details)
;      user_fullname - full name of user making request
;      user_email - email address of user making request
;      user_affiliation - affiliation of user making request
;      ignore_no_data - if 0 (the default), raises error if no data, If 1, ignores no data
;
; OUTPUT:  a two dimensional array of doubles.  Number of columns = number of parameters requested.  Number of rows
;         is the number of measurments.  Note that Madrigal often contains a number a measurments at the same time
;         (such as when a radar makes different range measurements at the same time), so two or more rows may have the same time.
; EXAMPLE: 
;     result = madPrint('http://madrigal.haystack.mit.edu/madrigal', '/opt/madrigal/blah/mlh980120g.001', $
;                                   'year,month,day,hour,min,sec,gdalt,ti', 'filter=gdalt,500,600 filter=ti,1900,2000' $
;                                   'Bill Rideout', 'brideout@haystack.mit.edu', 'MIT')
;
; $Id: madidl.html 3539 2011-08-26 18:00:05Z brideout $
;
FUNCTION madPrint, madurl,  fullFilename, parms, filters, user_fullname, user_email, user_affiliation, ignore_no_data
   ***********************************************************
;+
; NAME: 
;    maddownloadfile, madurl, fullFilename, outputFile, user_fullname, user_email, user_affiliation
;
; PURPOSE: 
;    downloads a Madrigal file in simple ascii format to local computer
;
; INPUTS: 
;      madurl - scalar string giving a fully qualified url to the Madrigal site
;      fullFilename - full path to experiment file as returned by madGetExperimentFiles.
;      outputFile - path to save file locally
;      user_fullname - full name of user making request
;      user_email - email address of user making request
;      user_affiliation - affiliation of user making request
;
; EXAMPLE: 
;     maddownloadfile, 'http://madrigal.haystack.mit.edu/madrigal', '/opt/madrigal/blah/mlh980120g.001', '/tmp/junk.txt', &
;                                   'Bill Rideout', 'brideout@haystack.mit.edu', 'MIT'
;
; $Id: madidl.html 3539 2011-08-26 18:00:05Z brideout $
;
PRO maddownloadfile, madurl,  fullFilename, outputFile, user_fullname, user_email, user_affiliation



   ***********************************************************
   
;+
; NAME: 
;    madglobaldownload, madurl, outputDir, user_fullname, user_email, user_affiliation, startDate, endDate, 
;                          inst, kindats, expName,  fileDesc
;
; PURPOSE
;     madglobaldownload is a procedure to search through the entire Madrigal database
;     for appropriate data files to download to outputDir
;
; INPUTS: 
;
;        madurl - url to homepage of site to be searched (Example: 
;              'http://www.haystack.mit.edu/madrigal/'
;
;        outputDir - the local directory to store the downloaded files in. Must exist.
;                 (Example: '/tmp/isprint.txt')
;
;        user_fullname - the full user name (Example: 'Bill Rideout')
;
;        user email -  Example: 'brideout@haystack.mit.edu'
;
;        user_affiliation - Example: 'MIT'
;
;        startDate - a time in IDL Julian Date format at which to begin the search
;
;        endDate - a time in IDL Julian Date format at which to end the search
;
;        inst - instrument code (integer).  See 
;            http://madrigal.haystack.mit.edu/cgi-bin/madrigal/getMetadata
;            "Instrument Table" for this list. Examples: 30 for Millstone
;            Hill Incoherent Scatter Radar, 80 for Sondrestrom Incoherent 
;            Scatter Radar
;
;    Optional inputs
;
;        kindats - is an optional array of kindat (kinds of data) codes to accept.
;           The default is a null pointer, which will accept all kindats.
;  
;        expName - a case insensitive string as used by strmatch that matches the experiment
;           name.  Default is zero-length string, which matches all experiment names.
;
;        fileDesc - a case insensitive string as used by strmatch that matches the file description.
;           Default is zero-length string, which matches all file descriptions.
;
; OUTPUT: Nothing.
;
;    Affects: Writes downloaded files to outputDir
;    
;
; EXAMPLE: 
;   madglobaldownload, 'http://www.haystack.mit.edu/madrigal/',  $
;                         '/tmp/downloads', $
;                         'Bill Rideout',  'brideout@haystack.mit.edu', 'MIT', $
;                         julday(1,19,1998,0,0,0),  julday(1,21,1998,23,59,59), 30
;
;  $Id: madglobaldownload.pro 4061 2012-08-24 17:16:34Z brideout $
;
pro madglobaldownload, madurl, outputDir, user_fullname, user_email, user_affiliation, startDate, endDate, $
                           inst, kindats, expName,  fileDesc
   ***********************************************************
; globalIsprint is a procedure to search through the entire Madrigal database
; for appropriate data to print in ascii to a file
;
;    Inputs:
;
;        madurl - url to homepage of site to be searched (Example: 
;              'http://www.haystack.mit.edu/madrigal/'
;
;        parms - a comma delimited string listing the desired Madrigal 
;            parameters in mnemonic form.  
;            (Example: 'year,month,day,hour,min,sec,gdalt,dte,te').  
;            Ascii space-separated data will be returned in the same  
;            order as given in this string. See 
;            http://madrigal.haystack.mit.edu/cgi-bin/madrigal/getMetadata
;            "Parameter code table" for all possible parameters.
;
;        output - the local file name to store the resulting ascii data.
;                 (Example: '/tmp/isprint.txt')
;
;        user_fullname - the full user name (Example: 'Bill Rideout')
;
;        user email -  Example: 'brideout@haystack.mit.edu'
;
;        user_affiliation - Example: 'MIT'
;
;        startDate - a time in IDL Julian Date format at which to begin the search
;
;        endDate - a time in IDL Julian Date format at which to end the search
;
;        inst - instrument code (integer).  See 
;            http://madrigal.haystack.mit.edu/cgi-bin/madrigal/getMetadata
;            "Instrument Table" for this list. Examples: 30 for Millstone
;            Hill Incoherent Scatter Radar, 80 for Sondrestrom Incoherent 
;            Scatter Radar
;
;    Optional inputs
;
;        filters - is the optional filters requested in exactly the form given in isprint
;         command line (example = 'filter=gdalt,,500 filter=ti,500,1000')
;         See:  http://www.haystack.mit.edu/madrigal/ug_commandLine.html for details
;
;        kindats - is an optional array of kindat (kinds of data) codes to accept.
;           The default is a null pointer, which will accept all kindats.
;  
;        expName - a case insensitive string as used by strmatch that matches the experiment
;           name.  Default is zero-length string, which matches all experiment names.
;
;        fileDesc - a case insensitive string as used by strmatch that matches the file description.
;           Default is zero-length string, which matches all file descriptions.
;
;    Returns: Nothing.
;
;    Affects: Writes results to output file
;    
;
;  Example: 
;   madglobalprint, 'http://www.haystack.mit.edu/madrigal/',  $
;                         'year,month,day,hour,min,sec,gdalt,dte,te', $
;                         '/tmp/isprint.txt', $
;                         'Bill Rideout',  'brideout@haystack.mit.edu', 'MIT', $
;                         julday(1,19,1998,0,0,0),  julday(1,21,1998,23,59,59), 30
;
;  $Id: madidl.html 3539 2011-08-26 18:00:05Z brideout $
;
pro madglobalprint, madurl, parms, output, user_fullname, user_email, user_affiliation, startDate, endDate, $
                           inst, filters, kindats, expName,  fileDesc
   ***********************************************************
;+
; NAME: 
;    madcalculator(madurl, year,  month, day, hour, min, sec, startLat, endLat, stepLat,
;                             startLong, endLong, stepLong, startAlt, endAlt, stepAlt,  parms)
;
; PURPOSE: 
;    madcalculator calculates requested derived parameter for a given time and grid of spatial points
;
; INPUTS: 
;      madurl - scalar string giving a fully qualified url to the Madrigal site
;      year,  month, day, hour, min, sec - time at which to run calculation
;      startLat - Starting geodetic latitude, -90 to 90 (float)
;      endLat - Ending geodetic latitude, -90 to 90 (float)
;      stepLat - Latitude step (0.1 to 90) (float)
;      startLong - Starting geodetic longitude, -180 to 180  (float)
;      endLong - Ending geodetic longitude, -180 to 180 (float)
;      stepLong - Longitude step (0.1 to 180) (float)
;      startAlt - Starting geodetic altitude, >= 0 (float)
;      endAlt - Ending geodetic altitude, > 0 (float)
;      stepAlt - Altitude step (>= 0.1) (float)
;      parms - comma delimited string of Madrigal parameters desired
;
; OUTPUT: a two dimensional array of doubles.  Number of columns = 3 + number of parameters requested, because the
;                first three parameters are always geodetic latitude, longitude, and altitude.  Number of rows
;                is the number of latitudes * number of longitudes * number of altitudes.
; EXAMPLE: 
;     result = madcalculator(1999,2,15,12,30,0,45,55,5,-170,-150,10,200,200,0,'bmag,bn')
;
; $Id: madidl.html 3539 2011-08-26 18:00:05Z brideout $
;
FUNCTION madcalculator, madurl, year,  month, day, hour, min, sec, startLat, endLat, stepLat,  $
                       startLong, endLong, stepLong, startAlt, endAlt, stepAlt,  parms

Example IDL code

  ; Simple example of all methods in the remote IDL interface to Madrigal (madidl)
;
; $Id: madidl.html 3539 2011-08-26 18:00:05Z brideout $

PRO example_madidl

     ; these are the instrument codes for the Millstone ISR.  Run the function madGetAllInstruments
     ; to see a list of all instruments available in Madrigal
     instList = [30,31,32]
     
     ; run this test for two different sites (Haystack and SRI), using the test files in each site
     siteList = ['http://madrigal.haystack.mit.edu/madrigal', $
                      'http://isr.sri.com/madrigal/' ]
     for i=0,  n_elements(siteList)-1  do begin
         site = siteList[i]
         print, 'Testing Madrigal url ', site

         ; the highest level of Madrigal data is the instrument - here we get a list of all available instruments
         ; See madgetallinstruments.pro for a definition of the instrument structure returned in an array
         result = madGetAllInstruments(site)
         print, 'madGetAllInstruments returned', result
         print, ""
         
         ; the next level of Madrigal is the experiment.  An experiment has only one instrument associated with
         ; it.  In this case, we ask for all experiments in Jan 1998 for any Millstone ISR radar.  sonce this
         ; is a standard test file, its found at both Millstone and SRI
         ; See madgetexperiments.pro for a definition of the experiment structure returned in an array
         result = madGetExperiments(site, instList, 1998,1,19,0,0,0, 1998,1,20,23,59,59,1)
         print, 'madGetExperiments returned ', result
         print, ""
         ; verify at least one returned - note a null pointer is returned since IDL does not support empty arrays
         resultSize = size(result)
         if (resultSize[1] eq 10) then continue ; no experiments found
         
         ; next we get an array of files in that experiment.  See madgetexperimentfiles.pro for a definition 
         ; of the experimentfile structure returned in an array
         expId = long64(result[0].strid)
         result = madGetExperimentFiles(site, expid, 1)
         print, 'madGetExperimentFiles returned ', result
         print, ""
         ; verify at least one returned - note a null pointer is returned since IDL does not support empty arrays
         resultSize = size(result)
         if (resultSize[1] eq 10) then continue ; no experiment files found
         
         ; next we want to get a list of all parameters.  Note Madrigal allows access to both parameters in the file
         ; (called measured parameters) and parameters derivable form the measured ones (called derived parameters).
         ; See madgetexperimentfileparameters.pro for a definition of the parmeter structure returned in an array
         fullFilename = result[0].name
         result = madGetExperimentFileParameters(site, fullFilename)
         print, 'madGetExperimentFileParameters returned ', result
         print, ""
         
         ; the next method effectively downloads the file into an IDL 2-D array.  It only downloads measured parameters
         ; in the file.  See more advanced method madPrint to choose measured or derived parameters and/or to filter data.
         ; See madsimpleprint.pro for details of how the data is returned
         result = madSimplePrint(site, fullFilename,  'Bill Rideout', 'brideout@haystack.mit.edu', 'MIT')
         print, 'size of data returned by madSimplePrint is ', size(result.data)
         print, 'madSimplePrint parameters are ', result.parameters
         print, ""
         
         ; the next method is similar to madsimpleprint, except the user chooses which measured or derived parameters 
         ; to download, and can filter data using the filter string as discussed in 
         ;    http://madrigal.haystack.mit.edu/madrigal/ug_commandLine.html#isprint
         ; See madprint.pro for details of how the data is returned
         result = madPrint(site, fullFilename,  'year,month,day,hour,min,sec,gdalt,ti', 'filter=recno,5,6 filter=ti,1000,', $
                                         'Bill Rideout', 'brideout@haystack.mit.edu', 'MIT')
         print, 'size of data returned by madPrint is ', size(result)
         print, ""
         
         ; the next method is similar to madSimplePrint, except that it downloads the data from
         ; a file into a simple column delimited formated file on your local computer
         maddownloadfile, site, fullFilename, '/tmp/junk5.txt', 'Bill Rideout',  'brideout@haystack.mit.edu',  'MIT'
         print, 'file has been downloaded to /tmp/junk5.txt'
         
         ; The next method is used to run the Madrigal derivation engine for any random time and array of geodetic points in space
         ; In other words, things such as Magnetic field and geophysicals indices can be calculated without a Madrigal file.
         ; See madCalculator.pro for details of how the data is returned
         result = madCalculator(site, 1999,2,15,12,30,0,45,55,5,-170,-150,10,200,200,0,'bmag,bn')
         print, 'madCalculator returned ', result
         print, ""
         
         ; The final procedure globalPrint allows you to get data from multiple files in a particular Madrigal datbase
         ; at once.  Since it can return very large data sets, it writes to a file rather than an IDL array
         madglobalPrint, site,  'year,month,day,hour,min,sec,gdalt,dte,te', $
                               '/tmp/isprint.txt', 'Bill Rideout',  'brideout@haystack.mit.edu',  'MIT', $
                               julday(1,19,1998,0,0,0),  julday(1,21,1998,23,59,59), 30, $
                               '', ptr_new(), '*world*'
                              
           
                               
      endfor ; next Madrigal site


      ; an example of getting data from a site from a different Madrigal site - search for Poker Flat data from Millstone site
      site = 'http://madrigal.haystack.mit.edu/madrigal'
      instList = [61] ; Poker Flat ISR
      result = madGetExperiments(site, instList, 2008,4,1,0,0,0, 2008,4,30,23,59,59,0)
      print, 'madGetExperiments returned ', result
      ; handle a non-local experiment
      if (result[0].isLocal eq 0) then begin
          site = result[0].madrigalUrl
          ; call madGetExperiments again with local url
          result = madGetExperiments(site, instList, 2008,4,1,0,0,0, 2008,4,30,23,59,59,0)
      endif
      expId = long64(result[0].strid)
      result = madGetExperimentFiles(site, expid, 1)
      print, 'madGetExperimentFiles returned ', result
      print, ""
      
END

  
Previous: Remote access using python   Up: Remote access programming tutorial toc   Next: Remote access reference toc