Previous: Madrigal web services   Up: Remote access programming tutorial toc   Next: Remote access using python

Remote access using Matlab


This page describes the remote Matlab API, and gives two examples of using this API. The first example uses all the basic methods, and outputs text. The second example creates a simple pcolor plot.

The remote Matlab API is organized in the same way as the Madrigal data model, from Instrument at the highest level, down to the level of data values. Readers who are not familiar with the Madrigal data model should read the material in that section before proceeding with this tutorial.

This API and example have been tested on both Windows and Linux, and require only access to the internet and Matlab 5 or greater to run. It is available for download here.

Remote Matlab methods

A good way to learn how to use this Matlab API is to run this example.

   ***********************************************************
function cgiUrl = getMadrigalCgiUrl(url)
  getMadrigalCgiUrl  	parse the main madrigal page to get the cgi url

  input: url to Madrigal

  output: cgi url for that Madrigal Site

  Note: parses the homepage for the accessData link
   ***********************************************************
function instArray = getInstrumentsWeb(cgiurl)
  getInstrumentsWeb  	returns an array of instrument structs of instruments found on remote Madrigal server.

  inputs:  cgiurl (string) to Madrigal site cgi directory 
      (Example: 'http://www.haystack.mit.edu/cgi-bin/madrigal/') 
      Note that method getMadrigalCgiUrl converts homepage url into cgiurl. 

  output:
     instArray - array of instrument structs found

       instrument struct has the fields:

     instrument.name (string) Example: 'Millstone Hill Incoherent Scatter Radar'
     instrument.code (int) Example: 30
     instrument.mnemonic (3 char string) Example: 'mlh'
     instrument.latitude (double) Example: 45.0
     instrument.longitude (double) Example: 110.0
     instrument.altitude (double)  Example: 0.015 (km) 

  Raises error if unable to return instrument array
 
  Example:
  getInstrumentsWeb('http://www.haystack.mit.edu/cgi-bin/madrigal/')
   ***********************************************************
function expArray = getExperimentsWeb(cgiurl, instCodeArray, starttime, endtime, localFlag)
  getExperimentsWeb  	returns an array of experiment structs given input filter arguments from a remote Madrigal server.

  Inputs:

      1. cgiurl (string) to Madrigal site cgi directory 
        (Example: 'http://www.haystack.mit.edu/cgi-bin/madrigal/') 
         Note that method getMadrigalCgiUrl converts homepage url into cgiurl. 
 
      2. instCodeArray - a 1 X N array of ints containing selected instrument codes.  Special value of 0 selects all instruments.
 
      3. starttime - Matlab datenum double (must be UTC)
 
      4. endtime - Matlab datenum double (must be UTC)

      5. localFlag - 1 if local experiments only, 0 if all experiments 

   Return array of Experiment struct (May be empty):
   
   experiment.id (int) Example: 10000111
   experiment.url (string) Example: 'http://www.haystack.mit.edu/cgi-bin/madtoc/1997/mlh/03dec97'
   experiment.name (string) Example: 'Wide Latitude Substorm Study'
   experiment.siteid (int) Example: 1
   experiment.sitename (string) Example: 'Millstone Hill Observatory'
   experiment.instcode (int) Code of instrument. Example: 30
   experiment.instname (string) Instrument name. Example: 'Millstone Hill Incoherent Scatter Radar'
   experiment.starttime (double) Matlab datenum of experiment start
   experiment.endtime (double) Matlab datenum of experiment end
   experiment.isLocal (int) 1 if local, 0 if not 
   experiment.madrigalUrl (string) - home url of Madrigal site with this
       experiment. Example 'http://madrigal.haystack.mit.edu/madrigal'

  Raises error if unable to return experiment array
 
  Example: expArray = getExperimentsWeb('http://www.haystack.mit.edu/cgi-bin/madrigal/', ...
                                         30, datenum('01/01/1998'), datenum('12/31/1998'), 1);

   Note that if the returned
   experiment is not local, the experiment.id will be -1.  This means that you
   will need to call getExperimentsWeb a second time with the cgiurl of the 
   non-local experiment (getCgiurlForExperiment(experiment.madrigalUrl)).  This is because 
   while Madrigal sites share metadata about experiments, the real experiment ids are only
   known by the individual Madrigal sites.  See testMadmatlab.m
   for an example of this.
   ***********************************************************
function cgiurl = getCgiurlForExperiment(experiment)
  getCgiurlForExperiment  	returns cgiurl of experiment.url as returned by getExperimentsWeb.

  inputs:  experiment struct as returned by getExperimentsWeb or getExperiments.


  output:
     cgiurl of experiment

  Simply truncates experiment.url to remove /madtoc///
 
  Example: If expArray is the value returned in the getExperimentsWeb example, and
           expArray(1).url = 'http://madrigal.haystack.mit.edu/cgi-bin/madrigal/madtoc/1998/mlh/07jan98', then

     getCgiurlForExperiment(expArray(1))

     returns: 

          'http://madrigal.haystack.mit.edu/cgi-bin/madrigal/'
   ***********************************************************
function expFileArray = getExperimentFilesWeb(cgiurl, experimentId)
  getExperimentFilesWeb  	returns an array of experiment file structs given experiment id from a remote Madrigal server.

  Note that it is assumed that experiment is local to cgiurl.  If not,
  empty list will be returned. Use getCgiurlForExperiment to get the correct
  cgiurl for any given experiment struct.

  Inputs:

      1. cgiurl (string) to Madrigal site cgi directory that has that
      experiment.
        (Example: 'http://www.haystack.mit.edu/cgi-bin/madrigal/') 
         Note that method getMadrigalCgiUrl converts homepage url into cgiurl. 
 
      2. experiment id (int) as returned by getExperiments or
         getExperimentsWeb

   Return array of Experiment File struct (May be empty):
   
   file.name (string) Example '/opt/mdarigal/blah/mlh980120g.001'
   file.kindat (int) Kindat code.  Example: 3001
   file.kindatdesc (string) Kindat description: Example 'Basic Derived Parameters'
   file.category (int) (1=default, 2=variant, 3=history, 4=real-time)
   file.status (string)('preliminary', 'final', or any other description)
   file.permission (int)  0 for public, 1 for private 

  Raises error if unable to return experiment file array
 
  Example: expFileArray =
  getExperimentFilesWeb('http://www.haystack.mit.edu/cgi-bin/madrigal/', 10001686);
   ***********************************************************
function parmArray = getParametersWeb(cgiurl, filename)
  getParametesWeb  	returns an array of parameter structs given filename from a remote Madrigal server.

  Note that it is assumed that filename is local to cgiurl.  If not,
  empty list will be returned.

  Inputs:

      1. cgiurl (string) to Madrigal site cgi directory that has that
      filename.
        (Example: 'http://www.haystack.mit.edu/cgi-bin/madrigal/') 
         Note that method getMadrigalCgiUrl converts homepage url into cgiurl. 
 
      2. filename (string) as returned by getExperimentFiles or
         getExperimentFilesWeb

   Return array of Parameter struct:
   
       parameter.mnemonic (string) Example 'dti'
       parameter.description (string) Example:
          "F10.7 Multiday average observed (Ott) - Units: W/m2/Hz"
       parameter.isError (int) 1 if error parameter, 0 if not
       parameter.units (string) Example "W/m2/Hz"
       parameter.isMeasured (int) 1 if measured, 0 if derivable
       parameter.category (string) Example: "Time Related Parameter" 
       parameter.isSure (int) 1 if can be found for all records, 0 if only
           for some records (implies not all records have same measured
           parameters)

  Raises error if unable to return parameter array
 
  Example: parmArray = getParametersWeb('http://www.haystack.mit.edu/cgi-bin/madrigal/', ...
                                        '/opt/madrigal/experiments/1998/mlh/07jan98/mil980107g.001')
   ***********************************************************
function records = isprintWeb(cgiUrl, file, parms, user_fullname, user_email, user_affiliation, filters, missing, assumed, knownbad)
  isprintWeb  	Create an isprint-like 3D array of doubles via a command similar to the isprint command-line application, but access data via the web
  
  The calling syntax is:
  
  		[records] = isprintWeb(cgiurl, file, parms, user_fullname, user_email, user_affiliation, [filters, [missing, [assumed, [knownbad] ] ] ])
  
   where 

     cgiurl (string) to Madrigal site cgi directory that has that
      filename.
        (Example: 'http://www.haystack.mit.edu/cgi-bin/madrigal/') 
         Note that method getMadrigalCgiUrl converts homepage url into cgiurl.
  
     file is path to file
         (example = '/home/brideout/data/mlh980120g.001')
  
     parms is the desired parameters in the form of a comma-delimited
         string of Madrigal mnemonics (example = 'gdlat,ti,dti')

     user_fullname - is user name (string)

     user_email - is user email address (string)

     user_affiliation - is user affiliation (string) 
  
     filters is the optional filters requested in exactly the form given in isprint
         command line (example = 'time1=15:00:00 date1=01/20/1998 
                       time2=15:30:00 date2=01/20/1998 filter=ti,500,1000')
         See:  http://www.haystack.mit.edu/madrigal/ug_commandLine.html for details

     missing is an optional double to represent missing values.  Defaults to NaN

     assumed is an optional double to represent assumed values.  Defaults to NaN

     knownbad is an optional double to represent knownbad values.  Defaults to NaN

     The returned records is a three dimensional array of double with the dimensions:

         [Number of rows, number of parameters requested, number of records]

     If error or no data returned, will return error explanation string instead.

   Example: data = isprintWeb('http://www.haystack.mit.edu/cgi-bin/madrigal/', ...
                                '/opt/madrigal/experiments/1998/mlh/07jan98/mil980107g.001', ...
                                'gdlat,ti,dti', ...
                                'Bill Rideout', 'wrideout@haystack.mit.edu', 'MIT');

   For now avoids limits with urlread by only asking for maxRecs records at once.  Repeatedly
   calls isprintUnguarded, which is a method without any additional filtering.

    $Id: rt_matlab.html 4103 2012-11-13 21:40:59Z brideout $
   ***********************************************************
function records = isprintUnguarded(cgiUrl, file, parms, user_fullname, user_email, user_affiliation, filters, missing, assumed, knownbad)
  isprintUnguarded - Create an isprint-like 3D array of doubles via a command similar to the isprint command-line application, but access data via the web

  This method differs from isprintWeb in that no protection against
  downloading too much data is provided.  If too much data is requested,
  urlread will fail.
  
  The calling syntax for this method is:
  
  		[records] = isprintUnguarded(cgiurl, file, parms, user_fullname, user_email, user_affiliation, [filters, [missing, [assumed, [knownbad] ] ] ])
  
   where 

     cgiurl (string) to Madrigal site cgi directory that has that
      filename.
        (Example: 'http://www.haystack.mit.edu/cgi-bin/madrigal/') 
         Note that method getMadrigalCgiUrl converts homepage url into cgiurl.
  
     file is path to file
         (example = '/home/brideout/data/mlh980120g.001')
  
     parms is the desired parameters in the form of a comma-delimited
         string of Madrigal mnemonics (example = 'gdlat,ti,dti')

     user_fullname - is user name (string)

     user_email - is user email address (string)

     user_affiliation - is user affiliation (string) 
  
     filters is the optional filters requested in exactly the form given in isprint
         command line (example = 'time1=15:00:00 date1=01/20/1998 
                       time2=15:30:00 date2=01/20/1998 filter=ti,500,1000')
         See:  http://www.haystack.mit.edu/madrigal/ug_commandLine.html for details

     missing is an optional double to represent missing values.  Defaults to NaN

     assumed is an optional double to represent assumed values.  Defaults to NaN

     knownbad is an optional double to represent knownbad values.  Defaults to NaN

     The returned records is a three dimensional array of double with the dimensions:

         [Number of rows, number of parameters requested, number of records]

     If error or no data returned, will return error explanation string instead.

   Example: data = isprintUnguarded('http://www.haystack.mit.edu/cgi-bin/madrigal/', ...
                                '/opt/madrigal/experiments/1998/mlh/07jan98/mil980107g.001', ...
                                'gdlat,ti,dti', ...
                                'Bill Rideout', 'wrideout@haystack.mit.edu', 'MIT');

   Unlike isprintWeb, no effort is made to protect the user from
   requesting too more data than the Matlab method urlread can handle

    $Id: rt_matlab.html 4103 2012-11-13 21:40:59Z brideout $
   ***********************************************************
function  result  = madDownloadFile(cgiurl,  fullFilename, outputFile, format )
madDownloadFile downloads a Madrigal file to local computer in various
formats

    The calling syntax is:
       result  = madDownloadFile(cgiurl,  fullFilename, outputFile, [ format ] )
   
 Inputs: 

      1. cgiurl (string) to Madrigal site cgi directory 
        (Example: 'http://www.haystack.mit.edu/cgi-bin/madrigal/') 
         Note that method getMadrigalCgiUrl converts homepage url into
         cgiurl. 

      2. fullFilename - file to download as returned by getExperimentFilesWeb.m

      3. outputFile - name to save new file to

      4. format - one of the following strings:
          'simple', 'cedar_ascii'
              'simple' is the default if not specified, and
              is recommended for all but the most advanced users
              because it is a simple ascii space delimited column 
              format, and is the easiest to parse.  See Cedar
              standard for description of cedar_ascii

    $Id: rt_matlab.html 4103 2012-11-13 21:40:59Z brideout $

   ***********************************************************
function record = madCalculatorWeb(cgiUrl, ...
                                   time, ...
                                   startLat, ...
                                   endLat, ...
                                   stepLat, ...
                                   startLong, ...
                                   endLong, ...
                                   stepLong, ... 
                                   startAlt, ...
                                   endAlt, ...
                                   stepAlt, ...
                                   parms)
  madCalculatorWeb  	Create a matrix of doubles via a the Madrigal derivation engine for a time and range of lat, long, and alt
  
  The calling syntax is:
  
  		[record] = madCalculatorWeb(cgiurl, time, startLat, endLat, stepLat, startLong, endLong, stepLong, 
                                   startAlt, endAlt, stepAlt, parms)
  
   where 

     cgiurl (string) to Madrigal site cgi directory that has that
      filename.
        (Example: 'http://www.haystack.mit.edu/cgi-bin/madrigal/') 
         Note that method getMadrigalCgiUrl converts homepage url into cgiurl.
  
     time - Matlab datenum double (must be UTC)	7. startLat - Starting geodetic latitude, -90 to 90 (required)
     
     endLat - Ending geodetic latitude, -90 to 90 (required)
     
     stepLat - Latitude step (0.1 to 90) (required)
     
     startLong - Starting geodetic longitude, -180 to 180 (required)
     
     endLong - Ending geodetic longitude, -180 to 180 (required)
     
     stepLong - Longitude step (0.1 to 180) (required)
     
     startAlt - Starting geodetic altitude, >= 0 (required)
     
     endAlt - Ending geodetic altitude, > 0 (required)
     
     stepAlt - Altitude step (>= 0.1) (required)
  
     parms is the desired parameters in the form of a comma-delimited
         string of Madrigal mnemonics (example = 'gdlat,ti,dti')
  

     The returned record is a matrix of doubles with the dimensions:

         [(num lat steps * num long steps * num alt steps), 3 + num of parms]

     The first three columns will always be lat, long, and alt, so there are three
     additional columns to the number of parameters requested via the parms argument.

     If error or no data returned, will return error explanation string instead.

     Example: result = madCalculatorWeb('http:/grail.haystack.mit.edu/cgi-bin/madrigal', ...
                                         now,45,55,5,45,55,5,200,300,50,'bmag,bn');
   ***********************************************************
function [] = globalIsprint(url, ...
    parms, ...
    output, ...
    user_fullname, ...
    user_email, ...
    user_affiliation, ...
    startTime, ...
    endTime, ...
    inst, ...
    filters, ...
    kindats, ...
    expName, ...
    fileDesc)
 globalIsprint is a script to search through the entire Madrigal database
 for appropriate data to print in ascii to a file

    Inputs:

        url - url to homepage of site to be searched (Example: 
              'http://www.haystack.mit.edu/madrigal/'

        parms - a comma delimited string listing the desired Madrigal 
            parameters in mnemonic form.  
            (Example: 'year,month,day,hour,min,sec,gdalt,dte,te').  
            Ascii space-separated data will be returned in the same  
            order as given in this string. See 
            http://madrigal.haystack.mit.edu/cgi-bin/madrigal/getMetadata
            "Parameter code table" for all possible parameters.

        output - the local file name to store the resulting ascii data.
                 (Example: '/tmp/isprint.txt')

        user_fullname - the full user name (Example: 'Bill Rideout')

        user email -  Example: 'brideout@haystack.mit.edu'

        user_affiliation - Example: 'MIT'

        startTime - a Matlab time to begin search at. Example:
                    datenum('20-Jan-1998 00:00:00') Time in UT

        endTime - a Matlab time to end search at. Example:
                  datenum('21-Jan-1998 23:59:59') Time in UT

        inst - instrument code (integer).  See 
            http://madrigal.haystack.mit.edu/cgi-bin/madrigal/getMetadata
            "Instrument Table" for this list. Examples: 30 for Millstone
            Hill Incoherent Scatter Radar, 80 for Sondrestrom Incoherent 
            Scatter Radar

    Optional inputs

        filters - is the optional filters requested in exactly the form given in isprint
         command line (example = 'filter=gdalt,,500 filter=ti,500,1000')
         See:  http://www.haystack.mit.edu/madrigal/ug_commandLine.html for details

        kindats - is an optional array of kindat (kinds of data) codes to accept.
           The default is an empty array, which will accept all kindats.
  
        expName - a case insensitive regular expression that matches the experiment
           name.  Default is zero-length string, which matches all experiment names.

        fileDesc - a case insensitive regular expression that matches the file description.
           Default is zero-length string, which matches all file descriptions.

    Returns: Nothing.

    Affects: Writes results to output file

        

  Example: globalIsprint('http://www.haystack.mit.edu/madrigal/', ...
                         'year,month,day,hour,min,sec,gdalt,dte,te', ...
                         '/tmp/isprint.txt', ...
                         'Bill Rideout', ...
                         'brideout@haystack.mit.edu', ...
                         'MIT', ...
                         datenum('20-Jan-1998 00:00:00'), ...
                         datenum('21-Jan-1998 23:59:59'), ...
                         30);

  $Id: rt_matlab.html 4103 2012-11-13 21:40:59Z brideout $
  
   ***********************************************************


function [] = globalDownload(url, ...
    outputDir, ...
    user_fullname, ...
    user_email, ...
    user_affiliation, ...
    format, ...
    startTime, ...
    endTime, ...
    inst, ...
    kindats, ...
    expName, ...
    fileDesc)
globalDownload is a script to search through the entire Madrigal database
for appropriate files in ascii or hdf5 format to store locally
 
     Inputs:
 
         url - url to homepage of site to be searched (Example: 
               'http://www.haystack.mit.edu/madrigal/'
 
         outputDir - the local directory to store the downloaded files.
                  (Example: '/tmp/isprint.txt')
 
         user_fullname - the full user name (Example: 'Bill Rideout')
 
         user email -  Example: 'brideout@haystack.mit.edu'
 
         user_affiliation - Example: 'MIT'
 
         format - either "ascii" or "hdf5"  Ascii is simple column
         delimited acsii
 
         startTime - a Matlab time to begin search at. Example:
                     datenum('20-Jan-1998 00:00:00') Time in UT
 
         endTime - a Matlab time to end search at. Example:
                   datenum('21-Jan-1998 23:59:59') Time in UT
 
         inst - instrument code (integer).  See 
             http://madrigal.haystack.mit.edu/cgi-bin/madrigal/getMetadata
             "Instrument Table" for this list. Examples: 30 for Millstone
             Hill Incoherent Scatter Radar, 80 for Sondrestrom Incoherent 
             Scatter Radar
 
     Optional inputs
 
         kindats - is an optional array of kindat (kinds of data) codes to accept.
            The default is an empty array, which will accept all kindats.
   
         expName - a case insensitive regular expression that matches the experiment
            name.  Default is zero-length string, which matches all experiment names.
 
         fileDesc - a case insensitive regular expression that matches the file description.
            Default is zero-length string, which matches all file descriptions.
 
     Returns: Nothing.
 
     Affects: Writes downloaded files to outputDir
 
         
 
   Example: globalDownload('http://www.haystack.mit.edu/madrigal/', ...
                          '/tmp/download', ...
                          'Bill Rideout', ...
                          'brideout@haystack.mit.edu', ...
                          'MIT', ...
                          'ascii', ...
                          datenum('20-Jan-1998 00:00:00'), ...
                          datenum('21-Jan-1998 23:59:59'), ...
                          30);
 
   ***********************************************************

Example Matlab code use this API

  % demo program of madmatlab running on a pc or linux

madurl = 'http://madrigal.haystack.mit.edu/madrigal';

cgiurl = getMadrigalCgiUrl(madurl)

'List all instruments, and their latitudes and longitudes:'
instArray = getInstrumentsWeb(cgiurl);
for i = 1:length(instArray)
    [s,errmsg] = sprintf('Instrument: %s, at lat %f and long %f', ...
            instArray(i).name, ...
            instArray(i).latitude, ...
            instArray(i).longitude);
     s
end
% now list all experiments from local Madrigal site with mlh (code 30) in
% 1998 - should be data if default files installed...
startdate = datenum('01/01/1998');
enddate = datenum('12/31/1998');
expArray = getExperimentsWeb(cgiurl, 30, startdate, enddate, 1);
for i = 1:length(expArray)
    [s,errmsg] = sprintf('Experiment name: %s, at url %s and id %i', ...
            expArray(i).name, ...
            expArray(i).url, ...
            expArray(i).id);
     s
end

% now list all files in the first experiment
expFileArray = getExperimentFilesWeb(cgiurl, expArray(1).id);
for i = 1:length(expFileArray)
    [s,errmsg] = sprintf('File name: %s, with kindat %i', ...
            expFileArray(i).name, ...
            expFileArray(i).kindat);
     s
end
% now first 2 parameters in the last file
parmArray = getParametersWeb(cgiurl, expFileArray(end).name)
for i = 1:10
    [s,errmsg] = sprintf('Parameter mnemonic: %s, description "%s" -- isMeasured = %i', ...
            parmArray(i).mnemonic, ...
            parmArray(i).description, ...
            parmArray(i).isMeasured);
     s
end
%  run isprintWeb for that file for first two parameters
parmStr = sprintf('%s,%s', parmArray(1).mnemonic, parmArray(2).mnemonic);
data = isprintWeb(cgiurl, expFileArray(1).name, parmStr, 'Bill Rideout', 'wrideout@haystack.mit.edu', 'MIT');
% print first 10 records
data(:,:,1:10)

% download that data file in simple format
result = madDownloadFile(cgiurl, expFileArray(1).name, '/tmp/junk.txt');
'downloaded file to /tmp/junk.txt'

% run globalIsprint, which can gather data from multiple files at once
globalIsprint('http://www.haystack.mit.edu/madrigal/', ...
              'year,month,day,hour,min,sec,gdalt,dte,te', ...
              '/tmp/isprint.txt', ...
              'Bill Rideout', 'brideout@haystack.mit.edu', 'MIT', ...
              datenum('20-Jan-1998 00:00:00'), datenum('21-Jan-1998 23:59:59'), 30);
'globalIsprint output saved to /tmp/isprint.txt'

% madCalculatorWeb runs the Madrigal derivation engine for any point
data = madCalculatorWeb(cgiurl, datenum(1999,2,15,12,30,0), 45,55,5,-170,-150,10,200,200,0,'sdwht,kp');
'madCalculator output'
% print data
data

'The following is an example of searching for non-local experiments'
% 61 is the instrument id of Poker Flat ISR - so this will return an
% experiment not from the Millstone Madrigal site
startdate = datenum('04/01/2008');
enddate = datenum('04/30/2008');
expArray = getExperimentsWeb(cgiurl, 61, startdate, enddate, 0);

% calling this now would fail: expFileArray = getExperimentFilesWeb(cgiurl, expArray(1).id);
% Instead, get the cgiurl of the non-local experiment
if (expArray(1).isLocal == 0)
    cgiurl = getMadrigalCgiUrl(expArray(1).madrigalUrl)
    expArray = getExperimentsWeb(cgiurl, 61, startdate, enddate, 0);
end

% now you can get the files
expFileArray = getExperimentFilesWeb(cgiurl, expArray(1).id);
for i = 1:length(expFileArray)
    [s,errmsg] = sprintf('File name: %s, with kindat %i', ...
            expFileArray(i).name, ...
            expFileArray(i).kindat);
     s
end

          
  
Previous: Madrigal web services   Up: Remote access programming tutorial toc   Next: Remote access using python