|
|
|
What's new in Madrigal 3.0? | Doc home | Madrigal home |
With Madrigal 3.1.1, the code is compatable with https. Instructions are given to install or convert Madrigal to https.
With Madrigal 3.1, all python code has been updated to use python 3 instead of python 2. The functionality is unchanged from Madrigal 3.0.
Migration to Hdf5 file format
With Madrigal 3.0, the old 16 bit integer based file format has been replaced with Hdf5. However, all the data model and well-defined parameters have been retained. The CEDAR Hdf5 format is designed to be completely self-contained. That is, any user with an Hdf5 reader can fully understand all data in the file without any reference to documentation. A full description of the format can be found here.
Web interface generates scripts to download any amount of data
With Madrigal 3, you can generate a script command to download a whole series of files with a few click. You can also create a script that will filter those same files and allow you to choose the parameters you want, again with just a few clicks. These scripts can be run with python, Matlab, or IDL.
Download files in a new format
Prior to Madrigal 3, files could be downloaded only as ascii or Hdf5 (or the difficult to understand CEDAR format). Now they can be downloaded as ascii, Hdf5, or netCDF4.
Get data with selected/derived parameters and filters in new formats
Prior to Madrigal 3, data with selected/derived parameters and filters was available only as ascii. Now it can be downloaded as ascii, Hdf5, or netCDF4.
Independent spatial parameters now built in to data model
Prior to Madrigal 3.0, there was no way to automatically tell what the independent spatial parameters were in vector data. With Madrigal 3.0, any data with vector data must define its independent spatial parameters, which can be found in the Hdf5 Metadata group. This allows Madrigal to automatically add array layouts of the data, making for easier plotting.
All new web interface
A much simplified web interface based on Django and bootstrap. Developed with much assistance from Jicamarca Observatory and Jose Antonio Sal y Rosas Celi.
Simple FTP-like web interface added
Designed for non-native English speakers. The url follows a very simple pattern that can be easily parsed if the user in unable to understand the Madrigal API's and automatically generated scripts.
Create CEDAR Hdf5 files with either Matlab or python
Prior to Madrigal 3, there was only a python API to easily create Madrigal files. Now both Matlab and python can be used to create CEDAR Hdf5 files.
All metadata easily accessible
The Access metadata menu item on the main navigation menu allows easy access to all Madrigal CEDAR metadata, including Madrigal sites, instruments, parameter definitions, and kind of data codes.
Added support to file download cgi scripts to allow logging of remote API requests to download files.
Added popup javascript window to warn users not to click "Download File" button multiple times.
Small bug fixes in 3 cgi scripts affecting plots: madExperiment.cgi, madDataDisplay, getMadplot.py
Small bug fix involving creating Hdf5 export files in data.py.
Small bug fix involving a memory leak in _Madrec.c and a rare problem creating Hdf5 export files in data.py.
New simple local web user interface
A new easy to use interface to access local Madrigal data was developed by Jicamarca with support from Millstone Hill staff. This interface in now very robust and has been tested with data from all Madrigal sites.
HDF5 file download availablity
Users can now download data files as HDF5, in addition to simple column delimited ascii and the more complex CEDAR formats. This is also the result of a Jicamarca/Millstone Hill collaborations.
Users can register interest in experiments or instruments
Users can now register interest in experiments or instruments, and get emails whenever that Madrigal experiment or instrument is updated.
Administrative improvements - can now add external disks to experiments
It is now easy to expand the Madrigal database by mounting additional hard disks to make more room for data.
New metadata - experiment PI's and analyst
Experiments now have direct email links to the experiment PI. These PI's can be updated on an experiment or intrument basis.
Global seach now more robust
The global search UI now generates scripts so you can run the global search right from your computer using either the python remote API, the Matlab remote API, or the IDL remote API.
A bug fix release - no new features.
This minor revision to Madrigal 2.5 was needed to support using Madrigal as an archiving database. The experiment metadata was expanded to allow archived experiments. Archived experiments are not shared between Madrigal sites, but are visible as local experiments.
Simplification of Web User Interface
Both the Browse for Individual Madrigal Experiments and the Global Madrigal Database Report web interface have been simplified. Searching for instruments under Browse for Individual Madrigal Experiments is now easier through the use of an instrument category selector.
One step file printing available
Under Browse for Individual Madrigal Experiments, users can now choose to print an ascii version of any Madrigal file with one click. With this option they can not include any derived parameters or data filters.
Installation simplified
Autotools is now used to compile all code, significantly reducing the number of parameters in the madrigal.cfg configuration file.
64-bit tested
Madrigal has now been fully tested as a 64-bit application. It is important that all Madrigal installations switch to 64-bit machines by the year 2037, because 32-bit unix cannot handle dates beyond then.
International Reference Ionosphere (IRI) derived parameters now available
All parameters calculated by the International Reference Ionosphere (IRI) model can now be selected as derived parameters.
Additional automatic sharing of metadata added
For administrators: Now when new sites or instruments are added to Madrigal, these metadata files are automatically added to your site.
Experiment level security
Previously, individual Madrigal files could be made public or private. Now entire experiments can be made public, private, or hidden altogether. See the script changeExpStatus.py for details.
Experiment directory naming convention modified
Previous to Madrigal 2.5, all Madrigal experiments had to be stored in directory paths in the form:
$MADROOT/experiments/YYYY/<3 letter inst mnemonic>/ddMMMyy[letter], Example: /opt/madrigal/experiments/1998/mlh/20jan98b.Under this convention, the date of the directory name represented the start date of the experiment, with one letter optionally added. This meant there was a limited number of experiments that could be created for a particular day for a particular experiment. This part of the directory naming convention has been dropped, and now the convention is:
$MADROOT/experiments/YYYY/<3 letter inst mnemonic>/*, Example: /opt/madrigal/experiments/1998/mlh/mlh_exp_234.
Simple web UI added
A new web user-interface has been added that allows easy printing and plotting of basic Madrigal data. To make it easy to use, advanced Madrigal features such as derived parameters and filtering of data have been removed.
On-demand plot creation
Madrigal now allows users to create basic scatter plots and pcolor plots versus range or altitude of any measured or derived parameter in a data set.
Logging of user data access
Madrigal now logs user's names, emails, and affiliations whenever data files are directly accessed in a file administrators can access.
Automatic updating of all geophysical data
Madrigal now automatically updates all its internal geophysical files (e.g., Kp, Fof2, Dst, Imf, etc) every time updateMaster is run.
Simple-to-use python module to create and edit Madrigal files
There is now a simple-to-use python module to create and edit Madrigal files.
New administrative scripts to manage Madrigal experiments
Administrators can now add or modify all Madrigal experiments using simple administrative scripts, instead of trying to edit Madrigal metadata files themselves or use the complex genExp script.
Complete documentation rewrite
Madrigal documentation has now been completely rewritten and reorganized into three manuals: one for users, one for administrators, and one for developers.
Automatic graphics conversion
Madrigal will now allow users to select any graphics format they prefer for graphics administrators place in experiments. This feature was contributed by Eiscat.
Update of IGRF/MSIS
The Madrigal derivation engine is now using the IGRF 2010 coefficients, and the MSIS 2000 model.
Limiting of disk space used for global search files
Administrators can now limit the maximum amount of disk space used to store temporary global search files. See the section on editing the madrigal.cfg file in the installation guide.
Remote programming access to Madrigal via web services using any platform
Madrigal now exposes all the information and capabilities it has as web services, which allows easy access to Madrigal from any computer on the internet using any platform (Unix, Windows, Mac, etc). Madrigal's web services are basically cgi scripts with simple output that allows easy parsing of the information. Any language that supports the HTTP standard can then access any Madrigal site. We have written remote API's using python and Matlab, but almost any language could be used. See the section on remote programming access for details of these APIs and the underlying web services.
Note that this approach of remotely accessing Madrigal data has been always possible before by parsing the html output meant to be displayed in a web browser (this general programming method is referred to as "screen scraping"). However, not only is this parsing difficult; but the code often breaks when the user interface is modified in any way. With web services the returned cgi scripts are designed to be both simple to parse and stable.
The web services are not implemented according to the SOAP or XMLRPC standard since not all scripting languages have support for these standards (or for XML parsing). Instead they use the simple approach of returning data requested via a query as a delimited text file. These web services are fully documented here.
Users who want only to write programs to remotely access Madrigal, and not to install a Madrigal server themselves, are now able to download the remote python and Matlab API's from the OpenMadrigal site.
Command-line global search
As an example of remote programming access to Madrigal via web services, an application globalIsprint was written using the python remote API that does a global search of data on any Madrigal site that has installed Madrigal version 2.3. This application is installed as part of Madrigal, and also when the standalone remote python API is installed. It has all the filtering ability of the web-based global search.
Calculate any derivable Madrigal parameter for any time and point(s) in space
By clicking on "Run Models", users can calculate any derived Madrigal parameter (such as magnetic fields, or geophysical parameters) for arbitrary times and ranges of position. Note that this capability is also available as a web service, and through the remote python and Matlab API's.
New derived parameters
- CGM_LAT: Corrected geomagnetic latitude (deg)
This parameter gives the location of a point in Corrected geomagnetic latitude. This method uses code developed by Vladimir Papitashvili. For more information on CGM coordinates and this code, click here.- CGM_LONG: Corrected geomagnetic longitude (deg)
This parameter gives the location of a point in Corrected geomagnetic longitude. This method uses code developed by Vladimir Papitashvili. For more information on CGM coordinates and this code, click here.- TSYG_EQ_XGSM: Tsyganenko field GSM XY plane X point (earth radii)
This parameter gives the X value in GSM coordinates of where the field line associated with a given input point in space and time crosses the GSM XY plane (the magnetic equatorial plane). GSM stands for Geocentric Solar Magnetospheric System, and its XY plane is the equatorial plane of the earth's magnetic dipole field. The field lines are traced using the Tsyganenko Magnetospheric model, so external effects on the earth's magnetic field such the solar wind are taken into account. This code uses the 2001 Tsyganenko model, which averages solar wind values over the past hour, instead of simply using present values.- TSYG_EQ_YGSM: Tsyganenko field GSM XY plane Y point (earth radii)
This parameter gives the Y value in GSM coordinates of where the field line associated with a given input point in space and time crosses the GSM XY plane (the magnetic equatorial plane). GSM stands for Geocentric Solar Magnetospheric System, and its XY plane is the equatorial plane of the earth's magnetic dipole field. The field lines are traced using the Tsyganenko Magnetospheric model, so external effects on the earth's magnetic field such the solar wind are taken into account. This code uses the 2001 Tsyganenko model, which averages solar wind values over the past hour, instead of simply using present values.- TSYG_EQ_XGSM: Tsyganenko field GSE XY plane X point (earth radii)
This parameter gives the X value in GSE coordinates of where the field line associated with a given input point in space and time crosses the GSE XY plane (the equatorial plane). GSE stands for Geocentric Solar Ecliptic System, and its XY plane is the equatorial plane of the earth's rotation. The field lines are traced using the Tsyganenko Magnetospheric model, so external effects on the earth's magnetic field such the solar wind are taken into account. This code uses the 2001 Tsyganenko model, which averages solar wind values over the past hour, instead of simply using present values.- TSYG_EQ_YGSM: Tsyganenko field GSE XY plane Y point (earth radii)
This parameter gives the Y value in GSE coordinates of where the field line associated with a given input point in space and time crosses the GSE XY plane (the equatorial plane). GSE stands for Geocentric Solar Ecliptic System, and its XY plane is the equatorial plane of the earth's rotation. The field lines are traced using the Tsyganenko Magnetospheric model, so external effects on the earth's magnetic field such the solar wind are taken into account. This code uses the 2001 Tsyganenko model, which averages solar wind values over the past hour, instead of simply using present values.- BHHMMSS and EHHMMSS: Start and end time in HHMMSS (suggested by Mary McCready at SRI)
Bug fixes
The Madrigal C API now no longer aborts when a Cedar file contains cycle marks (Cedar parameter 95) that are not in order. (Reported by Angela Li, SRI)
A problem launching the global search with the python module os.spawnlp was fixed. (Reported by Angela Li, SRI)
New derived parameters
- SUNRISE_HOUR - Ionospheric sunrise (hour)
This parameter gives the hour UT that sunrise occurs at that particular point in space that particular day. If that point in space is either in sunlight or in shadow the entire UT day, sunrise_hour will be missing. To find out which, display the Shadow height (SDWHT) parameter. If shadow height is less that the altitude of the point, its in sunlight; if shadow height is greater than the altitude, its in the earth's shadow.- SUNSET_HOUR - Ionospheric sunset (hour)
This parameter gives the hour UT that sunset occurs at that particular point in space that particular day. If that point in space is either in sunlight or in shadow the entire UT day, sunset_hour will be missing. To find out which, display the Shadow height (SDWHT) parameter. If shadow height is less that the altitude of the point, its in sunlight; if shadow height is greater than the altitude, its in the earth's shadow.- CONJ_SUNRISE_H - Magnetic conjugate point sunrise (hour)
This parameter gives the hour UT that sunrise occurs at the magnetic conjugate point of the particular point in space that particular day.- CONJ_SUNSET_H - Magnetic conjugate point sunset (hour)
This parameter gives the hour UT that sunset occurs at the magnetic conjugate point of the particular point in space that particular day.- SZEN - Solar zenith angle in measurement vol (deg)
This parameter gives the solar zenith angle in degrees. If 0 degrees, the sun is directly overhead. A solar zenith angle of between 90 and 180 degrees does not mean the sun is not visible, due to the finite solid angle of the sun and the altitude the point may be above the earth's surface.- SZENC - Conjugate solar zenith angle (deg)
This parameter gives the solar zenith angle at the magnetic conjugate point in degrees.- SDWHT - Shadow height (km)
This parameter gives the height above the earth's surface at which any part of the sun can be seen. It depends only on the time, and on the geodetic latitude and longitude. During the day shadow height will be zero. Since the sun is larger than the earth, the shadow height is always finite. If shadow height is less that the altitude of a given point in space, its in sunlight; if shadow height is greater than the altitude, its in the earth's shadow.- MAGCONJSDWHT - Magnetic conjugate shadow height (km)
This parameter gives the height above the earth's surface at the magnetic conjugate point's latitude and longitude at which any part of the sun can be seen.- 10 Interplanetary Magnetic Field parameters
Includes field strength in GSM or GSE coordinates, solar wind plasma density, speed, and measuring satellite id. Click on any parameter to see the definition of the two coordinate systems.Filtering using any parameter
- There are now also free-form filters at the end of the filter section, which allow you to set up filters based on any single parameter or on two parameters either added, subtracted, multiplied, or divided together. For example, you can now filter on Ti, the ratio Ti/dTi, or gdalt-sdwht (which is positive if the point is in sunlight). See the tutorial for more details.
Better help understanding what each parameter means
- Complex parameters now have full html descriptions accessible from the isprint page. Just click on the parameter name and you'll see the short description. For more complex parameters you'll also see a link to a more detailed explanation.
Improved data output
- If you select only 1D parameters, or derived parameters that depend only on other 1D parameters, isprint will only print a single line per record, making it easier to read.
- All filters used are printed at the beginning of the report. Trivial filters that don't exclude data (such as elevation from 0 to 90 degrees) are ignored.
Better consistency with Cedar standard
- All units are now consistent with the Cedar standard (when displaying Cedar parameters).
- The special Cedar values "missing", "assumed", and "known bad" are differentiated in isprint output, and not all lumped together as "missing" as before.
- Unknown parameter codes displayed with a scale factor of 1.0.
New derived parameters are simple to add
- The isprint web page is now based on the madc library, and has been designed to make it extremely simple to add new derived parameters. See the madc API documentation for details.
|
|
|
What's new in Madrigal 3.0? | Doc home | Madrigal home |