Upgrading Madrigal to version 3

The following directions assume you have a version of Madrigal 2 already running, and you want to upgrade it to Madrigal 3. It is recommended you back up your existing Madrigal installation before proceeding. You will also need to upgrade slightly any loading programs you are running to create the new Madrigal 3 Hdf4 CEDAR format. Contact Bill Rideout or go to Create Madrigal files to learn more.

Prerequisites

With Madrigal 3, the installation uses a version of python 3 you install yourself. The easiest way I have found to install python with scientific tools is to use either apt-get (for ubuntu), or the Anaconda python distribution, or pip. The following python modules need to be installed:

django (version 1.11 or greater)
pip and setuptools
numpy
scipy
matplotlib
netCDF4
h5py
sqlparse
bootstrap3 (can be installed via pip install django-bootstrap3)
aacgmv2 (can be installed via pip install aacgmv2)

In addition, you need to have Autotools (automake, autoconf, aclocal, libtoolize, autoheader) installed, along with h5repack (may be part of h5utils package).

Installation instructions

Be sure the environmental variable MADROOT is set to your main Madrigal directory (where madrigal.cfg is locatied).
Make sure $MADROOT/bin/python links to the version of python with the required modules listed above. For example, if you are using anaconda, you might run <ln -s //anaconda/bin/python $MADROOT/bin/python>.
Download the latest Madrigal 3 distribution file, madrigal*.tar.gz from the OpenMadrigal distribution page to MADROOT.
tar -xzf madrigal*.tar.gz
Configure your web server to run the Madrigal Django application. Instruction for apache can be found in the Configuring Apache for a Madrigal Django Application section below. If you are running a webserver other than apache, please consult the Django documentation for your particular webserver.
Edit your file $MADROOT/metadata/siteTab.txt. The format of siteTab.txt is described here. Note that if your site will be running https, the third field will begin https://. Make sure the last column of that line for your site ends ,3.0 . This tells all sites that you site is now a Madrigal 3 site. For example:
```
999,mad31,localhost,madrigal,cgi-bin/madrigal,madrigal/servlets,Bill Rideout,MIT Haystack Observatory,Route 40,,Westford,MA,01886,USA,1-781-981-5624,brideout@haystack.mit.edu,3.0
```
Be sure to cd to MADROOT before running the following step. Then you should be able to complete the installation simply by typing
```
bash installMadrigal  &> install.log &
```
Help with any installation errors is available from the OpenMadrigal administrator.
If there were no errors, your madrigal installation should be running at the url given by MADSERVERROOT.
Set the script madroot/bin/updateMaster up as a cron job to run once a day.
If you want to add any documentation pages specific to your site to the Madrigal documentation pages, see the other admin tasks section of this manual.
If you want to add your own rules of the road to the Madrigal experiment page, see the other admin tasks section of this manual.

Editing the madrigal.cfg file

The madrigal.cfg file contains all the configuration information specific to your installation. This section discusses how to edit that file for each parameter. The madrigal.cfg.template file contains examples of each parameter. For an upgrade, this file does not generally need to be edited.

MADROOT - Madrigal root directory (absolute). This must be set as an environmental variable.
MADSERVER - Web server for accessing Madrigal. For https, add https:// to the beginning. If not given, uses http
MADSERVERROOT - document virtual directory relative to MADSERVER. If the full Madrigal Url is the same as the MADSERVER field (for example, if the url is http://madrigal.hao.ucar.edu/), then set this field to a period. (Example: MADSERVERROOT = . )
SITEID - Site ID - Must be unique and same as in siteTab.txt
HTMLSTYLE - Body style of html pages
INDEXHEAD - Heading in the top-level Madrigal page
CONTACT - Mailto link of contact person(s) for this madrigal installation. Multiple email addresses may be included if separated by commas.
MAILSERVER - Name of mailserver (possibly localhost if running sendmail)
NOTESMANAGER - If you want someone to be notified whenever a user adds a note to an experiment, uncomment this optional line and add the email address. See below for a discussion of the optional notes feature and how to configure it after installation.

Configuring Apache for a Madrigal Django Application

Some configuration of apache is required to run a Django application like Madrigal. If you are running a webserver other than apache, please consult the Django documentation for your particular webserver. We will configure apache to run Madrigal in mod_wsgi daemon mode for improved performance.

The first step is to make sure mod_wsgi is installed as part of your apache webserver. If not, see mod_wsgi installation directions. Once installed, add the line

LoadModule wsgi_module modules/mod_wsgi.so

near the top of the httpd.conf file. Next add the follow section, with MADROOT, MADSERVERROOT and YOUR_SERVER replaced with values for your server from the madrigal.cfg file described below. Note that if MADSERVERROOT is zero length (that is, if the full Madrigal Url is the same as the MADSERVER field - for example, if the url is http://madrigal.hao.ucar.edu/) then /MADSERVERROOT just becomes / unlike in the madrigal.cfg file when it becomes a period. Set NUM_CPU to about one half available on your server.

SetEnv PYTHONPATH MADROOT/source/madpy/djangoMad/
WSGIDaemonProcess YOUR_SERVER python-path=MADROOT/source/madpy/djangoMad processes=NUM_CPU display-name=%{GROUP} lang='en_US.UTF-8' locale='en_US.UTF-8'
WSGIProcessGroup YOUR_SERVER
WSGIScriptAlias /MADSERVERROOT MADROOT/source/madpy/djangoMad/djangoMad/wsgi.py process-group=YOUR_SERVER
Alias /static/ MADROOT/source/madpy/djangoMad/madweb/static/
WSGIApplicationGroup %{GLOBAL}

 <Directory MADROOT/source/madpy/djangoMad/madweb/static>

Require all granted

</Directory>
<Directory MADROOT/source/madpy/djangoMad/djangoMad>

<Files wsgi.py>

Require all granted

</Files>

</Directory>

You may also need to add to your apache configuration file SetEnv LD_LIBRARY_PATH $MADROOT/lib, possibly along with other library paths.

Finally, you need to edit the file MADROOT/source/madpy/djangoMad/djangoMad/settings_production_template.py, and save it as MADROOT/source/madpy/djangoMad/djangoMad/settings_production.py. The lines to be edited are:

SECRET_KEY = '*****'
ALLOWED_HOSTS = ['madrigal3.haystack.mit.edu']
ADMINS = (('Bill Rideout', 'brideout@haystack.mit.edu'),)
EMAIL_HOST = 'hyperion.haystack.mit.edu'
MANAGERS = (('Bill Rideout', 'brideout@haystack.mit.edu'),)
STATIC_URL = '/static/'

The STATIC_URL line only needs to be edited if MADSERVERROOT is not empty; that is, if your madrigal is installed in a subdirectory. For example, if your url is http://madrigal.sri.com/madrigal, you would change that line to be STATIC_URL = '/madrigal/static/'. But if your url is http://madrigal.sri.com, you would leave that STATIC_URL line as is.

You can generate a new secret key for your site at http://www.miniwebtool.com/django-secret-key-generator/, which is recommended for security reasons. The other lines are self-explanatory.

When you are done, be sure to restart apache for the changes to take effect. Note that because this Madrigal Django application runs in daemon mode, it will be necessary to restart the web server if you ever update the python code.

Throttling Madrigal admin emails

Sometimes Madrigal users write scripts that cause numerous admin emails, and in addition malicious users can also generate numerous admin emails. If you would like to throttle the number of Madrigal admin emails you receive, see Throttling admin emails