Previous: Creating Madrigal experiments   Up: Madrigal admin guide   Next: User access logging

Other administrative tasks

Some other tasks a Madrigal administrator might need to do:

Contact the OpenMadrigal administrator if some other administrative issue arises.

Create cached text or netCDF4 versions of CEDAR Madrigal Hdf5 files

When users download ascii or netCDF4 versions of CEDAR Madrigal Hdf5 files as is (that is, just the parameters in the file without filtering), these alternative file formats by default are dynamically created. If this takes too long, the Madrigal admistrator can speed that up by creating cached files in the <experiment directory>/overview directory. The cached text file has '.txt.gz' appended to the basename of the CEDAR Madrigal Hdf5 file, and the netCDF4 file has '.nc' appended to the basename of the CEDAR Madrigal Hdf5 file. These cached files are created with the $MADROOT/bin/createCachedFiles.py script. Here's the usage:

createCachedFiles.py [--excludeText --excludeNetCDF4 --inst=  --path= 
--includeNonDefault  --overwrite --includeGeo --numCPU= -h --help]

By default, both text and netCDF4 files created.  Use --excludeText or --excludeNetCDF4 to only create one type.
By default all instruments will be included.  Use --inst= to only include some instruments.
By default, all experiment directories will be included.  Use --path to limit to a particular directory and all subdirectories.
By default only default files will be cached.  Use --includeNonDefault to include all files.
Use --overwrite to overwrite all Hdf5 cached files. Default is to skip existing cached files.
Set --includeGeo to also convert geophysical files. Default is to skip them.
numCPU by default is the maximum of (1, numCPUs available - 2).  Use --numCPU to override, but still will not be highter than default.
-h or --help - print usage and exit

Set private versus public access for Madrigal data files

This new feature of Madrigal allows data files to be made either publicly available, or restricted to a limited set of IP address listed in a file.

To disable this feature and make all your data public, run the following from the $MADROOT directory:

$MADROOT/bin/setAccess $MADROOT/experiments public 

To enable this feature, you must first create a file called "trustedIPs.txt" under MADROOT. This file should contain the IP addresses of hosts considered to be part of the private group, or partial IPs if whole sub-nets should be included. This file is not created during installation. Anyone whose browser's IP matches this list will be able to view files marked as private. For example, if trustedIPs.txt contains the line 132.197.*, any IP address that begins 132.197 will have private access.

Also, if the fileTab.txt file for that experiment can be overwritten by the web server, anyone whose browser's IP matches this list will be able to to change the access of any file between public and private from the madExperiments listing page.

To change a large number of experiments to be either public or private, run the following script from the $MADROOT directory:

$MADROOT/bin/setAccess dirPath [public || private] 

where dirPath is the full path name of any directory in $MADROOT/experiments, and the second argument is either public or private. This script will set all experiments in dirPath and below to be public or private.

The access permission for any given file is set in the fileTab.txt file. See metadata documentation for details.

Modifying the Madrigal homepage

You can easily modify the Madrigal homepage of you site by modifying the template file $MADROOT/source/madpy/djangoMad/madweb/templates/madweb/index.html.

Modify settings in the madrigal.cfg file

The following settings in the madroot/madrigal.cfg file can be modified at any time and the changes will take effect immediately:

Add documentation specific to your site to Madrigal documentation page

If you want to add any documentation pages specific to your site to the Madrigal documentation pages, simply put them at a url where they can be indexed. Then edit the file MADROOT/source/madpy/djangoMad/madweb/templates/madweb/madContents.html to modify the line

<a href="#">

to replace # with the url where your site specific documentation is.

Add site-specific rules-of the-road to the Madrigal experiment page

By default, users see only the Cedar rules-of-the-road on the data access page when they first enter the Madrigal site. If you want your users to see rules-of the-road specific to your Madrigal site when the reach the Madrigal experiment page, simply create a text file with your rules called local_rules_of_the_road.txt in the madroot directory.

Add static pages to your Madrigal site

Soemtimes its helpful to add static pages to your Madrigal site, such as a page describing your instrument, that you can then simply reference from each individual experiment page. To do this, simply copy the static file to MADROOT/source/madpy/djangoMad/madweb/static/siteSpecific, and use the url "/static/siteSpecific/[filename]" to refer to it.

Copy entire experiment directories from one Madrigal site to another

If you want to copy data from one Madrigal site into yours (such as is done during the standard installation with the test data from Haystack) copy all the desired data from another Madrigal site into the appropriate directory in $MADROOT/experiments. Then follow these steps:

  1. cd $MADROOT
  2. ./configureExperiments
  3. $MADROOT/bin/updateMaster

The script configureExperiments automatically edits the metadata files, changing the site id to your site id. This same procedure also works for moving experiments between experiments[0-9]* directories.

Reset experiment start and end times according to data in the experiment files

Sometimes the experiment start and end times in the metadata (the expTab.txt file) are manually edited, and do not match the start and end times of the data found in the files. The script updateExpTimes.py will examine the start and end times of all the files in an experiment, and set the experiment metadata start and end times to be the earliest and latest times found. Usage:

$MADROOT/bin/updateExpTimes.py [experiment_directory]

If optional argument experiment_directory given, then it will only update all experiments found in that directory. Default is to update the entire database. Only default or real time files are used to determine the times. Example:

$MADROOT/bin/updateExpTimes.py $MADROOT/experiments/1998

will update all experiment times for the year 1998.

 

Registering interest in an experiment or an instrument

With Madrigal 2.6, users can register interest in an existing experiment or in an instrument. When a data file is updated in that experiment using the standard admin scripts, by default all users registered for that experiment or instrument are sent an email. To ensure that is possible, you simply need to make sure the files $MADROOT/metadata/userdata/regExp.txt and $MADROOT/metadata/userdata/regExp.txt are writable by the web server. If you get a request from a user to unregister them from an experiment or an instrument, simply delete that line with their email from that file. With Madrigal 3.0, you can also use the utility $MADROOT/bin/registrationAdmin.py. Usage is:

registrationAdmin.py [--unregister | --register]
 [--inst=<kinst> | --exp=<expDir>] userEmail
Either unregister or register must be set.
   Either --inst=<instrument code> or --exp=<expDir> must be set.
   expDir is in form <experiments/2010/mlh/18jan10>

Rebuilding the madrigal python library

If for any reason you rebuild the madrigal python library by running MADROOT/bin/python setup.py install in the MADROOT/sopurce/madpy directory, you will need to restart the web server after you are done.

Mounting additional hard drives to expand the experiments disc capacity

With Madrigal 2.6, administrators can expand the disc space available for experiments by mounting a directory named experiments[0-9]* in the Madroot directory. For example, you can add a directory called experiments2 in the Madroot directory and use it for a mount point for additional space for Madrigal experiments. After you create this additional mount, you will need to create a soft link in your web server document root with a soft link to that new directory. For example, say you do the following:

You can now make use of this new directory in two ways:

  1. You can move experiments or whole directory trees of experiments between different experiments[0-9]* directrories as described in Copy entire experiement directories section (remember to run configureExperiments and updateMaster afterwards). Note, however, the experiments with geophysical data should not be moved (years 1950, 1957, and 1963).
  2. When running the scripts createExpWithFile.py or createRTExp.py, use the --experimentsDirNum argument to choose the new directory to use. By default the base experiments directory will be used. In the example above, the option --experimentsDirNum=2 would cause experiments2 to be used.

 

Throttling admin emails if too many sent

If you receive too many emails, you can use a third party tool to throttle the amount. The tool I have used successfully is django-email-throttler, which is described at https://github.com/robert-kisteleki/django-email-throttler. To install, use pip and then edit your settings_production.py file:

pip install django-email-throttler

Then add the following lines to the end of your file MADROOT/source/madpy/djangoMad/djangoMad/settings_production.py. The top lines can be adjusted to set exactly how many emails you get, but these numbers should work fine. When done, you will need to restart apache for the changes to take affect.

# stuff for email throttling and their defaults
EMAILTHROTTLER_TMPDIR = '/tmp'
EMAILTHROTTLER_PREFIX = 'emailthrottler-'
EMAILTHROTTLER_INTERVAL = 600
EMAILTHROTTLER_TRUNCATE_SUBJECT = None # 128 is also ok
EMAILTHROTTLER_SUBJECT_THRESHOLD = 3 # 0 for no-check
EMAILTHROTTLER_OVERALL_THRESHOLD = 6 # 0 for no-check

LOGGING = {
    'version': 1,
    'disable_existing_loggers': True,
    'handlers': {
        'mail_admins': {
            'level': 'ERROR',
            'class': 'django.utils.log.AdminEmailHandler',
            'email_backend': 'django_email_throttler.throttler.ThrottledEmailBackend',
            'include_html': False,
        }
    },
    'loggers': {
        'django': {
            'handlers': ['mail_admins'],
            'level': 'ERROR',
            'propagate': True,
        },
    }
}

Converting from http to https

The following are the steps needed to change an existing Madrigal site from http to https

  1. In $MADROOT/metadata/siteTab.txt, include https:// in the third field of your site. For example, change millstone.haystack.mit.edu to https://millstone.haystack.mit.edu
  2. Change the MADSERVER line in $MADROOT/madrigal.cfg to include https:// For example, change millstone.haystack.mit.edu to https://millstone.haystack.mit.edu
  3. cd $MADROOT/source/madpy
  4. $MADROOT/bin/python setup.py install
  5. $MADROOT/configureExperiments
  6. $MADROOT/bin/updateMaster
  7. Let Bill Rideout (brideout@mit.edu) know about the change.
Previous: Creating Madrigal experiments   Up: Madrigal admin guide   Next: User access logging