Environmental Plotting Documentation
Adam Orin
Updates:
- 7-18-06: Converted env_readme.txt into this htm file.
Sections:
|
|
|
Introduction
|
|
We have written a series of scripts and programs to get the houston control environment log files (housctl*.log) and create plots of the various parameters for viewing on the web.
Currently, these scripts reside and function on Cocoa in /home/environment. An identical ccopy should also be kept on Grlab in /home/apollo/src/environment. The scrips are run by a cron job. They get the logfiles from Houston, which is mounted on Cocoa, generate plots (in the form of png image files) and web pages, which are viewable from /home/www/html/environment.
We also have scripts that archive the plot images. They copy the plot images and put them in /home/www/html/environment/archive. These scripts are also run by a cron job.
|
|
|
Relevant files
|
|
Scripts for getting log files, generating plots and webpages |
update.script | Shell script that runs all the other scripts. |
logclip.py | Python script that creates a log file with the past couple days of housctl*log record in it |
env_parse.py | Python script that parses log file, and outputs several whitespace-delimited data files |
env_plot.py | Python script that takes those whitespace-delimited data files and makes plots |
env_webpage.py | Python script that creates updated htm page (just changes the time updated) |
Scripts for archiving the plots and generating archive webpages |
env_archive.py | Python script that moves plot images into archive folder, and creates html pages for the archives, run by a cronjob |
Utility scripts |
dataparse.py | Python function that parses a whitespace-delimited data file, used by env_plot.py |
keyvalparse.py | Python function that parses keyword=value records, used by env_parse.py and logclip.py |
date_convert.py | Python functions that convert a date into days since the begnining of year, and converse, used by logclip.py, env_parse.py, env_plot.py |
Obsolete Files |
getlog.py | Python script that used to ssh into Houston, and copy the most recent logfile to the local machine using scp. Now that Houston's disk is mounted to Cocoa, this isn't needed. |
env_archive.script | Shell script that used to move the images into the archive folder and generate a very basic htm page for them. Now env_archive.py does it all. |
|
|
|
Required apps
|
|
Linux. These scripts will not work on a Windows machine becuase they use the "ls" command an in python the op.popen command.
python
matplotlib for python
|
|
|
How environmental plot generation works
|
|
- Run update.script on Cocoa
This shell script automatically launches the other scripts and updates the environment plot webpage. The following is pretty much a line by line explanation of what's going on.
- cd /home/apollo/environment
This makes all the scripts run in the '/home/apollo/environment' folder. This is what I refer to in this help file as the 'local folder'.
- getlog.py
This looks at houston, figures out the name of the newest housctl*log file, then uses 'scp' to copy the file to the local folder on the Cocoa. It used to (but no longer does) take the 2 most recent logfiles and stick them together with the 'cat' command in a file called 'current_log.log'.
- ./logclip.py 1.5 current_log.log
This creates a file 'current_log.log' that contains the most recent 1.5 days worth of housctl*log records. It currently assumes the logfiles are contained in the '/home/apollo/environment/' folder.
- ./env_parse.py current_log.log
This will parse the file 'current_log.log' (in the local folder) for parameters like the gps dac, RTD temperature readouts, flow rates, etc. It outputs the several whitespace delimited data files (into the local folder) : 'env_control.dat' 'env_dac.dat' 'env_flow.dat' 'env_temps.dat'.
- ./env_plot.py -dAgg
This uses dataparse.py to read-in the 'env*dat' files created above. It then uses matplotlib to generate plots 'env*.png' that are png image files. The '-dAgg' option makes matplotlib use the 'antigrain geometry backend', which doesn't require an X-Display, and outputs pngs.
- ./env_webpage.py
This simply generates the htm webpages that you can view the plots in: plots-overall.htm, plots-temps.htm, and plots-flow.htm.
- And the final commands in update.script just move the pngs and the htm files to the appropriate location for viewing on the WWW.
|
|
|
How archiving works - detailed description of env_archive.py
|
|
This is a Python script that copies the image plot files to the archive location, and generates the html files for the archive pages.
At the top pf the script, there are two strings defined. One is 'archiveloc' -- this is the folder you want the archive images and associated htm files to be put. The other is 'imagesloc' -- this is the folder that the plot images that you want to archive are located. I wrote the script to use absolute paths, so you should probably use absolute paths ('/home/environment' okay, '../environment' not okay). The script should run properly regardless of what folder it is run in. I think.
Here is how the archives are organized. Within the 'archiveloc' folder, there will be a folder for each month, like "0601/" for Jan 2006. And inside each folder are plot images for each day, like "060120env_temps.png" for a plot copied on Jan 20, 2006. In the 'archiveloc' folder, an 'index.html' file is created that has links to all the htm files for each month. And in each month folder there is an htm file, like "0601/0601.htm" that has links to the plot images in that folder.
To get lists of folders, and copy files around, we use the os.popen() function in the os module.
Here is a step by step rundown of the algorithm:
- Get a list of all the folders in 'archiveloc'. And if this month doesn't have a folder, create one. So the first time the script is run in July 2010, the script will create a folder called "1007/" in 'archiveloc'.
- Get a list of today's plot images located in 'imagesloc'. The script assumes that the plot images are png files, and end in the 'png' extension.
- Copy today's plot images to the appropriate folder, and add the date to their filename. So 'imagesloc/tst.png' would be copied to archiveloc/0607/060710tst.png' if the time the script ran was July 10 2006.
- Generate the 'archiveloc/index.html' page. This is done by the 'makeindex' function. It is passed a list of all the folders in 'archiveloc', like ["0604/","0605/",...]. It generates a table with link to the htm files inside each folder.
- Generate the htm files for each folder. In other words, generate 'archiveloc/0607/0607.htm'. This is done by the 'makemonthpage' function. You pass it a string of the folder you want to generate the htm file for. The script only calls the 'makemonthpage' function for the 2 most recent months, since there is no point in regenerating htm files for older months. This function uses the calendar module to know what day of the week a date like July 12, 2006 is.
|
|
|
How the scripts are run automatically by cron
|
|
Here is what my crontab looks like. The plots are generated once every 5 minutes, and they are archived once a day.
SHELL=/bin/sh
HOME=/home/aorin
PATH=/usr/local/bin:/usr/bin:/usr/sbin:/bin:/sbin:/usr/X11R6/bin:.
LOGNAME=aorin
*/5 * * * * /home/environment/update.script
0 0 * * * /home/environment/env_archive.py
|
|
|
Detailed description of the innards of each script
|
|
|
|
How to generate environmental plots manually
|
|
So, if you wanted to create plots for a custom time range, simply copy the logfile data you want plotted to some file 'junk.log'. Next, run 'env_parse.py junk.log' to produce the 'env*dat' files. Then, run 'env_plot.py' to produce the plot images from the 'env*dat' files.
|
|
|
How to update RTD/POWER mappings for plots
|
|