USGS Home Contact USGS Search USGS

dv_cleanup

Name:
dv_cleanup - Circumvent normal data-aging rules and delete old daily value statistics for any period/processor in a DD.

Information:
This cleanup script was written to help those needing to cleanup old, bad daily value statistics from DD's (and optionally prevent them from being stored into the future with the current processor). It is meant to delete a specific DD statistic (e.g. mean) stored in a DD for a user specified period (or a period a given processor is in effect for if the processor option is used), circumventing data-aging rules in the process to simplify and speed up the cleanup process. Script is primarily meant to be used to cleanup old daily values statistics that were stored in a DD, but were never appropriately QA/QC'd and likely never should have been stored in the first place. It is very useful in the cleanup tasks associated with the NWISWeb daily values project. Note that this tool is very powerful and must be used with caution as it can delete good data if not used carefully. A new database checkpoint is recommended prior to using this or any extensive database cleanup activity (manually making backup rdb files of the data to be deleted and watching your typing very carefully are also good ideas). It can only be run as user "nwis."

This is a "generic" cleanup script for any parameter code and statistic - for more specialized utilities dedicated to cleaning up mean daily stage data or max/min daily discharge data, see the stop_mean_stage_processing or stop_maxmin_q_processing scripts, which should be safer to use in those circumstances.

Usage:

dv_cleanup [-z db_no] [-a agency_cd] -n site_no -d dd_nu [-b start_dt (yyyymmdd)] [-e end_dt (yyyymmdd)] [-p] stat_cd stat_cd ...

Where:

db_no is database number (if omitted will default to database 01)
agency_cd is agency code (if omitted will default to USGS)
site_no is the 8-15 digit site ID (required)
dd_nu is the data descriptor (DD) number (required)
start_dt is the beginning date of the deletion (if omitted will default to the beginning of period of record)
end_dt is the ending date of the deletion (if omitted will default to the end of the period of record)
stat_cd is the DV statistic code(s) to delete (required)

if -p is specified, operates in processor mode:

If the DD specified is an input DD, the processor it is input for will be used.
The begin date argument will be used to find the processor that is in effect on the given date.
The end date argument will be ignored, if present.
Daily values for the specified stat codes will be deleted starting at the begin date of the processor through the end date of the processor.
The processor itself will have the stat code removed, if present, so that daily stat code will no longer be stored in the DD.

For example for a station number 12345678, to delete max and min data from DD 5 for the period 10-01-1950 to 09-30-1994 the command would be:

dv_cleanup -n 12345678 -d 5 -b 19501001 -e 19940930 1 2

To delete mean data from station 12345678, DD 5 for the entire period the current processor (as of 10-01-2011...) has been in effect for and prevent mean stage data from being stored into the future in that DD, the command would be:

dv_cleanup –n 12345678 -d 5 –b 20111001 –p 3

- Must be run as user "nwis"

Installation:
The scripts can be installed by unzipping the files in any Sun directory and making sure the script is executable (chmod +x dv_cleanup).

Script Language:
ksh

Requirements:

• NWIS 5.0 or higher

Original Author(s): Scott Bartholoma, NWIS

Note the original author is noted on this page for the purposes of giving credit only. Please address questions/comments to OSW (GS-W OSW Scripts@usgs.gov).

Scripts:

Version 5.0: (Posted 02/01/2013)
—Updated for NWIS 5.0/Oracle and a few minor other minor tweaks

dv_cleanup download (Zip file = 5.9K)