USGS Home Contact USGS Search USGS

tidal filter

Name:
tidal filter - A utility to remove tidal signals from time series data using the Godin filter.

Information:

This program uses the Godin filter to remove tidal signals from time-series discharge data, as called for in OSW memo 2010.08. Specifically the program,

• retrieves daily-value discharge data from NWIS as identified by the user
• runs the Godin filter to remove the tidal signal (using supplied program grfilt)
• processes the filtered data through Decodes and stores it back into NWIS

See that memo and appendix B in particular for more information.

In order to use tidal_filter, computed unit values of discharge must be available for the time period selected and a data descriptor for parameter code 72137 (tidally filtered discharge) must be available to store the filtered product data.

Filtering data requires the user to run the script tidal_filter, which executes four steps: (1) retrieves data, (2) filters data, (3) runs decodes, and (4) loads the results back into ADAPS. Processing times vary with the length of the time period, but one year of data can generally be filtered and loaded back into ADAPS in less than two minutes with approximately 70% of the time required to load the ADAPS Standard Input file back into the database and run primary computations. If run manually (interactively), messages requesting user input and messages describing computational progress are displayed on the user’s terminal. The step-by-step process is summarized below:

1. Retrieve Data: Data is retrieved from ADAPS using nwts2rdb based on the following user defined criteria:
   
   1. Invoke program: Type “tidal_filter” at the Unix command prompt.
   2. Station number: Type the station number at the prompt.
      
      1. Valid station number: the program will list all of the available data descriptors at the station
      2. Invalid or no station number input: the program will generate an error message and return to the “Enter station number:” prompt.
   3. Data descriptor identification number: Type the data descriptor number associated with the data requiring tidal filtering (usually the number for the primary discharge DD).
      
      1. Valid data descriptor: the program will prompt the user for the date range for filtering.
      2. If the parameter code for the chosen data descriptor is not 00060, the program will generate an error message and return to the “Enter station number:” prompt.
   4. Start and end dates: Type the date range.
      
      1. The required input format for the start and end dates is YYYYMMDD. Simple error checking on the date ranges are made at this time. For example, dates in the future are flagged and the user is asked to enter a valid date.
      2. There is a message reminding the user that the filter will cause 1.5 days to be trimmed from the beginning and end of the data set selected so a longer date range is necessary to compute tidally filtered values for a specific period.
2. Filter data: tidal_filter calls grfilt and the Godin filter is applied to the data. Several messages may be displayed to the user screen as follows:
   
   1. Gap in input data: if there is a gap in the input data, a message will be displayed showing the date and time of the beginning and end of the gap in the input series. It will also display the dates and times of the longer gap in the output series that will result. For example:
      Gap in input series from 2006/06/25 08:00 to 2006/06/26 22:15 will cause
      Gap in output series from 2006/06/23 21:00 to 2006/06/28 10:00 (4 days, 13 hours)
   2. Input data insufficient in length: if there is a segment of data that is less than 71 hours in length the Godin filter cannot generate a result. This is because a total of 35 hours is removed from the beginning and end of any time series. For example:
      Cannot filter segment of less than 71 hours between 2006/07/25 00:00 and 2006/07/26 00:00

3. Run DECODES: DECODES program cvtstd is run automatically and is currently programmed to use GRFILTF-USGSGR-001 only.
4. Load results into ADAPS.
   
   1. The names of the *.sum and *.std files that were generated are displayed on the screen and the user is reminded to review the *.sum file.
   2. The user has the opportunity to automatically load the data into ADAPS by typing “Y”. The default is “N”, in which case the user would like to review the Decodes summary file and load the *.std file manually.

Script Language:
The basic script is ksh, but the underlying program called by the script is grfilt, which is written in Java. Decodes software (also Java) is used re-enter the filtered data into NWIS.

Requirements:
Both the grfilt and tidal_filter scripts need to be installed for the process to function properly and a specific Decodes configuration must be set up. Optionally, tidal_filter can also be configured to run as an automated “cron job” processing a list of sites. See the "readme" file for detailed installation instructions.

Original Author(s): Joel Johnson (tidal filter) and John Donovan (grfilt) - California WSC

Note the original author is noted on this page for the purposes of giving credit only. Please address questions/comments to OSW (GS-W OSW Scripts@usgs.gov).

Known Problems/Limitations/Need Improvements:

• Currently only works on discharge data. Needs to be expanded to stage.

Installation:

Upgrade instructions for a previously installed version are below (please follow the full install directions in the tide_README.doc if doing a fresh install available from the link in the "Scripts:" section below).

1. If installing a new server, tar up the old server's gr area and transfer it to the new server (skip this step if not installing a new server)
   As root or nwis on old server cd to directory containing the gr area then:
   tar -cvf gr.tar./gr
   As root or nwis on new server place the tar file where you want it, cd to that area then:
   tar -xvf gr.tar
   rm gr.tar

2. Unzip and untar the upgrade folder downloaded from the link below in the "Scripts:" section and copy the included files into the appropriate area of your gr directory (after migrating it to the new server from the old server if necessary), replacing the ones already there.
   as nwis or root in the area where the /gr directory is located,
   gzcat TidalFilter5update.tar.gz | tar -xvf -
   cp TidalFilter5update/grfilt gr/
   cp TidalFilter5update/tidal_filter gr/
   cp TidalFilter5update/tidal_filter_auto gr/CRON/

3. Change permissions as needed to ensure all files are executable by the user running tidal_filter (chmod +x fileif necessary).

Scripts

Version 5.0-2: (Posted 11/19/12)
Initial NWIS 5.0 version updated for more recent java versions and to allow scripts to run as a cron process with NWIS 5.0/Oracle. Version 2 released to fix issues with CA-specific code left in script and new issues discovered with cvtstd command. The following files were updated
—tidal_filter
—grfilt
—tidal_filter_auto

Full install package (tidal_filter not installed on old or new NWIS server):
Tidal Filter Readme file download (Zip file = 94K)
TidalFilter5full_package_download (Zip file = 731K)

Upgrade package (migrate old gr area to new NWIS server if necessary first):
TidalFilter5update_package_download (Zip file = 5.9K)

(NOTE: Correct file name for packages should end in .tar.gz. If given a name ending in .tar.tar or any other name by your downloading software please rename the file prior to install/upgrade.)