ITAC Activities

http://web409.keynote.com/MyKeynote/Post/usgswink_051502_083018.asp look at the graph and note that the weekends don't have the same spikes as the weekdays, you can see the actual data in the table at the bottom.

http://web409.keynote.com/MyKeynote/Post/usgswink_051502_083720.asp this one shows all the data points whereas the first shows an average for each 4-hour period.

now we're looking at if performance for the last two days may be related to a POTENTIAL denial of service attack on water..... more coming.

------------ Additional Email Follow ------------

This email is serving several purposes. For the ITAC, please read this and I would like to discuss during my status briefing Wed afternoon. For others it is FYI and pondering and discussion (we should have a conference call soon). There is a lot of data included below -- I'm going home now.

NOTE:   We have procured an 8 processor v880 sun server to off-load the NWISWeb software onto. I am confident that it will improve things -- I am not confident it will be a long term solution. We have received the system with 4 of the cpus and are awaiting the other 4. The new system as is will be faster than the current water server and we are attempting to bring that system on-line ASAP and will add the add'l cpus when they arrive. It is possible that it'll happen this week, but I'm not positive.

The Keynote performance scatterplot graphs for yesterday and today showed ~140 seconds to deliver the water.usgs.gov/index.html page. I saved two scatterplot graphs this evening; the points are interactive and will display detailed information.

http://web405.keynote.com/MyKeynote/Post/usgswink_051402_163255.asp
http://web409.keynote.com/MyKeynote/Post/usgswink_051402_173143.asp

You can view and graph performance data available from Keynote:

url:	http://mykeynote.com
userid:	usgswink
password:	7ZPGMCZM

Following is an excerpt from the analog statistics about the NWISWeb system produced this morning. The statistics analyzed requests from Tue-14-May-2002 00:00 to Tue-14-May-2002 10:30 (0.44 days). We have been chasing down reasons for extremely poor performance on water for some time. The stats were generated to help find data miners conducting their activities in a currently unacceptable manner.

http://natweb.usgs.gov/server_stats/2002/usgs/nwisweb0514_i.html#host

reqs:	%bytes:	host
------:	----------:	------
448:	2.31%:	cms1.rsn.com
418:	2.11%:	mistral.atd.ucar.edu

*note from link above that other miners have >2600 reqs with <1% of bytes served.

The listed machines have been blocked from access to the water server (and waterdata). Once we block systems the following special error message is be generated by water(data).usgs.gov for subsequent access attempts (we did this several weeks back).

Your use of the USGS NWISWeb system is important to us. However, the server's usage statistics have indicated that your information request(s) have exceeded the server performance thresholds and is impacting the server's ability to provide information to all users. Your information request most likely exceeded the performance thresholds due to too many data requests within a very short time frame or a malformed query. Please contact the support team at gs-w_support _nwisweb@usgs.gov so that we can provide assistance with optimizing you requests/queries and return your access to our system.

I was able to resolve the names from today to webpages:

• rsn.com is serving http://rsn.com/water_conditions RSN is an undefined acronym
• http://www.ucar.edu/ucar/ University Corporation for Atmospheric Research

1. RSN:   Apparently provides data from across the Nation for the recreation community which has been mined and repurposed from various sources. This includes USGS stream gage, buoy (other agency), and other data. There is an inconsistent method of providing citations to the source providers of the data; I did not notice any citations for USGS even though they are providing hydrographs, etc. which have been produced by the NWISWeb software. (They are using the hydrographs that do not have the identifier and don't reference where the data comes from.) http://rsn.com/conditions/1,2849,226321,00.html.

Not much of a different situation then when AWA was first found to be mining our data. We can send them an email and ask someone to call (or they will contact us now that they can't get into the system).

Now, it gets sticky. Since there are some organizations (AWA for one) which mine our hydrographs and data and use them, is it appropriate that we cut off others. http://www.americanwhitewater.org/gauges/id/183/.

I am nervous about continuing to "recommend" that user's mine the presentation graphic (provide them with the most efficient means to mine us) because the performance on the server is now unacceptable.

2. For UCAR I couldn't locate exactly where they are using our data but I found the following presentation slide:   http://www.ucar.edu/governance/meetings/apr02/followup/unidata/sld022.htm

Yesterday we blocked another domain after we ran statistics due to very similar slow response time on water.usgs.gov. http://natweb.usgs.gov/server_stats/2002/usgs/nwisweb0513_i.html#host

We found the University of New Hampshire had several machines mining the system with collectively ~22% of all bytes retrieved for the reference time period.

ACTION:   We have established another cron job to run the nwisweb stats for each 24 hour period just after midnight. We will begin examining them in the morning along with the keynote data and try to be proactive about restricting excessive mining activity and watch from there. I will be putting a page up in the NatWeb area that will have
• links to these daily stats as evidence when we block any systems
• information about who we are blocking
• info about who may have contacted us about mining activities.

This information/page will need to be kept updated by the NWISWeb and NatWeb teams so we can get a good handle on what the solution is.

Bob Hirsch previously said that if mining was bringing the server to it's knees it was reasonable to block that activity, the system was not designed for that usage. Performance on the server is reaching an unacceptable level, and I'm not sure that the new server will solve the problem, very soon we'll have two choices which will need to be evaluated:

• provide another sun server that we point all data miners at (anyone making more than n requests of the machine in a 24 hr period).
• provide an ftp depot with the hydrographs and data available for miners to download and cut off all users requesting more than n requests in a 24 hr period.