Water Resources Discipline ITAC Activities

_____________________________________________

ITAC,

Below is the primary info for Wed's 10:45a agenda item, "Issue clarification of 24x7 coverage"

--Katherine and Scott

Scott,

Progress is actually being made on the 24x7 issue for USGS. To quickly sum:

1- The Enterprise Service Center NOSC will be taking all USGS calls related to Internet 1 access problems as soon as USGS can successfully complete DOI’s Connection Approval Process (CAP). We anticipate that we will complete by September 15, 2005. This will assist some USGS users who have clearly identified Internet 1 access problems.

2- The NOSC will be taking USGS Field Office calls for FO routers after we move the Field Office routers to NOSC monitoring. Bob Wakelee has worked with the ITSOT to assure that appropriate IP addresses and access are allowed for NOSC/MCI personnel to view and access the field routers. We are moving these routers over to NOSC monitoring starting with the ER and moving west. This process <hopefully> will be complete by the end of CY 2005.

3- In parallel, the GIO is establishing a 24x7 response Incident Manager approach for all types of infrastructure problems to include: WAN, LAN, network security, DNS and server problems. The basic concept is that if the USGS Contact Center and/or the ESC NOSC cannot resolve the infrastructure problem they will call the duty USGS Incident Manager on-call. This Incident Manager (senior staff like the RCTOs and technical leads) will, in turn, have an array of WAN, ITSOT, server and LAN staff on-call to respond to these after hours problems. Establishing the criteria for problem referral to the Incident Manager and the duty rooster will be critical.

Scott, I am sure that Kevin will elaborate on the above at the ITAC meeting in Maine. We are trying to move ahead with the Survey’s critical need for IT 24x7 and remain within existing budgets. Not always an easy task.

The NOSC numbers and escalation contacts will be:

Enterprise Service Center - Network Operations Center Contacts 24x7

Hope this helps a little.

--Elaine
703.648.7006

Hi Elaine,

The subject topic is on the agenda for next week's WRD Info Technology Advisory Committee (ITAC) meeting in Maine. Katherine and I are to give the Committee an update. Can you share with me a status on the plans in the Word doc below that I can, in turn, share with ITAC? Also, did the Webpage to describe trouble reporting ever stand up? I found the following, but don't know that it addresses 24x7:
http://internal.usgs.gov/estout/webGEONET3/contacts/contactHome.html

Thanks so much Elaine.

--Scott

Scott,

Slight rewrite of your bullets, and I've attached my 24x7 current and future paper. Hope it helps.

During that same meeting, the GIO's Telecom Chief, Elaine Stout, addressed the topic of 24x7 network support. From my notes and some other research:

Years ago, WRD formed a Network Services Team consisting of Bob Wakelee, Mark Sweny, Jim Morris, and Tim Brown. This Team deployed software that "pinged" all district routers and paged the Team member “on-call” whenever a router became unavailable. The on-call Team member carried a pager and arranged a 365-day on-call schedule for 18-hours each day. WRD maintained a contacts list of Site Admins at district offices for when the need arose to contact a district SA. They also established a 1-800 network status message.
The Bureau has adopted a similar approach as the GIO transferred Bob Wakelee and telecom support services from WRD to the GIO. Details are found at: http://wrdnet.er.usgs.gov/contacts/index.html
The NatWeb Project established its own process for monitoring the network. Some SAs use NatWeb's "Big Brother" network/computer monitoring software to page them when their district router becomes unavailable. Ref http://natweb.usgs.gov/monitor/

Both the MCI and Sprint FTS2001 current support for USGS is on a best effort basis. Both carriers can monitor “their” side of the network, but since the routers are owned and operated by USGS, the carriers cannot both monitor and manage and do not have true end-to-end control. Consequently, this is viewed as best effort basis for both the WAN backbone and field offices.
With the ESN Phase 1 in-place (that is, Internet via Sprint and Intranet with MCI), the ESC (enterprise services center) NOSC (Network Operations / Security Center) is operational 24x7x365, and currently supports BIA and NPS. With NPS, Park Service employees still need to call a regional person in Parks who, in turn, contacts the NOSC. At this juncture, the NOSC does not permit all users to call directly. Starting CY05, the NOSC will be supporting other bureaus as they transition to ESN; the call procedure will be similar to NPS for several months. When USGS turns over monitoring and management control of its backbone routers to MCI and Sprint (for Internet 1), the USGS will be allowed to call the NOSC. It is still unclear whether those calls can be from all users or pooled thru some central point (like the USGS Call Center). DOI plans are for USGS to be ready to transfer management responsibility of the backbone April, 2005.

To:
cc:
Subject: NWIS ESC Meeting Thoughts

Elaine,

I'm trying to give my boss, Bill Carswell (Central Region Executive for Water), a synopsis of our 24x7 network support situation. I failed to take careful notes during your address a couple weeks ago to ITAC. I wonder if you could review my notes below, as well as write about the ESC/NOSC and it's plans for Sprint to perform 24x7 monitoring starting this Jan, and GEONET plans to stand up a USGS call center in March. Thanks much.

--Scott

During that same meeting, the GIO's Telecom Chief, Elaine Stout, addressed the topic of 24x7 network support. From my notes and some other research:

Years ago, WRD formed a Network Services Team consisting of Bob Wakelee, Mark Sweny, Jim Morris, and (I forget the other member). This Team deployed software that "pinged" all district routers and paged the Team whenever a router became unavailable. The Team carried pagers and arranged a 365-day on-call schedule for 18-hours each day. WRD maintained a contacts list of Site Admins at district offices for when the need arose to contact a district SA. They also established a 1-800 network status message.

The Bureau has adopted a similar approach as the GIO transferred Bob Wakelee and telecom support services from WRD to the GIO. Details are found at: http://wrdnet.er.usgs.gov/contacts/index.html
The NatWeb Project established it's own process for monitoring the network. Some SAs use NatWeb's "Big Brother" network/computer monitoring software to page them when their district router becomes unavailable. Ref http://natweb.usgs.gov/monitor/
Sprint's FTS2001 contract is to make a best effort to keep our network up. I don't recall details about Sprint's obligations to respond to outages. However, it seems that Sprint's ability to detect and respond to network outages and router outages is very limited at our district offices. I'm asking Elaine Stout for clarification and details.