Loading...
HomeMy WebLinkAbout35-Fire Department CITY OF SAN BERNARDINO - REQUEST FOR COUNCIL ACTION From: Larry R. Pitzer Subject: Status Report of the City's Year 2000 (Y2K) preparedness and Y2K Emergency Plan. Dept: Fire Date: December 20,1999 ORIGINAL Synopsis of Previous Council Action: None RECOMMENDATION: Receive and file. Contact Person: Larry R. Pitzer Supporting data attached: Staff Report Phone: (909)384-5286 Ward: City Wide FUNDING REQUIREMENTS: Amount: N/A. Source: N/A. Finance: Council Notes: Agenda Item No. 35' '~I~)~" CITY OF SAN BERNARDINO - REQUEST FOR COUNCIL AC~ STAFF REPORT SUBJECT: Status report of the City's Year 2000 (Y2K) preparedness and Y2K Emergency Plan. BACKGROUND: During the past five years, the City of San Bernardino has been engaged in a thorough assessment and remediation process to examine and correct possible computer, embedded chip, and other technical problems related to the Year 2000 computer problem. In 1995, a plan for major server replacements was developed by the City's Management Information Services (MIS) Department to address the Y2K technology problem. Beginning in 1996, MIS began its remediation projects with Y2K upgrades. In 1998, the City Administrator established a Y2K Compliance Project Committee that was charged with developing an action plan for resolving the City's non-compliant Y2K computer and electronic systems and developing contingency plans in the event of system failures. This Committee is represented by key City departments impacted by the Y2K problem including: City Administrator's Office, Management Information Services, Fire, Police, Development Services, Finance, Facilities Management, Public Services, Water, and Parks and Recreation Departments. ASSESSMENT OF READINESS: The City of San Bernardino has analyzed all City hardware and software computer systems with the goal of bringing all systems into compliance. The City has also worked closely with other govemmental agencies and service providers, including Southern California Edison and GTE, to ensure Y2K issues have been addressed properly. Utilizing a project management focus, the City's Y2K Compliance Project Committee has successfully implemented its project milestones including: I. Awareness - developing an overall strategy and communicating the issue to our employees and City residents; 2. Inventory and Assessment - assessing the scope of work required to be millennium-ready or Y2K compliant; 3. Remediation - renovating or changing systems to make them Y2K compliant; 4. Testing - certification of new or repaired systems, including the testing of the critical connections and interfaces between systems; and 5. Implementation - putting new and repaired systems online, ready for use. Staff Report December 20, 1999 Page 2 The City has identified its critical systems that have been grouped into the following categories: Public Safety, City Business, Public Utilities, and Public Facilities. Public Safety systems consist of the Police and Fire Department computer-aided dispatch/record management system, 911 emergency telephone system, traffic signal operation, and other support functions. The major public safety systems have been remediated and tested. City Business systems consist of major services such as licensing, building permits, finance and accounting, payroll, personnel, utility billing, desktop computers, and other similar systems. These systems have also been remediated and tested, where necessary. The City's Public Utilities systems pertain to water and sewer issues. These systems have been remediated and tested. The City's Public Facilities systems include telephones and elevators (located within public facilities only), HV AC, fire protection and security systems. All critical systems have been remediated and tested. In addition to these actions the City has developed contingency plans to minimize the impact of potential service disruptions. These plans are consistent with the City's Emergency Plan and will be implemented in the event of any Y2K-related problems. Similarly, a comprehensive public education campaign has been implemented to ensure that residents are properly informed about ways to prepare for potential Y2K problems. Overall, the City's Y2K Compliance Committee is confident that our services to the community will continue into the Year 2000 without significant interruption or degradation. PROPOSED Y2K STAFFING: Two levels of City-wide staffing are proposed for the New Year's holiday weekend. The first level will be staff assigned to the City's Emergency Operations Center (EOC). The second level consists of staff designated to be working or "on call" for department/field response problems. These departments include: Animal Control, Development Services, Fire, MIS, Police, Public Services and the Water Department. The EOC will be partially activated from 9:00 p.m. on December 31, 1999 through 2:00 a.m. on January 1,2000. Staff from the following departments will report to the EOC: Facilities Management (1) Finance (1) Fire (2) MIS (9) -- located in "B" Building Parks, Recreation, and Community Services (1) Police (1) Staff Report December 20, 1999 Page 3 All other pre-designated EOC staff will be "on call" during this critical time period and should be available to respond within 30 minutes. In the event it is necessary to fully activate the EOC, additional personnel will be contacted via telephone and/or pager. Should communications be disrupted, EOC staff will follow the regular EOC staffing schedule assigned to each impacted department. Both Fire and Police Departments will have additional staff on duty and/or "on call" during the New Year's holiday weekend. The Fire Department will have one additional Division Chief on stand by at the EOC and two staffed reserve engines with a crew of three each from 8:00 p.m. on December 31,1999 to 8:00 a.m. on January 1, 2000. In addition to a partial activation of the Police Department's Operations Center on December 31, there will be a fifty-man Rapid Deployment Force in the field to respond to any type of civil incident that may arise. This Force is in addition to the normal patrols that will be on duty during the New Year's holiday weekend. RECOMMENDATION: Receive and file. CITY OF SAN BERNARDINO YEAR 2000 (Y2K) COMPLIANCE PROJECT FINAL REPORT (DECEMBER 1999) PURPOSE The City of San Bernardino's Y2K Compliance Project has been established to ensure the continued functionality of computer systems (hardware and software), facilities! infrastructure and the delivery of vital services and supplies that support the City's business operations before, during and after January I, 2000. The reliable operation of these systems and the continued delivery of these services and supplies are critical components to the provision of essential public services. CITY'S Y2K COMPLIANCE PROJECT COMMITTEE In 1998, the City Administrator established a Y2K Compliance Project Committee that was charged with developing an action plan for resolving the City's non-compliant Y2K computer and electronic systems and developing contingency plans in the event of system failures. This Committee is represented by key City departments impacted by the Y2K problem including: City Administrator's Office, Management Information Services, Fire, Police, Development Services, Finance, Facilities Management, Public Services, Water, and Parks and Recreation. The objectives of the Y2K Compliance Project Committee include: I. To understand the impact of the Y2K problem. 2. To determine the effect on the City's key infrastructure areas. 3. To develop strategies for addressing the problem where there is greatest risk for significant disruptions. 4. To develop a public awareness campaign that demonstrates the City's commitment to preparing for this potential challenge and educates residents on preparedness measures. NOTE: It should also be noted that the MIS Department had initiated the process to ensure that the City's computer systems were Y2K compliant beginning in 1996. FIVE-STEP MANAGEMENT PROCESS To accomplish the objectives listed above, the Committee adopted an action plan based on a Five-Step Management Process consisting of: 1. Awareness Phase -- The purpose of this phase was to gain a general understanding of Year 2000 issues. City departments were briefed on the Committee's objectives and began disseminating information to staff. In this phase, each key department identified a Y2K advocate. This person has been the focal point for the department's Y2K activities. Part of this phase has been ongoing as new information has been obtained. Key activities during this phase have included: . Identification of a Department Y2K advocate . Defining the project's scope . Defining what will be reviewed (departmental systems, PC's, embedded systems, and vendors) . Defining project management structure . Defining tracking and reporting . Identifying other staff involvement . Communicating goals to Departments . Documenting awareness activities Y2K Compliance Project Final Report Page 2 2. Assessment Phase - The purpose of this phase was to understand how the Year 2000 may effect City operations. This began with a complete inventory followed by a review of risk, priorities, and solution alternatives. This led to a plan and cost estimates. Contingency planning was also considered. Departmental inventories were continually reviewed and every decision documented for due diligence. Key activities during this phase have included: . Completing departmental inventories . Reviewing inventories for risk, priorities, solution alternatives . Making remediation decisions . Developing contingency plans . Documenting decisions . Updating Y2K plan for remediation activities . Estimating time and cost of remediations 3. Remediation Phase -- The purpose of this phase was to finalize the plan for any corrective actions that needed to be taken to achieve Year 2000 compliancy. This included the actual fixing or replacing of items. Activities in this phase were also documented for due diligence. Key activities during this phase have included: . Determining remediation strategy . Replacing/fixing as needed . Documenting activities 4. Testing Phase - The purpose of this phase was to develop a test strategy and perform any required testing. Based on the item and type of remediation, this process was simple in some cases and very involved in other cases. It included unit and integration testing and the testing of interfaces. It was also necessary to involve vendors in some cases. Key activities during this phase have included: . Developing a test plan . Determining test strategy . Developing test data . Performing tests . Documenting results 5. Implementation Phase -- The purpose of this phase was to develop an implementation plan and to perform actual implementation tasks. In many instances, it was necessary to schedule implementations during weekends thus causing minimal disruption to daily operations. Key activities during this phase have included: . Developing implementation plan . Implementing replacements/fixes . Documentation SIGNIFICANT EVENTS Development Services - . Conducted a thorough inventory of embedded chip systems, including traffic signals, and examined each for Y2K compliance. . Implemented necessary upgrades to traffic signal system to ensure continued operability during the Y2K rollover date. . Coordinated efforts with Public Services to ensure a contingency plan was developed for back-up systems to sewer pump stations. Y2K Compliance Project Final Report Page 3 Facilities Management - . Conducted a thorough inventory of embedded chip devices citywide, such as security and environmental control systems, elevators, voice mail systems, and telephone instruments and examined each for Y2K compliance. . Coordinated with appropriate vendors to ensure communications and electronic equipment are Y2K compliant. Fire Department - . Conducted a thorough inventory of embedded chip devices including defibrillators, fire engine units and communications equipment and examined each for Y2K compliance. . Coordinated with vendors to ensure products and services are Y2K compliant. . Determined 9-1-1 emergency communications system is Y2K compliant. . Established a Y2K Web Site that is currently accessible from the City's main home page. The site contains comprehensive information on the City's compliance project and emergency preparedness measures for individuals and families. . The Disaster Preparedness Division conducted a series of training opportunities for City staff involved in emergency response roles. The training program culminated in a Citywide Y2K Readiness Exercise held on October 20, 1999. . The Disaster Preparedness Division coordinated a public information campaign, with the assistance of the Mayor's Office, to ensure the business community and the general public were aware of the City's compliance project and self preparedness measures. . Submitted a Y2K Preparedness CRN to the Telecommunications Department which will be run throughout the month of December 1999. . The Fire Department has coordinated with the City's primary ambulance provider, American Medical Response (AMR), to have an ambulance unit pre-positioned at key fire stations in the event of widespread communication failures. . Established a plan for staffing of emergency response personnel as well as staffmg for the City's Emergency Operations Center on New Year's Eve. . The Disaster Preparedness Division, in conjunction with the Water Department, developed a Y2K Preparedness insert for all San Bernardino residents that will be distributed with the December water bills. . The Disaster Preparedness Division has developed a draft Citywide Y2K Emergency Plan in the event of system failures during the rollover date. Management Information Services (MIS) - PC Hardware . MIS has visited each of the City's 1100 pes twice over the last six months to check and recheck for Y2K issues. . The minimal number ofY2K issues encountered were rectified. It is possible a few machines will surface with a problem after January 1, 2000. MIS will handle those on an individual basis as they are identified. PC Software and Overatinf! System . Early releases of Windows 95 were not Y2K compliant. The fix is readily available from Microsoft's web site. MIS made sure the Y2K compliant version was installed on the desktops as part of the Y2K checks noted above. Y2K Compliance Project Final Report Page 4 . A wide variety of software is in use on individual PCs throughout the City beyond the Microsoft products MIS supports. It is possible some of these products will experience Y2K related problems. MIS will attempt to assist any users who can identify the vendor contact and phone number of their particular piece of software. Server Based Software A1JTJlications & Overatiml Svstems CAD/RMS . Public Safety's major software applications which are CAD (Computer Aided Dispatching) and RMS (Records Management System) were not Y2K compliant. New Y2K compliant software would not run on the old hardware. Council approved a citywide technology upgrade in fiscal 96/97 that included total replacement of Public Safety CAD and RMS, hardware and software. . The Public Safety conversion effort was estimated to be a 2-year project. The new CAD is fully functional and in use by both Police and Fire. Police RMS is fully functional and in use. Fire RMS is scheduled to begin live usage December 13, 1999. . There are a number of minor bugs and requested enhancements that continue to be addressed by the Public Safety vendor on these systems, however, those items do not represent problems related to Y2K. . The new CAD and RMS utilize the Unix operating system. Unix required Y2K patches that the vendor installed on November 24, 1999. Sable . The City's business or financial server referred to as "Sable" has the ability to change the system clock to 2000 or beyond. . MIS scheduled a Y2K test, changed the year to 2000, and with the assistance of various departments, ran a subset of the most critical applications residing on Sable. Minimal issues were encountered and MIS technical staff were able to make the necessary program modifications. . The Unix operating system used by Sable had Y2K patches installed late November 99 and is expected to be fully compliant. A second day of tests on Sable occurred December 3, 1999, and verified that no new issues were created by the operating system upgrade. Utility Billing (UB) . MIS began support of the Water Department July I, 1998. Their utility billing system was very old, non-compliant software running on an equally old IBM system. . The department chose new hardware and software and MIS began the replacement project in fall of 1998. . Water cut over to their new system in October 99. There are a number of issues still being addressed, but they are not Y2K related. Water Financial System (WFS) . Beginning in the Spring of 1998, in anticipation of contracting with Water, MIS began assisting the Water Department in a conversion effort to new financial hardware and software. The new system is referred to as WFS (Water Financial System). Y2K Compliance Project Final Report Page 5 . The Water Department chose identical hardware to the City's Sable system. They also chose the same fInancial software vendor as the City. MIS has performed the same operating system (Unix) upgrade on WFS as on Sable. . The Water Department has the latest version of the fInancial software which is Y2K compliant. Document Imaging . The City's Imaging vendor identifIed some minor Y2K related upgrades and made those changes the week ofDecernber 6, 1999. Web Server, GIS, Office Automation Servers. Permits Servers . No issues. City Network Irifrastructure . The City's fIrewall software has required Y2K upgrades which have been installed by MIS. The rest of the backbone infrastructure has no known issues. Radios and 800 MHz Infrastructure . The County has tested to the extent they can and have found no Y2K issues. Motorola has certifIed all their equipment as Y2K compliant. Parks, Recreation and Community Services - . Conducted a thorough inventory of all lighting and irrigation systems and determined that systems are manual in most cases and others are not date sensitive in their operations. . One irrigation system that is computer controlled has had updated Y2K software installed by the vendor and a new computer has been installed by MIS. Police Department - . Conducted a thorough inventory of computer and embedded chip systems and examined each for Y2K compliance. . Coordinated with Facilities Management to ensure elevators, generators, fIre alarms and phone switches are Y2K compliant. . Coordinated with MIS to ensure Police Dispatch dictaphone is Y2K compliant. . Distributed a department-wide policy on Y2K staffIng levels for the period from December 31, 1999 through January 3, 2000. Public Services Department - . Conducted a thorough inventory of computer and embedded chip systems and examined each for Y2K compliance. . Public Services has acquired a 100 kw generator in the event of power failure and has arranged to rent a 50 kw generator to provide a back-up for sewer lifts and the City's garage. . Coordinated with appropriate vendors to ensure City's fleet vehicles are Y2K compliant. Y2K Compliance Project Final Report Page 6 Water Department- . Electrical controls on 55 welJs, 4 treatment plants, 32 boosting stations, and 24 chlorinating stations (City and IVDA) have been checked and verified for Y2K compliance. The new SCADA system is Y2K compliant. The old system cannot be totally verified; however, it is primarily for monitoring. The system is overridden by hand control. Water Department employees have extensive and continuous practice at doing this. . The Water Department has acquired one 500 kw and two 175 kw portable generators and has one 150 kw generator on order scheduled for delivery in late December 1999. In case of total power failure, the generators wilJ be placed at key high elevation welJs and/or moved around throughout the water system as needed. . Three welJs located in the north portion of the Water Department's service area have diesel and/or natural gas stand-by engines as power sources. . The Water Department has acquired three gasoline driven 500-1000 gpm portable booster pumps that can by-pass electronicalJy driven boosters in case of power failure. . The Water Department has quick electrical connects for the Operations Control SCADA system, emergency lighting and the gas pumps at 195 "D." . Key employees in Engineering, Water Treatment Operations, Electrical Pump and Control Maintenance, Water Distribution System Repair, Water Quality Control and Water Administration (Management) and Procurement wilJ be on stand-by status from December 31, 1999 through January 3,2000. . An adequate supply of water treatment chemicals are in stock to maintain the potability of the City water supply should normal supplies be temporarily disrupted. . The Water Department also maintains Mutual Aid (some are informal agreements) with several neighboring Water agencies (e.g., Upper Santa Ana Water Resources Association) to provide mutual assistance during emergencies which would include generators, water boosters, water treatment chemicals and certified Water Distribution and Treatment personnel as needed. PROPOSED Y2K STAFFING Two levels of City-wide staffing are proposed for the New Year holiday weekend. The first level wilJ be staff assigned to the City's Emergency Operations Center (EOC). The second level consists of staff designated to be working or "on call" for department/field response problems. These departments include: Animal Control, Development Services, Fire, MlS, Police, Public Services, and the Water Department. The EOC wilJ bepartialJy activated from 9:00 p.m. on December 31,1999 through 2:00 a.m. on January 1,2000. Staff from the folJowing departments wilJ report to the EOC: Facilities Management (I) Finance (I) Fire (2) Parks, Recreation & Community Services (I) Police (I) MIS (9) --located in "B" Building AlJ other pre-designated EOC staff wilJ be "on calJ" during this critical time period and should be available to respond within 30 minutes. In the event it is necessary to fulJy activate the EOC, additional personnel wilJ be contacted via telephone and/or pager. Should communications be disrupted, EOC staffwilJ folJow the regular EOC staffing schedule assigned to each impacted department. Y2K Compliance Project Final Report Page 7 Both Fire and Police Departments will have additional staff on duty and/or "on call" during the New Year's holiday weekend. The Fire Department will have one additional Division Chief on stand by at the EOC and two staffed reserve engines with a crew of three each from 8:00 p.m. on December 31, 1999 to 8:00 a.m. on January I, 2000. In addition to a partial activation of the Police Department's Operations Center on December 31, there will be a fifty-man Rapid Deployment Force in the field to respond to any type of civil incident that may arise. This Force is in addition to the normal patrols that will be on duty during the New Year's holiday weekend. STAFF ACKNOWLEDGEMENT The Co-Chairs of the City's Y2K Compliance Project wish to acknowledge the diligent efforts of the following Project Team members: Miguel scarrunz Disaster Preparedness Coordinator, Fire Department .t Y2K Compliance Project Co-Chair Dave Batson Tom Boggs Linda Ceballos June Durr B/C Matt Fratus Tom Marek John Murphy Margaret Park Lt. Roger Poyzer Off. Robert Richards Lori Sassoon Jim Sharer Robert Simmons Rita West Janet Mauren Respectfully submitted, Finance Department Parks, Recreation & Community Services Department Public Services Department Mayor's Office Fire Department Human Resources Department Water Department Development Services Department Police Department Police Department City Administrator's Office Facilities Management Department City Attorney's Office Finance Department Facilities Management Department ~~ Director, MIS Oep.rtmeut .t Y2K Complimce Project Co-Chair