When Hurricane Katrina and Hurricane Ike struck the Gulf Coast in 2005, the pipeline industry was greatly impacted. SCADA systems went down, the price of oil and gas spiked across the U.S., and there was an extended recovery period. Since 2005, midstream and pipeline operators have become increasingly more reliant upon their SCADA systems.
Most recently in 2017, Hurricane Harvey devastated Texas and parts of the Gulf Coast. Fortunately, the impact on the pipeline industry was not as significant as it could have been because of the business continuity, disaster recovery, and emergency response plans in place with the pipeline operators.
Despite the best efforts to minimize downtime, the pipeline industry continues to learn valuable lessons about the importance of business continuity (the ability to continue operations through a failure) and disaster recovery (the ability to return operations to normal). SCADA redundancy is critical to both.
SCADA Redundancy: Is Your System Still Operable During a Disaster?
The disasters in 2005 revealed that many SCADA systems were not prepared to continue operations if there was a systemwide failure. The computers and control systems used to operate pipeline operations were inoperable, which prolonged downtime and created massive costs for operators.
In the aftermath, pipeline operators realized the importance of SCADA redundancy to continue operating if there is a systemwide failure during a disaster. That led to more emphasis on backup systems, additional data servers, and enhanced communication protocols.
Twelve years later, the challenge was different. Rapid advances in technology, industry growth, and the increasing cybersecurity threat for pipeline operators made reliable and robust operations even more challenging.
Additionally, the need to reduce the cost of operations combined with the dropping cost of automation and telemetry meant that pipeline operators were more reliant on SCADA to operate. That picture pointed to a much more extensive system to manage during a natural disaster.
In 2017, the SCADA redundancy capability of these larger systems was put to the test. During Hurricane Harvey, some pipeline operators were forced to temporarily suspend operations and the U.S. Department of Energy tapped into the Strategic Petroleum Reserve to prevent the price of oil and gas from spiking.
The lesson learned was that each stage of pipeline system growth must include equal emphasis on SCADA and redundancy. The operator cannot rely on a system with a single point of failure.
SCADA Failure: What Are the Causes?
To properly design a SCADA system with effective redundancy, it is necessary to understand the typical causes of SCADA failure. Beyond natural disasters, there are many other factors to consider.
- Infrastructure
- Power: availability and quality
- Air conditioning required to prevent servers from overheating
- Communications, including cyber secure networks
- Geography: a backup facility that is not impacted by same natural disaster as the primary
- System Complexity
- Cascading Failure: a fault in one system causes a fault in another system
- Escalating Failure: loss of one system prohibits mitigation in a separate critical system
- Common Cause Failure: Loss of a power grid impacts a primary and backup facility
- Cybersecurity
SCADA Disaster Recovery: How to Manage the Aftermath
SCADA redundancy can be necessary to continue operations during a natural disaster. However, a system could still go down in the aftermath if there is a significant amount of stress on the supporting systems during the actual event.
The key for pipeline operators is building, testing, and refining your SCADA disaster recovery plan. This process will help reduce the risk of downtime, operating issues, leaks, and other damage that could be financially costly as well as harmful to communities.
What should go into your SCADA disaster recovery plan? A terrific example can be found in Pinellas County (Florida), where their Utility team created a plan that included diagrams of the SCADA system, a network device listing, location maps, and backup and restore procedures linked to data servers.
“By bringing the documents and drawings current with the existing SCADA system, and by developing a customized backup and restore procedure, the county is now prepared to return to operations should a condition happen to render part of, or the entire system, inoperable,” said Mike Skrzypek, the Pinellas County Utilities SCADA & security systems manager.
The lesson learned about SCADA disaster recovery is to bring each aspect of the system into alignment, have a method to track the status of each site, and regularly test your backup procedures to reduce the risk of becoming inoperable.
How EnerSys Supports Your SCADA Redundancy and Disaster Recovery Capabilities
As part of our POEMS software suite, EnerSys built the Intelligent Operator Console (IOC) that provides a high-performance HMI for pipeline and midstream operators. This software includes three key components — HMI Philosophy, HMI Style Guide, and HMI Design Guide — that support your SCADA system.
- HMI Philosophy: Includes policies for high-performance HMI style and design.
- HMI Style Guide: Defines the graphical elements, system behaviors, and animations.
- HMI Design Guide: Defines the SCADA architecture, security measures, network topology, and information dataflow.
The POEMS process creates and delivers a customer-specific set of documentation that becomes a key element of a SCADA Disaster Recovery Plan. These include a Functional Specification, Topology Diagram, Data Flow Diagram, and a System Administration Guide.
By linking together documents and the system design in one location, your SCADA system will be able to operate consistently under standard operating conditions and react appropriately during abnormal conditions and natural disasters.
We believe this step of implementing advanced planning actions to improve SCADA redundancy and disaster recovery will protect your pipeline operation during the next major event.
To schedule a demo of the POEMS IOC software or to discuss your operation’s current SCADA system, please complete our contact form, email our team at sales@enersyscorp.com, or call us directly at 281-598-7100.