2021 PHMSA CRM Enforcement Actions: Improve Alarm Management and Adequate Information

It’s time for many pipeline operators to consider implementing alarm management software to elevate the quality of their Alarm Management program. We have arrived at this conclusion after reviewing the 2021 PHMSA Enforcement Actions in the control room and the 2022 CRM findings through the first half of this year.

In 2021, there were 37 findings related to Alarm Management (Section E of the PHMSA Control Room Management Rule) in the CRM Enforcement Actions. One such finding was included in a Proposed Civil Penalty that came with a $54,900 proposed fine.

Additionally, pipeline operators need to focus on how to improve Adequate Information (Section C of the CRM Rule). In our research of 2021 findings, we identified 22 issues stemming from non-compliance with this section of the rule. One operator was subject to a Proposed Civil Penalty that included a $58,400 proposed fine for a violation.

In the continuation of our series covering recent PHMSA CRM Enforcement Actions, we will share lessons learned about Alarm Management and Adequate Information to help you identify areas where your control room can grow. We will also present our solutions for your operation to achieve Natural Compliance with regulations.

– Be sure to bookmark the EnerSys blog and subscribe to our email newsletter to continue receiving information from our series.

Section E – Alarm Management Lessons Learned

According to the CRM Rule, the baseline requirement for Alarm Management is that each operator using a SCADA system “must have a written alarm management plan to provide for effective controller response to alarms.”

As noted before, we identified 37 violations of this requirement in 2021. During the first half of 2022, we identified 23 additional Alarm Management findings. Some findings were noted in Warning Letters that included no proposed fine. However, others were more serious. Let’s review the lessons learned.

– One operator was subject to a $54,900 proposed fine due to failure to provide records that demonstrated compliance with the requirements of Section E.

Specifically, the operator was unable to provide records to verify that the correct safety-related alarm set point values and alarm descriptors when associated field instruments were calibrated or changed at least once each calendar year, but at intervals not to exceed 15 months. A total of three separate instances were identified for this operator.

– Three additional operators were subject to fines due to failures to perform monthly alarm reviews, which is something that we can address through the combination of our POEMS® CRM Suite CRMgr software module and CRM Compliance Support services. Consider these examples of alarm review violations.

1. An operator was subject to a Proposed Civil Penalty that included a $37,100 suggested fine for a Section E finding. PHMSA alleged that the operator failed to identify at least once each calendar month the points affecting safety that have had forced or manual values for periods of time exceeding that required for associated maintenance or operating activities.

2. An operator faced a $23,100 proposed fine for a similar finding. The operator was cited for failing to identify at least once each calendar month the points affecting safety that were taken off scan in the SCADA host, had alarms inhibited, generated false alarms, or that had forced or manual values for periods of time exceeding that required for associated maintenance or operating activities. PHMSA noted this violation occurred over a three-year period.

3. An operator faced a $19,300 proposed fine due to a lack of proper Alarm Management recordkeeping. PHMSA alleged that the operator’s records for the monthly review of points that were taken off scan, had alarms inhibited, generated false alarms, or that had forced or manual values failed to comply with the regulation.

PHMSA noted that while the operator’s control room staff performed daily reviews of alarms, the monthly alarm reporting did not reflect the review of false alarms or forced or manual values over a three-year period.

– We have covered the incidents that were accompanied by proposed fines. There were dozens of other findings that did not lead to proposed fines, but they were still significant because they captured common non-compliance themes. We have summarized additional Alarm Management findings that operators need to pay attention to:

• Alarm Management plan did not provide for an effective controller response to alarms.
• Procedures for SCADA and Alarm Management do not provide a method to quantify the amount of general activity a controller is responsible for managing during a shift.
• Failed to complete a workload analysis for a given year.
• CRMP (Control Room Management Plan) was inadequate because it did not address the monthly identification, recording, review, and analysis of bad actors.
• CRMP lacked criteria to determine when a controller has sufficient time to analyze and react to incoming alarms.
• Failed to demonstrate whether metrics were reviewed to determine the effectiveness of the Alarm Management plan.
• Failed to produce records to demonstrate that the operator performed a program review to identify any deficiencies.

We also identified Alarm Management findings that were coded to other sections of the CRM Rule. Consider these additional examples:

• Section J (Compliance and Deviations) finding: an operator failed to demonstrate compliance with its review of the Alarm Management Plan to determine its effectiveness.
• Section A (General Requirements) finding: an operator failed to review the Alarm Management plan to determine its effectiveness.
• Section A finding: an operator failed to follow its CRM Manual, which required a review of the Alarm Management Plan at least once each calendar year.
• Section A finding: an operator failed to provide a procedure to satisfy PHMSA requirements requiring the verification of the correct safety-related alarm set-point values and descriptions when the associated field instruments are calibrated or changed at least once each calendar year.

Section C – Adequate Information Lessons Learned

According to the CRM Rule, the baseline requirement for Adequate Information is that each operator must provide its controllers with “the information, tools, processes, and procedures necessary for the controllers to carry out the roles and responsibilities the operator has defined.”

As noted before, we identified 22 instances of non-compliance in 2021. Through the first half of 2022, we have already identified 25 additional findings. Some findings were nominal in nature that were noted in Warning Letters, while other violations were more severe in nature. Let’s review the lessons learned.

– PHMSA issued a Proposed Civil Penalty to an operator for multiple CRM Rule violations. One of the Adequate Information findings carried a proposed fine of $58,400 due to an Adequate Information violation of not providing personnel with necessary information when changes were initiated.

Specifically, PHMSA alleged that the operator failed to conduct point-to-point verification between SCADA displays and related field equipment when field equipment was added or moved and when other changes that affect pipeline safety were made to field equipment or SCADA displays. A total of three instances were identified.

– One operator received a Notice of Probable Violation, Proposed Civil Penalty, and Proposed Compliance Order covering CRM Rule findings and other pipeline safety violations. The grand total for all violations was millions of dollars. Within the CRM Rule findings, there was an Adequate Information violation.

PHMSA alleged that the operator failed to provide its controllers with the information, tools, processes, and procedures necessary for the controllers to carry out the roles and responsibilities that were defined for abnormal and emergency operating conditions. Specifically, the controllers had “inadequate information” to recognize a pipeline failure.

– We have covered the big-ticket items related to Adequate Information. There were numerous other Section E findings that are worth highlighting:

• CRMP was inadequate because it did not provide enough detail in the internal communication plan to guide safe manual operations.
• CRMP was inadequate because it did not require the outgoing and oncoming controllers to sign the shift log at the end of the shift turnover meeting.
• Failed to provide adequate clarity on what information is required to be exchanged when a different controller assumes responsibility.
• Records for SCADA testing drills failed to capture how the drills were conducted, what the scenarios for each drill included, and how the drills were documented.
• An operator failed to test and document their backup SCADA server.
• CRMP did not provide definitions or examples for the addition, expansion, or replacement of SCADA systems.
• CRMP was inadequate because it did not clearly define the types of changes to the SCADA system(s) that constitute additions, expansions, or replacements.
• An operator could not produce records indicating they evaluated their SCADA system according to the requirements of API RP 1165 (Recommended Practice for Pipeline SCADA Displays).
• CRMP did not provide directions to conduct and document a thorough point-to-point verification.
• Failed to complete an adequate point-to-point verification between SCADA displays and related field equipment when field equipment was added or moved.

Solutions to Align with the PHMSA CRM Rule

Through a combination of EnerSys software and services, we can help your operation incorporate these lessons learned, prevent non-compliance, and operate in a state of Natural Compliance with PHMSA CRM requirements.

We recommend utilizing our Control Room Management software suite (CRM Suite) and working with our team of expert consultants to help you close gaps and improve operations.

– Alarm Management solution: Recordkeeping and documentation were recurring themes in the PHMSA CRM Enforcement Actions pertaining to Alarm Management. These issues can easily be solved by implementing alarm management software that will streamline recordkeeping and provide your operation with records that satisfy a PHMSA audit.

Use the ALMgr module in our CRM Suite to conform to PHMSA CRM Rule requirements, support your pipeline alarm management program, and expedite the alarm rationalization process. You can also generate analytical reports in the software, identify bad actors, and implement corrective action.

– Adequate Information solution: We offer multiple solutions to address common themes related to point-to-point verification, SCADA system changes, shift handover, and compliance logging.

• Use our ComplyMgr software module to identify gaps between the CRM Rule and your processes, procedures, and implementation.
• Use our CRM Support Services to perform backup control room tests and log the results in your compliance log.
• Use our PointMgr software module to streamline the point-to-point verification process between the control room and field techs.
• Use our Pipeline SCADA solution to execute an API RP 1165 assessment to ensure alignment with the industry standard.
• Overall, we can help you track and manage deadlines to review whether you are in compliance with PHMSA CRM Rule requirements.

– Let’s get started with a consultation about your current compliance state and control room challenges. Call us at 281-598-7100 to speak with EnerSys General Manager Ross Adams or VP Business Development Dale Schafer. You can also contact us via email at sales@enersyscorp.com or complete our website contact form.

We look forward to supporting your control room as you incorporate the lessons learned from the 2021 PHMSA CRM Enforcement Actions related to Alarm Management and Adequate Information. Let’s identify solutions to help you achieve Natural Compliance.