Pipeline control room controller reviewing PHMSA pipeline Alarm Management lessons learned

PHMSA Enforcement Actions: Optimize Processes and Recordkeeping to Support Alarm Management

As part of our team’s review of the 2020 PHMSA Enforcement Actions related to the pipeline control room, we identified three pipeline SCADA alarm management violations of the PHMSA Control Room Management Rule (CRM Rule).

The violations were found in Section E (Alarm Management) and Section J (Compliance and Deviations) of the CRM Rule. The issues point to the importance of optimizing your alarm management processes and ensuring proper recordkeeping to achieve natural compliance.

The CRM Rule Alarm Management Violations We Identified

Consider the key takeaways from each of these violations to apply the lessons learned to your pipeline operation’s control room.

Section E Violation – Have The Appropriate Processes

One pipeline operator was found to be in violation of Section E (49 CFR – 192.631) because they failed to perform their Monthly Alarm Reviews, and as such, failed to identify and reconcile their false alarms.

This violation relates to the CRM Rule requirement that any operator that uses a SCADA system “must have a written alarm management plan to provide for effective controller response to alarms.” Specifically, according to (e)(2) in 192.631/195.446, PHMSA requires that the plan must include provisions to identify at least once each calendar month “the points affecting safety that have been taken off scan in the SCADA host, have had alarms inhibited, generated false alarms, or that have had forced or manual values for periods of time exceeding that required for associated maintenance or operating activities.”

This violation captures the need to have the appropriate processes built into your Alarm Management plan, Control Room Management Plan (CRMP), or broader Operations & Maintenance manual. The processes and systems used should specify what type of alarms should be reviewed and when they should be reviewed to help identify the alarms that could be inhibiting a controller’s ability to understand what is presented to them through the SCADA system.

PHMSA pointed out that the operator’s CRM procedure in their O&M required a monthly review of false alarms, yet the PHMSA inspector found that the operator was not following this procedure to identify all of the points in the SCADA host affecting safety that were generating false alarms. It points to a common issue where the operator’s policy was sound, but the implementation of their policy was not.

The specific issue was the operator’s reporting process was based on a review of the top, most-frequent alarms for the month. The operator then determined whether those alarms were actual events occurring frequently or false alarms. After reviewing the reports, PHMSA inspectors noted that some alarms were identified as “actual” while others were identified as “faulty.”

By only reviewing the highest volume of alarms for the month, the operator did not capture all the false alarms that could be occurring in the pipeline system and affecting safety. As a result, the operator was found to be in violation of Section E because they failed to identify the points affecting safety that generated false alarms due to following a procedure that did not conform
with the regulatory requirement.

Overall, the proposed civil penalty for this violation was $28,500.

Key Takeaway: Ensure that your procedures capture the need to review all safety-related points once per month to align with CRM Rule. There could also be an implementation issue, which can be prevented if the rule, policy, and implementation are reconciled to one another in a review process. We can help facilitate this through our combination of software (ComplyMgr) and consultation (Compliance Reviews).

Section J Violations – Maintain Records

We identified two separate violations of Section J (Compliance and Deviations) that relate to Alarm Management recordkeeping.

– One pipeline operator failed to maintain records that could be produced during an inspection to demonstrate compliance with 49 CFR 192.631/195.446(e)(2). Specifically, the operator could not produce records that showed they performed a monthly review of points in the SCADA system that affected safety.

The issue for the operator was that they transitioned to a new SCADA system, which they determined was not the proper fit for their operation. This resulted in a transition to a second SCADA system. In the midst of change, the operator compiled and stored their monthly alarm reports in various formats. During the transition to a new recordkeeping system, however, all of the historical records were completely lost. Furthermore, the operator reported they were unable to recover the records.

After a review of the lost records issue, PHMSA proposed a civil penalty of $19,600 for this violation.

– Another pipeline operator was found to be in violation of Section J (49 CFR 195.446) as it relates to alarm management because they failed to maintain records to demonstrate alarm management compliance.

The operator stated that they would maintain records to demonstrate compliance with an alarm management review (192.631/195.446(e4)). Specifically, the operator said they would maintain written reports that are required for the annual audits of the managerial and work practices associated with their alarm system to determine if those practices were adequate.

However, during the PHMSA inspection, no written reports were provided upon request by the PHMSA inspector. When pressed for records, the operator was only able to provide an agenda and sign-sign sheets for meetings covering a four-year period, but the operator was unable to provide any written reports, as required by their procedure and the CRM Rule to demonstrate compliance.

The proposed penalty for this violation was rolled into a larger fine levied against the operator covering thousands of dollars.

Key Takeaway: Both violations of Section J capture the need to generate records, be able to produce records, and have the appropriate software tools to make sure recordkeeping is happening as alarm management tasks are being performed in the control room. These issues can also be identified through our ComplyMgr software solution and Compliance Reviews.

An overall point of consideration for operators is that these violations speak to the need to identify control room requirements when making changes. For example, not involving the control room in a SCADA system change means that maintaining the records for CRM regulatory compliance may not have been factored into the transition by the SCADA team.

EnerSys Can Solve These Pipeline Alarm Management Challenges

As you can see, an effective alarm management program requires multiple elements coming together to support natural compliance. You need the appropriate processes and procedures, you need systems in place, you need the monthly review of alarms that could be causing false alarms, you need appropriate recordkeeping, you need the ability to produce records to demonstrate compliance with the CRM Rule, and you need to factor in how changes to the operation could affect the control room.

Here’s how EnerSys can support your complete alarm management program through the software modules in our POEMS Control Room Management Suite (CRM Suite):

  • ALMgr module: ALMgr provides tools and documentation for managing the alarm rationalization process, assisting with controller response to alarms, and generating monthly reviews and reports that comply with the CRM Rule.
  • CRMgr module: CRMgr provides operators with simplified recordkeeping to log critical actions in the control room, ensure proper recordkeeping, and simplify the audit process by producing reports that demonstrate compliance with the CRM Rule. Essentially, we solve the lost record problem through our compliance log capabilities.
  • ComplyMgr module: Combined with our Compliance Reviews, the ComplyMgr module identifies gaps between the policy and the implementation of the policy to drive toward alignment with the CRM Rule.

We believe that regulatory compliance is best achieved when it is in the natural outflow of the actions, attitudes, behavior, and culture of your operation. Through the use of our software tools, operators will naturally fall into compliance with PHMSA 49 CFR Parts 192.631 or 195.446 to comply with the CRM Rule and produce the appropriate records during an inspection. This will help your operation avoid violations, penalties, safety issues, and lost time from inefficiencies.

We would appreciate the opportunity to show you the capabilities of our software tools. To schedule an educational demo or consultation with our team, call us at 281-598-7100 and ask for Dale Schafer or Ross Adams. Alternatively, complete our contact form or email us directly at sales@enersyscorp.com to schedule your demo.