2022 PHMSA Enforcement Actions: Solutions for Alarm Management

EnerSys is committed to providing you with the knowledge and resources to improve safety, efficiency, and compliance in the control room. In this blog post, we’ll be diving deeper into the 2022 PHMSA Enforcement Action Report to help you understand how to avoid or remedy the findings highlighted in the report. Maintaining compliance is essential for a safe work environment, and having the necessary tools and knowledge is critical to achieving this. 

We’ve covered sections A, B, C, and D – you can find those blog posts here. If you’re looking for a high-level overview of the full 2022 report, including violations by section and level of enforcement, take a look at our Control Room Findings Overview.

In this article, we will discuss the 2022 findings from the PHMSA Enforcement Actions Report related to sections E of the CRM Rule. We’ll summarize the findings in a consumable format, and highlight some of the larger implications associated with the results from PHMSA. We will also explore how our control room management software and services can help your organization stay compliant with these regulations. 

The Pipeline and Hazardous Materials Safety Administration (PHMSA) established the Control Room Management (CRM) Rule to ensure the safe and efficient operation of pipeline systems. Compliance with Sections E and F of this rule, which deal with Alarm Management and Change Management, respectively, is critical for pipeline operators to maintain the highest standards of safety and environmental protection. Failure to comply with these regulations can result in serious consequences, including financial penalties, losses in productivity and efficiency, and, most importantly, safety risks to employees, the public, and the environment. 

E – Alarm Management: 50 findings

• 32 NOA
• 5 Warning Letter
• 8 Proposed Compliance Order
• 5 Proposed Civil Penalty

Section E: Alarm Management

Findings Related to Inadequate Definitions

Several organizations were found in violation when their written procedures lacked adequate definitions related to identifying, tracking, and prioritizing alarms. This also included organizations that failed to define the metrics used to establish the severity, impact, and response time for safety-related alarms. 

• One operation’s procedure for determining the correct alarm set-point is inadequate, as it does not thoroughly define metrics related to severity, impact, and response time to establish priority and set points for safety-related alarms. 
• Another organization’s CRMP procedures were inadequate because it failed to include a process for ensuring SCADA safety-related alarms were accurate. The written procedures lacked details such as identifying and tracking inaccurate or malfunctioning alarms, accounting for different alarm designs or types, and managing stale or unreliable data. 

Organizations were also found in violation when their written plan failed to fully define the review process for safety-related alarms. Some organizations failed to define how the review would be conducted, while others failed to specify the required documentation related to the reviews. 

• One operation’s Alarm Management Plan was inadequate because it lacked specificity on conducting and documenting the review of SCADA safety-related alarms and how to correct them.
• Another organization’s written SCADA procedure failed to include specific instructions and documentation requirements for monthly reviews of points affecting safety that have been taken off scan in the SCADA host, have had alarms inhibited, generated false alarms, or that have had forced or manual values (for periods of time exceeding that required for associated maintenance or operating activities).
• One organization’s CRMP procedure was found inadequate because it failed to provide details on how the monthly review analysis and documentation of safety-related points would be completed, including the identification and documentation of false alarms.
• An organization’s plan failed to include an adequate procedure for reviewing its alarm management plan at least once a year to determine its effectiveness. There was no definition of how the review will be conducted, or what metrics would be used to determine the effectiveness of the plan.

Findings Related to Controller Workload

One of the most common reasons organizations were cited under Section E is related to controller workload. Several findings indicated a lack of regular activity monitoring – for both content and volume of activities assigned to controllers – as well as a lack of related documentation. 

• One organization’s procedures were found inadequate because they did not monitor the content and volume of general activities assigned to controllers.
  • The procedure did not reference forms used, and it failed to demonstrate how the information collected would be used to demonstrate sufficient time for analysis and alarm response. 
  • The procedures also did not identify what records would be used to demonstrate compliance with workload analysis. 

Other organizations were at fault when their Alarm Management Plan failed to include provisions to monitor controller activity, or adequately define the methodology that would be used to analyze controller workload analysis.

• One organization’s written alarm management plan did not include the required regulatory provisions for monitoring the content and volume of controller activity.
  • The organization also did not include provisions in its written Alarm Management Plan to monitor the content and volume of activity assigned to a controller. Its plan lacked a process or methodology for ensuring adequate time for alarm response.
• Another operation’s procedure did not include a method to identify all tasks/work activities performed by the controller or a plan to review and update that task list.

Findings Related to Determining Deficiencies and Efficacy

Several organizations were found in violation of the CRM Rule for failure to establish methods or metrics for gauging the efficacy of their alarm management plan.

• One organization was cited by PHMSA due to inadequate written procedures for addressing deficiencies identified through the implementation of section E. The procedure failed to provide any information or instruction on how deficiencies will be identified or criteria or guidelines for prioritizing their resolution and correction. 

Some organizations failed to define how deficiencies would be addressed, and others failed to provide adequate documentation in accordance with their Alarm Management Plan. 

• One operation’s CRM Plan did not adequately define how deficiencies will be addressed. Even though the plan stated inaccurate or malfunctioning alarms should be reported and fixed – it did not define how this would be achieved.
  • It also failed to include a mechanism for recording the issue and resolution and failed to define how stale alarms and data would be recorded and addressed.
• An operation’s control room alarm management procedures lacked specific instructions for documenting and correcting deficiencies identified in its alarm management plan.
• One organization failed to provide documentation that confirmed it had conducted an annual review of its written alarm management plan to determine its effectiveness, as required by the regulation.

Findings Related to Inadequate Procedures

Multiple organizations were cited when they failed to provide adequate procedures in their Alarm Management Plan. 

• One organization’s procedures were found to be inadequate as they lacked a specific process to identify and correct inaccurate or malfunctioning alarms.
  • No instructions on how to manage the content of its internal defect log, which resulted in unresolved issues. 
• Another organization’s procedures for tracking and addressing inaccurate, malfunctioning, and out-of-service field equipment were inadequate. It did not include a method or tools for reporting, documenting, tracking, reviewing, and promptly correcting deficiencies, as well as guidelines for handling alarms affecting safety.
• Additionally, one operation’s procedures were inadequate because they did not include a method for requesting and documenting change requests to alarm configurations beyond those authorized by the procedure.

Additional Findings

Several organizations failed to identify safety-related points in their SCADA systems – and some instances included points that were falsely labeled as not safety-related. There were also multiple instances where organizations failed to conduct annual reviews of safety-related alarm set-point values and alarm descriptions. 

Other organizations were cited in violation when their plan did not include a procedure to verify the correct safety-related alarm set-point values and alarm descriptions when field instruments are calibrated or changed. 

Two organizations failed to include false alarms in their regular monthly review of safety-related alarms. 

Findings from the report also included instances where organizations failed to properly document their adherence to procedures in its CRMP or Alarm Management Plan.

Fines

Section E was one of the most-cited sections of the CRM Rule in 2022. While a majority of the findings resulted in warnings or notices of amendment, a few organizations had fines imposed for failure to comply with the CRM Rule related to Section E, totaling $104,600. 

• One operation received a $45,600 fine from PHMSA for failing to identify and record all points affecting safety that had been taken off scan in the SCADA host; all points that have had alarms inhibited; or that have had forced or manual values for periods of time exceeding that required for associated maintenance or operating activities.
  • Despite having a ‘Tracking Log’ in their internal file system, there were no records to indicate that a monthly review had been completed.

• Another organization received a $27,900 fine when PHMSA found that, although controllers were identifying and documenting false alarms in the log as they occurred, the organization was not inspecting the logs to identify the false alarms for review.

• A fine of $31,100 was imposed on an organization when it did not verify safety-related alarm setpoint values and descriptions during field calibrations and failed to include a procedure in its CRM plan to coordinate with the field to meet the requirements of Section E.
  • They also did not have a process to complete the required verifications after incorporating a hazardous liquid pipeline into the control room.

Implications

PHMSA focused heavily on violations related to Alarm Management in 2022, second in quantity to Section A (which covers a broad swath of topics and violations). This emphasis on how operations are performing with respect to the regulation and their own Alarm Management Plans further highlights the pattern we’ve seen with findings from other sections of the CRM Rule – Do you have a plan? Does that plan align with the regulation? Are you following that plan? Are you documenting your compliance?

It’s clear that PHMSA is focused on ensuring your organization has a plan in place and the plan is free from ambiguity – processes, procedures, required documentation, and contingencies need to be fully fleshed out and leave no room for interpretation or error. 

PHMSA also prioritized controller workload with regard to alarm management. Several organizations were found in violation for failure to either monitor controller workload and regular activities regularly or failure to define the process that the organization planned to use to monitor activities. 

Even with a detailed plan for monitoring workload and typical shift activities, organizations frequently failed to follow through with monitoring. This is especially important for the ‘Check’ in PDCA. If you’ve got a detailed plan in place and you’re executing on that plan, are you then checking? How does your plan stand up to reality? Are there gaps that can be addressed?

EnerSys Solutions

If you need help with processes and procedures, monthly alarm reviews, or appropriate recordkeeping and access to compliance documentation, EnerSys has the tools and services to help meet those needs.

Alarm Philosophy

EnerSys can help with your alarm philosophy by establishing alarm definitions unique to your operation, configuring and documenting alarms for your control room, and validating alarms in the SCADA system.

Use the ALMgr module in our CRM Suite to conform to PHMSA CRM Rule requirements, support your pipeline alarm management program, and expedite the alarm rationalization process. ALMgr helps you streamline recordkeeping and easily access documents that satisfy a PHMSA inquiry. You can also generate analytical reports in the software, identify bad actors, and implement corrective action.

ALMgr also supports safety in the Control Room for controllers. The software captures and presents safety-related alarms and generates Alarm Response Sheets that help each controller quickly and effectively respond to those alarms. Reporting and analysis also help to identify corrective actions for bad actors.

Additional Resources

If you’re looking for more information on Alarm Management best practices and want to dive deeper, there are several great resources available. We would recommend listening to Episode 163: Overcoming Challenges in Alarm Management with EnerSys General Manager Ross Adams.

The Pipeliners Podcast also has several other episodes that focus on Alarm Management: 

More Alarm Management Episodes

You can also download our whitepaper:

Optimizing Emergency Response Through Alarm Management. 

This paper will help you better understand the difference between alarm response and emergency response, how to optimize your alarm management program, how to better support alarm and emergency response, and how to identify actions to take during emergency response. 

Let’s Get Started

Our team is here to help you review and implement these solutions. If you’d like to set up a demo to see how ALMgr can help your organization – we’d love to talk you through your options. We can show you how ALMgr helps bring efficiency and additional functionality to your Alarm Management Plan and safety to your control room. Don’t wait until you’ve received notice from PHMSA about an upcoming inspection – take steps to be proactive and prepared. 

Call us at 281-598-7100 to speak with EnerSys General Manager Ross Adams or VP of Business Development Dale Schafer. You can also contact us via email at sales@enersyscorp.com or complete our website contact form.

Additional Solutions

EnerSys has worked to build a comprehensive set of tools designed to help you manage compliance and safety in your Control Room. Through our partnerships with our sister companies, we can offer even more help on your path to Natural Compliance.

P.I. Confluence

• P.I. Confluence (PIC) provides tools for managing your program, processes, workflow, communications, and information exchange in pipeline operations.
• The POEMS™ Program Suite consists of web-based process management software tools to validate and manage pipeline programs and processes.

GCI & Muddy Boots

• Gas Certification Institute offers field operations software, Muddy Boots, to help close the communication gaps between the field and the control room.
• The platform is ideal for addressing change management within your operation, and the POEMS CRM Suite allows for linking and communication between the programs to create, track, and update work orders.