2023 PHMSA Enforcement Actions - E-J | PHMSA CRM Rule Enforcements

PHMSA CRM Rule Enforcements: Sections E-J

Last month, we covered the first half of the CRM Rule violations from PHMSA in 2023 (Sections A-D). In part two of our deep dive into PHMSA CRM Rule enforcements, we’ll break down the findings from sections E through J related to the Control Room Management Rule (49 CFR ​​192.631 for the Transport of Natural Gas and 195.446 for the Transport of Hazardous Liquids) to help provide a better understanding of common violations.

We’ll provide a brief overview of each section and its requirements, interesting findings, and appropriate solutions for your organization.

2023 PHMSA Enforcement Actions Control Room Findings by Section

  • Section E (Alarm Management) – 23 Findings 
  • Section F (Change Management) – 4 Findings 
  • Section G (Operating Experience) – 2 Findings 
  • Section H (Training) – 23 Findings 
  • Section I (Compliance Validation) – 1 Finding
  • Section J (Compliance and Deviations) – 11 Findings

Overview of Section E [Alarm Management]

Section E Operators utilizing SCADA systems must have written alarm management plans to ensure effective controller response to alarms. When acquiring a new SCADA system post-compliance deadlines, integral alarm management capabilities must be fully functional upon system operation. Periodic alarm reviews are encouraged for improvement.

A written alarm management plan is crucial for the control room as it provides a structured framework for identifying, prioritizing, and responding to alarms, ensuring operational safety and efficiency. Additionally, it serves as a reference guide for controllers, facilitating consistent and effective decision-making during normal, abnormal, and emergency situations.

Section E (Alarm Management) – 23 Findings

  • Notice of Amendment – 14
  • Warning Letter – 3
  • Proposed Civil Penalty – 8
  • Proposed Compliance Order – 6

*Some of the findings included both a proposed compliance order and a proposed civil penalty

Additional Info from CRM FAQ
  • False Alarms: Any alarm presenting inaccurate operational conditions or misleading information to controllers is deemed false. Awareness during testing is crucial to prevent false alarms
  • Safety-Related Alarms: Operators designate and train controllers on safety-related alarms critical to operational safety, with setpoint values indicating equipment deviations.
  • Monitoring and Workload: Operators monitor all controller activities formally, including workload evaluation beyond alarms, ensuring periodic workload assessments.
  • Alarm Handling Practices: Operators establish alarm management plans without a specific alarm threshold, periodically reviewing alarms and prioritizing based on operational importance.
  • Alarm Response Strategies: Controllers are trained to promptly analyze alarm situations, guiding alarm response strategies based on operator procedures.
  • Addressing Deficiencies: Operators promptly address non-functional alarms, document corrective actions, and analyze systemic issues for improvement.
  • Alarm Set-Point Adjustments: Controllers may adjust operational alarm set-points but not critical alarms, with procedures reflecting this distinction.

Findings from Section E

There were several operators that were cited based inadequate procedures – their written policies were inadequate to fully comply with Section E of the CRM Rule.

  • One operator received a notice of amendment due to inadequate procedures for managing stale or unreliable data. The procedure lacked guidance for practices to manage stale or unreliable data.
  • Another operator was given a proposed compliance order when it failed to have developed a written alarm management plan as required by §192.631(e).
  • One organization was given a notice of amendment from PHMSA when its CRM procedures were inadequate for identifying and addressing deficiencies identified when implementing specific sections of the CRM Rule.

There were two instances where an operator was cited for inadequate procedures around monitoring controller workload.

  • One operator was given a notice of amendment when its procedures for monitoring controller workload lacked records and procedures for compliance with Section E.

Proposed Civil Penalties from Section E

There were six different findings under Section E that resulted in PHMSA issuing a proposed civil penalty.

  • One operator lacked a process for alarm rationalization, which led to inconsistent alarm management practices and ineffective controller response to alarms. Civil Penalty imposed: $16,700
  • Another organization failed to properly identify safety-related points in its database, resulting in inaccuracies and potential safety risks.
    • PHMSA also noted that the same organization failed to provide records for reviewing points affecting safety, inhibiting alarms, and managing forced or manual values, which inhibited effective controller response to alarms.
    • Additionally, the organization also lacked a procedure to verify correct safety-related alarm set-point values and alarm descriptions – which could lead to inaccuracies and safety risks
    • Lastly, the organization failed to provide detailed records demonstrating the annual review of its alarm management plan, as required by regulation.
    • This organization received a proposed civil penalty totaling $197,300 for the above-mentioned violations.
  • One organization’s control room failed to provide detailed records demonstrating its annual review of the alarm management plan – lacking documentation of effectiveness assessments and follow-up actions. Civil Penalty imposed: $39,100

Overview of Section F [Change Management]

Section F of the PHMSA CRM Rule focuses on Change Management, emphasizing the importance of operators coordinating changes that could impact control room operations with control room personnel promptly. This requirement necessitates timely communication once program procedures are established, allowing for ongoing updates and improvements during implementation.

Notably, relying solely on emergency procedures is insufficient; field personnel must inform the control room of any changes affecting its operations, extending to routine tasks like transmitter calibrations. Even if field changes don’t directly alter SCADA data, control room involvement remains crucial, as modifications may affect the pipeline, influencing control room operations beyond SCADA data.

Section F (Change Management) – 4 Findings 

  • Notice of Amendment – 4

Findings from Section F

Of the four instances found under section F, all resulted in a notice of amendment from PHMSA. The violations centered around a lack of procedure for communication when changes could impact the control room.

  • One operator lacked procedures for coordinating physical changes to pipeline equipment or configurations. There were no instructions for how control room personnel, management and field personnel would coordinate and manage the process.
  • Another organization’s procedures for mainline valve inspection and testing failed to mandate field personnel to contact the control room before making changes that could affect their operations.

Overview of Section G [Operating Experience]

Section G of the PHMSA CRM Rule emphasizes the integration of lessons learned from operating experiences into control room management procedures, with ongoing review and updates once program procedures are developed.

The training program must incorporate lessons from operator experiences, encompassing not only reportable incidents but also near misses, audit findings, and more. This training requirement applies broadly to all incidents and events, ensuring controllers are equipped to handle abnormal conditions, whether they caused the incident or not, and even when controller actions are directed or superseded by authorized individuals.

Section G (Provide Adequate Information) – 2 Findings

  • Notice of Amendment – 2

Findings from Section G

The findings related to Section G were not particularly notable – however, they both related to procedures in the Control Room Management Plan (CRMP).

  • One operator failed to include a process for assessing controller fatigue as a potential factor in incidents or accidents. Despite having various procedures that were interrelated, they did not reference one another, causing a disconnect in processes. Fatigue determination was also not objectively and consistently determined through the existing procedure.
  • Another operator failed to include a procedure for mandating controller training on lessons learned from its operating experiences.  

Overview of Section H [Training]

Section H of the PHMSA CRM Rule underscores the necessity of establishing a comprehensive controller training program, subject to annual review for potential enhancements. Training programs must encompass various operational setups, including seasonal parameters and start-up/shutdown procedures, not limited to routine configurations. Controllers passing qualifications meeting CRM plan requirements are exempt from re-qualification after the implementation date.

Additionally, operators should identify any additional covered tasks resulting from CRM rule actions for inclusion in their Operator Qualification (OQ) programs. Training methods must be evaluated and documented, with exercises incorporating realistic scenarios to challenge decision-making skills. Completion deadlines for team training programs must be established, specifying recurrence for full team training in written procedures, with at least one qualified controller participating in all exercises.

Section H (Training) – 22 Findings

  • Notice of Amendment – 13
  • Warning Letter – 5
  • Proposed Compliance Order – 7
  • Proposed Civil Penalty – 6

*Some of the findings included both a proposed compliance order and a proposed civil penalty

Findings from Section H

Section H was one of the sections that incurred the most violations from PHMSA. Overall, the findings in Section H generally followed one of three common themes: 1. Failure to conduct training 2. Failure to review training program(s) or 3. Lack of required content.

There were several operators that failed to conduct training, as outlined in their CRMP and required by regulations.

  • One organization failed to conduct control room team training and exercises as required, and provided no documentation to demonstrate compliance with the mandated frequence and inclusion of relevant personnel.
  • Another operator failed to conduct team training in 2019 and 2020. Despite citing awareness for 2019 and non-completion in 2020 due to COVID, the organization failed to provide any documentation of deviation, as required.

Operators were cited for a failure to review their training programs or review the criteria for personnel required to attend control room training.

  • One operator failed to identify personnel expected to collaborate with controllers during various operating situations.
  • Another organization failed to conduct an annual review of its controller training program, as required – and were unable to provide documentation for reviews for 2020 and 2021.
  • PHMSA also cited an operation for inadequate procedures for reviewing training program content. The procedure lacked specificity and documentation requirements. Utilization of various activities for gathering information was not mentioned in the procedure, and there were no working documents to support review outcomes.

Several organizations failed to train the appropriate personnel on key responsibilities and various operating conditions.

  • One operator violated regulations by failing to provide training to controllers on key responsibilities defined by the operator over the past three years, including responding to abnormal conditions, understanding the pipeline system, reviewing relevant procedures, and participating in control room team training and exercises.
  • Another organization failed to provide training content on abnormal operations occurring simultaneously or in sequence, as required by its CRMP. The CRMP’s lack of specific examples or a review of unique pipeline conditions led to inadequate training for controllers.
  • One control room did not establish specific training on the leak detection system’s operation and its integration with SCADA. Training content referred to additional documentation, which lacked actionable steps for controllers in the event of a possible leak. Insufficient training records were maintained.
  • Another operator failed to include team training and exercises that included both controllers and other individuals, as required by regulations. Their procedures lacked requirements for soft skills training and did not adequately address operational modes.

Overview of Section I [Compliance Validation] & Section J [Compliance Deviations]

Sections I and J of the PHMSA CRM Rule cover important aspects related to compliance and documentation.

In Section I, operators are required to submit their procedures to PHMSA or the appropriate state agency upon request. They must also maintain records demonstrating compliance with the CRM rule, ensuring these records contain sufficient detail to demonstrate thoroughness and authenticity. While there’s no specific deadline for submitting documents for compliance validation, operators should comply promptly with requests.

For Section J, operators are mandated to maintain records demonstrating compliance with the CRM rule for at least one year or the last two periodic tests or validations, whichever is longer. They must document deviations from their procedures, not the FAQs, and electronic records are acceptable if they maintain integrity and authenticity. New operators acquiring pipeline assets are responsible for demonstrating compliance, and documentation for authorized personnel and team training and exercises is also required.

These sections are grouped together as they both emphasize the importance of maintaining accurate records and demonstrating compliance with the CRM rule, ensuring safe and efficient control room operations.

Section I (Compliance Validation) – 1 Finding

  • Notice of Amendment – 1

Section J (Compliance Deviations) – 11 Findings

  • Warning Letter – 10
  • Proposed Compliance Order – 6
  • Proposed Civil Penalty – 4

*Some of the findings included both a proposed compliance order and a proposed civil penalty

Findings from Section I and Section J

Sections I and J carried various levels of enforcement, ranging from a notice of amendment all the way to several proposed civil penalties.

  • For the lone violation related to Section I – one operator’s procedures did not require that the organization submit its procedures to PHMSA, or in the case of an intrastate pipeline facility regulated by a state, to the appropriate state agency.  

Violations for Section J were all related to inadequate documentation or a complete lack of records. PHMSA cited multiple operators for failure to maintain records related to shift change, education and training, alarm verification, controller fatigue, and more.  

  • PHMSA issued a warning letter when an operator failed to maintain records demonstrating compliance with team training requirements, indicating deficiencies in record-keeping related to controller training.
  • Another warning letter was issued to an organization when it did not maintain records demonstrating compliance with deviation procedures to reduce the risk of controller fatigue, violating requirements of § 195.446(d)(4).
  • An organization received a warning letter and proposed compliance order for failure to maintain records of shift handover between controllers as required, indicating deficiencies in shift-handover documentation.

Proposed Civil Penalties from Section J

There were four findings from PHMSA that included a proposed civil penalty.

  • One operator failed to maintain shift change records demonstrating compliance with § 192.631, leading to a repeat violation. Civil Penalty imposed: $83,000
  • Another organization did not provide adequate records to demonstrate compliance with safety-related alarm set point verification requirements, indicating deficiencies in record-keeping practices. Civil Penalty imposed: $39,100
  • PHMSA found another operator in violation when it failed to maintain documentation of a training program for controllers and records demonstrating how controllers are operator qualified.
    • This operator also failed to maintain records demonstrating field personnel contacted the control room before making field changes affecting control room operations, violating requirements of § 192.631(f)(2). Civil Penalty imposed: $26,000

EnerSys Solutions for PHMSA CRM Rule Enforcements

Implement POEMS® Software Tools for Control Room Management

Here’s how EnerSys can support your complete alarm management program through the software modules in our POEMS Control Room Management Suite (CRM Suite):

Alarm Management

If you need help with processes and procedures, monthly alarm reviews, or appropriate recordkeeping and access to compliance documentation, EnerSys has the tools and services to help meet those needs.

  • ALMgr module: ALMgr provides tools and documentation for managing the alarm rationalization process, assisting with controller response to alarms, and generating monthly reviews and reports that comply with the CRM Rule.

Additionally, EnerSys offers consultations where we can get a detailed view of your program and identify areas of opportunity. We can work together to help you build team trainings, or we can review your CRMP and use that information to take the necessary steps to address gaps in your training.

Alarm Philosophy

EnerSys can help with your alarm philosophy by establishing alarm definitions unique to your operation, configuring and documenting alarms for your control room, and validating alarms in the SCADA system.

Documentation and Communication

For issues involving documentation and recordkeeping, the CRMgr tool from our POEMS CRM Suite is built to help you address many of the findings we have discussed.

  • CRMgr module: CRMgr provides operators with simplified recordkeeping to log critical actions in the control room, ensure proper recordkeeping, and simplify the audit process by producing reports that demonstrate compliance with the CRM Rule. Essentially, we solve the lost record problem through our compliance log capabilities.

CRMgr improves communication across shifts and breaks down barriers between the field and the control room, enabling operators to communicate key information and training material to all relevant personnel.

Articles, training documents, notice of incidents, and even URLs can be assigned for review through the tool. Once training documents are distributed, controllers must read and acknowledge that they have reviewed all assigned materials. All assigned documents are finalized with a time-stamped confirmation to the supervisor, ensuring information has been read.

Users have the ability to log all monthly, annual, and semi-regular compliance-related tasks to align with PHMSA regulatory requirements. Documents are easily accessible to satisfy inspection inquiries, and the assigned reading functionality supports closed-loop communication with the control room personnel. Users can capture valuable activities and use the records created for future analysis to support implementing lessons learned. 

Additional Resources & Solutions

ComplyMgr module: Combined with our Compliance Reviews, the ComplyMgr module identifies gaps between the policy and the implementation of the policy to drive toward alignment with the CRM Rule.

If you’re looking for more information on Alarm Management best practices and want to dive deeper, there are several great resources available. We would recommend listening to Episode 163: Overcoming Challenges in Alarm Management with EnerSys General Manager Ross Adams.

The Pipeliners Podcast also has several other episodes that focus on Alarm Management: More Alarm Management Episodes

You can also download our whitepaper: Optimizing Emergency Response Through Alarm Management.

This paper will help you better understand the difference between alarm response and emergency response, how to optimize your alarm management program, how to better support alarm and emergency response, and how to identify actions to take during emergency response.

The CRM Suite includes software modules for control room management, alarm management, fatigue mitigation, point-to-point verification, and compliance management. Our software ensures adherence to the CRM Rule, the ability to quickly generate reports using your data, and more time to focus on expanding the capabilities of your operation.

We would like to schedule a brief demo with your team to learn more about your operation and to show you how our software can help you achieve natural compliance, promote safety performance, and improve operational effectiveness. To get started, please complete our contact form, email our team at sales@enersyscorp.com, or call us directly at 281-598-7100.