PHMSA CRM Rule Enforcements: Sections A-D

PHMSA CRM Rule Enforcements: Sections A-D

2023 Enforcement Actions in the Control Room

Our last post covered a high-level overview of the control room-related findings from PHMSA audits in 2023. In this post, we’ll break down sections A through D of the Control Room Management Rule (49 CFR ​​192.631 for the Transport of Natural Gas and 195.446 for the Transport of Hazardous Liquids) to help provide a better understanding of common violations.

We’ll provide a brief overview of each section and its requirements, interesting findings, and appropriate solutions for your organization.

2023 PHMSA Enforcement Actions Control Room Findings by Section

  • Section A (General Requirements) – 13 Findings 
  • Section B (Roles and Responsibilities) – 13 Findings 
  • Section C (Provide Adequate Information) – 33 Findings
  • Section D (Fatigue Mitigation) – 7 Findings

Overview of Section A [General Requirements]

When reviewing findings related to Section A, it can seem like the section is just a ‘catch-all’ for violations that don’t neatly fit into one of the subsequent sections. Ultimately, violations under Section A are often related to a higher-level problem within the operation – the CRMP (Control Room Management Plan). Section A requires that each operator must have a CRMP, and that the CRMP’s policies and procedures must align with the CRM Rule.

2023 Section A (General Requirements) – 13 Findings

  • 4 – Warning Letters
  • 3 – Proposed Civil Penalty
  • 3 – NOA
  • 3 – PCO – Warning Item

Section A General Overview of Requirements:

  • Each operator must have a written Control Room Management Plan (CRMP).
  • The CRMP must have policy and procedures that align with the CRM Rule.
  • The operator must follow the policy and procedures that were set forth in the CRMP.
  • Operators need to formalize and control all documents, which should be made readily available to controllers at their consoles.
  • The CRMP should be readily available to, and understood by, controllers.
  • Operators must review the CRMP program annually, not to exceed 15 months.

Findings from Section A

Multiple findings under Section A can be tied back to the Control Room Management Plan (CRMP) of the organizations cited. In some instances, the CRMP failed to establish or fully define procedures.

  • One organization received a Warning Letter from PHMSA for failure to have a CRMP for specific pipelines
  • Another operator received a Notice of Amendment when its CRM procedures did not explicitly identify which of its facilities qualified as control rooms under §195.446

Other findings indicated that an operator was unable to provide documentation to demonstrate compliance or failed to follow the CRMP entirely.

  • A $222,504 civil penalty was issued to an operation for failing to follow its CRMP when investigating SCADA indications of a leak, leading to a delayed response to a potential leak

Additionally, a lack of annual review execution or documentation confirming those reviews caused multiple operators to be cited.

  • PHMSA issued a Warning Letter when an organization failed to review its alarm management plan to determine its effectiveness

Two operators were cited for failure to monitor controller workload and were unable to provide required documentation.

  • One control room received a Proposed Compliance Order when it failed to monitor controller activity to determine the content and volume of activity controllers experienced on-shift. Additionally, they were unable to provide records to demonstrate compliance from 2019-2021
  • PHMSA issued a Proposed Compliance Order when an organization failed to conduct workload analysis to monitor controller activity – as required by regulations – and was unable to provide records to ensure controllers had sufficient time to analyze and respond to incoming alarms

Multiple Considerations

  • After repeatedly failing to adhere to its CRMP – neglecting critical shift change documentation, violating shift length regulations, overlooking Alarm Management Plan reviews, and failing to identify safety critical issues as required by regulations – one organization received a proposed fine of $1,228,900.
    • The organization also received a proposed fine of $50,200 when it failed to meet the requirements for monitoring controller workload annually and providing relevant documentation.

EnerSys Solutions for Section A Violations

CRM Applicability Assessment

We can perform an applicability assessment to help you determine whether the CRM Rule applies to you. This assessment is designed to determine if you meet the definition of a Control Room as defined in the CRM Rule, taking your pipeline facilities and operating setup into consideration.

CRM Compliance Review

We can also perform a CRM Compliance Review using ComplyMgr, our software tool that allows us to measure your plan’s level of policy compliance compared to the PHMSA regulations. Our team will make observations, perform interviews, and review records to evaluate proper 192.631 and 195.446 control room management implementation. 

CRM Program Development

Our team has the knowledge and tools to design and implement a Control Room Management Plan specific to your operation. If necessary, we can also rewrite your CRMP to address all PHMSA requirements. In addition to meeting regulations, this is used as a framework for continuous improvement moving forward.

Overview of Section B [Roles & Responsibilities]

Organizations must clearly define roles and responsibilities in the control room. Any lack of clarity within your organization’s CRMP regarding personnel roles and duties introduces the risk of safety incidents. Given that controllers are often the initial responders to abnormal or emergency situations in operations, it’s imperative they have a precise understanding of their role in upholding pipeline safety.

Section B (Roles and Responsibilities) – 13 Findings 

  • 1 Warning Letter
  • 10 NOA
  • 2 Proposed Compliance Orders (One finding received a proposed civil penalty and two PCO’s)

Findings from Section B

The 2023 findings from PHMSA related to Section B generally fell into one of four categories – either because of the nature of the violation, or the content that the violation related to.

There were two operations that were cited because of a lack of documentation.

  • One organization’s failure to adequately record controller shift-changes and handover of responsibilities led to PHMSA issuing a Warning Letter

Another common gap in CRMPs was centered around evacuation – specifically, the lack of guidance or procedure for control room evacuation and what can trigger an evacuation. Evacuations are a high-stress situation, even more so when control room personnel do not have a clear understanding of the part they play in the process.

  • The organization’s Control Room Management Plan (CRMP) failed to address evacuation, and the Control Room Evacuation Procedure lacked clarity on evacuation triggers. This resulted in a notice of amendment from PHMSA
  • PHMSA issued a Notice of Amendment when one operator’s CRM procedures lacked necessary steps for control room evacuation, including guidance on muster points, decision-making regarding returning to primary or backup control rooms, and protocol for remotely monitoring the system by the qualified manager

Consoles/Controllers

PHMSA also cited operators who failed to define procedures related to consoles.

  • One operation’s procedures were inadequate in defining the controller’s physical domain of responsibility, resulting in a Notice of Amendment
  • PHMSA issued a Notice of Amendment when one operation’s CRMP lacked procedures for when controllers leave the control room console unattended, despite their duties frequently requiring them to do so
  • One operation received a Notice of Amendment when its procedures lacked clarity regarding the roles and responsibilities of controllers when multiple controllers are on shift

Authority

Operators that failed to fully define varying levels of authority in their CRMP were also found in violation. There were multiple operators who required modifications to their CRMP to address who has the authority to supersede controller actions.

  • One organization’s CRMP lacked defined roles and responsibilities, leading to a situation where the Superintendent directed a controller without proper documentation or qualification, resulting in a Proposed Compliance Order and a fine of $50,200

EnerSys Solutions for Section B Violations

In crafting an effective Control Room Management Program (CRMP), it’s crucial to thoroughly define roles and responsibilities for your team. Operators should be mindful that these responsibilities may shift depending on the prevailing operating conditions. Flexibility in workflow and decision-making is essential as conditions transition between normal, abnormal, or emergency states.

Recent assessments highlight PHMSA’s expectation for CRMPs to meticulously detail procedural variations. This includes specifying the control room personnel accountable for tasks across normal operations, abnormal operating conditions (AOCs), and emergency scenarios.

ComplyMgr

Our software tool, ComplyMgr, not only captures the processes for each pipeline operation, but also helps operators understand what records need to be kept and how the records need to be kept.

ComplyMgr streamlines pipeline compliance management and includes a centralized document repository for managing your organization’s governing documentation – policies and related procedures and forms.

Key Benefits of ComplyMgr

  • Dedicated Tools to View, Update, and Review your CRMP Policy
  • ComplyMgr software module allows for document development to ensure roles and responsibilities are adequately defined for all operating conditions

CRMgr

The CRMgr module of the POEMS CRM Suite enables pipeline operators to capture valuable operational and compliance-related activity in a manner that promotes Natural Compliance. Records are then readily available and useful for future analysis or PHMSA CRM audit response.

CRMgr improves situational awareness for controllers, improves communication across shifts, and breaks down barriers between the field and the control room, enabling operators to communicate key information to control room personnel.

Key Benefits of CRMgr

  • Unique Dashboards for Controllers and Control Room Managers
  • Tools and Permissions Built to Accommodate Roles & Responsibilities Based on Operating Conditions[GD1] 
  • Assigned Reading with Confirmation Reporting to Communicate Important Information, Documents, and Changes to Controllers
  • Record of Changing Rolls and Responsibilities via emailed SHO Reports

Overview of Section C [Provide Adequate Information]

Section C of the CRM Rule requires that, at a minimum, each operator must provide controllers with the tools, information, processes, and procedures that are necessary so that a controller is able to perform their duties as defined by the operator.

This includes enabling operators to have an accurate view of the system’s health and performance and ensuring compliance with API 1165 for applicable SCADA systems. The Recommended Practice for Pipeline SCADA Displays provides guidance on document structure, writing styles, graphics and tables, references, appendices, and a review process. It also requires operations to conduct point-to-point verification between SCADA displays and related field equipment when field equipment is added or moved, or when there are other changes that affect pipeline safety made to field equipment or the SCADA displays. Backup SCADA systems must also be tested once each calendar year, but not at intervals greater than 15 months.

Section C also states that an internal communication plan to provide adequate means for safe, manual operation of the pipeline must be tested and verified at least once each calendar year (at intervals not to exceed 15 months). Lastly, operations must establish and implement procedures for when a different controller assumes responsibility for a console, including what information needs to be exchanged during a shift handover. 

Section C (Provide Adequate Information) – 33 Findings

  • 15 NOA
    • 2 [Proposed Civil Penalty & PCO]
    • 3 [Proposed Civil Penalty, and 2 PCO]
    • 3 [Proposed Civil Penalty, PCO, Warning Item]
    • 7 [PCO, Warning Item]

Findings from Section C

Section C included the most control room findings of any section in 2023. We’ve broken down the findings into three main categories that summarize the common thread amongst all the information: Testing and Documentation Failures, Procedural Inadequacies, and Compliance with Industry Standards.

Testing and Documentation Failures

There were 17 findings from PHMSA that were related to a lack of documentation, a lack of testing, or both. The severity of the notices from PHMSA ranged from Warning Letters up to and including Proposed Civil penalties. These findings underscore the importance of following your control room’s CRMP and documenting actions that are taken, or not taken, as the procedures are executed.

  • One operator failed to complete tests of its internal communication system for manual operation. A situation occurred where a lightning strike knocked out communication and a terminal was switched to manual operation. The operator was unable to produce logbooks related to this incident. Another instance at the same organization occurred when an operator instructed field personnel to shut down part of the system during a hurricane – something that was not part of the manual operations test plan.
    • These actions resulted in a Proposed Civil Penalty from PHMSA of $16,700
  • Another example involves an operator that failed to complete a test to verify its internal communication plan for manual pipeline operation. The operator indicated that constraints during COVID and lock-down were the reason, despite not filing a deviation.
    • That same operator incurred another Proposed Civil Penalty after it lacked sufficient records to demonstrate compliance with testing backup SCADA control room systems/facilities.
      • PHMSA issued Proposed Civil Penalties of $39,100 and $36,100 respectively, as a result

Procedural Inadequacies

11 of the 33 findings under section C were a matter of inadequate procedures – either the procedures lacked clarity and relevant detail, or were missing entirely, causing gaps that could potentially lead to safety incidents.

  • An organization received a Notice of Amendment when its procedures lacked a process for evaluating controller fatigue and reviewing reportable events
  • Another operator’s CRMP failed to adequately define and prioritize leak alarms as safety-related

Compliance with Industry Standards

Lastly, there were five instances related to inadequate procedures or a lack of compliance with API RP 1165 or 1168.

  • One organization’s CRMP lacked clear definitions for addition, expansion, or replacement to demonstrate compliance when implementing API 1165.  The CRMP failed to include examples to provide support for future control room or SCADA activities.
    • PHMSA issued a Notice of Amendment for this violation
  • Another control room failed to implement API RP 1168 Section 5.3.1 by failing to include a review of AOCs and Emergencies during shift change. The forms used during shift change did not include a section to document this information, and there was no process to review ongoing AOCs that cross over multiple shifts.
    • The operator received a Warning Letter, Proposed Compliance Order, and a Proposed Civil Penalty of $39,100 as a result

EnerSys Solutions for Section C Violations

Lack of documentation proved to be a common fault for many of the organizations cited. At EnerSys, we build software tools that automatically creates records as the work is done, so compliance happens naturally.   

We offer multiple solutions to address common issues related to point-to-point verification, SCADA system changes, and hassle-free recordkeeping – like the Compliance Log that’s part of our CRMgr software tool.

Additionally:

  • Use our ComplyMgr software module to identify gaps between the CRM Rule and your processes, procedures, and implementation.
  • Use our CRM Support Services to perform backup control room tests and log the results in your compliance log.
  • Use our PointMgr software module to streamline the point-to-point verification process between the control room and field techs.
  • Use our Pipeline SCADA solution to ensure alignment with API RP 1165
  • EnerSys can perform an API RP 1165 Assessment to gauge your current level of compliance

Overview of Section D [Fatigue Mitigation]

Juggling efficient scheduling and complying with Section D of the CRM Rule can be quite tricky. Control Room Managers often must create efficient schedules while considering staffing and Hours of Service (HOS) limits.

Section D asks your organization to take steps to lessen controller fatigue, which could affect their ability to do their job during a shift.

This includes:

  • Establishing shift lengths and schedule rotations that provide controllers sufficient time off-duty to achieve eight hours of continuous sleep.
  • Educating controllers and supervisors in fatigue mitigation strategies, including how off-duty activities contribute to fatigue.
  • Training for controllers and supervisors to recognize the effects of fatigue.
  • Establish a maximum limit on controller hours-of-service, which may provide for an emergency deviation from the maximum limit if necessary for the safe operation of a pipeline facility.

Section D (Fatigue Mitigation) – 7 Findings

  • 5 NOA
  • 1 Proposed Civil Penalty
  • 1 Warning Letter

Control room managers face the challenge of crafting shift schedules that adhere to fatigue mitigation requirements while promoting business continuity and personnel satisfaction. The complexity is magnified by outdated tools in use, dating back to the CRM Rule’s inception over a decade ago.

We have published a white paper that dives deeper into this topic. Download it now to gain a deeper understanding of promoting natural compliance, operational effectiveness, and pipeline safety—essential elements for a resilient and forward-looking control room.

Findings from Section D

There were seven instances where PHMSA found an operator in violation of the CRM Rule under Section D. These findings could be broken down into three main descriptors, each dealing with different aspects of Fatigue Mitigation in the control Room.

Hours of Service (HOS) Limits

Operators must define, implement, track, and document controller’s hours of service to avoid deviations from HOS limitations. Failure to do so can result in negative impacts to fatigue management within the control room.

  • One organization’s procedure was deemed inadequate for failure to include a step-by-step practice for determining hours worked by a controller, preventing fatigue considerations when an unplanned absence occurs. The plan also failed to establish a procedure for emergency HOS deviation when filling a controller absence.

Defining Fatigue Risks and Mitigation Strategies

Three of the findings primarily related to an organization’s failure to define the risks associated with fatigue and offer comprehensive mitigation strategies.

  • One control room’s CRMP failed to identify all fatigue risks that controllers are subject to, including environmental, dietary, and health factors.
  • Another Operator failed to include a comprehensive list of fatigue mitigation strategies.

Lack of Training and Failure to Follow CRMP

  • One finding resulted in a Proposed Civil Penalty of $50,200 when an operator failed to follow its CRMP and did not provide periodic fatigue education and training for controllers and supervisors.

EnerSys Solutions for Section D Findings

We offer tools specifically crafted to assist you in overcoming the hurdles of creating a solid schedule while adhering to PHMSA HOS regulations. Our CRM Suite includes the FatigueMgr module, equipped with features tailored for scheduling and fatigue management for both controllers and Control Room managers.

With the built-in reporting features in FatigueMgr, managers can verify compliance and uphold safe operations. Control Room supervisors can monitor real-time activity to pinpoint potential risk zones and act accordingly in line with their organization’s fatigue mitigation plan.

FatigueMgr Key Features & Benefits

  • Customizable, simplified control room scheduling
  • Easily identify staffing needs and workload trends
  • Track HOS for all qualified personnel (actual and projected)
  • Identify, mitigate, and document HOS Deviations (both shift time and reset/off shift time)
  • Understand who to call in to fill in on shifts, while preventing HOS Deviation ripple effects
  • Efficiently communicate fatigue risk and mitigation tasks for HOS Deviations to controllers
  • Adjust staffing levels or workload across shifts
  • Document and review fatigue mitigation records
  • Analyze and mitigate workload peaks


Additionally, the CRMgr module in our CRM Suite allows controllers to log fatigue mitigation tactics, enabling them to actively participate in your operation’s fatigue mitigation program. The software automatically creates reports when users log fatigue mitigation efforts – a necessary piece needed to satisfy PHMSA inquiries.

  • CRMgr allows CRM Managers (or Fatigue Risk Managers) to log fatigue training records
  • Managers can assign relevant fatigue-related reading and fatigue policy documents for controllers to read

When combined with an effective fatigue mitigation program, our tools help ensure compliance with the CRM Rule.

Closing

EnerSys offers the CRM suite to help operators comply with the CRM Rule.

The CRM Suite includes software components for control room management, alarm management, fatigue mitigation, point-to-point verification, and compliance management. Our software ensures adherence to the CRM Rule, the ability to quickly generate reports using your data, and more time to focus on growing the capabilities of your pipeline.

We would like to schedule a brief, no-obligation demo with your team to show you how our software can help your operation. To get started, please complete our contact form, email our team at sales@enersyscorp.com, or call us directly at 281-598-7100.