2022 PHMSA Enforcement Actions: Solutions for Adequate Information and Fatigue Mitigation

At EnerSys, our goal is not just to educate and inform our industry, but also to equip you with tools and resources to improve safety, increase efficiency, and ensure compliance. In this blog post, we’re continuing our in-depth review of the findings from the 2022 PHMSA Enforcement Action Report. 

Having the knowledge and the tools necessary to avoid or remedy findings like the ones reviewed below is imperative to maintaining compliance and ensuring a safe work environment. 

If you’re curious about General Requirements or Roles and Responsibilities, read our previous blog post for a closer look at Sections A and B. For an overview of the full 2022 report, including a high-level review of violations by Section and level of enforcement, read our Control Room Findings Overview. 

This post will thoroughly review the findings from Section C and Section D, the implications and insights from those cases, as well as applicable EnerSys solutions. 

C – Adequate Information: 38 findings

• 14 NOAs
• 12 Warning Letter/Items
• 8 Proposed Compliance Orders
• 4 Proposed Civil Penalties

D – Fatigue Mitigation: 8 findings

• 6 NOAs
• 1 Warning Letter
• 1 Proposed Compliance Order

Section C: Adequate Information

Section C of the CRM Rule requires that, at a minimum, each operator must provide controllers with the tools, information, processes, and procedures that are necessary so that a controller is able to perform their duties as defined by the operator.

This includes enabling operators to have an accurate view of the system’s health and performance and ensuring compliance with API 1165 for applicable SCADA systems. The Recommended Practice for Pipeline SCADA Displays provides guidance on document structure, writing styles, graphics and tables, references, appendices, and a review process. 

It also requires operations to conduct point-to-point verification between SCADA displays and related field equipment when field equipment is added or moved, or when there are other changes that affect pipeline safety made to field equipment or the SCADA displays. Backup SCADA systems must also be tested once each calendar year, but not at intervals greater than 15 months.

Section C also states that an internal communication plan to provide adequate means for safe, manual operation of the pipeline must be tested and verified at least once each calendar year (at intervals not to exceed 15 months).

Lastly, operations must establish and implement procedures for when a different controller assumes responsibility for a console, including what information needs to be exchanged during a shift handover. 

Considering that Section C was among the most-cited violations from the 2022 PHMSA Enforcement Actions Report, adherence to this section of the rule is critical to maintaining compliance and ultimately, a safe operation.

Section C Findings

Violations Based On API RP 1165 Compliance

• A Proposed Compliance Order was issued when an organization failed to verify, through records, its implementation of API RP 1165 when a different SCADA system was added to the existing console.
• One organization received a Notice of Amendment for failure to comply with API 1130. Additionally, it failed to address training as roles and responsibilities of a controller to help detect an abnormal condition in development or identify an emergency condition.
• An operation received a Proposed Compliance Order when it failed to audit or document a comparison of their SCADA style guide to ensure it is designed in accordance with API RP 1165
• An operation failed to verify that the required sections of API RP 1165 had been implemented in its SCADA system design. This resulted in a case that included a Proposed Compliance Order, a Warning, and a Proposed Civil Penalty.
• Rather than evaluating its SCADA system in accordance with API RP 1165, one organization used the American National Standards Institute ANSI/ISA-18.2-2016 – resulting in a Proposed Compliance Order from PHMSA. 

Violations Based On Point-to-Point Procedures

• One organization’s procedure failed to identify what records would be used to demonstrate compliance regarding point-to-point verification when utilizing assets from the ROC control room – resulting in a Notice of Amendment.
• Multiple operations failed to adequately define the process for conducting and documenting thorough point-to-point verification.
• One organization captured inadequate information when completing point-to-point verification. The operator failed to include necessary info when filling out the report, and the company failed to provide follow-up and documentation of adequate verification. This resulted in a Warning from PHMSA. 

Violations Based On Internal Plan Testing

• A Proposed Civil Penalty was issued after an organization failed to test and verify its internal communication plan regarding the manual operation of the pipeline.
  • While their plan took a controlled shutdown into consideration, it failed to establish processes for a manual restart and manual operation of the pipeline.
  • Individual locations had internal communication plans and tests, but there was no plan or test for a complete or system-wide failure. 
• Multiple operators were cited for a failure to test and verify internal communication plans (ICP) at the required intervals.
  • These findings resulted in 4 Warnings and 1 Proposed Civil Penalty from PHMSA.
• One organization received a Warning after it failed to test and verify its internal communication plan regarding the manual operation of the pipeline at least once per calendar year – despite having previously suffered a SCADA and corporate network outage.
  • The organization had a second case for failure to “create, test, and verify an internal communication plan for manual operation of its pipeline safely for 2018 and 2019,” which also resulted in a Warning.
• One organization did not facilitate tests or drills to ensure the viability of its procedures for a manual shutdown of the pipeline. Processes that were documented in its plan for manual shutdown or operation were also not tested to verify the operation’s internal communication plan. A Proposed Compliance Order was issued by PHMSA in response to this violation.

Violations Based on Backup SCADA Testing

• One operation received a Warning for failing to test backup SCADA servers.
  • They are not exempt from testing, even if they’re considered redundant.

• Multiple operators failed to test their backup SCADA systems despite having a policy in place that required regular testing. The violations included 3 Warnings, a Proposed Compliance Order, and one instance of a Proposed Civil Penalty from PHMSA
• Multiple operators received a Notice of Amendment when their plan failed to establish a procedure for testing backup SCADA systems – resulting in backup SCADA systems not being tested. 
• Another operator received a Warning from PHMSA after they tested some, but not all, of their backup systems.

Violations Based On API RP 1165 (SCADA System Hardware Change)

• Multiple organizations were found in violation of the requirement for failing to have adequate policies in place to address changes to the SCADA system. Plans frequently failed to define the process of the addition, expansion, or replacement of a SCADA system. 

• Another Proposed Compliance Order was issued when an organization failed to conduct point-to-point verification between SCADA displays and related field equipment when the field equipment was moved, added, or experienced other changes that could affect pipeline safety. The records failed to document verification between SCADA displays and related field equipment.
• Another Notice of Amendment was issued to an organization because its procedures were insufficient because they lacked an adequate procedure for conducting point-to-point verifications between SCADA displays and related field equipment when changes are made to field equipment or SCADA displays.
  • The procedures did not sufficiently address scheduling point-to-point verifications for logic changes that affect safety-related points, confirming the accuracy of moved points, recording safety-related alarm setpoints, including all points that can impact safety in the protocol-specific register list, matching display range and point value range, and confirming the accuracy of device responses through the full range.  

Violations Based on Inadequate Information

• One organization failed to provide adequate information to controllers for recognizing pipeline failures – and instead, controllers responded to the failure only as an abnormal operating condition (AOC). This resulted in a Proposed Compliance Order.
  • Due to a lack of pressure indicators that account for the terrain of the pipeline, controllers were not given the information necessary to operate safely and respond appropriately during all operating conditions.
• PHMSA issued a Notice of Amendment for an operation when its plan failed to define the practice, including necessary reference documents, of manually operating the pipeline during a SCADA failure or outage.
• One organization failed to establish an adequate procedure for when a different controller assumes responsibility – including the content of information that needs to be exchanged. The procedure also lacks a definition of who is responsible for completing shift change forms. This failure to clearly define the procedure resulted in a Notice of Amendment.
• One organization failed to provide adequate clarity on the information required to be exchanged when a different controller assumes responsibility, resulting in a Notice of Amendment.
• During one inspection, personnel indicated that items were checked, but likely not documented, and in another instance, personnel ‘assumed’ the items were checked.
  • When documentation did not support that point-to-point verification had been completed properly, PHMSA issued a Warning to the organization. 

Fines

$953,100 in regulatory fines were associated with violations PHMSA identified under Section C of the CRM Rule. One organization received the bulk of these penalties – including one fine for over $846,000. 

This penalty was recommended when the organization failed to verify its plan related to the manual operation of the pipeline. Because there wasn’t an adequate plan in place, and that plan hadn’t been tested – there were potential subsequent issues that were left unaccounted for.

• PHMSA noted that the approach for manual restart puts the pipeline’s integrity at risk. In addition, it added delays to restarting – creating supply issues and societal impacts. $864,300 Fine

Implications

Based on the findings we’ve reviewed under Section C, PHMSA placed a heavy emphasis on adequate procedures for testing, implementation, and sound record-keeping for each required testing interval. 

We saw 8 violations that specifically cited a failure to either define a procedure for testing backup SCADA systems, or a failure to follow through (including documentation) on the testing requirements set forth in their CRMP.

It’s clear that PHMSA is ensuring organizations have both a detailed plan for testing backup SCADA systems, and that they’re documenting those tests as they occur. 

Multiple organizations failed to adequately define the processes around changing, moving, or replacing SCADA hardware. As hardware changes – whether from an upgrade, failed hardware, or a new acquisition to your organization – operations need to verify compliance with API RP 1165 when the system is changed.

Overall – testing, execution of that testing, and documentation to verify execution proved to be the most common reasons that organizations were found in violation by PHMSA. 

EnerSys Solutions

There were organizations cited that claimed the work was being done, but couldn’t provide supporting documentation. At EnerSys, we believe that compliance should come as a result of putting in the work. That’s why we build software tools to help close these gaps and ensure you’re tracking all the work that’s being done. 

We offer multiple solutions to address common themes related to point-to-point verification, SCADA system changes, and hassle-free recordkeeping.

A CRM Compliance Review using our software module, ComplyMgr, will compare your Control Room Management Plan (CRMP) to PHMSA regulations. In addition to our observations, performance reviews, and records review – our team will help you identify gaps that could lead to violations like the ones discussed above. 

Our team also has the skills and tools to build and implement a CRMP that is custom-fit to your Control room. We can also modify your current CRMP to address any regulation discrepancies that our CRM Compliance Review identifies.  

Additionally:

• Use our ComplyMgr software module to identify gaps between the CRM Rule and your processes, procedures, and implementation.
• Use our CRM Support Services to perform backup control room tests and log the results in your compliance log.
• Use our PointMgr software module to streamline the point-to-point verification process between the control room and field techs.
• Use our Pipeline SCADA solution to ensure alignment with API RP 1165
• EnerSys can perform an API RP 1165 Assessment to gauge your current level of compliance

Section D: Fatigue Mitigation

Balancing effective scheduling while maintaining compliance with Section D of the CRM Rule can be a complicated act. Many CRM Managers are tasked with building efficient schedules while also dealing with staffing and HOS (Hours of Service) limitations. 

Section D requires your organization to implement mitigation efforts to reduce controller fatigue that could inhibit a controller’s ability to carry out their role and responsibilities during a shift. This includes:

• Establishing shift lengths and schedule rotations that provide controllers sufficient time off-duty to achieve eight hours of continuous sleep.
• Educating controllers and supervisors in fatigue mitigation strategies, including how off-duty activities contribute to fatigue.
• Training for controllers and supervisors to recognize the effects of fatigue.
• Establish a maximum limit on controller hours-of-service, which may provide for an emergency deviation from the maximum limit if necessary for the safe operation of a pipeline facility.

Section D Findings

PHMSA cited 8 different violations related to Section D in their 2022 report. The findings ranged from a lack of documentation for fatigue mitigation processes to inadequate definitions of shift lengths related to HOS. 

Violations Based on HOS Tracking & Documentation

• One operation received a Notice of Amendment for failing to consider their controller’s commute times in their fatigue mitigation efforts. 
• A Warning Letter was given to an organization that failed to document and factor in HOS limitations for controllers that were on-call.
  • While the time responding on-call was considered for payroll, it was not taken into consideration for HOS limitations.
• One organization received a Proposed Compliance Order because they only checked personnel commute time at the time of hiring, but failed to check or reconfirm after their initial onboarding. Establishing shift schedules without confirming commute times prevents the organization from ensuring adequate off-duty time to achieve 8 hours of sleep. 
• Another operation received a Notice of Amendment when their written procedure failed to specify how hours of service (HOS) must be documented.

Violations for Inadequate Fatigue Mitigation Plan

• A Notice of Amendment was given when an organization’s plan didn’t include a process to track and monitor the time worked on shift and otherwise by controllers. Instead of evaluating hours spent responding to alarm calls or monitoring the system after hours and on weekends, they were reviewing paystubs to evaluate and approve overtime.
  • This method of reviewing time does not take the on-call work into consideration, and wouldn’t give the context needed to ensure a minimum of 8 hours of continuous sleep.
• One organization failed to include processes to collect, analyze, and review its fatigue mitigation efforts to determine efficacy, resulting in a Notice of Amendment from PHMSA.
• PHMSA cited one organization in violation for failing to provide instruction for managing controllers who self-identified as fatigued, resulting in a Notice of Amendment.
• Another violation was found when an operation didn’t have procedures in place for its supervisors to complete periodic ‘refresher’ fatigue training routinely. Additionally, supplemental fatigue training completed by controllers was not documented.

Fines

While there were no fines related to findings under Section D, the importance of effective fatigue mitigation efforts – including follow-through and documentation – cannot be overlooked. Maintaining vigilance during a shift is an important factor in maintaining a safe operation, especially in abnormal and emergency operating conditions.

Implications

A common theme with the findings from Section D related to ensuring your controllers have adequate time to achieve 8 hours of continuous sleep between shifts. 

Work that is done for the company beyond normal shift duties like meetings, training, support roles in the Control Room, and various administrative tasks needs to be taken into consideration when building schedules. This also includes when your controllers are on-call or working remotely. The time they’re spending responding to alarms, monitoring, or controlling also contributes to their HOS. 

It’s important to keep commute times and work done off-premises or outside of normal shift duties in consideration, and operations should have processes in place to review the time impacts of these activities regularly. 

EnerSys Solutions

We’ve got tools designed to help you meet the challenges of building an effective schedule while managing PHMSA HOS requirements. The FatigueMgr module of our CRM Suite provides tools to controllers and Control Room managers for scheduling and fatigue mitigation efforts. 

Managers can use FatigueMgr reporting to validate compliance and ensure safe operations, and Control Room supervisors can view activity in real-time to identify risk areas and take action in alignment with their organization’s fatigue mitigation strategy. 

FatigueMgr Key Features & Benefits

• Customizable, simplified control room scheduling
• Easily identify staffing needs and workload trends
• Track HOS for all qualified personnel (actual and projected)
• Identify, mitigate, and document HOS Deviations (both shift time and reset/off shift time)
• Understand who to call in to fill in on shifts, while preventing HOS Deviation ripple effects

• Efficiently communicate fatigue risk and mitigation tasks for HOS Deviations to controllers
• Adjust staffing levels or workload across shifts
• Document and review fatigue mitigation records
• Analyze and mitigate workload peaks

Additionally, the CRMgr module in our CRM Suite allows controllers to log fatigue mitigation tactics, enabling them to actively participate in your operation’s fatigue mitigation program. The software automatically creates reports when users log fatigue mitigation efforts – a necessary piece needed to satisfy PHMSA inquiries.

• CRMgr allows CRM Managers (or Fatigue Risk Managers) to log fatigue training records
• Managers can assign relevant fatigue-related reading and fatigue policy documents for controllers to read

When combined with an effective fatigue mitigation program, our tools help ensure compliance with the CRM Rule.

– We are here to help your operation implement these solutions. If you’re interested in setting up a demo to see how FatigueMgr and CRMgr can help you achieve Natural Compliance – let’s start the conversation. Don’t wait until you’ve received notice from PHMSA about an upcoming inspection – take steps to be proactive and prepared. 

Call us at 281-598-7100 to speak with EnerSys General Manager Ross Adams or VP of Business Development Dale Schafer. You can also contact us via email at sales@enersyscorp.com or complete our website contact form.

Additional Solutions

EnerSys has worked to build a comprehensive set of tools designed to help you manage compliance and safety in your Control Room. Through our partnerships with our sister companies, we can offer even more help on your path to Natural Compliance.

P.I. Confluence

• P.I. Confluence (PIC) provides tools for managing your program, processes, workflow, communications, and information exchange in pipeline operations.
• The POEMS™ Program Suite consists of web-based process management software tools to validate and manage pipeline programs and processes.

GCI & Muddy Boots

• Gas Certification Institute offers field operations software, Muddy Boots, to help close the communication gaps between the field and the control room.
• The platform is ideal for addressing change management within your operation, and the POEMS CRM Suite allows for linking and communication between the programs to create, track, and update work orders.