In our previous post, we discussed the 2022 PHMSA Enforcement Actions report and gave a high-level overview of the year’s findings. To provide a better understanding of the report, and add some helpful insights, we’re going to take a deeper dive into sections A & B of the Control Room Management Rule (49 CFR 192.631 for the Transport of Natural Gas and 195.446 for the Transport of Hazardous Liquids).
If you’re looking for a simple overview, or want to start fresh before getting into the detail of specific CRM Rule sections, check out our first article on the 2022 PHMSA Enforcement Actions. Be sure to subscribe to our monthly newsletter to be notified when we release our next thorough entry into this series.
Jump to Section A Insights
Jump to Section B Insignts
Section A
In our overview from January, we identified 66 findings in the 2022 PHMSA Enforcement Action report. Many of the findings under Section A could be considered violations of other sections in the CRM Rule but were categorized by PHMSA as general requirements.
For assets that are subject to regulations, PHMSA has identified certain actions that an operator must take to support the Section A General Requirements section of the CRM Rule:
- Each operator must have a written Control Room Management Plan (CRMP).
- The CRMP must have policy and procedures that align with the CRM Rule.
- The operator must follow the policy and procedures that were set forth in the CRMP.
- Operators need to formalize and control all documents, which should be made readily available to controllers at their consoles.
- The CRMP should be readily available to, and understood by, controllers.
- Operators must review the CRMP program annually, not to exceed 15 months.
Section A Findings
To make things easier to consume, we’ve taken the information provided by PHMSA and identified the notable occurrences for Section A-related findings. These results highlight the particular areas of interest during PHMSA inspections and indicate a heavy emphasis on your organization’s Control Room Management Plan, including documentation and follow-through. We recommend a further review of enforcement actions to ensure you are up to date on findings most pertinent to your business case.
Alarms
- Several operators failed to perform alarm setpoint and descriptor reviews when associated field instruments were calibrated or changed.
- Several operators failed to define the process and criteria by which alarms were determined to be safety-related, and by which alarm setpoints were defined
Shift Handover
- One operator failed to account for instances in which the oncoming controller was unable to claim their scheduled shift.
Inadequate Definition
- One operator’s definition of “Controller” did not account for their ability to perform indirect control actions (initiating corrective action by e.g. making a phone call and giving instructions to field personnel)
- One operator had a proper definition of a control room, they did not account for how that procedure would be used to assess current or new facilities.
- One controller, using a primary and secondary controller, did not provide sufficient definition for roles and responsibilities between the two. Specifically, “the procedure did not disallow anyone outside the Primary Controller to operate, monitor, and take action on the system. “
- Another operator failed to account for the fact that personnel at PLCs or RTUs on their system could perform control actions to manage the flow of gas outside the fence and how this would meet the criteria for that PLC/RTU as being a control room. Furthermore, the operator did not define how the person performing the control action would be qualified as a controller.
- One controller had ambiguity around when a shift handover or temporary relief needed to be performed when the console was to be left unattended. It was noted their use of terms like “reasonable” and “appropriate” provided insufficient direction.
- Several operators were noted as having ambiguity in their procedures. One of which noted that the responsibility to perform certain calls in the event of a control room evacuation was not assigned and that the information to communicate was not adequately defined.
Other Insights
- One operator failed to require its SCADA and field personnel to contact the control center when making field changes that affect control room operations.
- Several operators had issues with their Internal Communication Plans (for manual control). One inspection finding, associated with a significant financial penalty, indicated the ICP testing did not account for full system failures and how some assets would be shut down while others would be manually controlled.
- One operator identified their use of redundant servers, but did not test them in accordance with Backup SCADA System Testing requirements.
- Several operators had multiple findings in this section. One had as many as 11 findings.
- Multiple operators failed to provide adequate policies and procedures for monthly and annual reviews
- Multiple operators failed to document activity that was defined in their CRMP (e.g. shift handovers, monthly and annual reviews, point-to-point verifications, etc.)
- A new audit trend pertained to several operators’ use of 3rd Party Control Rooms and the gaps in policy and implementation between the two organizations (Note: the findings were not related to the use of 3rd Party Control Rooms as a practice.)
- Multiple operators failed to adequately reference policies and procedures from their CRMP that were to be used in implementing CRM activity.
Fines
Although there were several findings related to Section A, most of the operations cited in the report did not incur civil penalties. The financial burden of fines imposed on your operation is another factor to keep in mind when reviewing your level of compliance and taking steps to mitigate PHMSA audit findings. A total of $94,500 in penalties were levied against organizations for compliance violations under Section A of the CRM rule.
Implications
The findings under Section A indicate that PHMSA is heavily focused on your organization’s Control Room Management Plan. Both in whether or not you’re following the plan, and if you’re properly generating and retaining records as defined by your CRMP. Comprehensive policies and procedures within an organization’s Control Room Management Plan are an essential piece of ensuring a safe and compliant operation. However, these are not the only pieces of the puzzle. Continuous follow-through and management review of documentation (implementation) are critical so that you can validate that your organization has done what it established in that plan.
If your organization received notice of violation from PHMSA, or you want to ensure you stay ahead of the next inspection, here are a few questions to consider based on the 2022 findings:
- Are policies and procedures in your CRMP defined thoroughly, so as to remove any ambiguity during all operating conditions?
- Are the policies and procedures in your CRMP sufficient for compliance with applicable CRM regulations?
- Are you following through on the frequency of reviews, training, etc. (i.e. CRM implementation) as directed in your CRMP?
- Can you verify that you’ve done the work required, as outlined in your CRMP, and are all documents/records readily available?
- Are your training policies and procedures thoroughly established and documented?
EnerSys Control Room Solutions
Whether your operation needs support with establishing more comprehensive policies in your CRMP or you need help with program implementation, we have a control room solution for your operation.
If you’re having trouble determining whether or not the CRM Rule applies to you, we can perform an applicability assessment. This assessment is designed to determine whether you meet the definition of a Control Room as defined in the CRM Rule, taking your pipeline facilities and operating setup into consideration.
We can also perform CRM Compliance Review using our software module, ComplyMgr, to measure your plan’s level of policy compliance compared to the PHMSA regulations. Our team will make observations, perform interviews, and review records to evaluate proper 192.631 and 195.446 control room management implementation.
Our team has the knowledge and tools to design and implement a Control Room Management Plan specific to your operation. If necessary, we can also rewrite your CRMP to address all PHMSA requirements. In addition to meeting regulations, this is used as a framework for continuous improvement moving forward.
Section B
Adequately defining roles and responsibilities is vital to ensuring a safe operation. Any ambiguity in your organization’s CRMP around the who and what opens up the potential for safety incidents. When you consider that controllers are typically the first to detect and respond to abnormal or emergency operating conditions, they must be clear on their role in maintaining the safe operation of the pipeline.
Section B Findings
25 findings from PHMSA’s report related to Section B (Roles & Responsibilities) and we’ve highlighted the cases that were particularly interesting or noteworthy for other operations to consider.
Controller Assignments, Qualifications, and Permissions
- One operator utilizing a 3rd Party Control Room did provide context as to which pipeline facilities the same controller was responsible for in addition to their own. Furthermore, there were not sufficient controls in place to ensure only the qualified controller could perform control actions (and other relevant functions e.g. receive 811 calls) for the operator’s assets. The same operator lacked clarity as to who could take temporary relief for their assets given the fact there were other controllers in the room with various qualifications.
- One operator had multiple controllers with qualifications for only specific asset groups pertinent to their assigned console. However, the same controllers could access all asset areas when logging into the SCADA system. Here, policy was not enough for disallow access beyond the assigned area.
- Another operator allowed two controllers to log in to a SCADA system and control the same operating area. This behavior could lead to conflicting control actions. PHMSA noted the operator should limit the number of logins to each pipeline system and assign SCADA permissions accordingly.
- Similarly, one operator has primary and secondary controllers assigned to the same operating area. Both control the same assets from different consoles, and both can issue commands and acknowledge alarms simultaneously. The same operator failed to define the amount of time the primary controller could be away from the console before the secondary controller must take over the role of the primary controller.
Training
- Another operator had a controller training two other controllers at the same time, all three of whom were responsible for an individual console. As such, the trainer’s span of control was across all three consoles.
- The same operator rotated each trainer and trainee every 3 to 4 days. Lacking training program structure and documentation meant a new trainer was never able to fully grasp what competencies the trainee had or had not mastered.
Shift Handover
- One operator allowed controllers to be away from the console for 90 minutes before a shift handover would be required.
- Numerous operators lacked sufficient process and/or recordkeeping around shift handover
Other Insights
- One operator utilized sampling when completing point-to-points. They also failed to test setpoints during point-to-points. Nor did they define which screens were to be assessed when a point was present on multiple screens.
- Multiple operators failed to define roles and responsibilities for various operating conditions (some failed to define the nature of and difference between operating conditions) such that operators were unclear on how to respond to abnormal/emergency operating conditions, etc.
- Multiple operators failed to account for roles and responsibilities of “others” who have the authority to direct or supersede the specific technical actions of a controller.
- For those who do allow “others” to direct or supersede, these operators failed to define the training required, the way the list of “others” is maintained, or the process following the act of superseding.
Fines
The findings observed by PHMSA highlight the importance of defining roles and responsibilities to eliminate any uncertainty among your team. The lone Proposed Civil penalty under Section B was a result of one operation’s failure to provide adequate information. Initially, a fine of $55,800 was imposed, but the collected civil penalty was $27,900.
Implications
Findings related to Section B have increased exponentially from just 3-4 years ago. This indicates an emphasis on the importance of Roles and Responsibilities in your control room. There should be no ambiguity in your processes, and you should account for all operating conditions – if the actions/responses are clearly defined and trained on before the conditions arise, your team will be knowledgeable about the appropriate response. Improving clarity means your team can operate more efficiently, effectively, and ultimately – safer.
EnerSys Control Room Solutions
When building an effective Control Room Management Program, it’s important to adequately define roles and responsibilities for your team. Operators need to maintain awareness that responsibilities may vary depending on the operating conditions at a given time. The workflow and decisions must adapt as conditions change from normal to abnormal or emergency conditions.
It’s clear from the findings in the last few years that PHMSA expects your CRMP to explicitly define all variations to procedures. This includes the control room personnel responsible for those tasks – in normal operating conditions, abnormal operating conditions (AOCs), and emergency situations.
The CRMgr module of the POEMS CRM Suite enables pipeline operators to capture valuable operational policies and procedures, as well as compliance-related activity. With the ability to view, update, and review your CRMP, CRMgr can help you stay ahead of potential violations.
Key Benefits of CRMgr
- Dedicated Tools to View, Update, and Review your CRMP Policy
- ComplyMgr software module allows for document development to ensure roles and responsibilities are adequately defined for all operating conditions
- Unique Dashboards for Controllers and Control Room Managers
- Tools and Permissions Built to Accommodate Roles & Responsibilities Based on Operating Conditions
- Assigned Reading with Confirmation Reporting to Communicate Important Information, Documents, and Changes to Controllers
- Record of Changing Rolls and Responsibilities via emailed SHO Reports
Additional Solutions
EnerSys has worked to build a comprehensive set of tools designed to help you manage compliance and safety in your Control Room. Through our partnerships with our sister companies, we can offer even more help on your path to Natural Compliance.
P.I. Confluence
- P.I. Confluence (PIC) provides tools for managing your program, processes, workflow, communications, and information exchange in pipeline operations.
- The POEMS™ Program Suite consists of web-based process management software tools to validate and manage pipeline programs and processes
GCI & Muddy Boots
- Gas Certification Institute offers field operations software, Muddy Boots, to help close the communication gaps between the field and the control room.
- The platform is ideal for addressing change management within your operation, and the POEMS CRM Suite allows for linking and communication between the programs to create, track, and update work orders.