April 2023, Vol. 250, No. 4
Features
Control Rooms: Achieving SCADA Operational Readiness, Compliance
By Payam Yeganeh and Michael Nushart, Black & Veatch
(P&GJ) — Implementing a comprehensive control room management (CRM) plan in today’s energy market can be a challenging task in that integrating it with modern supervisory control and data acquisition (SCADA) systems adds layers of complexity to the project’s implementation and compliance requirements.
Taking a holistic view of control room management shows there are multiple facets, depending on your situational point-of-view. Some aspects are specific to pipeline safety code compliance while others are related to pipeline operations. The nexus of these various perspectives is the safe, efficient transfer of product (gas or hazardous liquid).
Figure 1 illustrates the functional grouping of CRM components. Commercial operation, which includes product purchase, sales and supply, represents the primary commercial function of the pipeline, with pipeline control and measurement playing critical support functions.
CRM compliance and alarm management overlay and affect all the other commercial and functional components to ensure safe operation. Because all the components work in concert, stakeholder requirements for each need to be defined and integrated for operational readiness.
The Challenge: Defining, Integrating Functional Components
By examining the drivers for the individual CRM components, it becomes evident that each has its origins in a different business discipline (e.g., engineering, compliance, and control and measurement, along with operations). Each business discipline brings unique requirements to the control room and controllers.
Additionally, each component has different data requirements. This set of conditions creates a need for multi-dimensional thinking when designing SCADA functionality to support CRM and fully achieve operational readiness.
In the following sections, we will discuss the key components and recommended consideration for each of these functional components.
Commercial Operation
The purpose of the pipeline system is the throughput that supports product purchase, sales and supply functions. System controllers are responsible for safely moving specified volumes of products through the pipeline system and providing delivery within specific parameters.
The required operator qualification skills, supported by specific training, ensure that operations are conducted safely. The controllers’ understanding of system operation and safe operating pressure and flow enable the effective movement of products, including line-packing, when necessary. Real-time data is the controllers’ primary tool.
Effective measurement data – pressure, flow and volume – is critical to the system’s commercial operation. Understanding where and how much product is within the system ensures that the controller effectively moves the product to where it’s needed. The measurement data also is essential for product custody transfer purposes, both into and out of the pipeline system.
Codes & Standards
The Pipeline and Hazardous Materials Safety Administration (PHMSA) has published requirements for CRM for natural gas systems (49 CFR 192.631) and hazardous liquids systems (49 CFR 195.446). Because the safety of people and property near pipelines is paramount to system operations – and documented plans are mandated – the CRM plan is a logical starting point for SCADA system design, including architecture, capabilities and functions.
In addition to the requirements published in the code sections, several industry consensus standards have been incorporated by reference (IBR), in part or in their entirety. They serve as a valuable guide for the design and configuration of SCADA systems and related control room activities.
Some examples of IBR consensus standards are:
American Petroleum Institute (API) Standards and Recommended Practices
- API RP 1165 – Recommended practice for pipeline SCADA displays
- API RP 1168 – Pipeline CRM
Other consensus standards for design guidance include:
- API 1164 – Pipeline SCADA security
- API RP 1167 – Pipeline SCADA alarm management
- ANSI GPTC Z380.1 – Guide for gas Transmission, distribution and gathering piping systems
Pipeline Safety Code
The primary program elements for CRM for natural gas systems (192.631) and hazardous liquid pipeline systems (195.446) have the same topic headings with minor differences to the specific requirements, as follows:
Roles and responsibilities:
- Controllers’ authority and responsibilities under normal, abnormal and emergency operating conditions
- Requirements for formal inter- and intra-shift handover procedures
Provide adequate information:
- SCADA displays design requirements under API RP 1165 (IBR)
- Point-to-point verification requirements
- Testing and verification of an internal communication plan
- Backup SCADA system testing requirements
Fatigue mitigation:
- Establishment of shift schedules to provide an opportunity for eight hours of continuous sleep
- Establish maximum hours of service (HOS) for controllers
- Fatigue recognition and mitigation training for controllers and supervisors
Alarm management
- Establish safety-related alarm operations
- Monthly review and analysis of safety-related alarms that have had alarms inhibited, generated false alarms or have had forced or manual values
- Annual review and verification of safety-related alarm values
- Annual review and verification of the alarm management plan
- Annual review and analysis of controller workload content and volume (best tracked monthly for annual review)
- Address deficiencies identified by reviews and analysis
Operating experience:
- Review reportable incidents (49 CFR 191) to determine if control room actions contributed to the incident
- Integration and implementation of lessons learned
- Inclusion of lessons learned in the training program
Training:
- Recognition of and response to abnormal and emergency operating conditions
- Use of a computerized simulator or non-computerized (tabletop) method for training controllers to recognize abnormal operating conditions
- Communication responsibilities
- Pipeline system working knowledge
- Provisions for review of infrequently used pipeline operating set-ups
- Participation in team training exercises involving other operating groups who would operationally collaborate with controllers during normal, abnormal or emergency situations.
Compliance and deviations:
- Documentation that demonstrates compliance with the requirements of this section
- Documentation to demonstrate that any deviation from the procedures required by this section was necessary for the safe operation of a pipeline facility
All the elements require documentation showing completion. As a favorite PHMSA saying goes, “If you didn’t document it, you didn’t do it!” This adds to the importance of designing SCADA architecture, its functional components, and data flows to store and retrieve information required for review and analysis.
In addition to these training requirements, controllers of gas and liquids pipeline systems require qualification under their respective OQ code sections. Many controller functions meet the four-part test for OQ:
- It’s performed on a pipeline facility
- It’s an operations or maintenance task
- It’s performed as a requirement of this part
- It affects the operation or integrity of the pipeline
Point-to-Point Testing
The CRM rule requires a point-to-point verification for all safety-related points whenever there are changes made to the field equipment or SCADA displays and alarms.
The verification requires confirmation that the input-output of each field instrument or device is accurately presented to the controller. The confirmation process requires documenting measured field parameters and the corresponding SCADA information displays and alarm set-points.
This detailed process is labor-intensive, and accurate documentation is necessary for audit and incident investigation purposes. Management and oversight of the process is needed to ensure accuracy.
Analysis, Documentation
PHMSA requirements for natural gas and hazardous liquid control rooms result in the need to routinely review, monitor and analyze critical data points such as safety-related points from SCADA and develop mitigative and preventive actions for anomalous conditions identified by the review and analysis process.
These key data points are in addition to the alarm management functions of SCADA and involve collecting information on Controller workload, hours of service, and alarm categorization.
Documentation of the review and analysis becomes part of pipeline safety audits, making data availability and accuracy essential.
The SCADA system is the core operational technology that is integral to the pipeline control and measurement. Figure 2 shows the relationship between SCADA functions and CRM’s components. A multi-dimensional thinking is recommended when designing SCADA functionality to support control room management to fully achieve operational readiness.
Integrating SCADA and CRM Data Flows
By integrating systems, applications and data, both pipeline control operations processes and CRM business processes can be greatly enhanced. The following high-level business flow diagram is an example of how a CRM business process (alarm management use case) is tightly integrated with SCADA systems functions and how systems and data integration improve the abnormal operation conditions’ management.
In this example use case (alarm management), the benefits of the systems and data integration include:
- Improving situational awareness for the controllers and compliance team
- Enhancing the process of alarm management
- Automating the process of data logging, capture and audit trails required for adherence to CRM rule (analysis and documentation)
As part of the design process, the interaction and data flow requirement for all CRM business use cases and SCADA functions need to be assessed and discussed with all relevant stakeholders.
To achieve better integration between your business layer data flows (CRM use cases) and system layer (SCADA and control room applications), an architectural concept or framework can be especially useful as part of the SCADA operational readiness.
The architectural concepts can become a guide to identify applications, fundamental interactions, technical requirements, and organizational change management associated with implementing the new services.
Figure 4 illustrates an architectural concept where multiple applications in the control room are integrated. In this architecture, the SCADA platform is integrated with a third-party CRM application suite, and both are exchanging data with enterprise information delivery platform.
An emerging SCADA system architecture is shown in Figure 5. In this architecture, CRM-related applications are integrated part of the SCADA platform, and may consist of several software components:
- Electronic logbook (controller’s log, shift handover log, compliance log)
- Controller’s work volume logging and analysis
- Alarm anomaly logging and analysis
- Information linking (to system operating procedures repository)
- Point-to-point verification tool
Although both architectures can provide the guidelines and framework for integration of the business layer (CRM business use cases) with the system layer (SCADA platform), the emerging SCADA architectural has some fundamental benefits:
- Fewer integration across OT-IT systems
- Similar user Interface look and feel among users (controllers, supervisors, compliance team)
- Reduced operations, maintenance and support costs (one application versus two separate applications, one software support agreement, etc.)
- Lower implementation, training and change management costs
Alarm Management
PHMSA CRM rule requires an alarm management plan to be in place and being reviewed at least once each calendar year, but at intervals not exceeding 15 months. However, prior to CRM rule, both API and ANSI/ISA have both set the stage by publishing the following guidelines:
- API 1167 – Recommended practice for pipeline Alarm management
- ANSI/ISA 18.2 – Management of alarm systems for the process industries
Neither of these documents are included by reference (IBR) in PHMSA CRM rule, but they are consensus standards for design guidance. There are several steps involved in the alarm management and operational readiness including defining alarm philosophy, rationalization and detailed design. However, there is an emphasis on the safety-related alarms.
Putting It Together
The following steps are recommended during implementation of a pipeline SCADA to streamline the compliance with PHMSA CRM:
- Review your business processes and identify the area of improvements
- Define the functional components to address the diverse stakeholders’ needs
- Ensure compliance through integration of applicable codes, standards and recommended practices
- Establish a design discipline around CRM early in the project
- Integrate SCADA and CRM data flows to enhance business processes
- Develop test cases to validate stakeholders’ needs
The essential theme in the discussion of SCADA operational readiness and CRM design is the complexity of requirements to meet all stakeholder needs. A parallax view can be avoided by looking at SCADA and CRM from the viewpoint of the stakeholder needs.
While there is a commonly-held belief that PHMSA rules strictly govern CRM, the pipeline’s primary purpose needs to be considered to fully understand the relationship between the commercial value and purpose of the pipeline and the role that PHMSA rules play in safe operation.
Putting these requirements in perspective and including all stakeholders’ needs will shorten and improve the SCADA-CRM design phase and ensure compliance and operational requirements are met.
Author: Payam Yeganeh is a managing director in Black & Veatch Management Consulting LLC. and Michael Nushart in a principal consultant in Black & Veatch Management Consulting LLC
Comments