February 2011, Vol. 238 No. 2
Features
Deciphering 49 CFR 195.446 Control Room Management Regulations When Choosing A SCADA System
Choosing a SCADA system based only on the human-machine interface (HMI) and a few performance features of the remote terminal unit (RTU) could paint you into an expensive corner when attempting to comply with 49 CFR 195.446. Understanding those regulations and keeping their requirements in mind when evaluating and choosing a SCADA system could save time, money and a lot of headaches.
In developing the new regulations, the U.S. Pipeline and Hazardous Materials Safety Administration (PHMSA) has recognized that pipelines carrying hazardous liquids have unique characteristics as compared to other applications requiring SCADA systems. Regulation 49 CFR 195.446 addresses these by requiring specific improvements in control room management and change management. Compliance with the regulation also requires adopting American Petroleum Institute’s API 1165 standard, which documents best practices in displaying, monitoring and controlling information on pipeline SCADA systems, and API 1168, which documents best practices in control room management.
The closer one looks at these regulations, the clearer it becomes that meeting them cost effectively requires a much more tightly integrated SCADA solution than that which most pipeline operators now deploy. It will require 1) open architectures, 2) improved HMI and integrated workflow, 3) alarm management, 4) maintenance, 5) operating training simulators and 6) online portals. The paradigm of a traditional RTU-based SCADA system has now changed to an enterprise-server-based SCADA system.
49 CFR 195.446 Overview
In the language of PHMSA, pipeline operators are the companies that manage the pipeline and pipeline controllers are the people who interact most directly with the SCADA technology — configuring it, receiving data and acting on it. The combined regulations require attention to key functions of information and control room management targeted at operators and controllers as described in the following sections: 1) General control room management, 2) Roles and responsibilities, 3) Adequate information, 4) Fatigue mitigation, 5) Alarm management, 6) Change management, 7) Operator Experience, 8) Training, 9) Compliance validation, and 10) Compliance and deviations.
The Integration Imperative
A truly integrated SCADA system must handle the complete set of actions within the pipeline environment and be easily and affordably maintainable. Maintainability is the ability to handle changes and associated cascading functions. The way to make the system easy and affordable is to integrate all applications within an open, object- oriented development and run-time environment. The environment illustrated in Figure 1, for example, is based on an application engine that drives tag management across multiple applications. Users access data through a common HMI that can be proliferated throughout the enterprise and can share data according to established business process management rules. Alarm banners, faceplates, specialized control tools and customized application objects are embedded directly into the system, which improves control room productivity significantly.
Minimally, an integrated, server-based SCADA environment should support communications among all the related applications. Meeting 49 CFR 195 requirements would be most effective, however, if all applications shared the same database and used the same tag name definitions as the base SCADA system. Compliance would become even easier to attain if the extensions to that tag name — such as historization, alarm set points, scripts, security, web links and more — could carry across all the applications in an object-oriented environment. This is only possible in a system in which one developer provides all applications and has architected their technology with this in mind or in a system that is built on an open, object-oriented integration platform, which would enable users to mix and match best-in-class applications.
Everything derives from the base SCADA system. In a truly integrated environment, changing one attribute in the SCADA system object should deploy to all related objects, regardless of the application module. If you change the SCADA tag for a valve in the primary HMI, the changed tag and its attributes should propagate instantly to any workflow application, two-way training simulation environment, remote field operator device applications, lessons-learned section of a web portal, or any other application that might incorporate that tag. Figure 1 shows how changing an alarm setpoint cascades to the following applications shown in the diagram: 1) The associated workflow for the valve will be modified based on the new setpoint, 2) The field operator notification to act on the alarm for that valve, 3) The operator response to the alarm during a training exercise, 4) The lessons learned about how the alarm was handled and the cause/effect, and 5) The shift manager’s notes based on the new setpoint.
Figure 1 shows that the Application Object contains attributes of all 49 CFR applications. The VALV_001 object shown as the nucleus in the left section of the diagram contains all associated attributes. The right half of the diagram shows how the VALV_001 object is applied to key SCADA system components. A change in the alarm and event attribute, for example, would be deployed immediately to the workflow, alarm management and all other relevant application components.(end of caption for Fig. 1)
Unless integrated SCADA systems work this way, maintaining the relationships between all applications at the level necessary for affordable compliance with the new regulations will be all but impossible. The SCADA system object structure must be built on multiple compliance-driven attributes and applications that flow from the application engine. A typical object might, for example, represent a tag like “pump001_pressure,” which describes the pressure transmitter on pump 001. The pump object, then, contains a variety of elements that would be reproduced for every pump and in some cases reused for other tags. Each time the attribute for tag pump001_press is updated, all related modules update automatically. A tagging strategy of this sort is significantly easier to maintain than a relational database. Figure 1 also represents examples of object attributes and how they might be “pushed and deployed” to SCADA modules that support one or another of the 49 CFR requirements.
Integrating Your SCADA System To Comply With 49 CFR Sections
Roles and Responsibilities. Section (b) of 49 CFR 195.446 requires pipeline managers to clearly define roles and responsibilities for controllers during normal, abnormal and emergency operating conditions. Workflow, remote operator, shift handover portals and simulation modules of integrated SCADA systems work together to meet this requirement and help train operators to respond to various condition.
The workflow modules provide operators and field technicians with automated procedures to follow during various conditions, cascading directly off of system tags. Shift handover portals provide an environment in which controllers can log activity and define responsibilities for the next shift. Simulation and training modules not only help operators learn and perform their role in pipeline operations but also help them to understand the roles and responsibilities of downstream actors so they can respond in context of subsequent actions. Integrated SCADA systems drive all such modules from a single engine.
Adequate information. Section (c) of the regulation requires operators to provide controllers with the information, tools, processes and procedures necessary for them to carry out defined roles and responsibilities. Integrated SCADA systems support this requirement by providing workflow and remote operator applications, training simulations and all base SCADA functionality through a common HMI, which can be displayed across company Intranets or web portals.
One component of the information that section (c) requires is point-to-point verification for new equipment. Integrated SCADA systems can meet this by providing utilities to keep track of all I/O points by uploading new or updated I/O tables from the processing engine. Section (c) also calls for testing and verification of communication plans, which integrated training systems support by enabling instructor led simulations. It is even possible to create highly iterative programmable logic controller (PLC) operating simulations, in which SCADA operators interact with logic that the simulation model receives from the PLC based on a trainee’s actions.
Section (c) also requires testing back-up systems, which integrated SCADA systems provide by maintaining a centralized location at which all system components can be recorded and linked to the engine. Capturing and providing easy access to “lessons learned” is also part of this section of the regulation, and the integrated, open system supports that by enabling entry of lessons learned and associating them with specific equipment and processes through tags.
Section (c) is where adherence to API 1168 Section 5 comes into the picture. This requires a shift handover solution and an efficient method for exchanging information that can be met by aligning a web portal to the application engine. This allows key stakeholders or casual users to access the activities and information. API 1168 section 5.3.3 also calls for structured management of daily operation information, status of scheduled or unscheduled maintenance activities, incident Information; changes to physical assets, procedures and or responsibilities; alarm reviews and third-party incidents potentially impacting operations. All of these represent content that is better stored and retrieved via an object-oriented tagging structure rather than a relational database.
Alarm Management. Section (e) of 49 CFR 195.446 prescribes a very specific alarm-management process. Operators must write an alarm-management plan, monitor various alarms, match alarm descriptions and setpoint values after changes, monitor the volume of activity for the controller, review alarm-management plans and address deficiencies. A vendor of an integrated SCADA system should provide alarm-management applications that support a very rich analytical environment and provide a consultative organization to support initial alarm rationalization. The following services are essential to extracting the maximum alarm-management performance from an integrated SCADA system: 1) Alarm Management Assessment, including performance assessment; 2) Alarm Management Design, including a philosophy review, functional design specification, rationalization, and HMI design specification; 3) Alarm Management Implementation, including rationalization implementation and final alarm performance assessment; and 4) Alarm Management Optimization, including alarm system audits, chattering alarm reduction, and additional rationalization and implementation.
Modern, integrated SCADA systems can enhance alarm management further with rich, intelligent graphical interfaces that record and analyze procedures in a classic Activity/Decision style template. This leads to a very comprehensive view of how an alarm is handled throughout the organization. To ensure safety, alarm management should not stop at traditional alarm acknowledgement by the controller in the control center. Section (e)1 of the code states “review SCADA safety-related alarm operations using a process that ensure alarms are accurate and support safe pipeline operations.” The SCADA system should do more for the controller than simply acknowledge alarms. It should also show the controller the downstream activities of all stakeholders.
In an alarm flood, for example, automated workflow is really the only way the controller can handle the dozens of required actions adequately. Support in alarm situations can also be extended further by extracting lessons learned about what trips an alarm, documenting them, and perhaps making them accessible on an intranet portal, all of which would contribute to compliance within section (e)6 of the regulation, which calls for actions to address operating deficiencies that diligent alarm management might reveal.
Figure 2: Workflow Tool used to meet Section 49 CFR 195.446 section (e)1 Alarm Management.. Modern, integrated SCADA systems can enhance alarm management further with rich, intelligent graphical interfaces that record and analyze procedures in a classic Activity/Decision style template.
Integrated SCADA systems also make it easier to define and enforce company-specific alarm-management operations and policies, and some suppliers may bundle SCADA integration and alarm management consulting services.
Change Management. Section (f) of the new regulation requires that operators assure coordination of all changes that could affect control room operations with control room personnel, as well as field personnel. To address this, an integrated SCADA system must include modules for workflow, remote operations, training, alarm management and leak detection. To automate change management fully, the SCADA system should also incorporate a field communications module that enables field technicians to communicate information back to the control room.
Section (f) also discusses adherence to Section 7 of API 1168, which focuses on change management. An integrated workflow module supports change management by assuring that updates to SCADA tags proliferate across all similar and related processes. Embedded automation in a workflow module makes this especially relevant to the management of change (MOC) practices that API 1168 promotes. Other API 1168 change management recommendations, which an open, tag-driven integrated SCADA system supports, include 1) field maintenance, which benefits directly from workflow modules; 2) notification and training, which assures ability to implement changes; and 3) emergency management of change, which requires immediate implementation and commissioning of changes, along with management and tracking of leaks and other equipment failures.
Operating experience. Section (g) of 49 CFR requires operators to incorporate lessons learned from operating experiences into their control room management procedures. Workflow and training simulations both play roles here. Past experience certainly shapes workflow definitions, which might be structured and made available in a common arena to which all stakeholders can contribute lessons learned or post documents supporting various scenarios. Simulated training systems might come into play in instructor-created scenarios, which can be updated regularly to train controllers on how to react to an event as it progresses through the entire organization.
Training. Section (h) requires operators to establish programs that train controller to carry out their defined roles and responsibilities. The training program should also provide the operator with working knowledge of the system, especially operations during emergency situations. An integrated SCADA system should include a training module, and to meet the requirements, the training module should be expanded well beyond the traditional walk-through of basic SCADA system functionality to incorporate the workflow module, field operator module, leak detection system and alarm management modules.
The section requires a working knowledge of the pipeline and automated workflows. By incorporating workflows that extend to field personnel, emergency responders and relevant managers, the pipeline operator meets Section (h)3, which calls for clear definitions of “responsibilities for communication under emergency response procedures” and Section (e)4, which calls for provision of “a working knowledge of the pipeline system.” Section (e)5 dictates training on “operating setups, … for controllers to review relevant procedures in advance of their application.” Using the training system to teach about new system logic, about the characteristics of a new pipe section or about the addition of new equipment is implied clearly in Section (h)5. But what might not be so apparent is the possibility of using the training system to test the efficacy of a new workflow that is under consideration. This can be used on- or off-line to work out the kinks of the entire process. Training simulations also provide an additional measure of compliance for Section (f) Change management, and section 7 of API RP 1168, which complement each other in coordinating changes that affect operations with the control room representatives, management and associated field personnel.
The same can be said for tying the leak-detection program to the primary SCADA database because they pull values and alarms directly from the primary SCADA system. Applying an integrated system here may be one of the most critical areas for improved database management in the entire regulation, since the controller’s ability to react to the real SCADA system as it changes affects safety directly. An integrated simulation and training system also provides a great area to test out any changes to the SCADA system before they are put into operation. Trying to change all the attributes of existing tags and adding new tags to the training/simulation system without a simple way to deploy those changes would be daunting. Moreover, this minimizes the risk of incorrectly matching the system changes, as well as reducing the time and the cost of implementation.
Compliance Validation. Section (i) of the code requires operators to submit their procedures to PHMSA or state agencies. The Integrated SCADA system should be able to document workflows, databases, alarm analyses, field procedures, maintenance activities, back-up procedures and point to point verifications as a few examples in the regulation.
Compliance and Deviations. Section (j) requires operators to document all compliance and deviation corrections. Compliance here requires a workflow module, a training module, an alarm management module and a web portal module. The training module should provide a method for grading controllers over time and a workflow module should define actions that have been automated and how those procedures are documented. The training module should provide an environment for the controllers to continue to receive training as the system is modified. The web portal should deliver a lessons learned environment tied to each tag name in the database. If regulators see a SCADA system integrated on a single database and sharing a common namespace within an object-oriented environment, it will be instantly clear that changing fundamental attributes in the SCADA system will be deployed to all related applications. Without such a tightly integrated SCADA system, convincing regulators that the logical connection between the applications and the attributes match the cascaded applications correctly will be difficult. It will also be more difficult to convince them that data exchange between disparate vendor applications integrates consistently as the SCADA system undergoes regular tweaks.
Conclusion
If pipeline operators, their management team and their evaluation team look only at the base SCADA system functionality, they will put themselves in jeopardy when trying to meet 49 CFR 195.446. As they struggle to achieve compliance, they might indeed be able to connect the various modules necessary to provide the information and control room management functionality they need, but they will find themselves facing a complex jumble of database relationships that will be all but impossible to maintain. Operators always have the choice to build a SCADA system by integrating modules from various vendors and that might be an appealing choice. But systems changes complicate maintenance, which inevitably leads to finger-pointing and conflicts between vendors and the operator.
The fundamental imperative of maintainability and basic functionality of the modules should not be compromised because the lifespan of a SCADA system is typically many years. If excellence in pipeline safety and compliance with 49 CFR 195.446 are goals, a tightly integrated SCADA system in which all applications have been developed to operate on a common and an object-oriented platform from one vendor is the best way to achieve them. The paradigm for SCADA has shifted from a standalone unit to a true SCADA enterprise server-based system that participates in the production and safety of all pipeline operations. Pipeline operators must look at an enterprise SCADA system as the platform from which to serve the applications required to meet 49 CFR 195.446. Contact Ph: 469-365-6400, http://iom.invensys.com/49CFR.
Comments