Developing New Generation of Process Control Systems - CiteSeerX

IEEE Real-Time Embedded System Workshop, Dec. 3, 2001

1

Developing New Generation of Process Control Systems Deji Chen and Aloysius K. Mok

Abstract—The process control industry is bracing for the third generation control systems. Influenced by the latest development in this Internet era, many new paradigm shifts occur in control system development. In this paper, we describe the evolution of process control systems over the years and list challenges ahead. We share some of our thoughts drawn from both the experiences in participating in the development of these third generation control systems and the observation of this industry as a whole. We believe the everincreasing time-to-market pressure not only changes the world of conventional non-real-time system development, but also has profound impact on process control system development. The time pressure inevitably leads to software/hardware reuse, which may not yet be well object/component-oriented. We shall talk about the challenges of component reuse in this paper. While real-time system development is considered mostly different from non-real-time system development, we believe they are not as dissimilar as we thought. The process control system developers constantly adapt the latest idea from its counterpart in the non real-time field. In the mean time, challenges still lie ahead to bridge the gap between real-time research and the process control system development. Index Terms — process control system, research and industry experience, software/hardware reuse, object/componentoriented,

1

INTRODUCTION

I

ndustrial process control systems have evolved into the third generation. The first generation was simple loop control system. In the 1970s, Honeywell [10] led the second generation in offering distributed control systems. Since the 1990s, the field centric system, in which the intelligent field devices perform more than just sensing and actuation, has become the third generation. In the third generation, many of the controls are truly distributed in the field. Currently the process control industry is dominated by the secondgeneration systems, with third generation systems overtaking the lead. New generation poses new challenges. One difference this time, however, is that it comes with the Internet revolution. In this age, a new product used to take years must be finished in months; the product cycle is shrinking Deji Chen is with Fisher-Rosemount Systems, Inc., 8627 Mopac North, Austin, TX 78759. E-mail: [email protected]. Aloysius K. Mok is with the Department of Computer Sciences, University of Texas at Austin, Austin, TX 78712. E-mail: [email protected]

so rapidly that people are already working on the next release before the current one is really “released”. This phenomenon is particularly true in the computer industry. This shortened product cycle is not because the system developers want to, but have to due to the market pressure. In many cases projects that were not in the development plan are suddenly brought back in and must be included in the next release due to customer requirements. The other industries feel the same pressure. In more than two decades, the second generation control system stays relatively unchanged. Since the introduction of third generation systems, we have seen on average one major release a year, each with significant increase in functionality and scale. A process control system controls pharmaceutical plants, oil refineries, etc. System safety is of utmost concern. The system must pass numerous safety standards. It is not surprising that a second-generation system took years to perfect. Now in this Internet age, the third generation system development is not immune to the time-to-market pressures. To develop reliable control systems in a shorter period of time, new thoughts and new approaches must be found. In this paper, challenges to find the solution are analyzed. We shall describe the evolution of process control systems and the current challenges we face in the next section. In Section 3 we discuss the effect of the time-tomarket pressure on product development and new approaches taken. In the section afterwards, we shall talk about real-time analysis and the gap between research and practice. The paper is concluded in section 5. 2

EVOLUTION OF THE PROCESS CONTROL SYSTEMS

Figure 1 is a typical closed process control loop. The flow meter measures the fluid flow rate in the pipe. The information is sent to the controller, which in turn calculates the desired valve position to produce required flow rate. The controller sends the calculated value to the actuator, which re-positions the valve. In the first generation control systems, The controller is mechanical. The operator sets the desired flow rate by turning the knobs on the control panel. And the signal is sent from and to the controller pneumatically. Not much could be done besides setting the flow rate. Due to the physical limitation, the controller usually is installed in the field close to the devices.

IEEE Real-Time Embedded System Workshop, Dec. 3, 2001

Figure 1. A Simple Loop Control System In the 1970’s, Honeywell [10] introduced TDC 2000 system and started the second generation of control systems. In this second generation, the controller is replaced with an embedded digital computer that runs desired control in software. Analog electrical signals are passed between the controller and the field devices. One physical line conveys one signal. TDC 2000 started a trend of truly distributed system in which the operator remotely controls the plant while sitting in the off the field control room. The operator works on a computer console connected to the controller. For the past decades, the second-generation system dominated the field. It was continuously improved and perfected. In the 1990’s, a third generation system began to emerge. See Figure 2. In this new generation, devices are empowered with computers. The smart device digitally communicates with the controller through field buses. One physical line conveys multiple control signals plus diagnostic/maintenance information. Personal computers replace expensive mini-computers. Standards are created to provide customers choices. Field bus standards include Foundation Fieldbus [10], Profibus [11], DeviceNet [12], etc. OPC [13] was created to pass control information over computer networks.

2

Figure 2. A Field Centric Control System Figure 3 shows the disruptive improvements of three generations. Each line represents a generation. The rate of return over investment has greatly accelerated since the third generation. With the ever-increasing power of computer processing power, many advanced features are introduced as an integrated part of the whole system, such as automatic control loop tuning, predictive maintenance, fuzzy logic control, etc. We are in the early stage of this third generation, which means new challenges come up daily. Putting every new feature together is itself a big challenge.

IEEE Real-Time Embedded System Workshop, Dec. 3, 2001

3

Figure 3. Three Generations of Process Control Systems In the PlantWeb [15] architecture, the whole process plant is connected together not only horizontally but also vertically. As is shown in Figure 4, the process control is connected within the whole plant, which is connected with the corporate management layer, which in turn is connected to its suppliers and customers. In the future, a customer order will automatically generate raw material purchase order and the plant will automatically reconfigure the process to fill the customer order. In the second-generation systems, only the lower two layers in Figure 4 are connected.

Figure 4. PlantWeb 3

DEVELOPING PROCESS CONTROL SYSTEM WITH INTERNET SPEED

The short product cycle often conflicts with building a project from scratch, which takes substantial amount of time. A process control system requires extra work for safety purpose compared with conventional computer system. Often, some kind of component reuse is employed, both in hardware and software. Unfortunately, even the most recent technology in non-real-time system does not fully realize software reuse. We shall discuss this in Subsection 3.2, but first we shall look at the practical means of hardware/software reuse in Subsection 3.1. One bright side of fast developing pace is that the processors are getting faster and the memory chips are getting denser quickly. This brings significant impact in the embedded system design. A few years ago, 64K bytes memory is a luxury, now several megabytes can be put in an MP player. Except for really tiny simple embedded systems, processor and memory limit becomes a relative term. The limit today could easily become a non-issue in less than a year. This is especially true for process control systems. If you look at Figure 2 and figure 4, the sizes of the controller and the smart devices are relatively unchanged overtime, largely because the physical plant they control does not

IEEE Real-Time Embedded System Workshop, Dec. 3, 2001

4

shrink in size. So the net result is we could package more and more functionality into the controller and the devices. A typical hypothesized example goes like this. Even though one megabyte of embedded software was not even heard of a while ago, now a product requires about two megabytes memory for the embedded system. In the beginning the hardware design team struggles to put two 1megabyte memory chips on the circuit board. Before the board design is finalized, they find a new memory chip of the same size with 8 megabytes. The single new chip was put in the design. Now the software guys can fit in not only required functionality, but also additional enhancement. Short product cycle makes it possible that latest hardware and software can be put in the product. Short product cycle also means that the hardware and software used to build process control system are also upgraded and enhanced fast. This in many ways gets rid of many hassles in the early ages of embedded world, where developers have to fight the limit of the processor speed and the memory space, and by the time the produce is released, the components are already out dated. With long product cycle, the same system to be completed at the same time must be started much earlier, when even required hardware did not exist, or existed but too expensive. In the old days, this limit ironically also makes embedded development more like an art and the developers proud of themselves. The processor was so slow then that the interrupt service routine (ISR) contains a loop to serve all interrupts that come along to save ISR context switch. In a faster processor, this loop becomes unnecessary and can be removed from the code. Another example of the “art” is a trick to read elements of a C language array A[i] of size N and type T. The ith element’s address is calculated by A[0]+i´ size_of(T), which translates into lots of assembly code. The trick is to declare a function GetA(i): T GetA(int i) { switch (i) { case 0: return A[0]; case 1: return A[1]; . . . case N-1: return A[N-1]; } } GetA(i) reduces the run time instruction set to locate the array element to speed up the software. A faster processor will save the entire hassle to define GetA(i). Of course, in many cases including some of the process control systems, the rapid development in processor speed and memory density do not lead to increased functionality, rather, it drives the miniaturization of systems with unchanged functionality. We shall not focus on this aspect. While additional code is used to speed up the execution, with faster processors additional code sometimes has to be

used to communicate with slower peripherals. A temperature sensor needs time to get the accurate measuring. If a faster processor is used in the smart temperature transmitter device, the software has to be adjusted to accommodate the sensor. 3.1 Hardware/Software Reuse Hardware/software reuse used to be an option, for which a product is built from ground up if no existing usable parts are available or readily suitable. Given the time pressure, hardware/software reuse is now more and more prevalent. The question becomes how to efficiently manage it. While it is impossible to develop everything from the ground up, many existing hardware/software products can be adapted for new projects. For example, there are countless TCP/IP stack packages developed for the ISO network communication standard, when a project asks for a communication port, writing a new stack does not seem worth the effort. The proponent of software reuse foresees a future where software component are built and traded like brick and mortar parts [4]. For the time being, we have to settle on more primitive forms of reuse, let alone in the embedded world. Hardware and software/firmware together completes a product. The trend is that more and more load is put on the software side whereas hardware is built of standard parts. This again makes it possible to build new product based on previous ones. As an example, a hardware team might modify some existing hardware with one type of functionality to create a totally different piece of hardware with a different set of functionality for the software team to use, several months before the first new prototype hardware becomes reality. This gives software team an early head start. Software reuse includes internal and external software reuse. It can be source code level reuse, or binary code (library, component) reuse. The advantage of internal software reuse is that the source code is available and the expertise about the code is handy. The advantage of external software is that it is well packaged and documented, but a serious bug in the package renders you at the mercy of the vendor, whose priority differs from yours. For important projects, purchasing extra right to the source code is also an option, especially when the package needs to be dramatically enhanced to fit into your own final product. In reality, most of the projects reuse source code, and almost all projects have some sort of reuse. To distinguish from the term component reuse, we call this code “part” reuse. Challenges in code part reuse are: · Code Part Comprehension. It is impossible to fully comprehend the source code in the beginning, sometimes not even after the new product is released. It is not uncommon to reuse hundreds of source files in millions lines of source code. To handle this situation, skills are required to quickly locate the important places to work on. The challenge is how to

IEEE Real-Time Embedded System Workshop, Dec. 3, 2001

·

·

get the work done while not to require every developer to fully understand every detail of the reused source code part. Platform Change. Code part reuse entails recompilation. Often it is done with a different compiler onto a different target, which includes the processor and the operating system (OS). None of them are easy. For example, a prominent real-time operating system (RTOS) does not support waiting on multiple semaphores. The original RTOS supports it and it is used in the reused code part. To reuse the code in the new project, a fake kernel method has to be invented by using existing single semaphore waiting kernel methods. Different compilers also make differences. Data communicated between hardware parts are normally packed. In one case a code part was developed with a compiler that have a switch to pack all structures in a file. So all structures in the code part are defined without the “packed” keyword. When it is reused in another project, the new compiler no longer has this switch. The developer had to manually search for and pack structures individually. Here is an example of problems with regard to different processors. ++ operator in one processor is atomic. The code part developed does not protect the counter increment/decrement that is use to manage the queues, relying on the atomicity of the ++ operator. When this code part is reused on a new processor that does not preserve this property, this subtlety could easily be missed. Extra code needs to be added to make the counter safe. Often the reused code part is compiled with a different RTOS; the compiler and debugger are different; and the processor is hundreds of times faster. Putting Parts Together. Putting debugged parts together does not mean the whole will work. In an ideal component world, care still has to be taken. It is more true for this lower level of reuse. In one example, the stack was tested to work when its interrupts are the main concern in the system. Once it is put in the whole system, it has to compete with other important system interrupts. The developers have to modify many places in the ISRs to reduce the blocking time. When two software parts in previous products are combined to build a new product, a new application programming interface (API) should be defined and implemented so two parts can call into each other. The API could be adapted from one of the original ones, or a new one in between. In one example, one code part only supports asynchronous access in its API. A new middle layer of API was created to accept synchronous access from the other part, call the asynchronous method, block the current task, and continue when the asynchronous reply comes back.

5 ·

New Feature Implementation. A new product has additional features in addition to those supported in the reused parts. Often new features are intertwined with the old parts. This is more challenging when the reused source code is huge. Normally a process control bus has different types of devices, like master device that configures the bus, and slave device that responds to the master. Reusing the code part implemented for a slave device with inferior capability in the master device with superior capability is a good example of adding new features. Another example would be adding redundancy support in the code part developed for simplex application. · Code Part Maintenance. The maintenance of the master copy is a big issue. Except for binary reuse, the source code will be changed somehow for the new project. Some times the original source code evolves at the same time. There can be a new release of the source code part, or a bug discovered in the new system and conveyed back to the original code part developer. Once a new copy of the source code arrives, the developer has to merge it into the new product, saving the new code from the code part and new implementation for the new product. Reconciliation should be resolved if both are at the same place in the code. Sometimes there is the need to maintain a single copy of source code with difference chucks of code compiled out for different products. Another case of code part reused is functionality reduction. A full-fledged system could be stripped into a small tool. The tool keeps the original system backbone, combines several features into one, reduces hardware requirement, and can be distributed independently as an assistant application for the original system. None of the above challenges are easy at the source code level. We would like better ways of hardware/software reuse. We like readily reusable components in the process control systems. In the next section, we shall argue that component reuse, the Holy Grail of non-real-time software development, should also be the goal in process control system development. 3.2

Object/Component-Oriented Paradigm in Process Control System Based on our observation that processor speed and memory size may not be a constraint anymore, we suggest that it is beneficial to bring object/component-oriented (OO) ideas into real-time system development. People have already been preaching this idea for a while [4, 8]. We shall give in this section some practical reasons. First of all OO design methodology has been used in process control system development for years. The advantage of it for non-real-time system development equally applies to the real-time system development. Consider a simple state machine with 2 states S1 and S2.

IEEE Real-Time Embedded System Workshop, Dec. 3, 2001

6

There are 3 events e1, e2, and e3. There are also 4 types of condition t1, t2, t3, and t4. Figure 5 shows the state machine.

procedure CSM::e2_on_S1() is executed and the state is changed to S2. With C language CSM member methods have to be defined as global functions. CSM class nicely encapsulates the state machine's state and behavior. Note state machine is often encountered in the embedded world. In fact, most of the reasons why OO methodology is preferred apply to the embedded world as well. Another example is the implementation of lock() to protect critical sections. Critical section protection is very important in embedded systems. In practice it is easy to forget to call unlock() after lock(). Using the following mylock class, we no longer need to call unlock(). We could define a local variable of mylock and call its lockup() member function. Unlock will be called automatically in the class destructer when the local variable is out of scope. Class mylock { critical_section *p; mylock() { p = null; } /* constructor */ ~mylock() { if (p != null) unlock(p); } /* destructor */ lockup(critical_section *q) { if (q != null) lock(p=q);} } Second, the disadvantage of OO methodology to the embedded world are no longer that obvious. The extra dereference in class member/data access can be ignored with faster processors. Since predictability instead of speed is the correctness criteria in real-time systems, the slight slowness introduced by classes shouldn't be the hindrance for adopting OO. In one occasion, one code part was written in C while the other was written in C++. Because of function name mangling in C++, the interface between these two do not work directly. This can be resolved with a middle layer of C declaration in the C++ part. Again, this only introduces a few extra de-references. Last, OO does not conflict with real-time analysis and design. Priority assignment, critical section protection, interrupt service, etc. all work well with classes. A class may be associated with a thread, hence affiliated with a priority; a critical section is logically consistent to be put in a class. Another point could be made that, as we shall explain in the next section, in many case real-time constraints are enforced by checking conditions after constraints are violated and then fixing the code to avoid such condition to occur in the future. This post priori approach diminishes, at the early stages of design and coding phase, difference between using OO and non-OO methods for real-time related concerns. Towards software/component reuse is also a necessity. As more and more products are built by reusing, ways must be found to facilitate this. We should be able to use third party components without having to buy the source code. The future in the non-real-time development suggests the future of real-time development.

Figure 5. A State Machine Example When an event occurs, a transition is taken if certain condition satisfies. The state behavior can be implemented in a C++ state machine class CSM as illustrated in the following table: FiniteStateMachine finiteStateMachine[2][3] = { {/* S1 state */ /*event trigger nextState transition */ {e1, CSM::t1, S1, CSM::e1_on_S1}, {e2, CSM::t2, S2, CSM::e2_on_S1}, {e3, CSM::t1, S1, CSM::e3_on_S1}, } {/* S2 state */ /*event trigger nextState transition */ {e1, CSM::t3, S2, CSM::e1_on_S2}, {e2, CSM::t2, S2, CSM::e2_on_S2}, {e3, CSM::t4, S1, CSM::e3_on_S2}, } } If in state S1 and event e2 arrives, the state machine will check condition CSM::t2(). If the condition is true,

IEEE Real-Time Embedded System Workshop, Dec. 3, 2001

7

While pointers are very powerful and flexible, they are also often the cause of bugs. A story goes like this. People spent 10 days to find and fix a bug that only occurs in release build, not in debug build. When a buffer is full, the insert request fails and a negative confirmation is generated. However, the buggy code still passes the buffer pointer in the confirmation function, which happens to point to the memory space beyond the buffer. The buggy confirmation function faithfully frees the buffer by resetting the value the bogus pointer points to. Any weird result can thus happen. This kind of story is common in conventional software, but could well happen in a real-time system. OO proponents preach that pointers should be got rid of from a programming language. In conclusion, we believe there is no barrier to bring object/component-oriented technology to real-time development. Real-time industry, including OS venders and application vendors should strive to componentize their products. Borrowing the phrases in [5], in the future there will be components with added real-time attributes and there will be real-time component vendors, system builders that use these components, vendors that offer tools to support component reuse, brokers that market the components, and standard organizations setting the rules.

end of the development. In most cases, this crash dump support is left in the final product. Deadlock is sometimes also handled afterwards. In the beginning the system is designed to avoid any deadlocks. But if a deadlock does occur, a watchdog timer will be triggered. The watchdog timer will not reach the limit if it is constantly reset. So if tasks resetting the watchdog timer are deadlocked, we know it by getting this timer interruption. In some cases a single lock is used for multiple resources. This is brute-force but simple and reduces deadlock risks. One important task of a real-time system is ISR handling. Interrupt could not be blocked for too long. This requires careful coding. A good programmer only places minimum execution code in areas where interrupts are disabled. The ISR itself should also not take too long. In one project, two reused code parts A and B are put together, both have ISRs to communicate to the external hardware. The ISR running time is measured after the code is completed. It turns out the longest ISR time to serve A is bigger than the maximum allowable delay of B’s ISR. This is found from a system crash. The solution: that long ISR of A needed to be broken into two separate parts.

4

RESEARCH AND PRACTICE

In this section, we look at real-time issues in process control system development. We believe there is a gap to be bridged between research and practice. First, we shall look at how the issues are handled in practice (Section 4.1). Then we suggest ways to bring research results into practical use (Section 4.2). 4.1 The Practice In commercial software product development, testing still plays a pivotal role. Microsoft even delegates the test job to its customers by releasing countless service packs and hot fixes. A process control system is even more at stake. A failed system not only incurs financial lost, but may also cause disasters. It is not surprising to see the added effort in process controls system testing. A small design change or bug fix is often followed by a full system level test. Like non-real-time systems development, big effort is also put on post failure analysis. This is largely due to lack of tools that is capable of proving the system correctness. A real-time violation check is easy to define and implement; it is far more difficult to write a system that can be proven to meet real-time constraints. It is no wonder that in many practical situations complex system correctness proof is absent. During development of a typical system, special code is written to recover (often reboot) after a real-time violation is detected. The environment before the crash, called crash dump, is preserved in non-volatile memory. Once this occurs, the crash dump is analyzed and the reason is found. The fix is then put into the code. The cycle repeats until the

4.2 Bridging the Gap No doubt there is always gaps between research and practice. The current status of applying real-time research results to the real-time industry is still encouraging. Ratemonotonic scheduling, priority inversion handling, timetriggered protocol, etc. all find their way to commercial RTOSs. There are, however, many fine grain research results stays in the academia. If we do not consider their small footprint and fast response time, most commercial RTOSs only have more priority levels than their non-realtime counterparts. Some of them may have more subtleties in semaphore/mutex/event support; some of them may have implemented priority inversion prevention protocols. There are, however, countless research results in real-time academia that do not see their trace in the commercial world. There are three different cases. In the first case existing research results are not reflected in commercial RTOSs due to different reasons. For example, we have not seen a RTOS with built in automatic deadlock detection. In the second case, real-time research is still not mature enough to be commercialized, like program correctness proof, real-time logic, etc. Finally, there is just no research done yet on certain problems. Vertical integration of the third generation process control system with corporation management and business chains is still at its infant stage. We conclude this subsection with two examples. The famous Liu & Layland utilization bound [7] for fixed priority scheduling claims a system is schedulable if the total utilization is less than ln2. In practice, a real-time system remains utilized at a low rate. A process' free time could generally be above 50. At this low rate, The system is guaranteed schedulable. It is fruitless to look at so many fine-grained schedulability test techniques from the realtime research field.

IEEE Real-Time Embedded System Workshop, Dec. 3, 2001

8

Task priority assignment is one of the most important supports from RTOSs that are heavily used by real-time applications. Some RTOS support up to 256 levels of task priorities. Not only are these levels never used up, but also the research has already shown that too many priority levels degrade system performance. The number 256 could be the convenient result of defining priority levels in a byte. We believe lots of work could be done to apply research results. We could sophisticate RTOSs with deadlock detection/prevention support. For a large system, even a little bit real-time correctness proof is worthwhile.

reviews to this paper. The materials for this paper are the result of the collaborations with them in the past.

5

CONCLUSION

Market pressure brought a dramatic change in the process control industry. Currently in the third generation, the process control systems are developed in rapid speed, utilizing latest processors, memory chips, and other hardware, adapting latest software development technologies. In this paper, we presented our experiences and observations. We described the status of process control system development. We discussed the hardware/software reuse issues. We looked at miscellaneous challenges facing the process control system development in source code reuse. We argued that the latest technology in non-real-time software development could be applied to real-time software development also partly because processor and memory limitations are relaxed. We also looked at the real-time analysis in practice and believe there are lots to be done to bridge the gap between real-time research and practice. 6

ACKNOWLEDGEMENT

Deji Chen would like to thank his colleagues, who want to remain anonymous, for their invaluable contributions and

7

REFERENCES

[1] D. Chen, Real-Time Data Management in the Distributed Environment, PhD Thesis, The University of Texas at Austin, 1999 [2] D. Chen and A. K. Mok, SRDE -- Application of Data Similarity to Process Control, IEEE Real-Time Systems Symposium, December 1999 [3] D. Chen, A.K. Mok, and M. Nixon, Providing Real-time Support through Component Object Model, Microprocessors and Microsystems, 23(3), pages 145-154, October 1999 [4] B. P. Douglass, Real-Time UML - Developing Efficient Objects for Embedded Systems, Addison-Wesley Publishing Company, 1998 [5] I. Jacobson and M. L. Griss, Approaching the Promised Land of Component Reuse, Application Development Trends, pages 27-39, June 2001 [6] J. P. Lehoczky and L. Sha, and Y. Ding, The Rate Monotonic Scheduling Algorithm - Exact Characterization and Average Case Behavior, IEEE Real-Time Systems Symposium, December 1989 [7] C. L. Liu and J. W. Layland, Scheduling Algorithms for Multiprogramming in a Hard-Real-Time Environment, Journal of ACM, 20(1), January 1973 [8] D. Locke, Object-Oriented Techniques for Real-Time Systems, Real-Time Technology and Applications Symposium, June 1997 [9] L. Sha, R. Rajkumar, and J. P. Lehoczky, Priority Inheritance Protocols: an Approach to Real-Time Synchronization, IEEE Transactions on Computers, 39, 1990 [10] Foundation Fieldbus, http://www.foundationfieldbus.org [11] Profibus Standard, http://www.profibus.org [12] DeviceNet, http://www.odva.org/ [13] OPC- Ole for Process Control, http://www.opcfoundation.org [14] Honeywell Industrial Control, http://www.honeywell.com [15] PlantWeb Architecture, Emerson Process Group, http://www.plantweb.com [16] Siemens Energy & Automation, http://www. sea.siemens.com