Design and implementation of a standard framework for KSTAR control system Woongryol Leea, Sangwon Yuna, Tagu Leea, Jinsup Parka, Sangil Leea , Ali Kashif Bashira, Kaprae Parka and Mikyung Parkb a
National Fusion Research Institute (NFRI), Republic of Korea b ITER Organization, France
Standardization is an important issue in KSTAR which is organized with various heterogeneous systems. By 2012, many control systems adopt a new application frame in KSTAR. This application was launched for easy development a data acquisition system initially but now it is extended to as a base framework of a machine controller and real time controller in KSTAR. This EPICS based framework is provided single library and database, template, and descriptor files. Distributed library has several threads, data structure and standard user functions. The framework based controller has non-blocking control command method with thread. The sequence handler in framework makes a system can be synchronized operation in KSTAR. It supports ring buffer pool mechanism for streaming input data handling. There are two important functional improvements to the framework recent. Processor embedded FPGA was proposed as a standard hardware platform for specific application in KSTAR. Framework based embedded application controls them. This approach gives low level distributed control ability to a single board system under the EPICS environments. The framework also supports a real time control system development. We developed real time monitoring system as a real time network inspection tool in 2012 campaign. Keywords: KSTAR, Standardization, EPICS, embedded IOC, real time controller.
1. Introduction After completed a configuration about control and data communication infrastructure, the KSTAR finished first operation phase and start the second phase. Meanwhile, various control systems have been developed in KSTAR. Some of them are introduced in the name of collaboration. These systems are not ruled in KSTAR infrastructure. In order to meet the fast experiment cycle, we should have very fast design and implementation methods for new system, also should be consider the lack of human resources. The installed systems in KSTAR also have been increasing on a continuous scale. Until a reliable operating strategy is to be established, the experiment schedule is to be dependent on the ability of operator. For this reason, the exact state information and automatic sequence operation logic was needed to avoid mistakes of the operator. Reasons as described above, the KSTAR standard framework designed from the end of 2009 and has been applied incrementally. KSTAR already has been adopted a middleware toolkit named Experimental Physics and Industrial Control System (EPICS) [1]. However for the convenience of control system management, general purposed frame were needed. A framework includes general functions in KSTAR operation and supports development efficiency for high-level programmer who feels difficult to hardware related programming. In this paper, we describe the framework architecture, key _______________________________________________________________________________ author’s email:
[email protected]
features and implementation results in KSTAR. We also discuss about the future plan of the framework for stream archiving function.
2. KSTAR standard software framework 2.1 Overview of the sfwLib The framework is distributed in the form of an EPICS library named sfwLib, and provides template files, record instance file and database definition files. It also provides an environment descriptor that contains the IP address, MDSplus information. Fig 1 shows concept of framework based EPICS Input Output Controller (IOC) and components. Basic structure of the framework is based on the EPICS device / driver support routines and it adopted the asynchronous data flow for non-blocking command handling. The sfwLib has various data structures for a general control system which is including an Analog to Digital Converter (ADC), Digital to Analog Converter (DAC) and Digital Input Output (DIO) modules. It contains template file for minimize of iterative development process. The framework makes connection between a user command and hardware by using EPICS INST_IO link, and it has a total 35 Process Variables (PV). Except soft PVs for internal processing, framework provides 27 PVs.
Ethernet Typical IOC by Framework
sfwEnv
Device support routine DB processing
Driver support routine
Template file for command dispatcher
StopEvent thread
RingBuf thread
Device controller
Channel controller
libsfwLib
CatchEnd thread event
Template file
Master device
KSTAR sequence handler
M essageQueue
Dynamic register routine
None blocking command flow
Child List
Record instances file, Database definition file
MDSplus handler
Ring Buffer Pool
Predefined records
Logic of state notification
CA
EPICS Standard Layer
KSTAR environment and MDSplus connection descripter
DAQ thread
EPICS library
RTCORE thread
Operating System Hardware
Fig.1 Layout of KSTAR standard software framework and components
2.2 Software layout The sfwLib based controller can be conceptually divided as a master and peripheral devices. Though the both master and device module organized with device support routines, the master device has a virtual layer on top of the actual physical device. The device controller manipulates a physical hardware and, if necessary, has child channel controllers. The MDSplus handler, sequence handler and state notification module are generated automatically after creation of the master device. The device and channel controller are generated dynamically depending on the number of installed hardware without any modification of the program from the start scripter. The data structures of master/device module are composed of several threads, structure of user function pointer and variables for MDSplus. The device controller placed in master controller as a slave module. Basically, control commands from an external system are delivered to the user function of device module through the master device supplied interface logic. In this process, master/device control thread using a message queue to
prevent blocking of EPICS DB processing from a bulk command input. The master device can serve as a dispatcher of command without device controller. This function supports a synchronized operation with KSTAR sequential experiment to an introduced system from external site.
2.3 Data Acquisition (DAQ) System Most of the control system in KSTAR includes digitizer modules. In order to control them, it is used sleep and wake-up scheme after request a kernel function in general. Some digitizer uses double-buffering technique. In a more complex manner, some devices need chained Direct Memory Access (DMA) descriptor. In case of very high speed digitizer, some of them take a post-processing method for data acquisition by using a high-capacity internal memory. Our framework supports ring buffer pool mechanism to control various ADC, DAC and DIO boards. Most of DAQ system in KSTAR configured with “DAQ thread”, “RingBuf thread” and buffer pool data structure. Fig 2 shows a data flow of the
H-alpha DAQ system using DMA chain. This approach will be applied to continuous data transfer scheme in the
future.
Fig.2 Layout of stream data handling in sfwLib
2.4 Sequential operation and state notification The sfwLib was designed for pulse based operation system. The sequence handler in sfwLib controls the experiment start events from Central Control System (CCS) and Time Synchronization System (TSS). These operation events occurs DB processing of corresponding PVs. At this point, user registered function can be executed before or after the event. This control flow is depended on system characteristic. The event handler had been used the “Sequence start” and "Pulse start” events. The “Tree create” event was added as a new event from the 5th campaign. The framework uses the state notification component to display system state information. Each device controller has their own hardware status. Master controller determines the state of a system and assigns a status bit to a single PV. The final system state was determined after DB processing with predefined order. The state of the system is represented as seven.
3. Functional improvements 3.1 embedded IOC New concept of hardware platform is proposed for control system standardization recently. Frequency sweeping reflectometer and Thomson Scattering diagnostic systems has Field Programmable Gate Array (FPGA) based customized boards which was realized as an embedded IOC [2,3]. Reflectometer used FPGA Mezzanine Card (FMC) for the analogue interface part
[4]. FMC provide modularization of physical interface in digitizer. These embedded IOC were configured with the framework within hard-core processor in FPGA. The device controller in the framework controls the FPGA logic block directly by Processor Local Bus (PLB) that memory mapped between user space and kernel space. Therefore, embedded IOC does not need communication mediator or middle layer between hardware and application. We can establish high bandwidth data processing logic in FPGA for high performance required system.
3.1 Real time controller Variety of methods has been introduced for improving the real time performance. Among other things, PREEMPT_RT and CPU isolation technology has been applied to the development of systems that require deterministic processing time. We tried to apply the CPU affinity at our framework, in terms of research for the anticipated enhancement of plasma control system design. The EPICS is already able to use POSIX thread priority scheduling. But it does not provide CPU affinity mechanism and scheduling policy. However, real time support patch was introduced with the support of the ITER organization at recent EPICS community and more effective control scheme has been studied for multi-core architecture like the MCoreUtils. We try to add a real time control capability in the framework using EPICS rt-patch [5] and MRG-Realtime. We configured “RTcore thread” have high priority in sfwLib. The CPU affinity, scheduling policy and priority of “RTcore thread” are changeable on run time. The
TUNA supports fine optimization ability. The Real Time Monitoring system (RTMON) [6] is the first system to apply these techniques for real time monitoring of Reflective Memory (RM) interface at 2012. The RTMON saves RM data to a local storage device by synchronized with KSTAR clock. In our application, we use FPGA modified old Local Timing Unit (LTU) [7] as an event generator, able to generate TTL out and performs PCI interrupt generation with input KSTAR clock. This modified LTU and RM board are mounted on a PC workstation (Intel ® Core™ i7-3930K @ 3.2GHz with 12MB of cache memory and 16GB of RAM). The used OS is MRG-Realtime (2.6.33.9rt31.74.el6rt. x86_64) where the kernel has been compiled with PREEMPT_RT option enable. Digital output delay time of the event generator takes approximately 151 ns to generation single TTL pulse. The application is possible to program the module to generate a digital output upon the occurrence at a wide variety of conditions. We confirmed that the RTMON keeps 10KHz stable operation and up to 20KHz in our system configuration. The CPU affinity has a major impact on the stabilization of the system and reduces the jittering. This approach can perform a big role in the plasma control system improvements.
4. Implementation and results As shown in table 2, a large number of control systems are already using the sfwLib in KSTAR. The applicability of the framework is widened from a simple command dispatcher to the real time application. It does not apply to the nonstop operating system as like vacuum and cooling system.
Table 2: Framework based control system in KSTAR
SMBI
2011
DAQ for supersonic molecular beam injection system DAQ for electron cyclotron emission radiometer
ECE radiometer
2012
H-alpha
2012
DAQ for H-alpha system
ECCD
2012
DAQ for electron cyclotron current drive system
RTMON
2012
Real time monitoring for RM interface
The framework has a significant impact on the operating results. New device installation and channel expansion has been performed in last few years. The development and commissioning of a system shall be completed within about six month. The framework provides generic package for control systems development in KSTAR. Using framework, we can reduce period of control system development and surplus time can be assigned to the system commissioning. The aggregation of KSTAR faults shows incidence rate has been reduced after applying sfwLib despite new installation system increased [8].
5. Future works KSTAR operation entered second phase. This means high-efficiency plasma is generated for a longer period. In second phase, it is requires many considerations on data storage, reliable and fast interlock, the structure of the plasma control system. To derive a solution for these issues, KSTAR control team has been survey a new technology with various perspectives. A streaming data transmission function will be granted in the framework. We consider it based on 10G Ethernet infrastructure with remote direct memory access. We are also doing a conceptual design for fault tolerance system about real time plasma control based on Advance Telecommunications Computing Architecture (ATCA) platform.
system
year
Features
DDS1
2009
DAQ for Magnetic diagnostic
Thomson Scattering diagnostic
2010
FPGA based embedded IOC, DAQ
Frequency swept reflectometer
2009
FMC based modularized digitizer
RMP controller
2011
BES
2012
Power supply control system for Resonant Magnetic Perturbation Command dispatcher
VVS
2012
DAQ for vacuum vessel monitoring
References
Millimeter wave interferometer
2011
Real time calculation
[1]
GCDS
2012
DAQ for ground current detection
Fueling system
2011
Gas controlling and DAQ
NBI
2011
Integrated control for neutron beam injection system
RBA
2011
DAQ for resistive bolometer array
VBS
2012
IRTV
2011
DAQ for visible bremsstrahlung system Infrared TV control system
Acknowledgments
plasma
density
This work was supported by the Korean Ministry of Education Science and Technology. We would like to thank KSTAR teams.
[2]
[3]
[4]
See www.aps.anl.gov/epics/index.php for experimental physics and industrial control system. W. Lee, M. Park et al., Development of the EPICS-based data acquisition system for Thomson scattering diagnostic, Fusion Eng. Des. 86 (2011) 1294-1297. W. Lee, H. Kim, M. Park, J. Lee, K. Kim, Synchronized operation by field programmable gate array based signal controller for the Thomson scattering diagnostic system in KSTAR, Rev. Sci. Instrum. 83, 093505 (2012). W. Lee, M. Park, S. Seo, H. Kim, New design approach of FPGA based control system and implementation result in
[5] [6] [7]
[8]
KSTAR, 27th Symposium on Fusion Technology, 2012. See http://pub.cosylab.com/EPICS/EPICS_Real_Time.pdf. Woongryol Lee, Application with RT-patched EPICS for real-time monitoring, EPICS collaboration meeting, 2012. M. Park, W. Lee, H. Na, The upgrade of KSTAR timing system to support long-pulse operation and high-speed data acquisition, Fusion Eng. Des. 87 (2012) 1967-1971. J. Hong, M. Kim, W. Lee et al., Operation result of the KSTAR Interlock System and Shot Sequence Control on the 5th KSTAR Experimental Campaign, KSTAR conference, 2013.
