The challenges of developing embedded real-time

The challenges of developing embedded real-time aerospace applications on next generation multi-core processors Paul J. Parkinson Principal Systems Architect Wind River Swindon, United Kingdom [email protected]

Abstract— The advent of multi-core processor architectures has provided multiple performance benefits for enterprise and embedded systems. However, it has also disrupted the continued availability of single-core processor architectures traditionally used in embedded aerospace applications in both civil and defence sectors. In this paper, potential criteria for the selection of commercial-off-the-shelf (COTS) multi-core processors for use in sensor-related, mission-critical and safety-critical avionics applications will be considered; and the challenges of implementing support in the VxWorks real-time operating system (RTOS) on next generation multi-core processors will be presented. Keywords—avionics; certification; multicore;

I.

FACTORS INFLUENCING PROCESSOR SELECTION IN THE AVIONICS MARKET

During the last two decades, there has been dramatic growth in the number and complexity of aviation electronics (or avionics) systems in civil and military aircraft, driven by the need to implement new capabilities to meet operational requirements. The selection of microprocessors for avionics systems during this period has been influenced by a number of factors:

and type of line replaceable units (LRUs, as shown in figure 1), and as a consequence fewer processors of potentially different architectures when compared to federated systems. 3) Advent of Multi-core processors The advent of multi-core processor architectures and proliferation of numerous multi-core architecture variants, combined with the diminishing availability of single-core processor architectures, has led to a fragmentation of processor selection in the avionics market segment. The factors have had an impact on avionics software architectures, and this was previously discussed at length in “Safety, Security, Multicore” [2]. However, in this paper we will consider these factors and the impact on the selection of processor architectures for avionics applications, and the consequences for software provisioning and avionics safety certification.

1) US DOD directive on use of COTS The directive by US Defense Secretary William J. Perry in 1994 [1] paved the way for the use of commercial-off-the-shelf (COTS) technologies on US Department of Defense (DOD) programmes. This also led to the decline in the availability of military-grade processors, and more recently has resulted in programme challenges in terms of obsolescence management due to the shorter lifecycles of commercial microprocessors which are incompatible with the long lifecycles of aerospace programmes. 2) Adoption of Integrated Modular Avionics architectures The increasingly widespread adoption of integrated modular avionics (IMA) architectures has provided benefits in terms of reducing Size, Weight and Power (SWaP) requirements for airborne platforms. This has been achieved through the use of common computing platforms with highperformance processors to host multiple applications concurrently. This has resulted in a reduction in the number

© Copyright 2016 Wind River. All rights reserved.

Fig. 1. Republic F-105B with avionics layout, (Source: US Air Force – public domain)

1

B. Historical Processor Selection Aerospace suppliers historically have selected microprocessors for avionics programmes based on criteria which have included (but not limited to) performance, power dissipation, availability of extended temperature range, and longevity. Safety-critical avionics programmes have also had additional selection criteria related to potential for access to proprietary information on processor design, manufacture and testing to support RTCA DO-254 and EUROCAE ED-80 [3] objectives for hardware safety certification. Many avionics projects have selected for COTS and custom board designs which have used PowerPC™ 750 and PowerPC MPC7410, MPC7448 and MPC7457 processors with external devices for I/O. In addition, PowerPC architectures with integrated on-chip peripherals (such as MPC8349E, MPC8548E, MPC8572 and MPC8641D) need fewer external devices, and have been used in applications with more stringent SWaP requirements. A number of these avionics applications have successfully completed avionics hardware certification under DO-254 and ED-80, and avionics software safety certification under RTCA DO-178 and EUROCAE ED-12 [4] and the later revisions, DO-178C and ED-12C. Their successful deployments on safety-critical programmes has resulted in these microprocessors being regarded as a low technical risk for subsequent safety-critical programmes, creating a virtuous circle. C. RTOS Safety Certification Wind River has ported the VxWorks® real-time operating system (RTOS) to a broad range of processor architectures, including ARM®, MIPS™, PowerPC, SPARC® and x86. This has been undertaken due to the differing processor requirements of multiple vertical market segments (including aerospace & defence, automotive, industrial automation, medical, railway and telecommunications). However, undertaking DO-178B and ED-12B Level A software certification of an RTOS is extremely expensive, costing millions of Euros and is specific to an underlying processor architecture. So, it would have been cost-prohibitive for Wind River to undertake DO-178B and ED-12B safety certification on many different processor architectures, with no guarantee of being able to recoup the non-recurring engineering (NRE) costs. For these reasons, Wind River decided to develop the first DO-178B and ED-12B Level A safety COTS certification evidence package for VxWorks 5.4 Cert on the most widelyused processor in avionics, which at the time in 2000, was the PowerPC 750. In subsequent years, Wind River has developed DO-178 Level A COTS certification packages on other VxWorks releases and for additional processor architectures based on market trends and specific customer requirements.

© Copyright 2016 Wind River. All rights reserved.

This has included additional PowerPC variants (MPC74xx, MPC8245, MPC82xx, MPC8349E, MPC8548E, MPC8560 and MPC8641D) and Intel processors variants (Pentium III, Core and Atom) as shown in figure 2.

Fig. 2. VxWorks DO-178 COTS certification package timeline

This COTS evidence approach has enabled the significant DO-178 and ED-12 certification NRE costs to be amortised across multiple customers and programmes using the same processor architecture, reducing the cost of certification on each programme. This has also resulted in a virtuous circle, as the these processors have provided the lowest cost options for follow-on certification projects, due to the ability to reuse existing DO-178 and ED-12 certification evidence, rather than having to develop it for a new processor architecture and associated incremental costs. II.

THE CHALLENGES OF MULTI-CORE PROCESSOR SELECTION

Over the last decade, in order to meet the demands of ever increasing performance from the commercial market, and faced with the fundamental performance limit which could be achieved on a single-core processor due to clock speed ceiling, semi-conductor manufacturers transitioned to multi-core processor architectures to achieve performance gains. The introduction of multi-core processor architectures has provided performance gains for enterprise general purpose applications; it has also presented some unique challenges for their use in safety-critical avionics systems. This is because avionics applications have specific requirements, including (but not limited to) application isolation and determinism, and these are not the primary considerations of semiconductor manufacturers when designing multi-core processors for the commercial market. The avionics industry, academia and certification authorities have undertaken research projects into the use of multi-core processor architectures in avionics applications. A number of researchers have found that there is variation between multi-core processor designs in terms of their suitability for use in avionics applications, due to the impact of architectural design features on application isolation and determinism [5][6]. These relate to factors arising from shared resources on the device, which include use of a single memory

2

controller or shared bus is used by multiple cores (providing a risk of resource contention), and similarly use of separate or shared Level 2 caches per core. This uncertainty about the selection of multi-core processors for avionics programmes has been compounded by the following factors: i)

Although the avionics safety certification agencies EASA and FAA have published a research report [7] and a position paper [8] respectively, on the use of multi-core processors in avionics, this does not constitute formal policy or guidance.

ii) Single-core processors which have been used in safety-critical avionics applications are now nearing the end of silicon availability or are no longer available [9]. iii) The historical dominance of PowerPC in the embedded market appears to be somewhat in decline, and the long term future appears to be uncertain with NXP (formerly Freescale) developing ARM-based processors as well as to PowerPC. In addition, the large number of PowerPC QorIQ processor architecture variants makes it unclear if there will be a de facto choice for avionics. iv) The increasing performance of ARM-based processors means that they may be considered as a viable option for some types of avionics application where PowerPC processors had been used previously. v) Intel processors which historically were not widely considered for use in avionics applications due in part to their power dissipation requirements are now being considered due to Intel’s high-performance, lowpower 14nm processor devices [10]. These market dynamics have resulted in fragmentation of processor selection for avionics, resulting in a lack of an obvious, single successor for widely-deployed PowerPC single core processors. We are now facing a wide range of contenders in terms of ARM multi-core, PowerPC QorIQ architecture families and Intel Core and Atom architectures. A. The Challenge of Mission-Critical Systems Many aerospace applications are not safety-critical, as their failure may not directly impact the safety of the aircraft, but their failure could impact the success of the mission. So for convenience, we will refer to this broad category as mission-critical systems – this can include non-critical auxiliary systems, sensor payloads and other applications. These can have a very diverse set of processing requirements in terms of performance and power dissipation depending on end application and whether the application is deployed in a conduction-cooled or air-cooled environment.

© Copyright 2016 Wind River. All rights reserved.

Sensor payload applications performing signal processing or image processing have utilised the vector-processing capabilities of AltiVec™ enabled Freescale PowerPC processors (MPC74xx, MPC864x), and VMX from IBM (PowerPC 970) and PA Semiconductor (PA6T). The AltiVec provides a 128-bit vector processing unit which operates concurrently with the existing integer and floating point units, and provides the ability to accelerate parallel operations by performing up to 16 simultaneous operations in a single clock cycle using a SIMD (single instruction multiple data) architecture. Wind River first implemented support for AltiVec in VxWorks 5.4.1 in 2001, by extending the Wind River GNU and Diab C/C++ compilers to include language extensions for AltiVec, including new data type __vector, and also extending the VxWorks RTOS runtime and debugging environment. Although applications needed to be written in a way to specifically utilise the AltiVec vector unit, or employ a vectorising compiler front-end, many third-party signal processing and other libraries were developed and optimised for AltiVec. These enabled aerospace programmes to achieve significant performance speed-up compared to using a PowerPC processor without AltiVec. Many aerospace programmes that have developed applications with AltiVec technology want to continue to benefit from this investment in a transition to next generation multi-core processor architectures. Freescale (now NXP) launched the QorIQ T-series T208x processors with AltiVec support in 2013, providing a migration path for programmes using AltiVec from single-core to multi-core processor architectures. Although Intel processors have historically been bettersuited to ground-based applications due to their SWaP characteristics mentioned earlier, the most recent generations of Intel processors are increasingly being considered for use in aerospace applications due to their high performance and significant reduction in power dissipation. Intel Core i7 multicore processors incorporate Advanced Vector eXtensions (AVX™) which provide 512-bit vector processing capabilities, enabling the acceleration of signal processing applications [11] and providing the potential to consolidate legacy signal processing applications from single-core processors onto the Core i7’s multiple cores and therefore reduce SWaP footprint. Wind River introduced support for AVX vector processing in the VxWorks 6.9.3 Platform [12], and also implemented support for features which enabled programmes to utilise the performance potential of Intel processors and optimise their applications. This included the implementation of VxWorks support for Intel Hyper-Threading, through the VxWorks Symmetric Multi-Processing (SMP) Scheduler for Simultaneous Multi-Threading which utilises two threads per

3

physical core; and also support for native 64bit addressing using LP64 data model, providing increased address space overcoming the 4GByte memory address range limitations of the 32bit addressing model which can be an issue for memoryintensive sensor applications. These VxWorks runtime enhancements were augmented by enhancements on the host development side, with integration of the Intel C/C++ compiler (ICC) with Wind River Workbench, enabling architecture-level specific compiler optimisation to be performed. This can be further augmented with the use of the Intel Integrated Performance Primitives (IPP) providing a library for signal and image processor, data compression, cryptography and other functions. Additionally Intel CILK+, a C language extension which provides parallelization hints to the compiler to exploit the performance of multi-core architectures, and also be utilised. The Challenge of Fast Boot Avionics applications can have strict start-up time requirements; an example of this is electronic flight displays in the event of electrical transients where “recognizably valid pitch and roll data should be available within one second on the affected displays” [13].

Wind River has taken this approach a step further, by performing source-code level optimisation of FSP, by removing code not required for the specific hardware design. This has resulted in the achievement of application start-up times on Intel multi-core processors of around 600 milliseconds (measured from power-on, through FSP initialisation, load and start-up of the VxWorks RTOS to the first instruction of an application) as shown in figure 3. The start-up time can be reduced even further if the initialisation of the RTOS network stack is deferred to a later stage.

B.

This means that after power-failure, the processor must be re-initialised, run a boot loader, and load the real-time operating system (RTOS) and application, then start the RTOS and run the application with meaningful information on the display, all within one second. It has been possible to achieve these start-up times using VxWorks RTOS on PowerPC and other architectures for many years by configuring the processor to load and run VxWorks directly from flash memory on processor power-on reset, often without using a separate boot loader. However, on Intel architectures, the processor has traditionally run a BIOS before passing control to a boot loader to load an RTOS. In general, BIOS have been implemented as monolithic software component designed to support a broad range of hardware configurations and devices; BIOS have generally not been designed to meet strict initialisation time requirements. While this has meant that BIOS have provided the ability to support a broad range of hardware configurations and devices which may change between power-up cycles in an enterprise environment, this has had a drawback in terms of initialisation times which may take multiple seconds. This challenge has been addressed in recent years through the introduction of the Intel Firmware Support Package (FSP) [14] which provides binary firmware components for initialising Intel silicon, and can be integrated into the boot loader for an RTOS [15]. This provides the potential for reducing the start-up times of embedded applications running on Intel processors compared to using a traditional BIOS.

© Copyright 2016 Wind River. All rights reserved.

Fig. 3. VxWorks DO-178 COTS certification package timeline

C.

The Challenge of BIOS Certification A requirement which programmes may overlook when embarking on a DO-178 certification project, is the certification of the firmware initialisation code which runs from the processor’s reset address after a power reset and performs hardware initialisation before the boot loader loads and runs the RTOS. On PowerPC architectures, this is generally performed in processor-specific initialisation source code which can be compiled directly into the VxWorks boot loader or VxWorks kernel image and can undergo DO-178 certification with the RTOS. This is the approach that Wind River has previously taken on single-core PowerPC architectures, and is still valid for latest generation PowerPC QorIQ™ multi-core processors. For Intel architecture, the situation can vary, but generally a third-party BIOS firmware runs from processor reset, and performs initialisation of the processor and basic initialisation of other peripherals on the target board before passing control to a boot loader to load the operating system. The BIOS is usually only supplied in binary format, and access to the BIOS source code may not be available, which in this case would mean that the BIOS itself could not undergo DO-178 certification. Wind River faced this challenge on Intel architecture for the first time when developing the DO-178 certification package for VxWorks 5.4 Cert for the EGNOS programme [16]. As the source code for the BIOS was not available, Wind River Services needed to undertake development of some custom safety-critical software so that the VxWorks Cert RTOS did not need to rely on the correct initialisation of the hardware of the BIOS. This approach was accepted by the certification authority EASA for certification of the EGNOS

4

Integrity Processing Facility (IPF) to Level B under DO-178B and ED-12B. Using Wind River’s proven approach to source-level optimisations, it may now be possible to undertake sourcelevel optimisations of FSP to remove executable code and data for devices and peripherals which are not used on the specific hardware configuration in the certified system (which is known as dead code in DO-178). Using this optimised FSP source code, the relevant DO-178 artefacts, tests and results could then be generated, resulting in a fast boot COTS certified system. D. The Challenge of Multi-core Certification The route to multi-core certification currently presents a challenge to avionics programmes due to lack of formal policy / guidance published by FAA and EASA. However, the EASA MULCORS research report [7] and FAA CAST-32 position paper [8] should be taken into consideration when planning a safety-critical multi-core avionics project. Programmes may wish to consider the use of a multi-core processor in their next hardware platform even if their current processing requirements do not exceed that provided by a single core, in order to provide adequate processing capacity to meet future processing requirements. The selection of a multicore processor may also become a necessity due to the lack of availability of single core processors as mentioned earlier. Similarly, some programmes may wish to use multi-core processors which have more than two cores, as 4-core and 8core devices are now relatively common. However, CAST-32 does not consider multi-core processors with more than two active cores. Certifying multi-core processors will require substantial research and certification leadership to extend the guidance in the MULCORS and CAST-32 papers. In both of the above scenarios, programmes will need to be able to utilise certain processor cores and deactivate the unused cores. To meet the multi-core determinism objectives of CAST-32, programmes will need to demonstrate that a deactivated core cannot unexpectedly become active and interfere with the operation of the processor’s other cores. This could either use an approach of regularly reading control registers which are critical to safe operation and resetting the register value in the event of a change of state being detected; or by regularly overwriting the control registers to ensure that the desired state is maintained. Some processors may also provide performance monitoring units which enable the state of an individual core to be determined independently.

The ability of safety-critical avionics programmes to be able to deactivate individual cores and develop a safety-case which includes robust arguments for the deterministic operation of the process may depend on the ability to obtain detailed technical information on the design and operation of the processor from the semiconductor manufacturer. Some companies may make this information publicly available, while others may only provide certain levels of information under non-disclosure agreement. For programmes undertaking DO254 hardware certification, this will be a particularly important requirement, and will need to ensure that the selected semiconductor manufacturer will provide access to the required information, even if they do not formally support DO-254 certification in the same way as companies such as Altera [18]. III.

CONCLUSIONS

The avionics market is currently undergoing a significant transition from single-core to multi-core processor architectures, being driven by demands for greater system functionality and the semiconductor product lifecycles which primarily target the much larger commercial market segments. The advances made by semiconductor manufacturers now present a much broader range of viable processor choices for avionics applications than was available in the past. Although there currently appears to be some uncertainty about the best choice of processor for individual aerospace application use cases, it is likely that positive experiences gained by early adopters on multi-core programmes will result in a virtuous circle of support, further adoption and success, in a similar way to single-core avionics programmes of previous decades generated a rich supplier ecosystem of COTS avionics certification solutions. ACKNOWLEDGMENT The author wishes to thank the following Wind River colleagues for their input into this paper: A. Wilson, C. Downing.

The software implementation of core deactivation is processor-specific, and depends on whether individual processor architecture provides the ability for a core to be able to write to a control register to deactivate another core or not. For example, on the PowerPC QorIQ T2080™ processor, deactivation of an individual core can be achieved by setting the relevant bit field in the Core Disable Register during PreBoot Initialisation or when the core is in boot hold off mode, and once a core has been deactivated it can only be re-enabled via power-on, hard reset or core reset [17].

© Copyright 2016 Wind River. All rights reserved.

5

REFERENCES

http://www.intel.com/content/www/us/en/siliconinnovations/advancing-moores-law-in-2014-presentation.html

[1]

Perry, William J., US Secretary of State for Defense, “Specifications and Standards: – A New Way of Doing Business,” U.S. Department of Defense Memorandum, 29th June 1994.

[2]

Parkinson, Paul J. “Safety, Security and Multicore”, Proceedings of the Nineteenth Safety-Critical Systems Symposium, 8-10 February 2011. http://www.springerlink.com/content/w2751nx7l28mj35r/

[3]

“Design Assurance Guidance for Airborne Electronic Hardware”, DO254 (RTCA Inc.) and ED-80 (EUROCAE), 19th April 2002. http://www.rtca.org/store_product.asp?prodid=752

[4]

“Software Considerations in Airborne Systems and Equipment Certification” DO-178B (RTCA Inc.) and ED-12B (EUROCAE), 1st December 1992.

[13] “Electronic Flight Displays”, Advisory Circular AC25-11B, 7th October 2014, FAA. http://www.faa.gov/documentLibrary/media/Advisory_Circular/AC_2511B.pdf

[5]

Kinnan, Larry M., “Use of multicore processors in avionics systems and its potential impact on implementation and certification”, Digital Avionics Systems Conference, October 2009.

[14] Wong, Swee Heng; Sun, Jiming; Mahesh, Divya; “Intel® Firmware Support Package for Intel Architecture”, white paper, Intel, 2014.

[6]

“Microprocessor Evaluations for Safety-Critical, Real-Time Applications: Authority for Expenditure No. 43 Phase 5 Report”, US Federal Aviation Administration. DOT/FAA/AR-11/5, May 2011. https://www.faa.gov/aircraft/air_cert/design_approvals/air_software/med ia/11-5.pdf

[7]

Jean, Xavier; Gatti, Mark; Berthon, Guy; Fumey, Marc, “MULCORS Use of MULticore proCessORS in airborne systems”, Research Project EASA.2011/6, EASA, 8th November 2012. http://easa.europa.eu/system/files/dfu/CCC_12_006898-REV07%20%20MULCORS%20Final%20Report.pdf

[15] Yao, Jiewen; Zimmer, Vincent J.; Rangarajan, Ravi; Ma, Maurice; Estrada, David; Mudusuru, Giri; “A Tour Beyond BIOS Using the Intel® Firmware Support Package Version 1.1 with the EFI Developer Kit II”, Intel, April 2015. https://firmware.intel.com/sites/default/files/resources/A_Tour_Beyond_ BIOS_Creating_the_Intel_Firmware_Support_Package_Version_1_1_w ith_the_EFI_Developer_Kit_II.pdf

[8]

“Multi-core Processors”, Position Paper, Certification Authorities Software Team, CAST-32, FAA, May 2014. https://www.faa.gov/aircraft/air_cert/design_approvals/air_software/cast /cast_papers/media/cast-32.pdf

[9]

Product Longevity – Archived (September 2014), NXP website. http://www.nxp.com/pages/product-longevity-archived-september2014:LONGEVITY-ARCHIVED

[11] Santoni, Umberto; Long, Thomas; “Signal Processing on Intel Architecture: Performance Analysis Using Intel Performance Primitives”, White Paper, Intel website, 2011. http://www.intel.com/content/dam/doc/white-paper/signal-processingon-intel-architecture.pdf [12] Parkinson, Paul J.; “VxWorks 6.9.3 Platform – Optimised 64bit support for Intel architecture”, customer presentation, Wind River, 28th May 2013.

[16] Parkinson, Paul J., “European Geostationary Overlay System”, Case Study, Wind River website. http://www.windriver.com/customers/customersuccess/documents/CS_EGNOS_v2_0610.pdf. [17] QorIQ T2080 Family Reference Manual, T2080RM Rev 1, NXP, May 2015. https://www.nxp.com/webapp/Download?colCode=T2080RM. [18] DO-254 Safety Solutions, Altera website, https://www.altera.com/solutions/industry/military/applications/do254/mil-do-254.html.

[10] “Advancing Moore’s Law – The Road to 14nm”, presentation, Intel website, 11th August 2014.

© Copyright 2016 Wind River. All rights reserved.

6