Basics Of Embedded Linux - GEEks of Pune

Embedded Systems ~ Kiran Divekar

Agenda • • • •

Basic Embedded systems Embedded Linux Tools, Development Points to optimize

Embedded Linux • Run on Limited Resource • Real Time • Dedicated Device Drivers

Sixnet VersaTRAK IPm

• • • • •

Embedded Linux PowerPC CPU 16MB DRAM 16MB Flash Serial & Ethernet Ports Linux and IEC1131 Programming/Modbus

Sharp Zaurus • • • •

Lineo Embedix 206 MHz StrongARM 64 MB DRAM 16MB Flash 3.5” display (320x240) 64K colors • Opera browser & Qtopia • QT/Embedded GUI

Cell and Web Phones Telepong Mobile Phone

GITWiT Mobile Phone

Aplio/PRO IP Phone

TiVO Set Top Box

• Home grown port of Embedded Linux • 54MHz PowerPC • Multi GB hard disk

Axis 2120 Network Camera

• • • •

uCLinux Built-in Ethernet port 100 MHz ETRAX CPU 16 MB RAM

Humanoid Robots

• Isamu

• Univ. of Tokyo/Kawanda Ind. • Dual Pentium CPU • RT-Linux • Height: 53 inches • Weight: 121 lbs.

Humanoid Robots • HOAP • Fujitsu • RT-Linux • Height: 48 cm • Weight: 6 kg • 100 units/yr

What is an Embedded OS? • An "embedded system" is any computer system or computing device that performs a dedicated function or is designed for use with a specific embedded software application. • Embedded systems may use a ROM-based operating system or they may use a disk-based system, like a PC. But an embedded system is not usable as a commercially viable substitute for general purpose computers or devices.

What makes a good Embedded OS? • • • • • • •

Modular Scalable Configurable Small footprint CPU support Device drivers etc, etc, etc...

What is Real Time? “A real time system is one in which the correctness of the computations not only depends upon the logical correctness of the computation but also upon the time at which the result is produced. If the timing constraints of the system are not met, system failure is said to have occurred.” - Donald Gillies

What is Real Time? “Real time in operating systems: The ability of the operating system to provide a required level of service in a bounded response time.” - POSIX Standard 1003.1

Hard vs. Soft Real Time • Hard – guaranteed worst-case response times – absolutely, positively, first time every time

• Soft – Kinda, sorta, usually

What makes a good RTOS? • Multi-threaded and pre-emptible • Thread priority has to exist because no deadline driven OS exists • Must support predictable thread synchronization mechanisms • A system of priority inheritance must exist

Who are the Embedded OS players? • Wind River Systems – VxWorks – pSOS

• QNX Software Systems – QNX

• Green Hills Software – Integrity

• QualComm : REX OS

Who are the Embedded OS players? • Mentor Graphics – VRTX

• Palm Computing – PalmOS

• Symbian – SymbianOS, S60, S80, S90

Microsoft • Embedded NT/XP – “Real-time” control

• Windows CE (CE.NET) – Internet devices

• Pocket PC 2002 – Handheld PC’s and PDA’s

Commercial Embedded Linux • AMIRIX Embedded Linux – derived from Debian

• Coollogic Coollinux – combines Linux and Java for Internet apps

• Coventive Xlinux – kernel can be as small as 143KB

• Esfia RedBlue Linux – 400K, designed for wireless apps

Commercial Embedded Linux • KYZO Pizza Box Linux – SAMBA based file, print, CD server

• Lineo Embedix – supports real time and high availability apps

• LynuxWorks BlueCat – general purpose embedded solution

• MontaVista Linux – general purpose embedded solution

Commercial Embedded Linux • Neoware NeoLinux – Red Hat derived for information appliances

• PalmPalm Tynux – Internet appliuance and multimedia

• Red Hat Embedded Linux – general purpose embedded solution

• RedSonic Red-Ice Linux – runs from DiskonChip or CompactFlash

Commercial Embedded Linux • Bluecat Linux • TimeSys Linux GPL – low latency enhanced kernel

• Tuxia TASTE – distro targets Internet appliances

Open Source Embedded Linux • Embedded Debian Project – convert Debian to an embedded OS

• ETLinux – for PC104 SBC’s

• uCLinux – for microprocessors that don’t have MM

• uLinux (muLinux) – distro fits on a single floppy

Commercial Linux RTOS • • • • • •

FSMLabs - Open RT Linux Lineo - Embedix Realtime LynuxWorks - BlueCat RT MontaVista Software - Real Time Extensions REDSonic - REDICE Linux TimeSys - Linux/Real-Time

Open Source Linux RTOS • ART Linux - real time extension based on RTLinux • KURT - event schedules with 10us resolution • Linux-SRT - for soft real time apps like multimedia • Qlinux - provides Quality of Service guarantees • RTAI - “hard” Real Time Application Interface

RTLinux • • • •

A “hard real-time” mini operating system runs Linux as it’s lowest priority execution thread Linux thread completely preemptible Real time threads and interrupt handlers never delayed by non-realtime operations • Supports user level programming • MiniRTL implementation fits on a floppy

What’s so special about Linux? • • • • •

Multiple choices vs. sole source Source code freely available Robust and reliable Modular, configurable, scalable Superb support for networking and Internet • No runtime licenses • Large pool of skilled developers

What’s so special about Linux?

Process and Virtual Addressing Applications

GUI Applications GUI

Kernel Device Driver Hardware

Linux on PC # # #hello world :-( # #

PC

User Apps

Keybo

Kernel

ard

Disk File System

Memory

IO Devices File System File system on disk

Linux on Embedded System Memory

User Apps CPU RAM

ROM

IO

Kernel File System File system on chip

UI Printf

Kernel

Kernel Device Driver

Scanf

Display Adaptor # # #hello world :-( # #

Keyboard Driver

Keybo

ard

UI on Embedded System Printf Kernel Device Driver

Scanf

Kernel Display Adaptor

Any Output Device

Keyboard Driver

Any Input Device

Linux on Embedded System

Replacing Displayer

LCD

Button

UART

Touch screen

LED Network

Replacing Keyboard

UART Network

UI on UART Printf Kernel Device Driver

Scanf

Kernel UART Driver Txd

Rxd

Layout

System Memory Usage (Static)

Bootloader

ROM

Kernel Rootdisk

RAM

System Memory Usage (Running)

Bootloader

ROM

Kernel Rootdisk

Kernel

RAM

Rootdisk

User Memory

Debugging Using The Ethernet

Ethernet

RS232

Local Booting

Linux kernel

RAM RAM Disk

0x0xffe00000

u-boot

ROM

Zipped Linux kernel Zipped root image

Remote Booting (U-Boot on MPC860) Zipped Linux kernel

Linux kernel

192.168.0.3

DHCP server

RAM

TFTP server Zipped kernel Linux image (/home/TFTPdir/uImage)

NFS server

0x0xffe00000

u-boot

ROM

MyIP: 192.168.0.253 Server IP: 192.168.0.3 Kernel Name: uImage NFS position: /home/NFSdir

Root Image for MPC860 Linux (/home/NFSdir/)

Bootloader • • • •

Redboot U-boot Homemade bootloader Grub ???

Function of Bootloader

Bootloader

ROM

Rootdisk

• Init Board – Also support some basic debugging function

• Loading Kernel – From ROM or From Remote Server

• Load Root Image • Starting Kernel

Kernel

RAM

Root Disk • Host Target Development environment • Arm targets : Linux supports 10 distinct ARM Advanced RISC Machine

CPUs, 16 different platforms, and more than 200 related boards

• • • •

http://www.arm.linux.org.uk SCSI, USB, I2C buses Parallel, Serial ports MTD storage

Host Target • Host: Development machine • Target: Embedded target system • Make ARCH=arm CROSSCOMPILE=arm-linux-

Cross-compilation • • • • •

Contains: gcc, binutils, glibc headers uClibc based toolchains (create much smaller binaries): Download one from http://buildroot.uclibc.org/ Handhelds.org (glibc: bigger binaries): ftp://ftp.handhelds.org/projects/toolchain/ Build it yourself using crosstool (glibc) Code-sourcery.

Components • Kernel uImage • Loadable modules if non-GPL • RFS – Init scripts – Busybox (configrable) – Apps… – Done !!!

Kernel uImage • Configuration – Make allnoconfig ? – Arch//configs/ – B.S.P. [ Board Support Package ]

RFS

• Jffs2 images • Squashfs images • Initrd.gz for GRUB – – – – – –

# Create an empty ramdisk image dd if=/dev/zero of=/tmp/ramdisk.img bs=$BLKSIZE count=$RDSIZE # Make it an ext2 mountable file system /sbin/mke2fs -F -m 0 -b $BLKSIZE /tmp/ramdisk.img $RDSIZE # Mount it so that we can populate mount /tmp/ramdisk.img /mnt/initrd -t ext2 -o loop=/dev/loop0

– # Populate the filesystem (subdirectories) – ……………..# Finish up........... – umount /mnt/initrd – gzip -9 /tmp/ramdisk.img

PC 1010010 1101100 1111011 1110101 1101010

Download File To Development Board Development Board ROM

RFS • Jffs2, squashfs images • Netboot – setenv x_bootargs_root 'root=/dev/nfs rw nfsroot=$(serverip):$(rootpath) ip=$ (ipaddr):$(serverip):$(bootargs_end);' – setenv serverip 192.168.1.1 – setenv ipaddr 192.168.1.10 – setenv rootpath /home/test/exports/

• NFS exports : /etc/exports – Export directory with proper perms

NFS PC With NFS Server Ethernet 1111011 1110101 1101010

Cable

Development Board (Running Linux)

uclibc C program

Compiled with shared libraries glibC

Hello World

uClibc

Compiled statically glibC

uClibc

4.6 K

4.4 K

475 K

25 K

Busybox 245 K

231 K

843 K

311 K

Tools • • • • • •

Chroot: runs with specified root Minicom: Sshd : ssh server Lighttpd : HTTP server UI tookkits: I don’t know  Browsers: Webkit…

Files • ls Busybox • /dev/console Device files • myapp cross-compiled application • watchdog

stability module

Optimizations • • • • •

1. kernel optimizations 2. memory 3. performance 4. Size 5. Security

Startup Optimizations • System startup /etc/inittab

• SysV init: Starts services sequentially. Waits for the current startup script to be complete to start the next one! While dependencies exist, some tasks can be run in parallel! • Initng: http://initng.thinktux.net New alternative to SysV init, which can start services in parallel, as soon as their preconditions are met

Memory Optimizations • Upfront memory allocations • Implement and manage memory pools • This reduces the overheads of allocation and freeing of memory. • Adds pool maintenance overhead 

Security Tools • Selinux • Grsec • Lids • Lidsadm • Listconf

• Tools which can set capabilities, restrictions and access permissions on a file.

Network Performance • TCP / IP tuning • • • • •

/etc/sysctl.conf net/core/rmem_max = 8738000 net/core/wmem_max = 6553600 net/ipv4/tcp_rmem = 8192 873800 8738000 net/ipv4/tcp_wmem = 4096 655360 6553600

• NAPI for drivers.

Using processor acceleration instructions • liboil - http://liboil.freedesktop.org/ Library of functions optimized for special instructions from several processors (Altivec, MMX, SSE, etc.) • Mainly functions implementing loops on data arrays: type conversion, copying, simple arithmetics, direct cosine transform, random number generation...

Prelinking (1) Applies to executables using shared libraries • To load and start an executable, the dynamic linker has a significant amount of work to do (mainly address relocation) • It can take a lot of time for executables using many shared libraries! • In many systems in which executables and shared libraries never change, the same job is done every time the executable is started.

Use simpler Unix executables • Big, feature rich executables take time to load. •

Particularly true for shell scripts calling the bash shell! Idea: replace standard Unix / GNU executables by lightweight, simplified implementations by busybox (http://busybox.net).

Shells: reducing forking •fork / exec system calls are very heavy. • •

Because of this, calls to executables from shells are slow. Select Shells -> Standalone shell in busybox configuration to make the busybox shell call applets whenever possible. Pipes and back-quotes are also implemented by fork / exec. You can reduce their usage in scripts. E.g. cat /proc/cpuinfo | grep model Replace it with: grep model proc/cpuinfo

Use faster filesystems Run faster by using the most appropriate filesystems! • Compressed read-only filesystem (block device): use SquashFS (http:// squashfs.sourceforge.net) • NAND flash storage: if you do not need compression,

Use faster filesystems (2) • Use RAM filesystems for temporary,

•

speed critical files with no need for permanent storage. Documentation/ filesystems/tmpfs.txt Good to benchmark your system with JFS or XFS too. XFS is reported to be the fastest to mount (good for startup time), and JFS to have the lowest CPU utilization.

Speed up applications with tmpfs • When enough RAM is available, the OS keeps recently •

accessed files and applications in RAM (page cache). This significantly speeds up any new usage. However, depending on system activity, this may not last long. For programs that need fast startup even if they haven't been run for a long time: copy them to a tmpfs filesystem at system startup! Caution: don't use ramdisks instead! Ramdisks duplicate files in RAM and unused space cannot be reclaimed.

Use a profiler • Using a profiler can help to identify unexpected behavior degrading application performance. • For example, a profiler can tell you in which functions most of the time is spent. • Easy to do with Valgrind: http://valgrind.org/

Linux-Tiny Goal: reduce the disk footprint and RAM size of the Linux kernel Set of patches against the mainstream Linux kernel. Mergeability in mainstream is a priority. Many changes have already been merged in recent kernels. •All features can be selected in kernel configuration (CONFIG_EMBEDDED).

Linux-Tiny ideas (1) • Remove kernel messages (printk, BUG, • • • •

panic...) Hunt excess inlining (speed vs. size tradeoff) Hunt excess memory allocations Memory (slob instead of slab) allocator more space efficient for small systems. Simpler alternative implementations of kernel functionalities with less features, or not supporting special cases.

Linux-Tiny ideas (2) • Remove some features which may not be needed in some systems. • Compiling optimizations for size. • A smaller kernel executable also saves RAM (unless executed in place from storage).

Reducing kernel RAM

• Booting without sysfs support (if your apps can do without it):

• Kernel XIP

Kernel code is not copied into RAM, thus saving the size of the uncompressed kernel executable (at least 1 MB, often 2 or 3 MB).

Static or dynamic linking? (1) Static linking • All shared library code duplicated in the executables Allows not to copy the C library in the filesystem. Simpler and smaller when very few executables (busybox) Library code duplication: bad for systems with more executables, best for small systems with few executables!

Static or dynamic linking? (2) Dynamic linking • Shared library code not duplicated in the executables Makes much smaller executables Saves space in RAM (bigger executables take more RAM)

How to use uClibc? • Need to compile all your executables with a uClibc toolchain. • Ready-to-use toolchains • You also have to copy the uClibc files from the toolchain to the /lib directory in the target root filesystem.

Need for stripping • To remove debugging information, use the

•

strip command. This can save a very significant amount of space! gcc -o hello hello.c (output size: 4635 bytes) strip hello (output size: 2852 bytes, -38.5%) Don't forget to strip libraries too!

Are my executables stripped? You can use the file command to get the answer file hello hello: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs), not stripped strip hello hello: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs), stripped

How to strip • Some lightweight tools, like busybox, are automatically stripped when you build them. • Makefiles for many standard tools offer a special command: make install-strip • Caution: stripping is architecture dependent. Use the strip command from your crosscompiling toolchain: arm-linux-strip potato

sstrip: “super strip” • Goes beyond strip and can strip out a few

more bits that are not used by Linux to start an executable.

Library Optimizer http://libraryopt.sourceforge.net/ • Examines the complete target file system, resolves all shared library symbol references, and rebuilds the shared libraries with only the object files required to satisfy the symbol references. • Can also take care of stripping executables and libraries.

Compiler space optimizations • Regular compiler optimizations simplifying code also reduce size • You can also reduce the size of executables by asking gcc to optimize generated code size: gcc -Os -o husband husband.c • -Os corresponds to -O2 optimizations except the ones increasing size, plus extra size-specific ones.

Restartable applications • When RAM is scarce, can be useful to abort applications that are not in use. • Better to do it before the Linux Kernel OOM (Out Of Memory) killer comes and makes bad decisions. • To support this, design your programs with the capability to save their state when they are aborted and return to it when they are restarted. [ HA ???]

Compressing filesystems Can significantly increase your storage capacity • MTD (flash or rom) storage: use jffs2 • Block storage: use SquashFS ( http://squashfs.sourceforge.net) instead of CramFS for read-only partitions. It compresses much better and is much faster too.

Tickless kernel Kernel configuration: NO_HZ setting in Processor type and features To implement multitasking, the processor receives a timer interrupt at a given frequency (every 4 ms by default on Linux 2.6). On idle systems, this wakes up the processor all the time, just to realize there is nothing to do! Idea: when all processors are idle, disable the timer interrupt, and re-enable it when something happens (a real interrupt). This saves power in laptops, in embedded systems and with virtual servers!

•

•

Suspend hidden GUIs Idea: suspend hidden user interfaces to save CPU and power. • Send a suspend (stop) signal: kill -SIGTSTP • Send a continue signal: kill -SIGCONT

Software suspend http://www.suspend2.net/ • Lots of great features for notebook users, such as RAM suspend or hibernate to disk. Unfortunately, restricted on some Intel compatible processors and targeting only machines with APM or ACPI Not addressing the requirements of embedded systems

Reducing power consumption Summary • • • •

Tickless operation CPU frequency control Suspend hidden GUIs Software suspend (for PC embedded systems) • Kernel and application XIP • Power management in device drivers

Kernel Function Trace Kernel Function Trace (KFT) Kernel function tracing system, to answer questions such as: •What functions are called? •How long to the functions take? •How many times are the functions called? •In which functions is most time spent? Many details and a very good presentation: http://elinux.org/Kernel_Function_Trace

Use a swap partition For embedded systems with a hard disk • Even if your system never gets out of RAM, and swapping is not used much in embedded systems, it could make sense! • With a swap partition, the system can swap out memory pages which haven't been used recently, making more RAM available for caching frequently accessed files. • This should improve overall throughput, but may also reduce system responsiveness. Bad for real-time systems! • Caution: do not use flash storage as a swap partition!

Use mlockall() • Parts of the address space of a process may be swapped out or not loaded yet (on-demand paging: the kernel only loads process pages with are actually accessed, and loads other ones when actually accessed). This causes unpredictable delays in process execution. • The mlockall() system call can be used to make sure that the entire address space of a running process is loaded in RAM. • Useful for real-time applications needing predictable response time.

Debugging • Gdb server on Target • Target remote over serial/network • Unstripped binaries at host – Mechanisms to export cores. – Accidental reboots.

Remote Debugging Ethernet Cable

开发板

GBD Server

Remote PC Running Linux

GDB

Debugging Using RS232 Port

Ethernet

RS232

Unified tools • Android : software platform and operating system written by Google and the Open Handset Alliance • Openwrt • Scratchbox • Code-sourcery • Metroworks

Further info • Home page: •

http://handhelds.org/ Mailing lists: http://www.handhelds.org/email_lists.html

For Further Info Web Sites

• LinuxDevices.com http://www.linuxdevices.com/ • Embedded Linux Consortium http://www.embedded-linux.org/index.php3 • All Linux Devices http://alllinuxdevices.com/ • Embedded Linux StarGate http://linux-embedded.com/

For Further Info Web Sites/Magazines

• Dr. Dobbs Embedded Systems http://www.ddjembedded.com/about/ • Embedded Linux Journal http://embedded.linuxjournal.com/ • Embedded Systems Programming http://www.embedded.com/mag.htm

54 48 41 4E 4B 59 4F 55