Real Time in Linux Embedded Systems
Real Time
in Linux Embedded Systems
Michael Opdenacker
Free Electrons
http://free-electrons.com
Thanks to Nicolas Rougier (Copyright 2003, http://webloria.loria.fr/~rougier/) for the Tux image
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
1
http://free-electrons.com
Thanks
To the OpenOffice.org project, for their presentation and
word processor tools which satisfied all my needs.
To the Handhelds.org community, for giving me so
much help and so many opportunities to help.
To the members of the whole Free Software and Open
Source community, for sharing the best of themselves:
their work, their knowledge, their friendship.
To people who sent corrections:
Matti Aaltonen
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
2
http://free-electrons.com
Copying this document
© 2004, Michael Opdenacker
[e-mail]
This document is released under the GNU Free Documentation
License, with no invariant sections.
Permission is granted to copy and modify this document pro-
vided this license is kept.
See http://www.gnu.org/licenses/fdl.html for details
Document updates available
on http://free-electrons.com/articles/realtime
Corrections, suggestions and contributions are welcome!
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
3
http://free-electrons.com
Document history
Unless specified, contributions are from Michael Opdenacker
See http://free-electrons.com/doc/ChangeLog for detailed changes.
Sep 28, 2004. First public release
Sep 20-24, 2004. First session for Atmel, Rousset (France)
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
4
http://free-electrons.com
About this document
This document is first of all meant to be used as a visual
aid by a speaker or a trainer. Hence, this is just a
summary or a complement to what is said. Hence, the
explanations are not supposed to be exhaustive.
However, this document is also meant to become a
reference for the audience. It also targets readers
interested in self-training. So, a bit more details are
given, making the document a bit less visually attractive.
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
5
http://free-electrons.com
Contents
Introduction
Reducing latency in Linux
Linux hard real-time extensions
Commercial Linux real-time distributions
Conclusion
References
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
6
http://free-electrons.com
Real Time in Embedded Linux Systems
Introduction
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
7
http://free-electrons.com
Hard Real Time
A system is considered as a hard real time if it can answer to an
internal or external stimulus within a given maximum
amount of time. ª Guaranteed worst caseº
Hard real time systems are used wherever failing to react in
time can cause a system failure or damage, or put its users in
danger.
Typical examples
Industrial process control
Transportation
Medicine (pacemakers, etc.)
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
8
http://free-electrons.com
Soft Real Time
A system is considered as soft real time if it is built to react to
stimuli as quickly as it can. ª Best effortº
However, if the system loses events or fails to process them in
time, there is no catastrophic consequence on its operation.
There is just a degradation in quality.
Typical examples
Audio, video
Airline reservation systems
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
9
http://free-electrons.com
Linux and Real Time
Linux cannot be considered as a hard real time system
There are long sections of code where all interrupts are masked
Kernel code (system calls) not preemptible until they complete
or decide to release the processor by calling the scheduler.
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
10
http://free-electrons.com
Typical response time
A Linux system can be considered as soft real time if it
can react within a few hundreds of µs in most cases.
A Linux system can be considered as hard real time if it
can react within a few tens of µs in all cases.
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
11
http://free-electrons.com
Linux 2.6 improvements
Linux 2.6 improves this by offering a preemptible option
(CONFIG_PREEMPT)
Kernel code can be interrupted at almost any time (except when
spinlocks are held)
This reduces latency for high priority processes
Implications for driver writers
Anything can happen between 2 instructions
Uniprocessor machines behave like SMP ones. No issue for clean
drivers written with SMP in mind.
Caution with per-CPU variables
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
12
http://free-electrons.com
Other Linux 2.6 improvements
Much more efficient scheduler
Can be built with no virtual memory support
(CONFIG_SWAP=n)
Note: lots of RTOS systems don©t use an MMU, even if
one is available in the CPU. MMU management can add
too much overhead.
Improved POSIX threads (now available in mainstream)
POSIX signals can©t be lost
Make it easier to set task priorities and schedule periodic tasks
with precision (useful for polling resources)
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
13
http://free-electrons.com
Low latency patches
Linux 2.4 approach: add more scheduler calls throughout the code.
Addressed by the preemptible option in 2.6.
Voluntary preemption patch (Ingo Molnar)
http://people.redhat.com/mingo/voluntary-preempt/
Reduces latency by calling cond_reschedule() in the might_sleep()
calls (used for debugging).
However, Andrew Morton argued that it©s redundant with the
preemptible option. See http://lwn.net/Articles/92806/ for details.
August 2004: Some audio users still complain that Linux 2.6 latency
is still not sufficient, even with the low latency patches.
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
14
http://free-electrons.com
Real Time in Embedded Linux Systems
Linux hard real-time extensions
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
15
http://free-electrons.com
RTLinux
Original Linux real-time extension:
http://www.fsmlabs.com/products/openrtlinux/
However, the author filed a software patent covering the addition of real-
time support to general operating systems as implemented in RTLinux.
ª RTLinux Open Patent (!) Licenseº: 2 conditions for royalty-free use of
this license
Either using only GNU GPL software. No room for proprietary software or even
other free software (BSD, Apache...)
Or using an unmodified version of RTLinux.
See http://www.fsmlabs.com/products/rtlinuxpro/rtlinux_patent.html
RTLinux and distributing RTLinux violates the GPL license (FSF, 2001)
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
16
http://free-electrons.com
Other RTLinux issues
Community unfriendly. Example: http://www.rtlinux.org/ and
http://www.realtimelinux.org/ now redirect to http://www.fsmlabs.com/
The patent issue and uncertainty drew many developers away and
frightened users. Linux RTAI attracts much more attention, developers and
users.
Offers 2 products
RTLinuxPro: much more advertised. In particular, includes major
rewrites to the original common code base.
RTLinuxFree: doesn©t include the RTLinuxPro improvements... How
much focus does it get from FSM Labs?
If you need commercial RT distributions, better use the other ones (Real
Free Software with no patent burden).
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
17
http://free-electrons.com
The RTAI project
http://www.rtai.org/
Real-Time Application Interface for Linux
Dipartimento di Ingegneria Aerospaziale - Politecnico di
Milano (DIAPM)
Founded in 1999, by Prof. Paolo Montegazza (long time
contributor to RTLinux). 1.x series for Linux 2.2
Supports Linux 2.6 since April 2004 (3.1 series)
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
18
http://free-electrons.com
RTAI strengths
Community project (mainly European contributors now). Large user base.
Excellent documentation: beginners guide, articles, conference pages, user
and reference manuals...
Supported architectures (+ SMP support)
x86, PowerPC, ARM (StrongARM; ARM7: clps711x-family, Cirrus
Logic EP7xxx, CS89712, PXA25x), MIPS, CRIS (ongoing)
License: LGPL (only requires RTAI source changes to be shared)
Modularity
Support for POSIX and other RTOS APIs (through Xenomai)
Hard real-time support in user space
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
19
http://free-electrons.com
System architecture
User land
Linux process Linux process
(data acquisition) (GUI)
Real-time
Kernel modules
task 3
Real-time FIFO
Linux Kernel
Real-time Real-time
task 1 task 2
Linux interrupt handlers
Software interrupts
Real-time kernel
Hardware interrupts
Hardware. Interrupt Controller
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
20
http://free-electrons.com
RTAI concepts
RTAI adds a layer between the Linux kernel and the
hardware
The RTAI kernel manages real-time tasks according to
their priorities. The Linux kernel is also a real-time task,
with the lowest priority.
All non real-time interrupts are handed out to the Linux
kernel.
Very good introduction:
http://www.rtai.org/documentation/articles/guide.html
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
21
http://free-electrons.com
RTAI implementation
RTAI available as a kernel patch
RTAI implemented through Linux kernel modules:
rtai: core module
rtai_sched: real-time schedulers (3 available + other services)
rtai_fifos: real-time fifos for communication between real-time and Linux
user tasks.
rtai_shm: sharing memory between real-time and Linux tasks
lxrt: makes RTAI scheduler functions available to Linux
rtai_pqueue, rtai_pthread, rtai_utils: POSIX RTAI modules
Modular: you can only load the modules you need
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
22
http://free-electrons.com
RTAI usage
Download RTAI
Check the corresponding Linux version
Download Linux
Patch Linux with RTAI, compile it.
Reboot
Load the RTAI modules you need
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
23
http://free-electrons.com
RTAI example (1)
/* rt_process.c */
int init_module(void)
#include
{
#include
RTIME tick_period;
#include
rt_set_periodic_mode();
#include
rt_task_init(&rt_task, fun, 1, STACK_SIZE, TASK_PRIORITY, 1, 0);
#include
rtf_create(FIFO, 8000);
#include
tick_period = start_rt_timer(nano2count(TICK_PERIOD);
#define TICK_PERIOD 1000000
rt_task_make_periodic(&rt_task, rt_get_time() + tick_period, tick_period;
#define TASK_PRIORITY 1
return 0;
#define STACK_SIZE 10000
}
#define FIFO 0
void cleanup_module(void)
{
static RT_TASK rt_task;
stop_rt_timer();
rtf_destroy(FIFO);
static void fun(int t)
rt_task_delete(&rt_task);
{
return;
int counter = 0;
}
float sin_value;
while (1) {
sin_value = sin(2*M_PI*1*rt_get_cpu_time_ns()/1E9);
rtf_put(FIFO, &counter, sizeof(counter));
Real-time task
rtf_put(FIFO, &sin_value, sizeof(sin_value));
counter++;
rt_task_wait_period();
}
Caution: written for RTAI 1.4.
}
May not work with your version.
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
24
http://free-electrons.com
RTAI example (2)
/* scope.c */
/* Run */
#include
#include
sync
#include
insmod /home/rtai-1.4/modules/rtai.o
#include
insmod /home/rtai-1.4/modules/rtai_sched.o
#include
insmod /home/rtai-1.4/modules/rtai_shm.o
#include
insmod /home/rtai-1.4/modules/rtai_fifos.o
#include
insmod rt_process.o
static int end;
./scope
static void endme(int dummy) { end=1; }
rmmod rt_process
int main (void)
rmmod rtai_shm
{
rmmod rtai_fifos
int fifo, counter;
rmmod rtai_sched
float sin_value;
rmmod rtai
if ((fifo = open("/dev/rtf0", O_RDONLY)) < 0) {
fprintf(stderr, "Error opening /dev/rtf0\n");
exit(1);
}
signal(SIGINT, endme);
Execution script
while (!end) {
read(fifo, &counter, sizeof(counter));
read(fifo, &sin_value, sizeof(sin_value));
printf(" Counter : %d Seno : %f \n", counter, sin_value);
}
return 0;
}
Userland program
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
25
http://free-electrons.com
Adeos nanokernel
http://home.gna.org/adeos/index.en.html
Flexible environment to share hardware resources
between several operating systems (or several instances
of the same operating system).
Used by RTAI as a replacement Hardware Abstraction
Layer beneath the Linux kernel. Not impacted by the
RTLinux patent!
Also used for SMP clustering, patchless kernel
debugging...
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
26
http://free-electrons.com
Xenomai project
http://www.nongnu.org/xenomai/
Nanokernel mimicking APIs of traditional RTOS such as
VxWorks, pSOS+, and VRTXsa.
Can be plugged onto the Adeos layer, RTAI©s RTHAL or
the LinuxThreads POSIX interface.
Goal: facilitate the porting of programs from traditional
RTOS to RTAI on GNU / Linux.
Now part of the RTAI project.
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
27
http://free-electrons.com
Real-Time in Embedded Linux Systems
Commercial real-time distributions
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
28
http://free-electrons.com
Montavista (1)
Montavista Linux Professional Edition
http://www.mvista.com/products/pro/realtime.html
Employs some of the most active kernel hackers, in particular
on the ARM platform
All kernel development shared with the community kernel
core and drivers (Linux 2.6 example: preemption option,
many drivers...)
O(1) fixed overhead / fixed priority scheduler for real-time
running in front of the Linux scheduler. Handles real-time
POSIX pthreads, and leaves the other to the standard
scheduler.
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
29
http://free-electrons.com
Montavista (2)
Keeps the standard Linux API
Emulates VxWorks and pSOS API
Can be tried through a preview kit.
Supports an impressive list of processors
Tools to analyze performance and track down latency
sources.
Development tools seem to be proprietary
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
30
http://free-electrons.com
TimeSys
http://timesys.com
Exhaustive toolset as available from other vendors
(need to check details on their site)
Community friendly: share very interesting and generic
technical whitepapers and articles.
Free Software BSPs (Board Support Packages) available
True soft to hard real-time OS product
Development tools seem to be proprietary
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
31
http://free-electrons.com
LynuxWorks BlueCat
http://www.lynuxworks.com/products/bluecat/
Complete Embedded Linux distribution and toolset
Based on Linux 2.6 with standard soft real-time improvements
Comprehensive processor support.
Applications can run unmodified on LynuxWorks© LynxOS RTOS if
hard real time is needed.
LynuxWorks: a lot of experience in traditional RTOS. Same
development tools for BlueCat and LynxOS.
No visible contribution to the kernel. Looks like a traditional RTOS
vendor trying to surf the Linux wave (could be wrong!)
Proprietary development tools
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
32
http://free-electrons.com
Other real-time Linux vendors
Metrowerks
http://www.metrowerks.com/
Hardly mention Linux real-time support. More focus on
their non-Linux RTOS offerings
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
33
http://free-electrons.com
Benchmarking tools
LMBench: http://www.bitmover.com/lmbench/
Takes several latency measures on your system
dbench: ftp://samba.org/pub/tridge/dbench/
Produces filesystem load
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
34
http://free-electrons.com
Real-Time in Embedded Linux Systems
Conclusion
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
35
http://free-electrons.com
Choosing a real-time Linux OS
System
Latency
Development complexity
Simple
Standard Linux No real-time: Standard Linux API
10-100ms magnitude
Linux + patches Soft real-time: Applying patches
BlueCat 100 µs magnitude Standard Linux API
Montavista
Hard RT Linux Hard real-time: Applying patches
RTAI ~10 µs magnitude Loading specific modules
Montavista Implement drivers for POSIX
or specific API
Complex
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
36
http://free-electrons.com
References
http://www.realtimelinuxfoundation.org/
Community portal for real-time Linux
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
37
http://free-electrons.com
Related documents
This document belongs to the 500 page materials of an embedded GNU / Linux
training from Free Electrons, available under the GNU Free Documentation License.
Introduction to Unix and GNU / Linux
http://free-electrons.com/training/intro_unix_linux
Embedded Linux kernel and driver development
http://free-electrons.com/training/drivers
Development tools for embedded Linux systems
http://free-electrons.com/training/devtools
Java in embedded Linux systems
http://free-electrons.com/articles/java
What©s new in Linux 2.6?
http://free-electrons.com/articles/linux26
Introduction to uClinux
http://free-electrons.com/articles/uclinux
Linux real-time extensions
http://free-electrons.com/articles/realtime
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
38
http://free-electrons.com
Training and consulting services
This training or presentation is funded by Free Electrons
customers sending their people to our training or consulting
sessions.
If you are interested in attending training sessions performed by
the author of these documents, you are invited to ask your
organization to order such sessions.
See http:/free-electrons/training for more details.
If you just support this work, do not hesitate to speak about it to
your friends, colleagues and local Free Software community.
Real Time in Embedded Linux Systems
© Copyright 2004, Michael Opdenacker
GNU Free Documentation License
39
http://free-electrons.com