Embedded Linux Realtime

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

Real Time in Linux Embedded Systems

Real Time

in Linux Embedded Systems

Michael Opdenacker

Free Electrons

http://freeelectrons.com

http://webloria.loria.fr/~rougier/

) for the Tux image

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

Thanks

To the

OpenOffice.org

project, for their presentation and

word processor tools which satisfied all my needs.
To the

Handhelds.org

community, for giving me so

much help and so many opportunities to help.
To the members of the whole Free Software and Open
Source community, for sharing the best of themselves:
their work, their knowledge, their friendship.
To people who sent corrections:
Matti Aaltonen

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

Copying this document

michael@freeelectrons.com

This document is released under the GNU Free Documentation

License, with no invariant sections.
Permission is granted to copy and modify this document pro
vided this license is kept.
See

http://www.gnu.org/licenses/fdl.html

for details

Document updates available

http://freeelectrons.com/articles/realtime

Corrections, suggestions and contributions are welcome!

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

Document history

Unless specified, contributions are from Michael Opdenacker

See

http://freeelectrons.com/doc/ChangeLog

for detailed changes.

Sep 28, 2004. First public release
Sep 2024, 2004. First session for

Atmel

, Rousset (France)

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

About this document

This document is first of all meant to be used as a visual
aid by a speaker or a trainer. Hence, this is just a
summary or a complement to what is said. Hence, the
explanations are not supposed to be exhaustive.
However, this document is also meant to become a
reference for the audience. It also targets readers
interested in selftraining. So, a bit more details are
given, making the document a bit less visually attractive.

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

Contents

Introduction
Reducing latency in Linux
Linux hard realtime extensions
Commercial Linux realtime distributions
Conclusion
References

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

Real Time in Embedded Linux Systems

Introduction

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

Hard Real Time

A system is considered as a hard real time if it can answer to an

internal or external stimulus

within a given maximum

amount of time. “Guaranteed worst case”

Hard real time systems are used wherever failing to react in

time can cause a system failure or damage, or put its users in
danger.

Typical examples

Industrial process control
Transportation
Medicine (pacemakers, etc.)

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

Soft Real Time

A system is considered as soft real time if it is built to react to

stimuli as quickly as it can. “Best effort”

However, if the system loses events or fails to process them in

time, there is no catastrophic consequence on its operation.
There is just a degradation in quality.

Typical examples

Audio, video
Airline reservation systems

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

Linux and Real Time

Linux cannot be considered as a hard real time system

There are long sections of code where all interrupts are masked
Kernel code (system calls) not preemptible until they complete
or decide to release the processor by calling the scheduler.

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

Typical response time

A Linux system can be considered as soft real time if it
can react within a few hundreds of µs in most cases.
A Linux system can be considered as hard real time if it
can react within a few tens of µs in all cases.

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

Linux 2.6 improvements

Linux 2.6 improves this by offering a preemptible option

(CONFIG_PREEMPT)
Kernel code can be interrupted at almost any time (except when
spinlocks are held)
This reduces latency for high priority processes

Implications for driver writers

Anything can happen between 2 instructions
Uniprocessor machines behave like SMP ones. No issue for clean
drivers written with SMP in mind.
Caution with perCPU variables

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

Other Linux 2.6 improvements

Much more efficient scheduler
Can be built with no virtual memory support
(

CONFIG_SWAP=n

)

Note: lots of RTOS systems don't use an MMU, even if
one is available in the CPU. MMU management can add
too much overhead.
Improved POSIX threads (now available in mainstream)

POSIX signals can't be lost
Make it easier to set task priorities and schedule periodic tasks
with precision (useful for polling resources)

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

Low latency patches

Linux 2.4 approach: add more scheduler calls throughout the code.
Addressed by the preemptible option in 2.6.
Voluntary preemption patch (Ingo Molnar)

http://people.redhat.com/mingo/voluntarypreempt/

Reduces latency by calling cond_reschedule() in the might_sleep()
calls (used for debugging).
However, Andrew Morton argued that it's redundant with the
preemptible option. See

http://lwn.net/Articles/92806/

for details.

August 2004: Some audio users still complain that Linux 2.6 latency

is still not sufficient, even with the low latency patches.

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

Real Time in Embedded Linux Systems

Linux hard realtime extensions

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

RTLinux

Original Linux realtime extension:

http://www.fsmlabs.com/products/openrtlinux/

However, the author filed a software patent covering the addition of real
time support to general operating systems as implemented in RTLinux.
“RTLinux Open Patent (!) License”: 2 conditions for royaltyfree use of
this license

Either using only GNU GPL software. No room for proprietary software or even
other free software (BSD, Apache...)
Or using an unmodified version of RTLinux.
See

http://www.fsmlabs.com/products/rtlinuxpro/rtlinux_patent.html

RTLinux and distributing RTLinux violates the GPL license (FSF, 2001)

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

Other RTLinux issues

Community unfriendly. Example:

http://www.rtlinux.org/

and

http://www.realtimelinux.org/

now redirect to

http://www.fsmlabs.com/

The patent issue and uncertainty drew many developers away and
frightened users. Linux RTAI attracts much more attention, developers and
users.

Offers 2 products

RTLinuxPro: much more advertised. In particular, includes major
rewrites to the original common code base.
RTLinuxFree: doesn't include the RTLinuxPro improvements... How
much focus does it get from FSM Labs?

If you need commercial RT distributions, better use the other ones (Real
Free Software with no patent burden).

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

The RTAI project

http://www.rtai.org/

RealTime Application Interface for Linux
Dipartimento di Ingegneria Aerospaziale Politecnico di
Milano (DIAPM)
Founded in 1999, by Prof. Paolo Montegazza (long time
contributor to RTLinux). 1.x series for Linux 2.2
Supports Linux 2.6 since April 2004 (3.1 series)

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

RTAI strengths

Community project (mainly European contributors now). Large user base.
Excellent documentation: beginners guide, articles, conference pages, user
and reference manuals...
Supported architectures (+ SMP support)

x86, PowerPC, ARM (StrongARM; ARM7: clps711xfamily, Cirrus
Logic EP7xxx, CS89712, PXA25x), MIPS, CRIS (ongoing)
License: LGPL (only requires RTAI source changes to be shared)
Modularity
Support for POSIX and other RTOS APIs (through Xenomai)
Hard realtime support in user space

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

System architecture

Hardware. Interrupt Controller

Realtime kernel

Realtime

task 1

Realtime

task 2

Realtime

task 3

Linux Kernel

Kernel modules

Linux interrupt handlers

User land

Hardware interrupts

Software interrupts

Linux process

(data acquisition)

Linux process

(GUI)

Realtime FIFO

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

RTAI concepts

RTAI adds a layer between the Linux kernel and the
hardware
The RTAI kernel manages realtime tasks according to
their priorities. The Linux kernel is also a realtime task,
with the lowest priority.
All non realtime interrupts are handed out to the Linux
kernel.

Very good introduction:

http://www.rtai.org/documentation/articles/guide.html

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

RTAI implementation

RTAI available as a kernel patch
RTAI implemented through Linux kernel modules:

rtai: core module
rtai_sched: realtime schedulers (3 available + other services)
rtai_fifos: realtime fifos for communication between realtime and Linux
user tasks.
rtai_shm: sharing memory between realtime and Linux tasks
lxrt: makes RTAI scheduler functions available to Linux
rtai_pqueue, rtai_pthread, rtai_utils: POSIX RTAI modules

Modular: you can only load the modules you need

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

RTAI usage

Download RTAI
Check the corresponding Linux version
Download Linux
Patch Linux with RTAI, compile it.
Reboot
Load the RTAI modules you need

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

RTAI example (1)

/* rt_process.c */
#include <linux/module.h>
#include <asm/io.h>
#include <math.h>
#include <rtai.h>
#include <rtai_sched.h>
#include <rtai_fifos.h>
#define TICK_PERIOD 1000000
#define TASK_PRIORITY 1
#define STACK_SIZE 10000
#define FIFO 0

static RT_TASK rt_task;

static void fun(int t)
{
int counter = 0;
float sin_value;

    while (1) {
        sin_value = sin(2*M_PI*1*rt_get_cpu_time_ns()/1E9);
        rtf_put(FIFO, &counter, sizeof(counter));
        rtf_put(FIFO, &sin_value, sizeof(sin_value));
        counter++;
        rt_task_wait_period();
    }
}

int init_module(void)
{
    RTIME tick_period;
    rt_set_periodic_mode();
    rt_task_init(&rt_task, fun, 1, STACK_SIZE, TASK_PRIORITY, 1, 0);
    rtf_create(FIFO, 8000);
    tick_period = start_rt_timer(nano2count(TICK_PERIOD);
    rt_task_make_periodic(&rt_task, rt_get_time() + tick_period, tick_period;
    return 0;
}
void cleanup_module(void)
{
    stop_rt_timer();
    rtf_destroy(FIFO);
    rt_task_delete(&rt_task);
    return;
}

Realtime task

Caution: written for RTAI 1.4.
May not work with your version.

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

RTAI example (2)

/* scope.c */
#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <signal.h>
static int end;
static void endme(int dummy) { end=1; }
int main (void)
{
    int fifo, counter;
    float sin_value;
    if ((fifo = open("/dev/rtf0", O_RDONLY)) < 0) {
        fprintf(stderr, "Error opening /dev/rtf0\n");
        exit(1);
    }
    signal(SIGINT, endme);
    while (!end) {
        read(fifo, &counter, sizeof(counter));
        read(fifo, &sin_value, sizeof(sin_value));
        printf(" Counter : %d Seno : %f \n", counter, sin_value);
    }
    return 0;
}

/* Run */

sync
insmod /home/rtai1.4/modules/rtai.o
insmod /home/rtai1.4/modules/rtai_sched.o
insmod /home/rtai1.4/modules/rtai_shm.o
insmod /home/rtai1.4/modules/rtai_fifos.o
insmod rt_process.o
./scope
rmmod rt_process
rmmod rtai_shm
rmmod rtai_fifos
rmmod rtai_sched
rmmod rtai

Userland program

Execution script

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

Adeos nanokernel

http://home.gna.org/adeos/index.en.html

Flexible environment to share hardware resources
between several operating systems (or several instances
of the same operating system).
Used by RTAI as a replacement Hardware Abstraction
Layer beneath the Linux kernel. Not impacted by the
RTLinux patent!
Also used for SMP clustering, patchless kernel
debugging...

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

Xenomai project

http://www.nongnu.org/xenomai/

Nanokernel mimicking APIs of traditional RTOS such as
VxWorks, pSOS+, and VRTXsa.
Can be plugged onto the Adeos layer, RTAI's RTHAL or
the LinuxThreads POSIX interface.
Goal: facilitate the porting of programs from traditional
RTOS to RTAI on GNU / Linux.
Now part of the RTAI project.

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

RealTime in Embedded Linux Systems

Commercial realtime distributions

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

Montavista (1)

Montavista Linux Professional Edition

http://www.mvista.com/products/pro/realtime.html

Employs some of the most active kernel hackers, in particular
on the ARM platform
All kernel development shared with the community kernel
core and drivers (Linux 2.6 example: preemption option,
many drivers...)
O(1) fixed overhead / fixed priority scheduler for realtime
running in front of the Linux scheduler. Handles realtime
POSIX pthreads, and leaves the other to the standard
scheduler.

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

Montavista (2)

Keeps the standard Linux API
Emulates VxWorks and pSOS API
Can be tried through a preview kit.
Supports an impressive list of processors
Tools to analyze performance and track down latency
sources.
Development tools seem to be proprietary

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

TimeSys

http://timesys.com

Exhaustive toolset as available from other vendors
(need to check details on their site)
Community friendly: share very interesting and generic
technical whitepapers and articles.
Free Software BSPs (Board Support Packages) available
True soft to hard realtime OS product
Development tools seem to be proprietary

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

LynuxWorks BlueCat

http://www.lynuxworks.com/products/bluecat/

Complete Embedded Linux distribution and toolset

Based on Linux 2.6 with standard soft realtime improvements

Comprehensive processor support.

Applications can run unmodified on LynuxWorks' LynxOS RTOS if
hard real time is needed.

LynuxWorks: a lot of experience in traditional RTOS. Same
development tools for BlueCat and LynxOS.

No visible contribution to the kernel. Looks like a traditional RTOS
vendor trying to surf the Linux wave (could be wrong!)

Proprietary development tools

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

Other realtime Linux vendors

Metrowerks

http://www.metrowerks.com/

Hardly mention Linux realtime support. More focus on
their nonLinux RTOS offerings

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

Benchmarking tools

LMBench:

http://www.bitmover.com/lmbench/

Takes several latency measures on your system
dbench:

ftp://samba.org/pub/tridge/dbench/

Produces filesystem load

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

RealTime in Embedded Linux Systems

Conclusion

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

Choosing a realtime Linux OS

Linux + patches
BlueCat
Montavista

Hard RT Linux
RTAI
Montavista

Standard Linux

Standard Linux API

Soft realtime:
100 µs magnitude

No realtime:
10100ms magnitude

Hard realtime:
~10 µs magnitude

System

Development complexity

Applying patches
Standard Linux API

Applying patches
Loading specific modules
Implement drivers for POSIX
or specific API

Latency

Simple

Complex

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com

References

http://www.realtimelinuxfoundation.org/

Community portal for realtime Linux

Real Time in Embedded Linux Systems

GNU Free Documentation License

http://freeelectrons.com