LinuxUser/October 2001 5 1

inside

Linux

Why are so many developers now
choosing to build their embedded
applications with Linux? Bill
Weinberg reviews the reasons

F

rom the business parks

of Silicon

Valley, to Wall Street, Linux was the
catchword of 1999 and 2000. All

signs pointed to Linux continuing as both a
hot financial and technological prospect
into the new millennium. While today, in
2001, IPO fever seems a distant memory,
there are still strong technical reasons to
consider using Linux for your next embed-
ded development project.

Embedded Linux offers measurable engi-

neering advantages, like broad hardware
support, scalability, excellent performance,
high reliability, and open APIs. There are
enticing financial reasons to employ Linux:
no run-time royalties, free or low-cost avail-
able software components, and freely avail-
able source code. Add to these some less tan-
gible business reasons to put a penguin into
your next internet appliance, control sys-
tem, or communications switch: engineers
like Linux and want to put Linux experi-
ence on their resumes, easing today’s hiring
crunch - and so do marketers, who continue
to be eager to package products with ‘Linux

Inside’. The long and short of it is that
embedded Linux, before and after the
dot.com meltdown, is the embedded plat-
form with lowest overall cost and the fastest
time to market. This article reviews these
and other reasons for building your next
embedded application with Linux.

The elusive device driver

Finding device drivers for embedded
designs based on a traditional embedded
OS, running on commodity-off-the-shelf
hardware, can be a frustrating exercise. In
the desktop and server world, new devices
emerge every six months or less. Peripheral
vendors themselves augment the OS offer-
ing with drivers and utilities that accompa-
ny their video cards, network interfaces,
and other devices. Smaller, more resource-
constrained embedded OS vendors struggle
to keep a few dozen device drivers and
board support packages current. I once
spent three days searching for a single-board
computer whose complement of peripher-
als was fully supported by the OS in use!
Even more challenging is getting an embed-
ded OS to pass the ‘Currys Test’ – walking
down the aisle of the local electronics retail-
er looking to find which devices, if any, a
traditional embedded OS supports.

Traditional RTOS vendors employ two

strategies to support the hardware chosen
by their customers: either charge up to
US$40,000 in NRE/consulting fees (and
then retain the driver code themselves), or

‘borrow’ drivers from, you guessed it, Linux!
And, they still must invest substantial time
and (customers’) money to port the inter-
faces to their proprietary, closed systems.

Contrast the availability of drivers for

Linux: with very few exceptions, device
interface code for Linux appears quite regu-
larly along with or even prior to the release
of the new devices themselves. For example,
several years ago a client of mine in
Australia needed an Ethernet interface dri-
ver for the proprietary OS they were using.
The chip manufacturer, although already
shipping silicon, was keeping the specifica-
tion under wraps. Rather than haggle with
the vendor, together with a consultant I eas-
ily found a Linux driver for the device, and a
rapid port ensued. The client was so
impressed that last year they switched to
embedded Linux for their application plat-
form.

Scaling Linux

Traditional embedded operating systems
tout the size and efficiency of their kernels.
Even today, when memory is only available
economically in increments of 4MB or
16MB, vendors still extol their ‘complete
micro-kernel solutions’ in 50KB.
Realistically, viable commercial OS configu-

5 2

LinuxUser/October 2001

EMBEDDED LINUX

rations come in at 128-256KB for a reason-
ably configured kernel, another 100-200KB
for a TCP/IP stack and sockets library, and
for a web appliance, 50-300KB for a HTTP
server, plus a minimum 64KB of working
RAM. Thus, an embedded system software
profile of 800 KB to 1MB no longer looks
gargantuan.

These same vendors point out that a

desktop distribution of Linux runs into the
hundreds of megabytes. Well, they are right.
However, embedding an OS like Linux is
more like lunching at a good cafeteria than
dining at an upscale prix fixe establishment.
When you embed Linux, you choose only
those components that make sense for your
application. Don’t need a read/write file sys-
tem? Don’t use it! The same logic applies to
networking, GUI, shells, and countless
other utilities and libraries. If your project
does need more functionality than fits into
local non-volatile storage, you can craft
either a tiny Linux boot loader, a stand-
alone bootable Linux system, or a slim
Linux kernel that pulls down additional
modules and application code over a net-
work, often in 500KB or less.

Linux and real-time

Many (but not all) embedded applications
have some sort of performance or respon-
siveness requirement. However, the latency-
sensitive portions of embedded code are
usually dwarfed by purely throughput-sen-
sitive code that implements control algo-
rithms, communications protocols, and
user interfaces. Moreover, many of these
real-time requirements prove to be ‘soft’ –
missing a deadline once in a while does not
impact the overall system viability. Even
when ‘hard’ real-time deadlines do exist (in
less than 10 per cent of embedded applica-
tions), the scope of deterministic response
can be reduced to the driver level or over-
come by the ‘real-fast’ performance offered
by combining Linux with Pentium,
PowerPC, or MIPS silicon.

So, when developers choose to embed

Linux to leverage the wealth of available
networking, database, and interface soft-
ware, real-time concerns often take a back
seat. But should they? Using a general-pur-
pose operating system (GPOS), like
Windows NT, for real-time and embedded
applications, can spell disaster – recall the
US Navy’s having to tow their Aegis destroy-
ers back to port, repeatedly, because of
crashed Windows NT steering systems. A
GPOS typically suffers from several chal-
lenges to real-time applicability: determin-
ism in general, and response under load in
specific. GPOS schedulers, optimised for
time-sharing, can induce unpredictably
long blocking times (up to 500 millisec-
onds!); drivers developed by a mix of GPOS-
vendor engineers, peripheral-board ven-
dors, and other third parties add their own
variable latencies. The whole system may

respond in a sprightly fashion with a hand-
ful of threads and a couple of interrupting
devices, but scale either significantly and
watch responsiveness drop and jitter hit the
ceiling.

Linux, developed for desktops and

servers, is also a GPOS, but enjoys great suc-
cess in embedded designs with real-time
requirements. Three primary paths exist to
providing a real-time Linux: (1) tuning key
device drivers and kernel functions (ad
hoc), (2) inserting a second kernel into the
system, and (3) refining the standard Linux
scheduler and using a preemptible version
of the Linux 2.4 kernel, such as the open
source project sponsored by MontaVista.

Before even attempting to enhance

Linux responsiveness, it is key to measure its
real-time performance, thoroughly, in terms
familiar to real-time/embedded designers:
worst-case interrupt latency, context switch,
maximum blocking times, and most impor-
tantly, preemption or task response latency.
Simply knowing how well Linux responds
on a particular platform in many cases obvi-
ates the need for additional re-architecting.
Also, Linux already enjoys provably superi-
or compute and networking performance
throughput, even when compared to sup-
posedly lightweight RTOS products, with
good-to-excellent average response times,
even under load.

The first approach above (tuning) has

been applied to GPOS applications for years,
with deployed examples on Solaris, and
Windows NT; indeed such tuning is a mat-
ter of course for many so-called RTOS plat-
forms as well. The second path (kernel sub-
stitution), which has also been applied to
Windows NT (by Radisys and VenturCom),
presumes that to use Linux for real-time,
you must first ‘throw it out’. The addition of
a second OS, regardless of its putative real-
time characteristics, complicates both the
development and run-time considerations
of the embedded Linux developer. The sec-
ond OS introduces: a second set of RTOS-
specific APIs; the loss of Linux process and
threads programming model; a ‘flat’ memo-
ry model without benefit of memory pro-
tection; starvation of applications running
in the ‘normal’ Linux environment; and
stratification of Linux applications and dri-
vers into poorly-defined non-real-time and
real-time classes.

The much more sensible, third approach

is to optimise the existing, open Linux code
base to address the needs of actual applica-
tions. (For additional information on lever-
aging the Linux SMP kernel for preemptible,
deterministic performance, see Key Links).

Legacy code

Application code comes to an embedded
Linux environment from one of three
sources: existing code written for a commer-
cial RTOS, code written for an in-house pro-
prietary kernel or executive, and code devel-

Embedded Linux, present and
future

Key enabling technologies are now emerging and
maturing to foster the mainstreaming of Linux in
embedded applications.

Broadened processor support

Prior to the year 2000, most Linux distributions
focused on Intel Pentium. The last two years have
brought support for a range of embedded 32-bit
x86 and PowerPC CPUs, including the low power
Crusoe and Jeode, and high power and high integra-
tion PowerPC processors from Motorola (74xx, 8xx
and 82xx) and IBM (4xx). They’ve been joined by
ports for half a dozen MIPS-family and ARM archi-
tecture chips popular in networking, gaming, and
office automation, like Alchemy’s Au1000, NEC’s VR
family, Intel’s StrongARM, XScale, and the ARM
720/920. Add to this list emerging Japanese CPUs
like SH-3 and SH-4 and you see that Linux is begin-
ning to rival and even outstrip traditional RTOS plat-
forms for CPU coverage.

Open source flash file systems

Closed source enabling flash technology has been
joined by reliable, compressed and wear-levelled file
systems like CramFS and JFFS.

CompactPCI-based HA

Ports of Linux today exist for CPCI systems from
vendors like FORCE, Motorola, Radisys, and Ziatech,
now offer high availability frameworks with PCIMG
hot swap support, with maturing models for Linux
hot swap infrastructure (PICMG 2.14) and hot swap
aware drivers (HSADs), bonding redundant network
drivers, backplane communications, and multiple
options for clustering.

Enhanced real-time support

With the advent of the version 2.4 Linux kernel and
options for preemptibility, Linux has become more
real-time/deterministic in performance (see above)
with average preemption latencies under five
microseconds.

More options in tools

Traditional cross-development tool vendors and
open source projects are at last delivering both
workstation Linux hosting and embedded Linux tar-
geting tool sets for C, C++, Java, IDEs, and high-
level design tools that complement or compete with
GNU. Examples include IBM Visual Age Micro
Edition, Kdevelop, Greenhills Multi for Linux,
MetroWerks Code Warrior, I-Logix, Rational, and
others.

Embedded Linux tools

Developers are now leveraging tools targeted
specifically at embedded Linux, including GUI-
based scaling/configuration tools (like MontaVista’s
Target Configuration Tool and other vendors’ wiz-
ards), small GUIs (like TrollTech’s Qt/embedded and
MontaVista Hard Hat Graphics), and tiny web
servers and reduced footprint browsers (like
Netfront, Opera, and Mozilla)

These developments and others, technical and mar-
kets-based, have brought embedded Linux into the
traditional domains of the proprietary RTOS. Linux,
with its wealth of networking software, very natural-
ly accompanies the Internet revolution as host to
applications like Internet-enabled multimedia,
SOHO office appliances, and home network gate-
ways, as well as finding its way into both shelf and
card controllers for large communications switches,
routers, and other infrastructure equipment.
Industrial control, robotics, instrumentation - all tra-
ditional domains of the proprietary RTOS, are now
implementing and deploying Linux-based systems.

EMBEDDED LINUX

oped for Unix-compatible workstations and
servers.

Applications running on both commer-

cial and in-house kernels share a key charac-
teristic: proprietary APIs. While several off-
the-shelf kernel products enjoy extensive
design-in success, few if any offered pro-
grammers viable open applications pro-
gramming interfaces, like POSIX or the
cross-industry openness of the various
Linux APIs. Moreover, embedded OS ven-
dors had little interest in supporting open
APIs or any other tools that might facilitate
customers’ porting applications from their
kernels to those of other vendors or to open
platforms. In-house kernels, developed for
very specific target applications, either had
no perceived need for openness and porta-
bility, or arose with and were perpetuated
by NIH (‘not invented here’) sensibilities.

Today, embedded and real-time systems

designers, excited by the benefits of Linux
and the Linux explosion in general, are
eager to build their next-generation prod-
ucts on an embedded Linux. However excit-
ed, they still must face the challenge of port-
ing existing code to Linux.

Porting Unix applications to Linux often

is as simple as typing ‘make’ or using a GNU
configuration package. The availability of
POSIX (IEEE1003), System V, Berkeley, X11,
and other standard APIs on Linux makes
porting incredibly easier, if not always 100
per cent transparent. Challenges arise from

dependencies on semi-proprietary exten-
sions (vendor and platform specific) or from
use of superceded open APIs (like POSIX.4
or other POSIX draft APIs). Such challenges
are very much akin to those faced when
porting proprietary code.

While many embedded developers use

the occasion of moving to Linux as an
opportunity to rearchitect their code, there
still can arise the need to move existing pro-
grams with proprietary system and library
calls, ad hoc/in-line device access, explicit
interrupt/preemption disabling, and so on,
to Linux. Solutions exist for accommodat-
ing the most frequently-used calls and
programming practices (see box on this
page, Accommodation Solutions).

The embedded Linux business model

Embedded systems designers have always
reluctantly faced to prospect of purchasing
development tools. First came the sticker
shock from per-seat proprietary tool sets
priced from US$15,000 to US$20,000. Next
came often-obligatory service contacts, usu-
ally of dubious real value. And last came the
haggling over run-time pricing and over the
cost of source code (which can run over
US$250,000). While some typically low-end
embedded OS vendors offer run-time buy-
out options, most impose a heavy deploy-
ment royalty, not just on the kernel but on
every useful component. These run-time
costs significantly raise real development

costs, and impact the cost-
of-goods-sold of your prod-
uct, thinning sometimes-
precarious margins and
threatening your time-to-
volume.
Contrast the embedded
Linux business model.
Open tools, open kernel,
open utilities, greatly
reduced (but not eliminat-
ed) cost of development
and deployment.
Moreover, most commer-
cial embedded OS vendors
enjoy only a 25-30 per cent
support renewal rate (com-
pare the 85 per cent-plus
subscription renewal
enjoyed by open source

Accommodation solutions

It can be necessary to move existing programs to Linux complete with pro-
prietary system and library calls, ad hoc/in-line device access, explicit inter-
rupt/preemption disabling, and so on. Solutions for accommodating the sev-
eral dozen most frequently-used calls, as well as programming practices,
breaks out as follows:

Task creation, scheduling, management

use POSIX/Linux threads

IPCs (queues, mutexes, etc.)use POSIX .1 and SVR4 equivalents

Networking

use equivalent Berkeley sockets

Pre-emption disable

migrate to driver / rearchitect

In-line interrupt management

migrate to driver / rearchitect

Direct memory access

either migrate to driver or use POSIX .1b mmap()

For developers not wishing to re-code to track the above, inter-OS transla-
tion libraries do exist for at least large APIs subsets. MontaVista Software
and other companies provide RTOS porting and emulation kits, and a num-
ber of independent vendors, like Migratec, focus on the inter-OS migration
process.

vendors like MontaVista). Embedded Linux
suppliers cannot afford ‘drive-by’ selling
strategies – their commercial viability
accrues only from the quality of the distrib-
ution products and services they offer, not
customer lock-in from proprietary intellec-
tual property.

Proprietary embedded OS vendors

attempt to front-load their commercial rela-
tionships (even those charging royalties),
and thereby have little or no interest in see-
ing a project through to completion.
Embedded Linux suppliers must (and do)
prove their worth from the start to impart
value to their offerings (beyond what one
gets from the newsgroups), today, and
tomorrow. The benefit to you as an embed-
ded developer is a supplier who lives or dies
upon your project’s success.

Linux cachet

In the past, choice of embedded software
components was a closely-held trade secret;
implementation specifics, including the
embedded OS, could confer significant
competitive advantage. Marketers and
embedded OS vendors despaired of ever
being able to crow about design wins –
either the project languished, or the appli-
cation designer remained tight-lipped when
it came to PR.

Compare new designs based upon Linux

– when before have high-tech product mar-
keters touted their companies’ choice of
operating system? ‘Linux Inside’ is now
appearing on the outside of products.
When have manufacturers been able to use
their embedded OS choice as a hiring incen-
tive? Linux expertise continues to be in
high demand in the US, Europe, and Asia.
And when have inherent economies freed
the embedded developer from being a
hostage to imperious tools and OS vendors?
Today. That’s when, with embedded Linux.

Bill Weinberg is director of strategic marketing at
MontaVista Software

Key links

Leveraging the Linux SMP kernel for preemptible,
deterministic performance

www.mvista.com/realtime

Linux kernel preemption project

http://sourceforge.net/projects/kpreempt