Project co-funded by the European Commission as a Specific Support Action within the 6th Framework
Programme. ISSeG began in February 2006 and will run for 26 months.
Copyright © CERN, 2006. Member of the ISSeG Collaboration.
Project no: 026745
PUBLIC 1
/
59
Project No: 026745
I S S e G
Integrated Site Security for Grids
Specific Support Action
Information Society and Media
F
INAL REPORT ON DEPLOYMENT AT
CERN
EU DELIVERABLE: D1.1.4
Document identifier:
ISSeG-Del-D1.1.4-710058-v3.0.doc
Due date of deliverable:
End of Month 22 (30/11/2007)
Actual Submission Date:
10/01/2008
Lead Partner for this deliverable:
CERN
Document status:
FINAL
Abstract:
This final report documents the results obtained at the end of the 22 months of Integrated Site
Security (ISS) deployment at the CERN site. It describes results since D1.1.3 [R3], the
intermediate deployment report, which reported the status nine months into deployment.
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC 2
/
59
Copyright © CERN, 2006. Member of the ISSeG Collaboration.
ISSeG (“Integrated Site Security for Grids”) is a project co-funded by the European Commission as a
Specific Support Action within the 6th Framework Programme. ISSeG began in February 2006 and
will run for 26 months.
For more information on ISSeG, its partners and contributors please see
You are permitted to copy and distribute, for non-profit purposes, verbatim copies of this document
containing this copyright notice. This includes the right to copy this document in whole or in part, but
without modification, into other documents if you attach the following reference to the copied
elements: “Copyright © CERN, 2006. Member of the ISSeG Collaboration. See
for details”.
Using this document in a way and/or for purposes not foreseen in the paragraph above requires the
prior written permission of CERN.
The information contained in this document represents the views of CERN as of the date they are
published.
THE INFORMATION CONTAINED IN THIS DOCUMENT IS PROVIDED BY CERN “AS IS”
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL CERN, THE OTHER MEMBERS OF THE
ISSEG COLLABORATION OR THE EUROPEAN COMMISSION BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
ANY WAY OUT OF THE USE OF THE INFORMATION CONTAINED IN THIS DOCUMENT,
EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
All trademarks (registered or unregistered) mentioned in this document are the property of their
respective owners. Their use in this document is not intended in any way to infringe on the rights of the
trademark holders. Their use herewith is to merely describe the goods or services to which the mark
relates.
Delivery Slip
Name
Partner Date
Authored by
Matthais Schroder; Ivan Deloose; Stefan
Lueders; Rafal Otto; Judy Richards; Denise
Heagerty; Edoardo Martelli; Zbigniew Stanecki;
Lionel Cons; Tom Payne; Emmanuel Ormancey;
Kate Bradshaw; Sebastian Lopienski; Romain
Wartel; Jean-Michel Jouanigot; David Gutierrez
Rueda.
CERN 28/09/07
Edited by
Kate Bradshaw, Denise Heagerty
CERN
31/10/07
Reviewed by
Denise Heagerty; Emmanuel Ormancey;
Stefan Lueders; David Myers; Lionel Cons;
Tobias Koenig
Philippa Strange
CERN,
FZK,
STFC
28/11/07
Approved by
Project Board
28/11/07
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC 3
/
59
TABLE OF CONTENTS
1
Executive summary .......................................................................................................4
2
Introduction ....................................................................................................................5
3
Resource management extensions deployment results ...........................................6
3.1
Extend central systems management......................................................................................6
3.2
Enforce account policies and procedures .............................................................................10
4
Network connectivity management deployment results .........................................12
4.1
Strengthen network perimeter security.................................................................................12
4.2
Integrate and extend network perimeter management tools.................................................13
4.3
Protect devices on dedicated networks.................................................................................16
5
Security mechanisms and tools deployment results...............................................19
5.1
Investigate security assessment tools suitable for the CERN environment..........................19
5.2
Enhance CERN’s incident detection capability ...................................................................20
5.3
Evaluate multi-factor authentication technologies ...............................................................21
6
Administrative procedures and training deployment results..................................25
6.1
Document and strengthen account administration procedures .............................................25
6.2
Prepare and implement a training plan for improving knowledge of computer security
within the organization.........................................................................................................25
6.3
Review and update security policies and procedures ...........................................................26
7
The CERN final deployment report within the ISSeG project..................................28
8
References....................................................................................................................28
9
Acronyms and Abbreviations .....................................................................................29
Annex
A1
Resource management extensions............................................................................32
A1.1
Update on the CERN Computing and Network Infrastructure for Controls (CNIC).........32
A2
Network connectivity management............................................................................37
A2.1
Firewalling beyond 10Gbps...............................................................................................37
A3
Security mechanisms and tools .................................................................................49
A3.1
Security assessment tools and framework suitable for the CERN environment ................49
A4
Administrative procedures and training....................................................................53
A4.1
Web applications security, CNL Article, November 2007 ................................................ 53
A4.2
Suggestions for designing and writing secure software, CNL Article, September 2007 ... 56
A4.3
How to avoid a false sense of security, CERN Courier, April 2007 .................................. 59
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC 4
/
59
1 Executive
summary
This final report describes the tasks performed at CERN up until the end of November 2007.
It follows on from the intermediate report, D1.1.3 [R3] so that the full 22 months of ISS
deployment at CERN have been reported.
The tasks were categorised into the following four implementation areas:
I1 Resource
management
extensions
I2 Network connectivity management
I3 Security mechanisms and tools
I4 Administrative procedures and training
Experience from the tasks of I1 has confirmed that centralised management of resources
improves both the consistency and reliability of applying security patches and configurations.
This eases the process of keeping enforcement in line with policies. For large groups of
homogeneous systems a gain in overall resources can also be achieved as fewer system
administrators are required.
The tasks of I2 have strengthened the perimeter firewall and protected sensitive devices.
Dedicated networks have been implemented as part of a ‘defense in depth’ security model
targeting control systems. A well designed set of firewall management tools and workflow
allowed the smooth closure of firewall ports without impacting existing services. The
dynamic nature of Grid collaboration requires firewall configurations to be regularly updated
and the tools have proved well adapted to this requirement.
The tasks of I3 have provided a necessary adaptation to evolving technology and attacker
techniques. Netflow data has proved to be a successful basis for IDS on multiple 10Gbps
links. A framework has been designed and implemented so that existing and new
vulnerability assessment tools can be easily integrated. Web applications, in particular, have
been identified as a growing target for attackers. Multi-factor authentication has been
implemented using three different mechanisms: smartcards with certificates, OTP (One Time
Password) tokens, and OTP sent as SMS messages to mobile phones. Each mechanism is
well adapted to the community it serves, demonstrating the importance of the requirements
phase before choosing a solution.
The tasks of I4 have updated security procedures and improved awareness of security best
practices. A training plan was implemented and integrated into the organization’s structures.
Presentations and FAQs have been well received and adapted to their target audiences.
Checklists have provided a useful entry point to more detailed training for users, system
administrator and software developers. A single repository from which to link all policies and
procedures has proved to be an essential starting point.
Through this experience, the ISSeG project is able to provide lessons learnt and advice for
other sites wishing to implement Integrated Site Security (ISS). This advice will take the form
of recommendations and training material available on
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC 5
/
59
2 Introduction
This document contains additional lessons learnt since D1.1.3 [R3]. Where useful for
readability and consistency, some background information is repeated.
Each deployment task follows a clear structure in this final report:
• Security reason: describes the security problem being addressed and the security
improvements the task in question is intending to achieve.
• Implementation experience: describes the task and the experience gained from its
implementation. It can include the benefits of a particular approach, as well as difficulties
encountered and how they were overcome.
• Lessons learnt: provides a bulleted list of ‘hints’ that other sites should consider when
implementing this task. Where relevant, these include the resources required and the
suitability for other sites. The aim of these lessons learnt is that they will be extracted and
used within the recommendations of the ISSeG website
• Useful links: lists any useful URLs for someone implementing the task
• Training: lists any useful training and background information for this task.
For background on the CERN site and ISS deployment, see the CERN strategy,
implementation and intermediate deployment reports [R1-3].
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC 6
/
59
3
Resource management extensions deployment results
Below are the results achieved for Implementation Area I1: Resource management
extensions. Where relevant, tasks have been divided into sub-tasks and these are reported
either separately or together, as appropriate.
3.1
Extend central systems management
This task contains the following sub-tasks:
• Provide tools for managing homogeneous groups of Linux desktop systems
• Investigate management tools to enforce security for Linux control systems
• Extend the flexibility of the central management of Windows systems
• Investigate management tools to enforce security for Windows control systems
• Reduce the need for Windows administrator privileges on centrally managed systems
3.1.1
Provide tools for managing homogeneous groups of Linux desktop systems
Security reason
Individually managed systems offer no mechanism for automatically applying security
policies, such as firewall parameters or versions of packages to install. Configuring each
system separately has led to errors, inconsistencies and security incidents.
Implementation experience
The task of adapting the quattor central management system for use with homogeneous
groups of Linux desktops has been described in D1.1.3 [R3]. The templates developed for
that purpose have proved useful for configuration management, such as NFS mounting of
home directories. For software distribution however the gain is less clear. A tool, such as
yum, provides software distribution functionality with little overhead for the initial setup. In
practice, it has been more convenient to distribute new versions of software using yum
instead of modifying the quattor templates. This does not however give the global overview
of the software status across the desktops being managed.
From a security perspective, central configuration management provides a mechanism for
ensuring a consistent implementation of policies. For a very dynamic environment where the
configuration of a group of desktop computers regularly changes the initial investment
overhead is easily compensated for. For mainly static environments, the investment may not
pay off unless the number of homogeneous computers is high, e.g. at least more than 20.
Lessons learnt
• Central management tools improve the consistency of configurations across a group of
homogeneous computers.
• The initial investment to learn a central management tool is quite high.
• System administrators need to be motivated to move from a known environment to a
central management tool.
• Software distribution is best managed by standard Linux tools such as yum or apt.
Useful links
• quattor
http://linux.duke.edu/projects/yum/
• APT
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC 7
/
59
Training
Training is required to learn the central management tool both for its initial setup and for
making configuration updates.
3.1.2
Investigate management tools to enforce security for Linux control systems
This has been fully reported in D1.1.3 [R3].
3.1.3
Extend the flexibility of the central management of Windows systems
Security reason
Individually managed systems have been a common source of security incidents mainly due
to lack of patches, anti-virus protections, insecure configurations and end-user habits. The
centrally managed Windows service automates and improves security but has been viewed as
too rigid for CERN’s academic environment and so has had limited adoption by parts of the
organization. The central management needs to become more flexible in order to support a
wider range of requirements and thus reduce the number of individually managed systems.
Implementation experience
CERN has designed and implemented a management system for Windows computers called
Computer Management Framework (CMF). As described in D1.1.3 [R3], CMF provides
improved security, such as stricter control on patch deployment, reboot actions and on instant
reporting. Machines can be grouped into Named Sets of Computers (NSCs) and assigned
roles by means of package assignment. A package is a collection of applications, policy
settings and scheduled tasks. A secure delegation mechanism allows administrators to partly
or globally delegate their groups of machines and packages to other people. A computer can
therefore be completely locally managed by the administrator of a specific activity or can be
centrally managed including additional specific functionality if required. CMF provides an
intuitive user interface to schedule installations within the limitations defined by the
administrator.
CMF was initially intended to be used in the controls environment, but due to its success,
CMF has been implemented on all CERN Windows computers since the end of 2006. Some
minor changes were required to CMF to make it Terminal Servers compliant. It was also
adapted to run on 64-bit platforms. It now completely replaces a commercial deployment
technology that was used beforehand on the centrally managed desktop Windows PCs, with
the result that CMF packages can be easily shared between the controls, standard desktop and
other specific environments, such as Computed Aided Engineering.
Lessons learnt
• Commercial central management systems may provide sufficient functionality for other
sites.
• The overhead of developing and maintaining a more flexible system needs to be
considered and sufficient resources allocated.
Useful links
• The CMF web site, describing the full functionality, is available at
• More information about Windows at CERN can be obtained at
• Microsoft Active Directory Wikipedia article
http://en.wikipedia.org/wiki/Active_Directory
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC 8
/
59
Training
Although CMF requires minimal prior training, a series of awareness campaigns and training
sessions have been held for users, operators, and system experts at CERN. Monthly meetings
have provided a useful forum for questions and discussions.
Specific information for CMF administration is available at
http://cern.ch/cmf/help/?fdid=4
3.1.4
Investigate management tools to enforce security for Windows control systems
Security reason
The security risk for controls equipment is rising with the change of trend from dedicated
hardware and protocols, only known by specialists, to standard off-the-shelf equipment and
protocols, vulnerable to the ‘standard’ cyber attacks. Control systems, based on the Windows
operating system, are used to drive and monitor sensitive equipment.
The experience at CERN has shown that office PCs that are centrally managed/patched are
much less likely to be infected or compromised than office PCs that are managed by
individual users. However, central methods to manage controls PCs have failed in the past to
cope with the different nature of controls PCs compared to office PCs. Controls PCs cannot
be switched off or rebooted without proper scheduling by the corresponding system expert.
Also, patches cannot be applied that easily since controls applications might not be compliant
with a particular patch, or software licenses to run controls applications might become
invalid.
Implementation experience
Since none of Microsoft’s genuine installation tools provided the flexibility needed, CERN
has developed its own installation scheme called Computer Management Framework (CMF).
While the operating systems, antivirus software, and basic software applications should
continue to be centrally managed and maintained by the IT department, it is up to the control
system experts to take over full flexibility of the configuration of the PCs of their control
system, and full responsibility for securing them. Such a scheme will also help the experts to
recover e.g. from a security incident.
CMF has proven to be a big success: many control system experts at CERN are now using
CMF to install, configure, and manage their Windows control PCs. Corresponding users have
expressed great satisfaction. As a result, CMF is now even used to install standard office PCs,
and Windows Terminal Servers. However, the complexity of such a development and the
resources required must not be underestimated. Furthermore, since control system experts
should have the ultimate expertise on their system, a tight communication between them and
the IT department must be established. This avoids IT experts applying changes to their
scheme without tight scheduling and coordination (e.g. changing CMF itself or the
underlying structure of the Windows Active Directory, global group policies, or central
settings).
Experience with central management as well as other security improvements for control
systems is reported in Annex A1.
Lessons learnt
• Computer installation schemes must cover all major platforms used.
• For synergy reasons, use a single system to manage office PCs, control PCs and other
types of computers.
• The responsibilities of the IT department and the control system experts must be clearly
defined.
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC 9
/
59
Useful links
• CMF Computer Management Framework
• The CNIC presentation at the ICALEPCS 2007 Conference
http://ics-web4.sns.ornl.gov/icalepcs07/WPPB38/WPPB38.PDF
(included in Annex,
section A1.1)
Training
The initial setup requires knowledge of the structure of CMF, and the work flow for
installation and configuration changes as well as for patching. The control systems experts
will then need user level training.
3.1.5
Reduce the need for Windows administrator privileges on centrally managed
systems
Security reason
When a user has administrative privileges, there is a risk of compromising the computer at
every execution of code from a non-trusted origin (e.g. mail attachments or web browsing).
The malware executing with the user’s credentials has rights to install other applications,
change registry keys and other system settings. In this case the only reliable way to dispose of
the malware is to reinstall the compromised computer from scratch.
Therefore, there is a need to adopt the principle of least privilege and try to avoid putting
users into the local Administrators group of a Windows client computer.
Implementation experience
In 2005 CERN developed a solution called ‘CERN Non-Admin’, which was deployed on
Windows XP computers. The solution is well described in D1.1.3 [R3]. Detailed
documentation can be also found at
https://cern.ch/winservices/Help/?kbid=010121
. The idea
was to avoid putting users into the local Administrators group by default and just to provide
them with an easy way to elevate privileges temporarily when required. As this new
configuration significantly changes user habits, it was decided not to deploy it CERN-wide in
one go. It was considered that such change should be combined with other important changes
that the user would be making anyway. Therefore, the ‘CERN Non-Admin’ solution was
deployed only when the Windows XP operating system was reinstalled on a computer. With
such a deployment policy it is now configured on around 1/5 of CERN centrally managed
computers.
During the summer of 2007 CERN introduced Windows Vista. With this new version of the
operating system, there is no more need for the ‘CERN Non-Admin’ solution. Vista comes
with User Account Control (UAC) technology, which implements the same functionality.
UAC is a component of the Vista kernel and is properly configured with the default Vista
settings. A user who is a member of local administrators group runs all processes with
standard user privileges. Whenever any process needs some administrative rights, the UAC
prompts the user to approve elevation of privileges. A user explicitly sees which process
requires some additional rights. In cases where a malware may be trying to compromise the
computer, a user can stop the elevation of privileges.
Lessons learnt
• Working without administrative privileges on Windows XP is quite acceptable for
‘Office Users’, who normally never need administrative privileges, while for developers
or administrators it can be quite difficult. Therefore it is recommended to introduce the
solution on a voluntary basis.
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
10 / 59
• Several issues have been identified for Windows XP. These are documented together
with workaround solutions in Annex A1.3 of D1.1.3 [R3].
• Start deploying Windows Vista with UAC enabled, which will teach users the new way
of working from the beginning.
Useful links
• The ‘CERN Non-Admin’ documentation
https://cern.ch/winservices/Help/?kbid=010121
• Windows Vista User Account Control Step-by- Step Guide
• Windows Vista’s UAC functionality Wikipedia article
http://en.wikipedia.org/wiki/User_Account_Control
• CERN UAC documentation page
https://cern.ch/winservices/Help/?kbid=020201#_Toc175137651
Training
• For ‘CERN Non-Admin’ for Windows XP a developer needs to know C# programming.
In addition, developers require Windows explorer shell knowledge and Windows
administration skills. Additional training is also recommended for the end users so that
they are aware of the need for the change.
• For Windows Vista UAC implementation general Windows Vista administration
knowledge is required. End users should receive training on UAC together with general
Vista training, as UAC is one of the most visible new components in this new version of
the Windows operating system.
3.2
Enforce account policies and procedures
This task contains the following sub-tasks:
• Implement strengthened account procedures for creation, deletion and block actions
• Close accounts that do not conform to the strengthened policies
3.2.1
Implement strengthened account procedures for creation, deletion and block
actions
This has been fully reported in D1.1.3 [R3].
3.2.2
Close accounts that do not conform to the strengthened policies
Security reason
User accounts provide access both to computer systems themselves as well as protected
information and applications. It is essential to ensure that accounts are de-activated once the
owner is no longer eligible for the account.
Implementation experience
In order for accounts to be closed in a timely manner, it is almost essential to have automatic
procedures that can access the relevant HR data concerning contract termination. When
introducing such procedures there will probably be a significant number of historical cases to
clean up. At CERN the introduction was done in 3 phases. The initial phase, closing of
accounts that appeared inactive, was reported in D1.1.3 [R3]. Since then the policies have
been enforced for all accounts:
• Automatically closing accounts as people reached the end of their contract.
• Closing accounts that were still active for people who were no longer eligible for them.
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
11 / 59
Globally no serious problems were encountered. The increasing security risks and consequent
need for stricter rules and automatic enforcement policy were presented to key user
communities before implementation started and reminders were given before deployment.
The procedure was also aided by the following features:
• An email warning was sent to the user and his/her supervisor a month and a week in
advance. This also reminds the supervisor that provisions must be made for the
professional data that the user may have.
• The user could set up an email forwarding address before (s)he left.
• For certain categories of staff, a grace period of 2 months was given to allow for a
smooth transfer. A significant proportion of people leaving would have a physical re-
location to another country and/or continue their collaboration with CERN in another
form.
Users accepted the need for stricter rules. The main problems came from people leaving on
retirement. CERN’s policy allows limited personal use of its computing facilities and this has
been particularly used by a generation who have had limited use of computers at home and
continued to use CERN facilities after their retirement.
Lessons learnt
• If you are implementing a significant change in policy or practice and/or have a major
backlog to clear up, start your publicity early and repeat it. Today most people accept the
principle but can get very upset if they are unaware that they will be affected.
• Providing forwarding of email is very much appreciated.
• The procedures for deactivating accounts need to also define and document what happens
to resources (e.g. home directories, data files, web sites, network devices) owned by these
accounts.
• If you have allowed private use, pay special attention to the case of people who are
retiring, who may have devoted almost all their professional life to your organization and
may be partly using your services as their Internet service provider.
• Account closure procedures need to be carefully defined and implemented.
Useful links
• CERN’s account management procedures
http://cern.ch/ComputingRules/procedures/accountmanagement.php
Training
• You need to understand the different categories of people affected and eventually adapt
the procedures to their and the organizations requirements.
• On the technical side the task requires scripting skills and the ability to write small
applications that interrogate a database.
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
12 / 59
4
Network connectivity management deployment results
Below are the results achieved for Implementation Area I2: Network connectivity
management. Where relevant, tasks have been divided into sub-tasks and these are reported
either separately or together, as appropriate.
4.1
Strengthen network perimeter security
This task contains the following sub-tasks:
• Review and remove obsolete perimeter access entries
• Propose a mechanism for maintaining the validity of perimeter access entries
• Extend the number of TCP (Transmission Control Protocol) and UDP (User Datagram
Protocol) ports closed by default from off-site
4.1.1
Review and remove obsolete perimeter access entries
This has been fully reported in D1.1.3 [R3].
4.1.2
Propose a mechanism for maintaining the validity of perimeter access entries
This has been fully reported in D1.1.3 [R3].
4.1.3
Extend the number of TCP (Transmission Control Protocol) and UDP (User
Datagram Protocol) ports closed by default from off-site
Security reason
There are constant attacks from the Internet on computers without perimeter firewall
protection. In addition, attackers can use open firewalls at a site to gain access to ‘backdoors’
created by a virus and other malicious software. Implementing a firewall at the site perimeter
with TCP and UDP ports closed by default, provides an essential layer of protection while
leaving the flexibility to open ports for authorised applications.
Implementation experience
As described in D1.1.3 [R3], a schedule was defined for closing ports in the perimeter
firewall. An analysis was made of the existing traffic in order to identify the computers most
likely affected by the port closures. The people responsible for those computers were then
notified and given the possibility to make a firewall authorisation request (see Annex A2.2 of
D1.1.3 [R3] for an example of a firewall authorization request form). Most of the port
numbers below 1024 had been closed historically. For the remaining ports, the schedule
targeted first those considered the highest risk. For example, attacks had been experienced on
some server ports and some other rarely used ports had been used as backdoors by viruses in
the past. Having ensured that the closure procedures worked well, popular ports (mainly non-
standard ports used by web servers) were targeted next. Finally, dates were set for closing all
remaining ports.
The port closures were spread over a period of several months and went very smoothly. The
success of this possibly disruptive change was certainly due to the good collaboration and
communication channels set up with the service managers. Strong management support
ensured that sufficient priority was given to the task. Announcements were made through the
official communication channels of the organization as well as directly to those involved. The
schedule was agreed by a site-wide committee mandated to communicate information within
their departments. Technical experts were available to investigate complaints of broken
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
13 / 59
applications and an exception mechanism was foreseen to allow critical applications to run
under old firewall settings. This allowed investigations while minimising disruption to
services.
While deep technical knowledge is required to debug firewall related problems for specific
applications, the key to a smooth closure of previously opened ports in a firewall is
management and communication of the formally agreed process.
Lessons learnt
• A policy and procedures are required to define which ports are closed and the mechanism
for opening/closing ports in a perimeter firewall.
• It is recommended to start with a default of ports closed at a site and then only open what
is needed. Allowing access by default reduces the protection at a site. It also creates the
risk of disruption to services when previously open ports are closed.
• Management support is essential to ensure site-wide support for a perimeter firewall
closure policy.
• It is recommended to involve at least those responsible for Internet services, users and
site security to agree and oversee the firewall closure procedures.
• Make a plan for a fall-back solution in case important applications fail after the firewall
closures.
• Network level debugging may be required to determine the port number requirements of
an application.
Useful links
• RSS feed for Security in a Grid Environment
http://rss-grid-security.cern.ch/rss.xml
• Joint Security Policy Group documents
http://cern.ch/proj-lcg-security/documents.html
• Port Knowledgebase - list of frequently seen TCP and UDP ports and what they mean
http://www.iss.net/security_center/advice/Exploits/Ports/default.htm
• SANS top ten ports being threatened
http://isc.sans.org/trends.html
Training
• Technical knowledge is required for the firewall configuration and the applications
affected.
• Management skills are required to plan and oversee the process.
4.2
Integrate and extend network perimeter management tools
This task consists of the following sub-tasks:
• Design a network perimeter management system fully integrated with the existing
network database
• Implement the perimeter management system according to a prioritized schedule
• Provide tools to verify and debug network perimeter security configurations
4.2.1
Design a network perimeter management system fully integrated with the
existing network database
This has been fully reported in D1.1.3 [R3].
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
14 / 59
4.2.2
Implement the perimeter management system according to a prioritized
schedule
Security reason
The CERN computer infrastructure serves the scientific community that builds the LHC
accelerator and detectors and will use the LHC for their physics experiments. To accomplish
this task well, access from outside CERN to data, applications and documents must be simple
and straightforward for everyone. Thus the CERN network is built so that every computer
connected to the campus network can be accessible from anywhere. At the same time,
security threats are increasing both in quantity and in the damage they can cause. The
common practice to reduce the risk is to hide a network as much as possible, but this conflicts
with the goal of providing easy access.
Due to the large number of hosts connected to the CERN campus network and to the high
frequency that they change configuration and function, CERN developed a software
framework that manages and implements the policies applied to the firewall at its perimeter.
The framework has several functions:
• It allows the firewall administrators to configure and manage the hardware.
• It allows the security personnel to implement and change the access policies.
• It allows end users to request and remove policies for their own hosts.
• It periodically implements all the approved changes and checks the consistency of the
configurations.
Implementation experience
The implementation of the software framework required a complete revision of the policies
that were applied to the previous firewall. All the unnecessary rules were removed, the
necessary ones were optimized whenever possible and the missing ones were added.
The rules were moved into a software database, adding information about the requester, the
approver, the scope, an expiration date and its history. The database tool allows a better
management of the rules: the rules are easily categorized, found and understood and they can
be re-used in other firewalls protecting CERN.
The software is mostly independent of hardware vendors. Only the backend needs to know
the configuration commands in order to correctly generate the device configurations. The
frontend is hardware agnostic, freeing the security personnel from having to know vendor-
specific details.
Lessons learnt
• Carefully evaluate the functionalities provided by the firewall hardware in order to
implement the necessary fields in the database tables.
• Exploit all the already existing information about the network, in particular devices and
IP addresses.
• Collect requirements and required functionality from all the categories of users at the
very beginning of the design phase.
• Implement a software module that verifies the consistency of the generated configuration
before uploading them on the production device. The same software should also make an
estimation of the memory resource needed and protect the system from configurations
that are too much of a burden.
• Restrict access to the firewall configuration and management tools, as well as its
documentation, to avoid exposing such information to potential attackers.
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
15 / 59
Useful links
• Paper presented to the Terena conference, Firewalling beyond 10Gbps
http://tnc2007.terena.org/programme/presentations/show.php?pres_id=119
(included in
Annex, section A2.1)
• Article on the CERN Computing Newsletter, CERN upgrades firewall to meet
http://cerncourier.com/cws/article/cnl/27610
Training
Technical and training documentation has been produced. Due to the sensitive nature of its
contents, access is restricted.
4.2.3
Provide tools to verify and debug network perimeter security configurations.
Security reason
Modern firewalls have a lot of functionality and can be configured in many different ways.
Their configurations are often complex and long, so it can be hard to debug them.
Misconfigured firewalls can deny legitimate traffic or expose internal services that should
have been protected.
Implementation experience
The CERN Firewall Management System (cf. D1.1.3 [R3] section 6.2) uses an abstraction of
the firewall configuration that lies in a set of tables in a relational database. The tools
interacting with the configuration abstraction all have independent checks in place to make
sure that the data makes sense for them. These tools are:
• The graphical user interface used to edit the firewall configuration.
• The programs extracting the information and translating it to the language used by the
network device.
• Other querying and debugging tools.
These checks include for instance:
• Resource estimations, to make sure the network device will accept the configuration.
• Consistency checks, to make sure rules are not in contradiction (e.g. deny before permit
for the same port).
• Change control, to make sure a new firewall configuration is not too different from the
previous one.
Debugging tools allow checking of both the expected configuration (querying the database)
and the actual configuration (querying the network devices). Open source tools like Hping,
Tcpdump and Firewalk are very useful to send arbitrary packets to see if they cross the
firewall.
Lessons learnt
• Software dealing with firewall configurations should be very strict and check as many
things as possible, even at the cost of reimplementing the same checks in different places
(maybe using different programming languages or algorithms).
• It is much easier to check a firewall configuration using an abstraction rather than the
configuration syntax used in the network devices.
• It is very convenient to have access to test machines on both sides of the firewall to see
which packets go through.
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
16 / 59
Useful links
• Hping
• Tcpdump
• Firewalk
http://www.packetfactory.net/projects/firewalk
Training
Software dealing with firewall configurations should be written by experienced programmers
who program defensively. A programming error can have disastrous consequences.
4.3
Protect devices on dedicated networks
This task contains the following sub-tasks:
• Enforce the defined network policies
• Implement application level gateways able to restrict access to authorized users and
tasks
4.3.1
Enforce the defined network policies
Security reason
Large, interconnected or flat networks provide an ideal ground for attackers to spread viruses,
worms and other malware. If a network hosts a large number of devices, one infected or
compromised device can easily spread its infection. Generally, the more devices connected to
a Domain, the lower the security will be.
A mixture of office PCs and controls devices is particularly dangerous, as office PCs often
face threats directly (e.g. through malicious web sites, malicious email attachments etc),
while control devices may lack security protections because of configuration or scheduling
reasons.
Implementation experience
To reduce the risk of infections spreading, the CERN network has been segregated into
separate ‘Network Domains’. (Groups of) control systems whose failure can pose a risk have
been separated from the CERN office network and grouped into one or several dedicated
Domains. Each Domain is then dealing with communications of a dedicated and exclusive set
of particular systems, defined so that the Domain serves a common purpose. Communication
inside that Domain is extensive and complex, but communication to the outside is small and
clearly defined. Connection rules have been established, which are surveyed by an appointed
Domain Administrator.
Segregation of existing Domains has been shown to be very difficult. On one hand, an initial
assessment was needed in order to evaluate the dependencies between the new Domains, and
to locate and regroup already connected devices. On the other hand, full user commitment is
needed for such a task, since the segregation also implies moving (‘sorting’) many of their
devices from one Domain to another. The creation of Domains from scratch has proven to be
very effective, since connection rules can be easily applied, once the corresponding users
have been made aware of the security implications.
The enforcement of network connection rules has proven to be difficult, in particular to detect
modems or wireless access points not permitted by the policy.
Lessons learnt
• Split Domains according to functionality and avoid Domains that are too large.
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
17 / 59
• Never directly inter-connect office networks with controls networks.
• Clearly define inter-connection points between Domains, taking into account the
possibilities of data exchange (files, emailing, web pages).
• Produce connection rules that include definitions of the use of laptops and wireless access
points on a Domain.
• Introduce checks for unauthorised connections and configurations e.g. modems, wireless
etc.
Useful links
• Centre for the Protection of the National Infrastructure (CPNI), Good Practice Guidelines
Parts 1-7, 2006,
http://www.cpni.gov.uk/Products/guidelines.aspx
Training
• Network segregation, for newly created Domains or for existing Domains, needs the full
support of the users involved. Awareness raising is needed for users so that they know the
emerging security threats. Additional training on connection rules, local and remote
access methods etc. is mandatory.
4.3.2
Implement application level gateways able to restrict access to authorized
users and tasks
Security reason
A well controlled method is required for accessing devices on dedicated networks.
Application gateways provide a good compromise between security and convenience, making
them a suitable choice as a gateway front-end to sensitive devices on dedicated networks.
Access can be restricted to a defined set of users for performing a restricted set of tasks.
Implementation experience
More than 50 application gateways are running in production as front-ends to services on the
CERN site. The technology used is a mixture of Windows Terminal Servers (WTS) and
Linux SSH gateways. They are used as the principle means for remote access to the
organisation from off-site, as well as for access to devices on dedicated networks with stricter
security policies.
Many of the WTS have been installed using the central installation scheme for Windows PCs
(see section 3.1.3). The IT department is managing the hardware maintenance and the basic
installation of the operating system, while local experts manage the dedicated software
applications running on the application gateways and the management of access rights.
Access control on the WTS is currently performed using Microsoft Active Directory.
The application gateways have so far been a successful mechanism for preventing incidents
on dedicated networks while still allowing an essential set of users to perform actions from
inter-connected networks, including from off-site.
Lessons learnt
• Application gateways need to allow for scalability, so that they can grow with increasing
usage or an increasing user community.
• Requiring use of application gateways to access devices on dedicated networks provides
a reasonable compromise between security and convenience.
• User authentication and authorization should be part of a site-wide scheme.
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
18 / 59
Useful links
• Microsoft Windows Terminal Servers
http://www.microsoft.com/windowsserver2003/technologies/terminalservices
• Terminal Services Wikipedia article
http://en.wikipedia.org/wiki/Terminal_Services
• SSH port forwarding
http://www.ssh.com/support/documentation/online/ssh/adminguide/32/Port_Forwarding.
html
Training
• The initial setup of application gateways requires deep knowledge of the requirements
and workflow as well as the technology chosen. User level training is also required.
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
19 / 59
5
Security mechanisms and tools deployment results
Below are the results achieved for Implementation Area I3: Security mechanisms and tools.
Where relevant, tasks have been divided into sub-tasks and these are reported either
separately or together, as appropriate.
5.1
Investigate security assessment tools suitable for the CERN environment
This task contains the following sub-task:
• Survey existing security assessment tools and assess their suitability for the CERN
environment
• Design and implement a framework to integrate a minimum set of selected tools
• Develop modules to extend the capabilities of the above integrated security assessment
framework
To avoid information repetition, the results of the above sub-tasks are combined and reported
below.
Security reason
Large sites typically have a diverse range of devices attached to their network, which are not
all centrally managed. As new vulnerabilities are discovered, it is essential that the whole site
can be scanned quickly so that vulnerable machines can be identified and patched.
Implementation experience
Simple brute-force scans may cause problems for some devices, so a site-wide scanning tool
must respect exception lists. Off-the-shelf vulnerability scanners tend to focus on scanning
single devices in detail and lack the fine-grained control required to simultaneously scan a
large number of devices for a specific vulnerability. Output from such scanners can be
voluminous and require expertise to interpret.
Sites might have thousands or tens of thousands of devices and the scanner must be able to
cope with networks of this scale. Typically, this means that the scanning process will need to
be distributed across several machines if the scan is to be completed in a reasonable time.
Large sites will also have a variety of devices managed by different teams, connected to the
network and running a variety of services. A continuous scanning process, running in the
background, can simultaneously build a database of observed running services that can be
used to quickly launch a targeted scan if a new vulnerability is discovered.
A vulnerability scanning framework has been developed into which site-specific rules (such
as classes of devices to be scanned) and existing specialised vulnerability scanners can be
integrated. This framework is required to take a very high level view of the overall site-wide
scanning process. A plug-in architecture was developed to allow modules to be written to
perform several functions:
• Generate a list of devices to be scanned from site-specific data sources
• Scan machines to determine which services each machine is running
• Perform service-specific vulnerability tests
• Report results in various forms, such as human-readable or computer-readable, for
storage in site-specific databases
With this architecture, the state of the scanning process can be saved to a file. This allows the
scanning to be paused and resumed, parts of the scan to be distributed across several
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
20 / 59
machines, and distributed results to be centrally collated. For more details, see Annex section
A3.1.
Lessons learnt
• Vulnerability scanners are available commercially and in the public domain.
• Choose one or more vulnerability scanner depending on your needs e.g. to determine
ports open, crack passwords, attempt exploits (see links below).
• For a large site, consider a framework to manage and integrate the scanning process.
• Expect new types of scanning requirements as attacker techniques evolve.
• Some types of devices (e.g. Programmable Logic Controler) might fail during security or
vulnerability scans.
Useful links
• Nessus vulnerability scanner
• Nmap port scanner
• THC-Hydra
• Metasploit
http://framework.metasploit.com
Training
Programming skills and some experience of vulnerability scanners are required to develop a
site-specific framework.
5.2
Enhance CERN’s incident detection capability
This task contains the following sub-task:
• Investigate technologies for network based IDS (Intrusion Detection System) on 10
Gbps links and beyond
• Extend and integrate the existing set of tools to detect new attack vectors
• Investigate correlating host based and network based IDS
5.2.1
Investigate technologies for network based IDS on 10 Gbps links and beyond
This has been fully reported in D1.1.3 [R3].
5.2.2
Extend and integrate the existing set of tools to detect new attack vectors
Security reason
Security attacks are constantly evolving to target new services (e.g. web services) or to use
new technologies (e.g. P2P botnets or fast-flux DNS). As a consequence, security tools have
to evolve at the same pace. In addition, the computing environment is changing too. For
instance, the move from 1 Gbps links to 10 Gbps links has an impact on the way network
based intrusion detection can be used.
Implementation experience
As described in section 7.2.1 and Annex A3.2 of D1.1.3 [R3], CERN has developed a
NetFlow-based NIDS named DNIM, which is now in production. It replaced a previous tool
that was based on packet sniffing and could not scale to 10 Gbps networks.
The security tools have all been integrated and use a central database to store security
information. The database hides the tools internally and eases the modification of existing
tools and the addition of new ones. It also allows tools to use data produced by other tools
and improve their quality. For instance, one tool uses generic scan information (such as
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
21 / 59
‘machine X contacted more than N different machines’) and tries to identify which P2P
application was used during which precise time window. Accurate time information is critical
on DHCP, where the IP addresses get re-allocated.
Web Applications are number one on SANS’ Top-20 Internet Security Attack Targets, in the
‘Cross-Platform Applications’ category. It is very important to check for web applications
security holes, in particular the Cross-Site Scripting (XSS) and Cross-Site Request Forgery
(CSRF).
Lessons learnt
• Although it cannot replace NIDS using packet payloads such as Snort, a NetFlow-based
NIDS can be powerful, flexible and scalable to cope with very fast network links (tens of
Gbps).
• As tools will constantly evolve, it is necessary during the design phase to put emphasis on
their modularity, and use whenever possible output formats that are easy to change and/or
parse.
• Modular tools ease the automation of security incident handling.
• Handling incidents on DHCP networks requires access to the lease logs, but also requires
precise timing information to correctly identify the device owner.
Useful links
• SANS Top-20 Internet Security Attack Targets
• Securing Web applications
http://cern.ch/security/webapps
Training
• Security tools require experienced designers and programmers to build reliable and
extensible software.
5.2.3
Investigate correlating host based and network based IDS
Security reason
Host based and network based IDS are complementary and, when combined, should give a
better understanding of what really happens.
Implementation experience
Two commercial companies (one providing a host based IDS, the other a network based IDS
with customizable data correlation engine) were expected to try to combine their products in
a joint-project with CERN. For reasons independent from the ISSeG project, this did not
happen so their results cannot be reported here. CERN produces both host and network based
security incident alerts, which are analysed by security experts. Our experience is that either
one of the alerts is sufficient to indicate a potential incident, but correlating the alerts helps to
better identify the original cause and speed up incident follow up.
5.3
Evaluate multi-factor authentication technologies
This task contains the following sub-tasks:
• Prepare a written survey comparing multi-factor authentication technology choices
• Investigate the deployment of multi-factor authentication technology at CERN
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
22 / 59
5.3.1
Prepare a written survey comparing multi-factor authentication technology
choices
This has been fully reported in D1.1.3 [R3].
5.3.2
Investigate the deployment of multi-factor authentication technology at CERN
Three methods were investigated and evaluated: Cryptocard, Smartcards and OTP via SMS
(GSMauth). They are reported separately below:
5.3.2a Cryptocard
Security reason
The configurations of almost all the network devices and servers that form the infrastructure
of the CERN network are built and deployed by a software framework. This uses topology
information stored in an online database. Access to the systems that run the software and
store the information is very sensitive and it is granted only to users that own a cryptocard
that generates a one-time password.
Implementation experience
Several software and hardware options had to be carefully evaluated because of the multi-
vendor environment of CERN. The servers and the tokens had to work seamlessly with all the
operating systems, hardware and applications in use. Because of this, the one-time password
solution was chosen, rather than a hardware key (USB or similar).
Cryptocard was evaluated against RSA securID.
The advantages of Cryptocard were:
• Very good Linux support on the server side.
• The token generation was event-based and not time-based (time-based token get de-
synchronized)
• A good Developer Tool Kit allowed easy integration with web-based applications.
• Easy access to its database, since it can uses any sql server (RSA had a proprietary
format).
• More secure (the RSA token can be reused within 30 seconds).
On the other hand, RSA would probably scale better in a larger environment because it
provides web based tools for the end users to re-synchronize and unlock the tokens.
Lessons learnt
• Be supportive of the end users when the cryptocards are introduced and then become
compulsory as they may not be very enthusiastic at first.
• Don’t allow anyone to bypass the system.
• Tokens can get locked so always grant administrative permission to at least two people.
Both of them should have two active cryptocards.
Useful links
• Cryptocard product adopted
http://www.cryptocard.com/products/crypto-Dshield/
http://www.rsa.com/node.aspx?id=1156
Training
• Cryptocard technical documentation
http://www.cryptocard.com/support/technicaldocumentation/?cat=7
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
23 / 59
5.3.2b Smartcards
Security reason
Authentication on different CERN resources and on Grid facilities relies more and more on
certificates, especially as CERN has deployed its online Certification Authority allowing
every CERN user to request and use a digital Certificate as proof of identity. The next step in
strengthening the authentication security is to store these certificates on hardware tokens,
possibly stored on a device that CERN users would always keep with them.
Implementation experience
GemPlus Smartcard hardware provider was selected for evaluation. As the market leader,
GemPlus was able to provide customized CERN Access Cards, including the Smartcard chip,
the magnetic stripe, the new RFID (Radio Frequency IDentification) chip and the customized
printing (including photo and barcode).
Windows drivers were provided by GemPlus, and the Linux drivers are to be delivered later.
After 2 years of testing, the Windows pilots are running fine, despite some minor glitches
with drivers and card readers (reader freezing from time to time). One CERN experiment has
implemented this Smartcard platform for critical authentication of Windows based control
applications.
Investigations need to be made in the Linux area, to find a Smartcard vendor providing cards
with open source drivers available, which for the moment seems difficult to find.
Lessons learnt
• Define the target platforms for Smartcard usage (mostly desktops).
• Define how the smartcard will be embedded: to an existing access card, on a separate
token (USB token, blank card, etc...).
• Take the drivers licensing price into account, as it will cost more than the hardware
token.
• For Windows Vista users, GemAlto (former GemPlus and Axalto) is providing an
interesting Smartcard.NET solution with Microsoft, allowing users to authenticate either
by using their Smartcard or by using a one-time password if no reader is available (for
temporary locations).
Useful links
Some Smartcard vendors and solutions are:
• GemPlus
• GemAlto
• PrimeKey solutions
5.3.2c GSMauth
Security reason
Access to sensitive machines must be tightly controlled to prevent unauthorized use. One way
to improve access control is to use multi-factor authentication based on OTP (One Time
Passwords).
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
24 / 59
Implementation experience
In some cases, simple security solutions may prove to be very effective. This is the case for
GSMauth, which uses mobile phones to provide a second authentication factor for some
sensitive machines at CERN.
GSMauth is a program that is executed after the normal authentication but before the user can
run any command. It generates a random Personal Identification Number (PIN) and sends it
to the mobile phone of the user trying to login via Short Message Service (SMS). The user
must then type in this PIN within a short time window.
In fact, GSMauth does more than simply PIN checking. Via a flexible configuration file, it
can also grant access based on the IP address of the client (e.g. no need to type your PIN if
the connection comes from a trusted machine). It can also perform CryptoCard One-Time
Password (OTP) verification.
Currently, this program is connected to the operating system using shell wrappers and uses an
email-to-SMS gateway. It is very simple (200 lines of code in total) and easy to audit. Yet it
fulfils the user requirements of the team of users using it daily.
Lessons learnt
• Although SMS has no guaranteed delivery, it can be good enough for specific contexts,
e.g. if one SMS text message gets lost, the user can simply re-try to login later. An
alternative solution is needed where no mobile phone access is present.
• Pluggable Authentication Module (PAM) support was tested but it was abandoned
because of its lack of portability and its poor integration with SSH and Kerberos.
• For multi-factor authentication, there is no universal technology fulfilling all possible
user requirements. When possible, simple solutions are preferable to address simple
requirements.
Useful links
• Two-Factor Authentication with Cell Phones
http://www.schneier.com/blog/archives/2004/11/twofactor_authe.html
• Secure Web Authentication with Mobile Phones
http://www.mit.edu/~minwu/dimacs.ppt
Training
• The development of software such as GSMauth requires a programmer with a strong
understanding of security implications.
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
25 / 59
6
Administrative procedures and training deployment results
Below are the results achieved for Implementation Area I4: Administrative procedures and
training. Where relevant, tasks have been divided into sub-tasks and these are reported either
separately or together, as appropriate.
6.1
Document and strengthen account administration procedures
This task contains the following sub-tasks:
• Document categories of users with a formal relationship with the organization
• Document strengthened account procedures that ensure a formal relationship with the
organization
6.1.1
Document categories of users with a formal relationship with the organization
This has been fully reported in D1.1.3 [R3].
6.1.2
Document strengthened account procedures that ensure a formal relationship
with the organization
This has been fully reported in D1.1.3 [R3].
6.2
Prepare and implement a training plan for improving knowledge of computer
security within the organization
This task contains the following sub-tasks:
• Identify training needs based on security incident analysis and known risks
• Define target audiences based on their knowledge levels and training needs
• Integrate security training and awareness into the organization’s structures
6.2.1
Identify training needs based on security incident analysis and known risks
This has been fully reported in D1.1.3 [R3].
6.2.2
Define target audiences based on their knowledge levels and training needs
This has been fully reported in D1.1.3 [R3].
6.2.3
Integrate security training and awareness into the organization’s structures.
Security reason
Security training and awareness needs to be part of an organization’s structure to help reduce
the risk of computer security incidents. Computer users need to be aware of incident threats
and their consequences, and need to know security related rules and best practices.
Implementation experience
A training plan (documented in the Annex of D1.1.3 [R3]) set out ways to integrate security
training and awareness. The following actions have been achieved:
• A FAQ for CERN’s computing rules and best practices has been created.
• Security awareness has been raised using CERN’s dissemination channels: e.g. articles
in CERN Computer Newsletter (CNL) and the external CERN Courier (see Annex A4).
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
26 / 59
• A security card explaining key security messages has been designed for distribution at
strategic points on the CERN site.
• Security training presentations were given to targeted groups at CERN.
• A quiz has been prepared for incorporation into automated account creation. The quiz
uses an integrated training tool already in place for safety training. This tool records
results and allows for follow-up.
• Pages on the CERN security web site have been updated with Best Practice advice for
system administrators and software developers. A security checklist for developers has
been submitted to SANS in response to their request for examples.
• A software developer training session on security was held at the CERN School of
Computing.
• Relevant external training courses have been identified.
Lessons learnt
• Have security web pages within your organization’s online repository that are linked
from relevant entry points (e.g. the main IT pages of your Intranet, introductory pages for
newcomers, etc).
• Promote security awareness campaigns within your organization, e.g. via presentations to
targeted audiences, posters, quizzes, leaflets etc.
• Create security related articles for your organization’s internal newsletter to ensure users
are made aware of any security measures that will affect them and are also informed of
security policies, best practises, and web addresses of more information.
• Develop security training material, either for presentation to computer users or for them
to follow online.
• When new computer accounts are created, ensure that users are fully informed of the
security rules of your organization and pass on key security messages to them, including
relevant security web addresses.
• Integrate awareness-raising into incident response so that users failing to comply with
security policies are given best practice and policy information to help prevent future
incidents.
• Develop a set of Frequently Asked Questions with clear, concise answers.
• Target security information to different audiences in your organization, e.g. managers,
system administrators, software developers and general users.
Useful links
• For example training material for managers, system administrators, software developers
and general users, see
and select the ‘Training’ tab.
• For an example of a security home page, see
http://www.jmu.edu/computing/runsafe
(James Madison University).
• For an example security article, see
http://cerncourier.com/cws/article/cern/29863
• For example security information for software developers, see
Training
• You need to understand the different audiences within your organization and the different
mechanisms in place to target them. You need to get feedback from these audiences
throughout the creation and maintenance of training and awareness material to ensure
maximum effectiveness.
6.3
Review and update security policies and procedures
This task contains the following sub-task:
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
27 / 59
• Update perimeter security policy for default access from off-site
• Create a structure for documenting and maintaining security policies and procedures for
IT Services
6.3.1
Update perimeter security policy for default access from off-site
This has been fully reported in D1.1.3 [R3].
6.3.2
Create a structure for documenting and maintaining security policies and
procedures for IT Services
Security reason
Security policies and procedures need to be documented to provide a common understanding
of both the theoretical policy and the practical procedure. Maintenance is required so that
policies and procedures stay in step with evolving security needs.
Implementation experience
As part of I1.2: Enforce account policies and procedures, many procedures were documented
e.g. who is authorised to have accounts and procedures for closing accounts. As part of this
process, other policies and procedures were gathered together. All information was added to
the Computing Rules web site, which was restructured so material was easy to find and to
maintain. The Security home page was also restructured to easily direct people to
documentation. Frequently Asked Questions, within an easily maintainable Sharepoint
system, were also written so that material could be easily accessed and updated.
Lessons learnt
• Gather together existing policies and procedures within your organization.
• For any undocumented policies and procedures, discuss with the relevant contacts what is
required and get documentation fully approved.
• Establish a repository for policy and procedure documentation that can easily be accessed
by the relevant computer users e.g. well organised web pages within an existing structure,
or a list of Frequently Asked Questions in an existing structure linking to relevant
documentation.
• Ensure policies and procedures are maintained by setting timeframes for assessment and
assigning responsibility to a relevant member of personnel.
• Publicise policies and procedures and where to find them via awareness campaigns
within the organization e.g. presentations to new staff members, leaflets, posters, training,
etc.
Useful links
• The Joint Security Policy Group (JSPG) has established Grid policies, which can be
http://cern.ch/proj-lcg-security/documents.html
Training
• Knowledge of the institute’s existing regulations is necessary. In addition, knowledge of
who to contact within the organization to establish policies and procedures, which can
then be documented.
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
28 / 59
7
The CERN final deployment report within the ISSeG project
This report marks the end of the deployment of Integrated Site Security (ISS) at the CERN
site.
CERN’s strategy document, D1.1.1 [R1], contained proposed strategic directions to improve
CERN’s computer security. D1.1.2r [R2] then took these proposals and converted them into
concrete implementation tasks with deadlines and responsible personnel. D1.1.3 [R3], written
nine months into deployment reported the progress and the lessons learnt so that they could
be fed into WP3’s recommendations and WP4’s training materials. This final deployment
report continues from where D1.1.3 [R3] left off, charting the full site deployment and, where
necessary, referring back to D1.1.3 [R3].
Deliverable D1.2.4, FZK’s final deployment report, will follow a similar structure to this
document. Together these two final reports will complete the practical basis for the
recommendations and training resources.
This final deployment report, like the intermediate reports before it, illustrates the usefulness
of tangible results of ISS deployment, allowing the ISSeG project to disseminate practical site
security advice to other Grid sites.
8 References
[R1] Deliverable D1.1.1 Description of CERN ISS Strategy, ISSeG EU Project no: 026745
[R2] Deliverable D1.1.2r Implementation Plan for CERN ISS, ISSeG EU Project no: 026745
[R3] Deliverable D1.1.3 Intermediate report on deployment at CERN and input on lessons
learnt to WP3 and WP4, ISSeG EU Project no: 026745
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
29 / 59
9 Acronyms
and
Abbreviations
Acronym/
Abbreviation
Name/Definition
ACL
Access Control List
APT
Advanced Packaging Tool
CDB Configuration
DataBases
CERN
European Organization for Nuclear Research
CMF
Computer Management Framework
CNIC
Computer and Network Infrastructure for Controls
CPNI
Centre for the Protection of the National Infrastructure
CPU
Central Processing Unit
CSRF
Cross-Site Request Forgery
CVE
Common Vulnerabilities and Exposures
DDoS
Distributed Denial of Service
DHCP
Dynamic Host Configuration Protocol
DoS
Denial of Service
EGEE
Enabling Grids for e-Science
EU European
Union
FAQ
Frequently Asked Question(s)
FZK
Forschungszentrum Karlsruhe GmbH
GEANT2
Pan-European Gigabit Research and Education Network
GSM
Global System for Mobile Communications
HEP High
Energy
Physics
HR Human
Resources
HTTP
HyperText Transfer Protocol
HTTPS Secure
HTTP
ICALEPCS
International Conference on Accelerator and Large Experimental
Physics Control Systems
IDS
Intrusion Detection System
IGP
Interior Gateway Protocol
IP Internet
Protocol
IPS
Intrusion Prevention System
ISS
Integrated Site Security
ISSeG
Integrated Site Security for Grids
JSPG
Joint Security Policy Group
L4C
Linux for Controls
LCG
LHC Computer Grid
LHC
Large Hadron Collider
NIDS
Network Intrusion Detection System
NSC
Named Set of Computers (within CMF)
NTP
Network Time Protocol
OTP One-Time
Password
P2P Peer-to-Peer
PAM Pluggable
Authentication
Module
PC Personal
Computer
PIN
Personal Identification Number
PLC
Programmable Logic Controller
PXE
Preboot Execution Environment
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
30 / 59
quattor
QUattor is an Administration ToolkiT for Optimizing Resources
RFID
Radio Frequency IDentification
RSA
An algorithm for public-key cryptography
RSS
Rich Site Summary
SANS
SysAdmin, Audit, Networking, and Security
SCADA
Supervisory Control And Data Acquisition
SMS
Short Messaging Service
SQL
Structured Query Language
SSH Secure
SHell
TCP
Transmission Control Protocol
TERENA
Trans-European Research and Education Networking Association
UAC
User Account Control
UDP
User Datagram Protocol
URL
Uniform Resource Locator
USB
Universal Serial Bus
WAN
Wide Area Network
WP Work
Package
WTS
Windows Terminal Servers
XSS Cross-Site
Scripting
YUM
Yellow dog Updater, Modified
Project co-funded by the European Commission as a Specific Support Action within the 6th Framework
Programme. ISSeG began in February 2006 and will run for 26 months.
Copyright © CERN, 2006. Member of the ISSeG Collaboration.
Project no: 026745
PUBLIC
31 / 59
Project No: 026745
I S S
e
G
I
ntegrated
S
ite
S
ecurity for
G
rids
Specific Support Action
Information Society and Media
A
NNEX FOR
F
INAL REPORT ON DEPLOYMENT AT
CERN
TABLE OF CONTENTS
A1
Resource management extensions............................................................................32
A1.1
Update on the CERN Computing and Network Infrastructure for Controls (CNIC).........32
A2
Network connectivity management............................................................................37
A2.1
Firewalling beyond 10Gbps...............................................................................................37
A3
Security mechanisms and tools .................................................................................49
A3.1
Security assessment tools and framework suitable for the CERN environment ................49
A4
Administrative procedures and training....................................................................53
A4.1
Web applications security, CNL Article, November 2007 ................................................ 53
A4.2
Suggestions for designing and writing secure software, CNL Article, September 2007 ... 56
A4.3
How to avoid a false sense of security, CERN Courier, April 2007 .................................. 59
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
ANNEX FOR
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
32 / 59
A1 Resource management extensions
The following paper was presented at the 11th International Conference on Accelerator and
Large Experimental Physics Control Systems ICALEPCS 2007, Knoxville, TN, USA, 15 - 19
Oct 2007
http://ics-web4.sns.ornl.gov/icalepcs07/WPPB38/WPPB38.PDF
.
A1.1
Update on the CERN Computing and Network Infrastructure for Controls (CNIC)
Abstract
Over the last few years modern accelerator and experiment control systems have increasingly
been based on commercial-off-the-shelf products (VME crates, PLCs, SCADA systems, etc.),
on Windows or Linux PCs, and on communication infrastructures using Ethernet and TCP/IP.
Despite the benefits coming with this (r)evolution, new vulnerabilities are inherited too:
Worms and viruses spread within seconds via the Ethernet cable, and attackers are becoming
interested in control systems. Unfortunately, control PCs cannot be patched as fast as office
PCs. Even worse, vulnerability scans at CERN using standard IT tools have shown that
commercial automation systems lack fundamental security precautions: Some systems
crashed during the scan, others could easily be stopped or their process data be altered [1].
During the two years following the presentation of the CNIC Security Policy at
ICALEPCS2005 [2], a ‘Defense-in-Depth’ approach has been applied to protect CERN’s
control systems. This presentation will give a review of its thorough implementation and its
deployment. Particularly, measures to secure the controls network and tools for user-driven
management of Windows and Linux control PCs will be discussed.
Introduction
The enormous growth of the worldwide interconnectivity of computing devices (the
‘Internet’) during the last decade offers computer users new means to share and distribute
information and data. In industry, this results in an adoption of modern Information
Technologies (IT) to their plants and, subsequently, in an increasing integration of the
production facilities, i.e. their process control and automation systems, and the data
warehouses. Thus, information from the factory floor is now directly available at the
management level (‘From Shop-Floor to Top-Floor’) and can be manipulated from there.
Unfortunately, the adoption of standard modern IT in distributed process control and
automation systems
also exposes their inherent vulnerabilities to the world [1]. Furthermore,
this world is by far more hostile than a local private controls network as the number and
power of worms and viruses increase, and attackers start to become interested in control
systems. Partial protection can be obtained through the usage of properly configured firewalls
and through well-defined network architectures. * Commonly denoted in the following as
‘control systems’, where a ‘system expert’ has the expertise in its configuration. However,
some means of security incorporated into standard IT equipment cannot be directly applied to
controls equipment since both differ in hardware but also in the concepts of availability and
manageability.
At CERN, control systems are used for the operation of the whole accelerator complex,
including the Large Hadron Collider (LHC), for the LHC experiments, as well as for the
∗
Commonly denoted in the following as ‘control systems’, where a ‘system expert’ has the expertise in its configuration.
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
ANNEX FOR
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
33 / 59
technical infrastructure (e.g. electricity, cooling & ventilation) and for fixed-target
experiments.
In order to cope with the growing usage of standard IT technologies in control systems at
CERN, the corresponding operation principles have been reviewed taking the aspect of
security into account. This paper will give an update on the implementation of the Security
Policy presented by the CERN Computing and Network Infrastructure for Controls (CNIC)
working group two years ago [2].
CNIC security policy
The CNIC Security Policy gives directions on how to secure CERN control systems. It is
necessary to reduce the overall risk to a minimum in order to obtain maximum security,
where ‘risk’ is defined as:
Risk = Threat × Vulnerability × Consequence
The major part of the factor ‘threat’ originates from outside CERN and cannot be
significantly reduced. Thus, protective measures have to be implemented to prevent external
threats like viruses & worms or attackers penetrating CERN control systems. These protective
measures should also prevent insiders from (deliberate or accidental) unauthorized access.
The ‘consequences’ are inherent to the design of CERN’s accelerators and the affiliated
experiments. All are running a wide variety of control systems, some of them complex, some
of them dealing with personnel safety, some of them controlling or protecting very expensive
or irreplaceable equipment. Thus, CERN’s assets and their proper operation are at stake. A
security incident can lead to loss of beam time and physics data, or — even worse — damage
to or destruction of unique equipment and hardware.
The ‘vulnerability’ factor can be either minimized by guaranteeing a prompt fixing of
published or known vulnerabilities, and/or by adding pro-active measures to secure the
unknown, potential or not-fixable vulnerabilities.
In order to protect such vulnerabilities against being exploited (either because there is no
patch available or a patch could not be applied), the Security Policy follows the
recommendations of the U.K. CPNI [3]. It is based on a ‘Defense-in-Depth’ approach, where
pro-active security measures must be applied to every possible level:
• …of the security of the device itself;
• …of the firmware and operating system;
• …of the network connections & protocols;
• …of the software applications;
• …of third party software;
• …together with users, developers, and operators.
‘Defense-in-Depth’ is, thus, based on four major pillars: ‘Network Security’, ‘Central
Installation Schemes’, ‘Authorization & Authentication’, and ‘User Training’.
However, sufficient protection should not provide false security. Incidents will happen.
Therefore, the Security Policy also defines rules to deal with ‘Incident Response & System
Recovery’, as well as with regular security audits. The next chapters will outline the major
implementations in detail.
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
ANNEX FOR
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
34 / 59
Network security
In order to contain and control the network traffic, the CERN network has been separated into
defined ‘Network Domains’. For example, each of the LHC experiments is now using such a
dedicated Domain. CERN’s accelerator complex and the control systems which monitor and
control the technical infrastructure (e.g. electricity distribution, cooling & ventilation, facility
management as well as safety and access control systems) use another. ‘Domain
Administrators’ were assigned to take full responsibility for a Domain. In particular, they
supervise the stringent rules for connecting devices to it. Additional network segregation
allows a system expert further to protect vulnerable devices like Programmable Logic
Controllers (PLCs).
Each Domain is interconnected with CERN’s ‘General Purpose Network’ used for office
computing, and other Domains via dedicated network routers. The traffic crossing any two
Domains is restricted to a minimum by the usage of routing tables, with only mandatory
traffic passing such boundaries. A large fraction of this traffic is either dedicated data
exchange with CERN’s Computing Centre, or currently inevitable due to the ongoing
commissioning of the LHC accelerator.
Windows Terminal Servers (WTS), and to a lesser extent Linux-based application gateways,
have become the only means for remote user access.
Central installation schemes
From experience at CERN, only centrally managed and patched PCs have shown to be secure
in the long run. However, since control PCs are very different in handling than office PCs, a
more flexible installation and management scheme was needed. While the operating systems,
antivirus software, and basic software applications should continue to be managed and
maintained by the IT service providers, it is up to the system expert to take over full
flexibility of the configuration of the PCs of his system ― and full responsibility for securing
it. Such a scheme will also help the expert to recover e.g. from a security incident.
Schemes for the central installation of CERN Scientific Linux and of Windows, respectively,
have been created.
Linux For Controls (L4C)
L4C is based on a bottom-up approach to the configuration of the PCs in a hierarchical
manner, and uses the same techniques having proven to be successful for managing Linux
clusters in the CERN Computing Centre, i.e. using quattor (http://quattor.web.cern.ch) for
configuring PCs (‘the nodes’). Settings that are specific to a node are defined in so-called
‘node templates’. The individual nodes join sets of PCs with a common configuration. These
sets in turn can join super-sets. L4C supports CERN Scientific Linux 3 and 4, including all
regular security updates and enhancements. Basic templates are maintained by L4C support,
all other templates by the system experts allowing him full flexibility. The templates can be
hosted in central or user owned configuration databases (CDBs).
The Linux installation of a PC from scratch uses CERN’s ‘Automated Installation
Management System’ (AIMS) service and is based on RedHat’s ‘Kickstart’ software. Booting
a Linux PC via the network using PXE Boot (Preboot Execution Environment) pulls the
‘Kickstart’ and configuration profile from the CDB. From this information, quattor is able to
perform a full automatic installation.
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
ANNEX FOR
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
35 / 59
From here, the CDB informs a node about any new changes in the configuration profile.
Triggered by CDB, a local daemon then downloads the modified profile and subsequently
applies the changes. For each node in the CDB, a webpage shows the names, versions,
hardware architecture and the repository from which all the packages for a node are to be
installed. However, in order to verify that all packages are installed as foreseen the system
expert has to log onto that node.
Computer Management Framework (CMF)
CMF [4], on the other hand, has implemented a topdown structure, focussing on sets of PCs
with a defined configuration. The installation of PCs is handled through a top-down tree of
so-called ‘Named Sets of Computers’ (NSCs). Each NSC assigns a list of applications to its
members, where these members can be individual PCs or other, nested NSCs. A PC that is
member of different NSCs will receive the applications from any of them. CMF is taking care
of clashes. Depending on the type of NSC, the administrator of the NSC, i.e. the system
expert who maintains that NSC, has full autonomy of his configuration (‘locally managed’),
or CERN’s IT department is still providing a basic configuration (i.e. that of an office PC),
and takes care of patches.
A long list of basic applications has been provided as CMF ‘packages’. Packages can be
applications but also group policy settings, regular scheduled tasks, or Visual Basic scripts.
NSC administrators can easily create additional packages using the CMF web-interface and
simple scripting. The installation of a package can be either forced (‘applied’), such that it is
installed automatically after a small notification period, ‘denied’, such that it cannot be
installed at all, or offered (‘published’) to the interactive user. In the latter case, the interactive
user can use the CMF web-interface to select/deselect packages he wants to install/de-install.
The installation of these packages is controlled via a small local program (the ‘CMF Agent’),
being installed on every CMF Windows PC. It handles all pending (de)installation tasks,
interacts with the user, and performs regular inventory checks which are passed back to a
central CMF configuration management database.
The initial installation from scratch is based on the Windows pre-installation environment and
PXE Boot. The preferred operating system (Windows XP SP2 or Windows 2003 terminal
server) can either be chosen from a list, or predefined for automatic installation on the CMF
web-interface. After the operating system has been installed, the CMF Agent controls the
subsequent installation of all packages being applied to that particular PC.
With PXE Boot and the proper configuration of his NSCs, the system expert has full liberty to
install sets of PCs in parallel or to run pilot installations prior to mass deployment. CMF
ensures that all members of a NSC are identically configured, and that corrections or
modifications are propagated accordingly. The configuration database provides always an up-
to-date status (e.g. PC hardware, operating system version, patch levels, installed applications
and their versions) via the CMF web-page. Queries can be run to assess a particular situation
(e.g. listing all PCs missing patch XYZ).
Authentication & authorization
Several dedicated authentication & authorization schemes have been developed at CERN
serving the accelerator complex [5] and LHC experiments [6]. These are based mainly on
general standards like Role-Based Access Control [5] or on commercial solutions [6]. Details
can be found in these proceedings [5][6].
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
ANNEX FOR
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
36 / 59
Major challenges still to be overcome are the generalization to one common central scheme at
CERN.
User training
A series of awareness campaigns and training sessions have been held for users, operators,
and system experts at CERN. Monthly CNIC meetings provide a forum for questions and
discussions.
Furthermore, CERN has raised aspects of Control System Cyber Security at several
conferences and workshops (e.g. at the CS2/HEP workshop [7]), interacted with major
vendors of control systems, and is now leading the ‘EuroSCSIE’, the European Information
Exchange on SCADA (Supervisory Control And Data Acquisition) Security, with members
from European governments, industry, and research institutions that are dependent upon
and/or whose responsibility it is to improve the security of SCADA and Control Systems.
Incident response & system recovery
Even with a stringent Security Policy incidents can never be prevented completely. Therefore,
handling incidents on a Domain have been and will be jointly performed by CERN’s
Computer Security Team and the corresponding Domain Administrator. The acting Computer
Security Officer has the right to take appropriate actions in justified emergency cases.
After incident analysis, the central installation schemes CMF and L4C allow for a rapid
system recovery by the corresponding system expert.
Summary
Due to the continuing integration of common IT technology into control systems, the
corresponding IT security vulnerabilities and cyber-attackers end up threatening control
systems, and, thus, CERN’s operation and assets. However, control systems demand a
different approach to security than office systems do.
This paper has presented a thorough rule-set to secure CERN’s control systems. Its
implementation uses a ‘Defense-in-Depth’ approach based on network segregation, central
installation schemes, authentication & authorization, user training, incident response &
system recovery, and security auditing.
References
[1] S. Lüders, Control Systems Under Attack?, ICALEPCS, Geneva, October 2005.
[2] U. Epting et al., Computing and Network Infrastructure for Controls, ICALEPCS, Geneva, October
2005.
[3] Centre for the Protection of the National Infrastructure (CPNI), Good Practice Guidelines Parts 1-
7, London, 2006.
[4] I. Deloose, The Evolution of Managing Windows Computers at CERN, HEPix, Rome, April 2006.
[5] S. Gysin et al., Role-Based Access Control for the Accelerator Control System at CERN; K. Kostro
et al., Role-Based Authorization in Equipment Access at CERN; and A. Petrov et al., User
Authentication for Role-Based Access Control, these proceedings.
[6] P. Chocula, Cyber Security in ALICE, these proceedings.
[7] S. Lüders, Summary of the Control System Cyber- Security CS2/HEP Workshop, these proceedings.
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
ANNEX FOR
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
37 / 59
A2 Network connectivity management
The following information has been published in the proceedings of the Trans-European
Research and Education Networking Association (TERENA) Networking Conference 2007,
which took place at the Technical University of Denmark, 21 – 24 May 2007
http://tnc2007.terena.org/programme/presentations/show.php?pres_id=119
A2.1
Firewalling beyond 10Gbps
Abstract
CERN is building the LHC, one of the most complex scientific instruments ever built. The
whole system relies on the LHC Computing Grid (LCG) to analyze all the data that will be
produced by the experiments. The LCG computing model requires high speed connectivity
between CERN and the remote institutes that will help with analyzing the produced data.
Thus, CERN had to re-design its whole network infrastructure, from the detector pits to the
connections to remote sites scattered around the world.
An aggregate WAN (Wide Area Network) bandwidth that already exceed 20Gbps, dozens of
servers accessible world wide, data to be pushed to remote sites at the highest possible
throughput, hundreds of freshly deployed applications, these are factors that have posed new
challenges to the computer security at CERN and that require a powerful but still flexible
firewall infrastructure. This paper describes how this security challenge has been tackled and
how the designed solution was deployed.
Terms definitions
In this paper the term ‘firewall’ refers to a generic device (or set of devices) that can perform
different actions on the flowing through traffic, from basic stateless packet filtering to deep
packet inspection.
The CERN network for the LHC
CERN is the centre of the LCG and the source of all the data coming from the LHC
experiments. The LCG is connected to the eleven Tier1 computer centres by mean of the
LHCOPN. Every Tier1 is required to be connected to CERN with a 10Gbps link, due to the
high volume of traffic expected. Still at the time of writing, the market cannot offer any
stateful firewall with such capabilities, therefore CERN decided to connect the LHCOPN
links directly to the so called ‘LCG Backbone’, and to protect its infrastructure using stateless
Access Control Lists (ACLs) on the boundary routers. Thus, the firewall described in this
paper is not filtering the Tier0-Tier1 traffic.
However, due to the routing architecture of CERN, the general purpose Internet is used to
provide last resort backup connectivity to the Tier1s, in case their LHCOPN links are down.
Furthermore, CERN is both Tier0 and Tier1, so it has to transfer data also to the Tier2
centres, and this happens via the general purpose Internet.
Furthermore, as soon as the LHC will start, hundreds of scientists are expected to be hosted
daily at CERN, everyone with his/her own network connectivity needs. These were among
the main reasons that pushed the upgrade of the main CERN firewall. The CERN network
and its connection to the Internet and to the Tier1/2 centres is depicted in this picture:
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
ANNEX FOR
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
Security challenges
The CERN computer infrastructure serves the scientific community that builds the LHC
accelerator and detectors and will use the LHC for their physics experiments. To better
accomplish this task, access to data, applications, documents from outside to CERN and must
be simple and straightforward for everyone. Thus the CERN network is built in a way that
every computer connected to the campus might be accessible from anywhere.
At the same time, security threats are increasing both in quantity and in the damage they can
cause. The common practice to reduce the risk is to hide a network as much as possible, but
this clashes with the goal of providing easy access.
CERN had to develop an ad-hoc strategy to conciliate these two requirements: the
accessibility is obtained using public IP addresses for every machine connected; security is
provided using a fine grained filtering on the firewall, allowing connectivity from outside
only to the strictly necessary ports in every single host. This means a big effort in collecting
and deploying all the users’ requests, thus the need of automate this process as much as
possible.
The size of the community using the CERN network posed another issue: the great amount of
required bandwidth to the Internet. The generic Internet is by far the major source of attacks,
so most of the security checks have to be done one the traffic coming from there. For this
PUBLIC
38 / 59
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
ANNEX FOR
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
39 / 59
purpose, CERN has been having a stateful firewall that interconnects the campus network
with the Internet, but the traffic load has always been greater than the capacity of any
commercial product could provide. CERN’s strategy to solve this problem is to offload some
well know and well defined traffic from the stateful firewall, with the double advantage to
reduce the load on it and also to avoid to introduce any sort of throttling to data transfer that
requires high bandwidth. In order to have this bypass granted, users must define their traffic
in term of source and destination IP addresses and application ports.
CERN Firewall upgrade
With the major challenges posed by the LCG and future LHC operation, the CERN IT
Department launched a project early 2006 for a major upgrade of the CERN main firewall. A
team of network engineers, software developers and security experts were dedicated to the
design and deployment of a new system that could tackle the challenges posed by the LHC.
Requirements for the hardware
The requirements for the new CERN main firewall were:
• Redundancy: two symmetrical paths must exist, one active and one passive, both with the
same capabilities and bandwidth capacity.
• Stateful firewalling: generic traffic must go through a stateful firewall for deep
inspection. It must handle at least 2Gbps full-duplex without service degradation.
• High speed traffic offload: it must be possible to offload very well defined high speed
data flows from the state-full inspection path, using policy based routing. The new system
must provide at least 40Gbps.
• Bandwidth throttle: it must be possible to throttle the bandwidth used by potentially
harmful and policy defined traffic.
• Possibility to add additional alternative paths and traffic mirroring capabilities, in order to
send part or all of the traffic to other screening devices, like IDS (Intrusion Detection
system) and IPS (Intrusion Prevention System).
• Flows information: the system must provide information about all the traffic flows.
• Scalability: the system must scale in order to handle the traffic growth.
• Modularity: single components should be easily upgraded as they become obsolete.
• Manageability: it must be possible to configure the system with an automatic software
framework
The desired architecture is depicted in this diagram:
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
ANNEX FOR
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
Requirements for the management framework
• allow the complete configuration of the system
• re-use of all the information regarding hosts and IP address assignments already stored in
the CERN’s
• Network Database
• vendor independent (able to configure any kind of equipment)
• architecture independent (to be re-used with any firewall)
• provide a web interface to the end-users to request firewall openings
CERN firewall architecture
The architecture designed and the equipment selected met all the requirements. The following
picture shows what has been implemented:
PUBLIC
40 / 59
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
ANNEX FOR
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
Routing through the Firewall
The Main CERN Firewall connects the Campus network to the External Network. The CERN
External Network provides the CERN community with the connectivity to the generic
Internet. All WAN links, except the ones that belong to the LHCOPN, are terminated to the
External Network routers: Geant2-IP, Esnet, Abilene and three commercial Internet Service
Providers guarantees reachability to any Internet destination.
Dynamic routing protocols
The External Network uses OSPF as IGP (Interior Gateway Protocol); this OSPF instance is
completely independent from the one used in the campus network.
The external network provides the default route to the CERN campus network by means of
iBGP peerings established through the primary and secondary gates. There is double full
iBGP mesh among the two main campus backbone routers and the two main External
Network routers. The primary mesh peerings are established using the loopback interfaces
number 1 and all go via the primary gate; the secondary mesh peerings are established using
the loopback interfaces number 2 and all go via the secondary firewall. The external routers
assign a MED of 100 (preferred) to the prefixes received and announced into the primary
mesh; they assign a MED of 50 to the prefixes received and announced into the secondary
mesh.
PUBLIC
41 / 59
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
ANNEX FOR
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
In case of a failure of one of the two gates, all the peerings of a mesh will go down, and so the
survived ones will be preferred. This mechanism is necessary because it was decided to
configure the secondary gate in hot-standby and not load-share, so to have the two firewalls
completely independent one from the other.
R01EXT and R02EXT also peers, and they are Route Reflectors in a cluster. R01GPN and
R02GPN are route reflector clients, thus they don’t need to peer one with the other.
Static routes
Static routes redistributed into OSPF are used to force each mesh to go via two different
paths. R01EXT has static routes to the Loopback1s of R01GPN and R02GPN pointing to
R01FWS, and it redistribute them into the external OSPF instance. R02EXT has statics to
Loopback2 of R01GPN and R02GPN pointing to R02FWS, and it redistribute them into the
external OSPF instance. R01FWS has static routes to loopback1 of R01EXT and R02EXT
pointing to R01EXT, and it redistributes them into the internal OSPF instance. R02FWS has
static routes to loopback2 of R01EXT and R02EXT pointing to R02EXT, and it redistributes
them into the internal OSPF instance. The redistribution is used to ensure that alternative
paths will be inside the two networks in case of link failures.
Default route
In the campus network, the two main routers generate a default route for all the campus and
distribute it with OSPF. The default route learnt by BGP is fact not redistributed, but used
locally to send the traffic to the active gate. The path is chosen using the different MED value
set by the External routers.
Internal prefixes.
The two main Campus Network routers announce five public CERN prefixes to the external
network via iBGP. These prefixes are necessary to the External network to decide which
firewall path to use. Also, they are used to announce them into the eBGP peerings.
PUBLIC
42 / 59
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
ANNEX FOR
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
Gate routing
The two firewalls, primary and secondary, behave in the same way, so only one will be
described here. Due to the double connection to F01FWS (the Stateful Firewall), the packets
in R01FWS have to be routed accordingly to their source and destination addresses, and not
only considering the destination as usually happen. This technique of routing packets not
accordingly to the routing table is called Policy Based Routing (PBR) and is realized using
ACLs. R01EXT route the packets in this way:
• all the packets coming from R01GPN and R02GPN are policy routed to F01FWS, except
the packets that have to go via the fast path that are policy routed to R01EXT.
• the packets coming from the external interface of F01FWS are policy routed to R01EXT
the packets coming from the internal interface of F01FWS are routed accordingly to the
OSPF routing table and sent to R01GPN or R02GPN.
• the packets received from R01EXT are policy routed to F01FWS, except the one the path
that has to go via the fast path that are routed accordingly to the OSPF routing table and
sent to R01GPN or R02GPN.
R01FWS doesn’t accept the default route it receives in OSPF and has a static default route
pointing to NULL. There are two reasons for this: the first reason is that there must be no
default route to the external network to avoid that any packet might be sent out if not coming
from the firewall or decided by the HTAR policy. The second reason is to avoid to send
packets coming from the external network and directed to CERN subnets not in use (the
OSPF routing table contains in fact only the sub-nets in use).
Management Framework
A research institute like CERN has a diverse user community running many applications
using different protocols. The environment is highly dynamic, notably Grid and physics
application software require the management of higher of port ranges that are rarely used on
other sites. Thanks to the database driven network management and dynamic work-flow
system, the CERN Computer Security team is now able to quickly open specific ports for
dedicated hosts upon user requests.
Transforming the generic ACLs into instructions for routers and firewall modules from
multiple vendors has been another major challenge. This is achieved by means of modular
PUBLIC
43 / 59
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
ANNEX FOR
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
44 / 59
software components, where the ACL-generation from firewall rules is done by an ACL-
generation module and then feed to a router-compiler with vendor-specific modules. The
system has been deployed step-wise onto different types of network equipment that is used
for gateways and firewalls within the CERN corporate network as well as the external
firewall.
Following the deployment of the high speed firewall, IDS and packet inspection also require
far more computing resources. A suite of web-based management applications for the
definition of the firewall filters, gates and rules has been implemented. In addition, software
has been developed that extracts the rules from the database and translates them in the
configuration commands for network equipment of different vendors. The main software
components of the framework deployed are:
• a Gate Model with a Database Schema that implements all the components needed to
represent the firewall
• a web based Management Framework that allows for the definition and the correlation of
all the components of a gate as well as the rules to be applied in the various interfaces of a
gate.
• a web based work-flow application that allows for End-user requests for changes to the
firewall configuration, and to easily and securely implement them.
• a Configuration Manager that understands all the information gathered in the database
and builds the configuration for all the network devices that compose the gate.
Gate Model
The combination of a highly dynamic environment and granular security policies have meant
that changes in the firewall configuration are requested on a regular daily basis. The core of
the CERN network is a database, which contains a complete description of the network
topology as well as all devices connected to the network. The database provides a framework
for network configuration management, ranging from end-user requests to connect new
devices to management of routers and switches. A flexible generic network interconnection
model has been designed and implemented within the network database to manage the
interconnection of different network domains and sites. The model can be applied to external
firewalls as well as internal firewalls and screening routers within a network site. The model
supports filtering of port and IP-based access control, as well as macros that create adhoc
rules for specific attacks and special configurations for high throughput computing. The main
components of the model are:
• Domain = A representation of a well defined network area.
• Gate = a system that interconnects two Domains by mean of Interfaces and capable of
filtering the traffic exchanged.
• Interface filter = relationship between a Filter and the Interface to which is applied.
• Filter = a set of Rules applied to one or more Interface Filters of a Gate.
• Rule = an unidirectional communication relationship between two IP prefixes belonging
to two Domains connected by a Gate.
• Interface = An IP interface belonging to a network equipment.
Domain
It represents a well defined network domain, i.e. a set of network devices and IP network
prefixes. IN the built system, the domain is just a name to help in the definition of the gate.
Gate
It is the set of devices that interconnect two domains and that filter the network traffic they
exchange. In order to simplify the creation of the configuration for the devices, the two
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
ANNEX FOR
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
45 / 59
domains are identified in Left Domain and Right Domain The gate provides the list of
Interface Filters used in the gate.
Interface filter
It is the definition of an action that a device has to do on the packet traversing an interface.
The action can be:
• Input Filter, i.e. an ACL applied to the packets entering the interface,
• Output Filter, i.e. an ACL applied to the packets leaving the interface
• PBR, i.e. a routing policy applied to the packet entering the interface
The Interface filter links to the interface where it has to be applied and to a set of Filters that
build the ACL. In order to allow the re-utilization of Filters in different directions, the
Interface Filters include the information of which domain is faced by the interface (Left or
Right)
Filter
It is a set of Rules.
Rule
It is a single Access List Entry, i.e. a set of characteristics that can match an IP packet. The
characteristics considered are: Layer3 or Layer4 protocol, source and destination IP
addresses, source and destination Layer4 protocol port.
Database schema
The model had to be represented with a database schema. This schema was designed with two
main goals in mind: to use as much information as possible about networks and hosts already
stored in the CERN network database; and also to provide a schema capable of representing
not only the planned CERN main firewall structure, but any firewall in general.
The first goal was achieved by using database ids for referencing firewalled CERN entities:
networks, services and hosts. This way, when a machine with some firewall privileges
changes its IP address, nothing changes in terms of accessing it through the firewall.
The second goal was achieved with the definition of the Filter concept. A Filter, logically, is a
set of rules that can be applied to any interface of any machine that is a part of any firewall.
For greater flexibility filter can consist not only of a set of rules, but can call database or perl
procedures which, in turn, will dynamically generate the filter content each time the compiler
runs.
Firewall privileges for large number of machines that require the same type of access through
the firewall (like the large clusters used for the LCG, or the pool of servers providing public
services) can be easily managed with the use of Sets, a database table that can group a list of
machines. Firewall rules can be defined using Sets as destination address.
The model also keeps track of the changes made to the rules during a Gate lifetime, by storing
historical information in dedicated tables.
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
ANNEX FOR
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
Management interface
The framework provides a web interface for network and security experts to manage the
firewalls. The management interface was developed extending an already existing platform in
use at CERN for the management of the network infrastructure.
The interface allows the definition of all the components needed for a gate configuration.
Configuration manager
The framework provides a tool called ‘cfmgr-gate’ that extract from the Network database all
the information related to any gate and use them to build the final configuration of all the
devices involved. cfmgr-gate is part of the software tool already used at CERN to configure
and manage the Network devices.
cfmgr-gate, before applying the computed configurations, checks that they can be supported
by the destination devices. This check is especially needed to avoid to exhaust the memory
resources of router and switches that have to host the ACLs. Router and switches, in order to
apply access list at wire speed without overloading their CPUs, have to store the ACLs in a
special memory used by the network processors. This memory, due to its characteristics of
fast speed and access, is very expensive, so limited in quantity. If an ACL exceeds the
available space, its process happen in the device’s CPU, degrading in this way the overall
performance. Since the software framework uses a database with virtually unlimited resources
that would allow the creation of ACLs of any size, special checks have to be performed in
order to avoid the performance degradation.
End-users requests
The framework provides a web interface that allow end-users to request any kind of firewall
opening for the hosts they own or manage.
The web interface was developed extending an already existing platform provided to the
CERN users to manage the network connectivity of their equipment.
Users who needs a firewall opening must fill the form provided by the framework. As a
consequence a request is sent to the Computer Security Team for approval. The Computer
Security team can accept or reject the request following the evaluation of the request and a
security scan of the involved equipment. The approval is made via another web interface that
PUBLIC
46 / 59
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
ANNEX FOR
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
47 / 59
automatically creates the correct rule in the Network database and allow the configuration
manager to apply it to the relevant firewall.
Implementation experience
The stateful firewall bypass has been implemented using a single switch-router with multiple
10Gbps ethernet interfaces and capable of policy based routing of traffic at wire-speed. The
switch-router is also capable of pre-filtering the traffic directed to the stateful firewall by
means of access control lists executed by the hardware. This is done in order to offload some
tasks from the stateful firewall. Also, it can export flow information without negative impact
on the performance. Several tests have been carried out in order to measure the number of
ACL entries supported by every device. The results have been used to enable the
configuration generator software to simulate resource depletion, thereby avoiding harmful
configuration.
Failover and BGP
The redundancy has been implemented using two identical sets of equipment; one set is the
active path, the other set is the hot-standby path. The failover mechanism is implemented
using iBGP routing among the two main campus backbone routers and the two main external
routers, as explained beforehand.
The reason of the double mesh is because iBGP peers need to be fully meshed to avoid
routing inconsistency. The use of eBGP would have reduced the number of peerings, but it
would have implied the use of a private ASN for the Campus network. Unfortunately, the
stripping of the private ASN and the inclusion of the Campus prefixes in the public CERN’s
ASN would have required many configuration changes in the External Network routers, so it
was abandoned.
Another option would have been the use of BGP confederations, but that’s would have also
required several changes in the running configurations of the External Network routers, so it
was also abandoned.
TCAM exhaustion
Multilayer security versus high performance and availability has been an important point of
discussion with our colleagues of the Computer Security team at CERN. Can we implement a
policy in every device participating in the network so that a breach in one of the links won’t
be enough to put the whole chain at risk? Concerning this project, the demand from the
Security Team was clear: apart from the stateful inspection and advanced filtering provided
by the firewall, implement in the routers access control policies which were granular enough
to filter based on application port numbers.
The Gate model defined above is wide enough to support application filtering on routers, but
because of the abstraction provided by the user interface, the gate administrators will not be
aware of the impact of the filters on the real hardware, in fact, they probably won’t know
what hardware is implementing the gate.
As responsible for the firewall system our concern is to guarantee that only configurations
that fit in the hardware resources will be configured on the routers. This sounds
straightforward, but unfortunately router manufacturers don’t provide information on how the
different elements in an access list entry impact the resources available. For example, an
access list entry containing layer 4 operators will consume more resources than another
without them, and even the resources consumed vary depending on the combination of these
operators. To accomplish this objective several scripts were developed to populate access lists
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
ANNEX FOR
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
48 / 59
with all possible combinations of operators and to read the resources consumed. The data
extracted was used to build the equations that matched the behaviour in the resource
consumption for every particular case, and these equations were implemented in algorithms
for every different router model.
Glossary
ACE = ACL Entry
ACL = Access Control List
ASN = Autonomous System Number
BGP = Border Gateway Protocol, inter domain routing protocol
CPU = Central Processing Unit
HTAR = High Throughput Application Route
Layer 4 operator = Operator used in an access list to select application ports, like http or ntp.
Common layer 4 operators are equal, greater/lower than, range and not equal
LCG = LHC Computer Grid
LHC = Large Hadron Collider
LHCOPN = LHC Optical Private Network
OSPF = Open Shortest Path First, intra domain routing protocol
MED = Multi Exit Discriminator, BGP parameter used in the route calculation algorithm
WAN = Wide Area Network
Reference
Integrated Site Security for Grids (ISSeG) EU FP6 Project no: 026745
Worldwide LHC Computing Grid (LCG),
Acknowledgments
The security related aspects of this work have been co-funded by the European Commission
ISSeG project,
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
ANNEX FOR
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
49 / 59
A3 Security mechanisms and tools
A3.1
Security assessment tools and framework suitable for the CERN environment
Requirements
The requirements for a security assessment tool suitable for the CERN environment were
developed from a number of identified use cases:
1) A site user requests that a service running on a machine managed by him/her be made
available off-site, thus necessitating an opening in the firewall. The machine needs to be
checked for known vulnerabilities before the firewall is opened to reduce the risk of
immediate infection.
2) A new vulnerability is discovered in a common service. A site-wide scan is quickly
needed to identify vulnerable machines on site. The check for the vulnerability might be
code written by a security expert on site or a third-party tool.
3) Ongoing background checks run continuously to build a database of services running on-
site and to detect known vulnerabilities in them.
Third party software
A number of third-party software packages were investigated.
1. Nessus
Nessus is the most popular vulnerability scanner, with plugins to test for tens of thousands of
vulnerabilities. However, license changes in the latest version, hard-to-interpret output and a
lack of fine-grained control over the scanning process make it hard to deploy in large sites. It
is, however, very useful as a model for how to write a vulnerability scanner and can be
incorporated as a plugin into a framework.
2. Nmap
Nmap is a port scanner, which has been extended to detect details of the services running on
each port and the host operating system. It performs these tasks well, which makes it ideal as
the port scanning component of a framework.
3. Metasploit Framework
The Metasploit Framework is a framework for creating exploits. It is very suitable as a basis
for creating vulnerability checks.
4. THC-Hydra
THC-Hydra is a brute-force password cracker for a variety of services. This also is valuable
as a plugin.
The above tools can all be integrated into a framework, giving the power of each specific
third-party tool within the control of the framework.
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
ANNEX FOR
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
50 / 59
Scanning process overview
The site scanning process was divided into a series of steps to be performed in strict
succession.
1. Host discovery
First, a list of targets to be scanned must be generated. A single or small number of hosts may
be specified directly by the user of the program, but for larger numbers an automatic process
is required. For example, the list might come from an existing asset management database,
and can be either inclusive (e.g. all known web servers) or exclusive (e.g. removing
computers that are in a ‘do not scan’ list).
2. Open port discovery
Once a list of target hosts has been created, the open ports on each host must be discovered.
This is often called port scanning, and is done well by existing software such as nmap. In
some cases, where the purpose of the scan is to test a service running on a known port, a full
port scan can be avoided.
3. Service discovery
When a host has been identified as accepting connections on a given port, the actual service
(application) running on that port must be determined. Although standard port numbers are
assigned to different services (e.g. web servers usually listen on port 80) there is no guarantee
that this corresponds to the actual service running on that port.
Recent versions of nmap include a number of heuristics for determining the service running
on an open port, including probing the application and version number of the service and
details of the application protocol used.
4. Service scanning
Ultimately the vulnerabilities are almost always in the application providing a service. After
the three previous steps have been completed, the host, port number and service are all known
and the application can be probed for vulnerabilities. This structure facilities the identification
of applications running on non-standard ports.
As vulnerabilities are tested the fact that the test was done is recorded (to facilitate the
identification of non-vulnerable hosts), along with the result of the scan. Certain tests might
not be 100% accurate and this is reflected in the recorded test result. For example, many
Linux distributions backport security fixes to applications, so discovering an ‘old’ version of
the software does not necessarily mean that the application is vulnerable. In general, the only
way to be 100% certain that an application is vulnerable to an exploit is to test the exploit
directly. However, this can be potentially destructive and so should only be done with care.
5. Reporting
Once the scan is complete the results need to be communicated. The same results might well
need to be reported in different formats, such as a simple text summary to the user for a single
scan, or a detailed machine-readable format for storage in a central database.
It is important to store alongside the scan itself the configuration of the scanning process.
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
ANNEX FOR
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
51 / 59
Scanning program design
The design chosen consists of a master program that performs the five stages described above
in order. The actual implementation of each stage is contained in plugins, small modules that
perform single functions and can be individually activated and deactivated. The plugin types
are as follows:
1. Host discovery plugins
These are responsible for generating a list of hosts to scan. One plugin might read a list of IP
addresses from a file, while a more complex one might query a site-specific database.
After the host discovery plugins have run, the control program has a list of potential targets.
2. Host filter plugins
To respect site-specific ‘do not scan’ lists, host filter plugins identify hosts that should not be
scanned further and mark the host as such. A simple one might query a site database, a more
complex one, suitable for automated scanning, might identify hosts that have been scanned
recently.
After the host filter plugin has run, the control program has the final list of targets.
3. Service discovery plugins
Service discovery plugins perform the open port and application identification (stages 2 and 3
above) for individual hosts. These two are merged into one plugin because modern port
scanning programs tend to perform both functions. A typical plugin might run nmap against
the host, but when only a specific port is being tested then the open port discovery can be
skipped.
After the service discovery plugin has run, the control program has a list of services with their
associated port numbers for each target host.
4. Service filter plugins
Service filter plugins identify services that do not need to be probed further. This is typically
for the case where the user is only interested in a single service. The service might be running
on a non-standard port, so the service discovery step is still required, but this allows non-
interesting services to be marked as done.
After the service filter plugin has run, the control program has a final list of services to test.
5. Vulnerability testing plugins
These plugins perform the actual vulnerability testing. It might be anything as simple as
checking the version number of the application running, to testing for default passwords, or
be as complex as running actual exploit code against the application.
Each vulnerability testing plugin records the fact that it has run alongside its result. After this
stage, the control program has a full list of hosts, services, and vulnerabilities.
6. Report plugins
Report plugins take the output from all previous steps and generate a report for either human
or computer consumption.
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
ANNEX FOR
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
52 / 59
Scanning process control
This plugin architecture allows new functionality, such as a third-party scanning code, to be
quickly added to the scanning process. The overall scanning process is controlled by selecting
which plugins to activate.
For example, to scan a single host for all known vulnerabilities, a simple host discovery
plugin is used that reads the host directly from the user (e.g. from the command line). All
service discovery and vulnerability plugins are activated, and the user can choose the form of
the output by his choice of the report plugin.
To scan an entire site for a single vulnerability, a more complex host discovery plugin is used
(e.g. to extract a list of hosts from an asset management database), a service filter plugin is
activated, and only the single vulnerability test plugin of interest is enabled. Again, the user
can choose the format of the output with the choice of report plugin.
To assist with plugin activation, plugins are marked with zero or more ‘tags’ that indicate
their behaviour. Example tags might be ‘aggressive’, meaning that this plugin behaves in a
way that might be considered aggressive by the target, such as a full port scan. Similarly,
vulnerability test plugins that risk to disrupt or damage the target machine (e.g. by causing
data loss or a reboot) might be tagged as ‘dangerous’. By choosing to activate or deactivate all
plugins with certain tags, the user can confidently retain high level control over the scan
without worrying about the details of individual plugins. On-going background scans, for
example, should probably not include running ‘dangerous’ plugins.
At any stage the state of the scanning process can be saved to a file. This file contains the list
of work already done (e.g. devices and services scanned so far and their results) and the work
to be done. This allows the scanning process to be paused and re-started, distributed over
several machines, and the distributed results merged together.
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
ANNEX FOR
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
53 / 59
A4 Administrative procedures and training
Below are three articles aimed at training application developers and/or general users:
• Web applications security, CERN Computer Newsletter (CNL) Article, November 2007,
http://cerncourier.com/cws/article/cnl/31988
• Suggestions for designing and writing secure software, CERN Computer Newsletter
http://cerncourier.com/cws/article/cnl/31106
• How to avoid a false sense of security, CERN Courier, April 2007
http://cerncourier.com/cws/article/cern/29863
A4.1
Web applications security, CNL Article, November 2007
There is a clear shift towards Web applications in the vulnerabilities trends. Exploits are easy
to build and targets easy to find. It is therefore important to adapt our programming and usage
practices to these threats.
The rise of Web applications
Web applications are commonly used from a Web browser and cover a wide range of
activities, such as e-banking, webmail, online-shopping, community websites, blogs, vlogs,
network monitoring and bulletin boards.
In recent years, the development of such applications has been significant and today rich
Internet applications offer complex, real-time interactions with users. For instance, Web
operating systems, such as eyeOS (
), offer many functionalities previously
available only using traditional operating systems.
While Web applications have become ubiquitous, they also present new security risks. It is
important to identify and understand them when developing, hosting or simply using these
applications.
Security risks related to Web applications
Firstly, it is generally difficult for the service manager to keep up-to-date with security
patches. This is a common issue for services in general, but it may be a particularly
challenging problem for Web applications. While this could be improved by a better design
and packaging, it is often impossible to upgrade Web applications automatically. This also
requires the service managers to actively monitor announcements lists of the application
vendors. In addition, customisation of the application may be required to match the need of
the community, which may result in additional delay to upgrade.
Secondly, Web applications are often easy targets for attackers. As a relatively recent
development, they use non-mature code compared to traditional network services.
Unfortunately exploits (malicious code exploiting software vulnerabilities) are generally easy
to prepare, remotely executable, cross-platform, and require no compilation. This helps
attackers designing effective and scalable automated attacks. The discovery of vulnerable
installations can be very fast, convenient and quite silent by using search engines to detect
known vulnerable patterns (generally filenames) of specific Web applications.
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
ANNEX FOR
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
54 / 59
Not only critical services are at risk
It is important to understand the threats against which Web applications need to be protected.
Whilst several years ago attackers were mostly attracted by challenge or fame, attacks have
become more professional and many attackers are now motivated simply by money (Phishing,
SPAM, extortion, DDoS: Distributed Denial of Service, Click fraud, etc.). Malware kits can
now be professional: user-friendly, modular (for instance, it is possible to buy extra plugins),
and some even include one year of support. For more information, there is an interesting
article about the Storm worm, available at
http://www.schneier.com/blog/archives/2007/10/the_storm_worm.html
A result of this professionalisation is that attackers need bandwidth and platforms to operate
from. In order to obtain and maintain a sufficient amount of compromised resources, attackers
usually choose the easier target. No matter how insignificant a particular service may be, it is
worth something for an attacker: a critical production service may be worth as much as an
unmaintained photo album.
For instance, once a Web application has been successfully compromised, the attacker may
gain the ability to:
• Cause damage to the reputation of the organization running the service
• Execute code remotely with the privileges of the user running the Web server. This is
sufficient to run Botnets and SPAM engines and gives the attacker additional
opportunities to try to spread the attack further on the system or against third party
services
• Access all the information hosted on the Web server
• Change the content of the website (defacement)
• Delete files or damage Web services provided by the host (DoS: Denial of Service)
Shift in the vulnerability trends
There are different types of Web application flaws, but the most common are caused by lack
of user input validation. Data coming from the client must indeed be filtered to ensure no
malicious content is passed to the server. If this is not performed correctly, the resulting
vulnerabilities include:
• Cross-Site Scripting (XSS): If a parameter is not sanitised correctly, an attacker may be
able to control the content of the vulnerable page. To perform the attack, the victim is
often tricked in clicking on a malicious link, but this is not always necessary for the attack
to be successful.
• SQL injection: Some parameters retrieved from the user’s Web browser may be used to
perform database queries. If a parameter passed from the user to the database is not
correctly filtered, an attacker may attempt to execute arbitrary SQL commands and/or to
gain privileges on the Web application.
• Remote File Inclusion (RFI): By exploiting insecure calls to local files (for example
templates), an attacker may attempt to upload arbitrary code on the server. The resulting
payload (for example a shell written in PHP) may be executed with the privileges of the
Web server.
CVE (Common Vulnerabilities and Exposures) is a standard enumeration for vulnerabilities
and provides yearly statistics about known problems (
http://cwe.mitre.org/documents/vuln-
). In 2001, well-known buffer overflow was the number one vulnerability
affecting software. Cross-Site Scripting, SQL injection and Remote File Inclusion were
almost non-existent. In 2006, a clear shift towards Web applications is visible, and buffer
overflow has become number four, while Cross-Site Scripting is number one, SQL injection
number two and Remote File Inclusion number three.
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
ANNEX FOR
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
55 / 59
Recommendations
Common ways to mitigate the risks related to Web applications are detailed below.
If you are developing Web applications:
• Do not trust any content, parameter, or variable coming from the Web browser; Be sure to
sanitise properly all input before making use of it
• Check all input by design, even if not directly visible to users
• Use the validation functions provided by your environment, avoid relying only on your
own filters
• If you are using a development framework that provides some input filtering, do not
solely rely on it
• Keep your framework up-to-date with security patches – it can be a target as well
• Beware of the information revealed or echoed by error messages/pages
• Requiring (re)-authentication for privileged operations is always a good idea
• Keep your support lists private – it may prevent leaks when a vulnerability is reported to
your team
• Try to reduce the exposure of your Web application and avoid directly exposing it in the
site firewall. For off-site access, whenever possible, require that users connect via a
gateway system, such as Windows Terminal Services or the LXPLUS Linux cluster, as
documented at
http://cern.ch/security/internet
If you are using Web applications:
• Take careful notes of security warnings from the Web browser
• Whenever possible, disable Javascript/Flash/ActiveX .
For example, the use of the NoScript extension (
) for Firefox may help
• Whenever possible, avoid following links to sensitive portals (for example e-banking) and
type the URL by hand
• Whenever possible, logout as soon as possible and/or close your browser when your
session is completed
• Whenever available, use the more secure HTTPS protocol instead of HTTP.
More information about Web applications security is available from
.
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
ANNEX FOR
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
56 / 59
A4.2
Suggestions for designing and writing secure software, CNL Article, September
2007
In this article we give some general tips for creating more secure software, which software
engineers should follow at the various phases of the software development cycle. The list
doesn’t contain all possible advice but it does include the most important – the suggestions
that are worth keeping in mind when designing and developing almost any kind of software
with any programming language.
Security should be seen as part of the system from the very beginning, and not added as a
layer at the end. The latter approach produces insecure code, may limit functionality, and will
cost much more (in both time and money).
Architecture
• Modularity: divide the program into semi-independent parts (small, well defined
interfaces to each module/function).
• Isolation: each part should work correctly even if others fail (return wrong results, send
requests with invalid arguments).
• Defense in depth: build multiple layers of defense instead of trusting just one protection
mechanism. For example, validate user input data at entry point, and check again all
values that are passed to sensitive parts of the code (like file handling etc).
• Simplicity: complex solutions are much more likely to be insecure.
• Redundancy: if possible avoid having a single point of failure.
Design
• Make security-sensitive parts of your code small.
• Least privilege principle: don’t require more privileges than you need. For example, run
your code as the least privileged user necessary (don’t run it as root, nor with SUID flag).
Make sure that the account on which you will run your code has only the file access and
execute privileges that your code really needs. Don’t connect to a database with
administrative privileges from your software.
• Choose safe defaults: for example, a random password that users are likely to change
rather than a standard default password that many won’t bother to change.
• Deny by default: for example, when validating user input accept only characters that you
expect rather than trying to block known ‘bad’ characters.
• Limit resource consumption, to limit the likelihood of a ‘denial of service’ attack.
• Fail securely: for example, if there is a runtime error when checking user’s access rights,
assume the user has none.
• In distributed or web applications don’t trust the client: don’t expect it to validate user
input, perform security checks or authenticate users – it all has to be done (again) on the
server side. Remember that HTTP response header fields (cookies, user-agent, referrer
etc) and HTTP query string values (from hidden fields or explicit links) may be forged or
manipulated.
• Cryptography: use trusted, public algorithms, protocols and products. Do not invent your
own cryptographic algorithms or protocols, nor implement existing ones.
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
ANNEX FOR
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
57 / 59
Implementation
• Read and follow guidelines for your programming language and software type.
• Think of the security implications of what your code does.
• Reuse trusted code (libraries, modules).
• Write good-quality, readable, maintainable code (bad code won’t ever be secure).
Coding
• Don’t trust input data – data coming from potentially malicious users is the single most
common reason for security-related incidents (buffer overflow, SQL injection, Cross Site
Scripting (XSS), code inside data etc). Input data includes command-line arguments,
configuration files (if accessible by not-trusted users), environment variables, cookies and
POST/GET arguments etc.
• Validate (sanitize) all input data: consider all input dangerous until proven valid, deny by
default if not sure, validate at different levels; for example, at input data entry point and
before really using that data.
• Don’t make any assumptions about the environment: make sure your code doesn’t break
with modified/malicious PATH, CLASSPATH and other environment variables, current
directory, @INC Perl variable, umask, signals, open file descriptors etc.
• Beware of the race condition: can your code run parallel? What if someone executes two
instances of your program at the same time, or changes environment in the middle of its
execution?
• Deal with errors and exceptions: don’t assume that everything will work (especially file
operations, system and network calls), catch exceptions, check result codes. Don’t display
internal error messages, failing SQL query, stack trace etc.
• Fail gracefully: if there is an unexpected error that you can’t recover from, then log
details, alert the administrator, clean the system (delete temporary files, clear memory)
and inform the user.
• Protect passwords and secret information: don’t hard-code passwords (it’s hard to change
them and easy to disclose), use external files instead (possibly encrypted) or already
existing credentials (like certificates or Kerberos tickets), or simply ask the user for the
password.
• Be careful when handling files: if you want to create it, report an error if it already exists;
when you create it, set file permissions; if you open a file to read data, don’t ask for write
access; check if the file you open is not a link with the lstat() function (before and after
opening the file); use absolute pathnames (for both commands and files); be extra careful
when the filename (or part of it) comes from a user.
• Temporary files: don’t fall for the symbolic link attack (someone guesses the name of
your temporary file and creates a link from it to another file e.g. ‘/bin/bash’ that your
program overwrites). Temporary files must have unique names that are hard to guess (use
tmpfile() for C/C++, mktemp shell command etc).
• Be careful with shell calls, eval functions etc: such functions evaluate the string argument
as code and interpret it, or run it on the shell. If an attacker managed to inject malicious
input to that argument, you’re executing his code.
After implementation
• Review your code and let others review it.
• When a (security) bug is found, search for similar ones.
• Use tools specific to your programming language: bounds checkers, memory testers, bug
finders etc.
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
ANNEX FOR
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
58 / 59
• Turn on compiler/interpreter warnings and read them (‘perl –w’, ‘gcc –Wall’).
• Disable debugging information (‘strip’ command, ‘javac -g:none’ etc).
Further information
Find useful links and language-specific tools at
software/SecurityChecklistForSoftwareDevelopers.pdf
. See also
Doc. Identifier:
ISSeG-Del-D1.1.4-
710058-v3.0.doc
ANNEX FOR
FINAL REPORT ON DEPLOYMENT AT
CERN
Date: 10-Jan-08
Project no: 026745
PUBLIC
59 / 59
A4.3
How to avoid a false sense of security, CERN Courier, April 2007
Even if a PC has up-to-date patches and the latest anti-virus software, and runs a local
firewall, it can still be infected. Technical solutions help, but they cannot prevent all security
problems. Computer users can help by taking simple precautions. The CERN computer-
security team has produced some advice, which is targeted at Windows users, but should be
useful for all platforms.
•
Do not expose your password
. Never use a work-related password for private use
and be wary of e-mails, instant messages and chat that request your password, including
via web links. This trick is known as phishing (password fishing). If you think your
password may have been exposed, change it.
•
Lock your screen when leaving your office.
Locking your screen prevents others
from accessing confidential material. From a Windows PC use [Control] [Alt] [Delete]
and select ‘Lock Computer’, or if you have a Windows keyboard, press [Windows] [L].
•
Be wary of web links and pop-ups.
Some web links and pop-ups can download
malicious software, so think before you click. Some pop-ups can still infect your machine
even if you click ‘Cancel’ or ‘No’ or close the window with the top-right ‘X’ . On a
Windows PC use [Alt] [F4] to close the active window.
•
Ensure software downloads respect copyright and licensing.
This is for legal
reasons and also because ‘free’ versions of copyrighted software can contain Trojan
horses, spyware or other malicious software that could infect a PC. Spyware is often
included in ‘free’ software and is used to trace your activity and even the data you type,
including passwords. Plug-ins may also contain malicious software. If a website requires
a plug-in to view it, avoid using it.
•
Be aware of social-engineering techniques.
Do not click on web links in
unexpected e-mails, spam, instant messages and chat. Do not open attachments that you
are not expecting.
•
Configure your machine to run without administrator privileges.
If you
accidentally execute malicious software, it can cause less damage if you are running
without administrator privileges. As many tasks do not need these, you are recommended
to run without them.
•
Keep yourself informed of your institute's computing rules.
There may be
restrictions concerning software for personal use. When computers are used for personal
as well as professional use, the chance of infections and other security incidents rises –
downloading films, games, music and other personal applications all have risks.
Document Outline
- 1 Executive summary
- 2 Introduction
- 3 Resource management extensions deployment results
- 3.1 Extend central systems management
- 3.1.1 Provide tools for managing homogeneous groups of Linux desktop systems
- 3.1.2 Investigate management tools to enforce security for Linux control systems
- 3.1.3 Extend the flexibility of the central management of Windows systems
- 3.1.4 Investigate management tools to enforce security for Windows control systems
- 3.1.5 Reduce the need for Windows administrator privileges on centrally managed systems
- 3.2 Enforce account policies and procedures
- 3.1 Extend central systems management
- 4 Network connectivity management deployment results
- 4.1 Strengthen network perimeter security
- 4.2 Integrate and extend network perimeter management tools
- 4.3 Protect devices on dedicated networks
- 5 Security mechanisms and tools deployment results
- 5.1 Investigate security assessment tools suitable for the CERN environment
- 5.2 Enhance CERN’s incident detection capability
- 5.3 Evaluate multi-factor authentication technologies
- 6 Administrative procedures and training deployment results
- 6.1 Document and strengthen account administration procedures
- 6.2 Prepare and implement a training plan for improving knowledge of computer security within the organization
- 6.3 Review and update security policies and procedures
- 7 The CERN final deployment report within the ISSeG project
- 8 References
- 9 Acronyms and Abbreviations
Wyszukiwarka
Podobne podstrony:
Referat 3 v3
br9000 mic v3
ITIL v3 Cykl życia usług IT
Cómo se dice Sugerencias y soluciones a las actividades del manual de A2
d1 -fizjo, materiały medycyna SUM, fizjologia, Fizjologia, test
Owi v3
EL SIMBOLISMO DEL AJEDREZ
proxima v3
5 min Intraday v3
LS PR D1
DSC PC1550 v3 0 obs
DSC PC4020A v3 3 ark
etn cwiczenia 3 D1
PBG(LATIN)2szt v3 Medalik Zbawienia
DSC PC1550 v3 0 inst
INSTRUKCJA do ćwiczenia pomiar temperatury obrabiarek v3 ver robocza
więcej podobnych podstron