www.redhat.com
USING aN oPEN SoURCE
fRaMEWoRK to CatCH tHE baD GUY
Built-in forensics, incident
response, and security with
red hat enterprise linux 6
red hat federal solutions architect team
By: norman mark st. laurent, senior solutions architect
editing and technical guidance: shawn wells, technical director
steve gruBB, red hat security lead
4
Part 1: Operational Use of the Red Hat
Enterprise Linux 6 Audit Log Management
Infrastructure
7
1.1 Establishing Policies and Procedures
for Log Management
8
1.1.1 RHEL 6 Log Storage and Rotation
9
1.1.1.1 Non auditd Log Files in /var/log
10
1.1.1.2 auditd Log Files in the Default /var/
log/audit Directory
12
1.1.2 RHEL 6 Remote Host Storage
12
1.1.2.1 Log Management with the /etc/audit/
auditd.conf File
17
1.1.2.1.1 Encryption over the Wire With SSH
Port Forwarding
20
1.1.2.2 Log Management with the /etc/
audisp/audispd.conf File
20
1.1.2.2.1 Log Management with the /etc/
audit/audisp-remote.conf File
27
1.1.3SpecificRHEL6LogGeneration
Settings
28
1.1.3.1 Log Management with the /etc/audit/
audit.rules File
34
1.2 Red Hat Enterprise Linux 6 Log
Management Operational Process
34
1.2.1DefiningRolesandResponsibilities
35
1.2.2 RHEL 6 Forensics and Incident
Response Log Analysis
39
Part 2: Host-based Intrusion
Detection System
40 Bibliography
2 www.redhat.com
abStRaCt
Every security policy provides guidance and requirements for ensuring adequate protection of information
and data, as well as high level technical and administrative security requirements for a system in a given
environment.Traditionally,providingsecurityforasystemhasfocusedontheconfidentiality
1
of the informa-
tion on it; however protecting the data integrity
2
and system and data availability
3
are just as important. For
example, for processing United States intelligence information there are three attributes that require protec-
tion:confidentiality,integrity,andavailability.
InordertorunonUnitedStatesGovernmentSystems,RedHat
®
Enterprise Linux
®
6 has met the a stringent
setoftechnicalsecurityrequirementsforconfidentiality,integrity,andavailabilitytoallowconformance
tobecertifiedandaccredited
4
.RedHatEnterpriseLinuxhasreceivedCommonCriteriacertificationat
EnterpriseAssuranceLevel4(EAL4+)undertheControlledAccessProtectionProfile(CAPP),LabelSecurity
ProtectionProfile(LSPP),andtheRole-BasedAccessProtectionProfile(RBACPP).
Security for Red Hat Enterprise Linux 6 begins with a core feature known as SELinux. SELinux delivers
astrongandflexibleMandatoryAccessControl(MAC)frameworktoenforcerole-basedaccesscontrol
and multi-level security. Security-Enhanced Linux support has been woven into all parts of the platform,
including virtualization, to provide critical guest separation regardless of the guest operating system.
SuccessfulsecurityusesaDefense-in-Depthstrategy,soRHEL6includessystemfirewalls
5
, host base intru-
siontools,systempackageandfileintegrityverificationtools,andasdiscussedinPART 1 of this whitepaper,
audit capabilities for a complete security architecture that covers deployment models ranging from Internet-
facing servers to trusted computing.
Common Criteria security event auditing requirements are covered in both the CAPP and LSPP protection
profiles.CAPPwasderivedfromtheOrangeBook
6
C2criteriaanddefinesaudittoprovidecomprehensive
logging of security events that are reliable and robust. LSPP extends audit requiring “enhanced security
event auditing” to include Mandatory Access Control (MAC) labeling and decision information. LSPP was
derived from the Orange Book B1 criteria. table 1: capp requirements provides a detailed description of the
CAPP requirement for audit. Audit must be non-bypassable, and the right to add records to the audit trail
mustbecontrolled.Therequirementsalsonotethatbothsetting/viewingtheauditconfigurationmustbe
controlled and that audit review must be controlled and assignable. It must have the ability to fail-stop the
system. The Linux syslog
7
facility has none of these properties.
1 Confidentialityisdefinedasensuringthatinformationisaccessibleonlytothoseauthorizedtohaveaccess.
2 IntegrityisdefinedasqualityofanITsystemreflectingthelogicalcorrectnessandreliabilityoftheoperatingsystem;thelogical
completeness of the hardware and software implementing the protection mechanisms; the consistency of the data structures and
occurrence of the data stored.
3 Availabilityisdefinedastimely,reliableaccesstodataandinformationservicesforauthorizedusers.
4 RHEL has passed the Common Criteria Process 13 times on four different hardware platforms.
5 RHEL 6 also includes Ethernet bridge frame table administration ( ebtables).This application program is used to set up and maintain the
table of rules inside the kernel that inspects Ethernet frames. It works just like the iptables application, which inspects the IP protocol;
ebtables inspects the Ethernet protocol.
6 Trusted Computer System Evaluation Criteria (TCSEC), referred to as the Orange Book, is a DOD standard that sets basic requirements for
assessing the effectiveness of computer security controls built into a computer system.
7 RHEL 6 uses rsyslogd, which is a reliable and extended syslogd. It is a system utility providing support for message logging. Local and
remote logging is supported, but it is not as granular as auditd and does not meet the strict requirements of CAPP and LSPP.
Using an open source framework to catch the bad guy
www.redhat.com 3
TheFutureCommonCriteriaProtectionProfile:the Operating System perspective has changed from single
isolated systems to more complex distributed and networked environments (e.g., virtualization and cloud),
thusrenderingseveraloftheoriginalprotectionprofiles,includingLSPP,RBAC,andCAPP,lessrobuston
therequirements.OnthehorizonisasecondgenerationcertifiedOperatingSystemProtectionProfile
(OSPP). Red Hat is currently meeting the standards and requirements of OSPP from the networked systems
approach and will meet the functional and assurance requirements that are applicable. In addition, applica-
tions executing on operating systems depend upon a secure platform. The security assurance provided by
many modern operating systems has been raised over the last decade with EAL4 being the norm for this
technology and Red Hat raising the bar higher.
8
taBle 1: capp requirements
aUDIt toolS
DEfINItIoN
Audit data generation
The Target of Evaluation Security Functions (TSF )shall be able to generate
an audit record of the auditable events listed in column “Event” of Table
(Auditable Events). This includes all auditable events for the basic level of
audit, except FIA_UID.1’s user identity during failures.
Audit data generation
The TSF shall record within each audit record at least the following informa-
tion: (a) date and time of the event, type of the event, subject identity, and the
outcome(successorfailure)oftheevent;(b)additionalinformationspecified
in Table 1 .
User identity association
The TSF shall be able to associate each auditable event with the identity of the
user that caused the event.
Audit review
The TSF shall provide authorized administrators with the capability to read the
audit information from the audit records.
Audit review
The TSF shall provide authorized administrators with the capability to read all
audit information from the audit records.
Restricted audit review
The TSF shall prohibit all users read access to the audit records, except those
users that have been granted explicit read-access.
Prevention of audit data loss
ThiscomponentspecifiesthebehavioroftheTargetofEvaluation(TOE).If
the audit trail is full: either audit records are ignored, or the TOE is frozen such
that no auditable events can take place. The requirement also states that no
matterhowtherequirementisinstantiated,theauthorizeduserwithspecific
rights to this effect can continue to generate auditable events (actions).
Federal security policies also mention that there must be an in-depth strategy that provides appropriate
degrees of protection to all computing environments, hosts, and applications. Information systems should be
monitored in order to detect, isolate, and react to intrusions, disruption of services, or other incidents that
threaten the security. Requirements and recommendations for audit should be created in alignment with the
security implications as well as the regulations and laws to which the organization are subject. A number of
laws, policies, and regulations compel organizations to store and review audit data
9
.
Security policy requirements also state that there be a way to collect and retain audit data to support foren-
sics and incident response relating to misuse, penetration, reconstruction, or other investigations. During a
forensics investigation, law enforcement and analysts will need to rely on audit logs as a source of evidence.
Along with this, there must also be proof that a malicious person has not altered those logs and that the
8 Red Hat Enterprise Linux includes the openscap-utils package. This package is the Security Content Automation Protocol (SCAP)
toolkit based on the NSA/NIST OpenSCAP library (to include the Open Vulnerability and Assessment Language (oval), the eXtensible
ConfigurationChecklistDescriptionFormat(xccdf)theCommonePlatformEnumeration(CPE),andtheCommonVulnerabilityScoring
System (CVSS). For more information on Open SCAP visit: http://www.open-scap.org/doc/
9 LawsandRegulations:Commercial-FISMA,HIPAA,GLBA,SOX,PCIDSS//Government-CNSSDirectiveNo.502,DoDDirective8500
Series, NSD 32, DCID 6/3, DOD 5200.
Using an open source framework to catch the bad guy
4 www.redhat.com
logs are creditable. Logs produced by a computer are not admissible as evidence unless it can be shown that
there is no reasonable ground for believing them to be inaccurate, and the computer was operating properly
duringthecollectionofdata.Thelogfilescannotbetamperedwith,ortheyarenotadmissibleasevidence.
This two-part whitepaper series covers the critical areas of information protection for Red Hat Enterprise
Linux 6:
• Part1:UsingtheRedHatEnterpriseLinuxAuditSubSystems for forensics and incident response to
meet security requirement objectives and goals. This section of the white paper closely follows and maps
NIST Special Publication 800-92 GuidetoComputerSecurityLogManagement written by Karen Kent
andMurugiahSouppayawithRHEL6auditspecifics.
• Part2:IntegritycheckingwithRedHatEnterpriseLinux6, which involves calculating a message
digestforeachfileandstoringthemessagedigestsecurelytoensurethatchangestoarchivedlogsare
detected.Amessagedigest(alsocalledadigitalsignature)uniquelyidentifiesdataandhastheproperty
that changing a single bit in the data causes a completely different message digest to be generated.
aboUt tHE aUtHoRS
norman mark st. laurent, principal author
Norman Mark St. Laurent is a Senior Solutions Architect on the Red Hat Federal Team. Early in his 20 year
Computer Security/Forensics career, Mr. St. Laurent evaluated HPUX 10.10 as a Senior Computer Scientist for
the Trust Technology Assessment Program (TTAP) (Prelude to Common Criteria). Norman was also a Senior
Network Intrusion Analyst and Senior Information Systems Security Engineer (ISSE) for the NSA. In addition,
he was a Senior Computer Forensics Examiner (Unix/Linux Lead) for the FBI’s Computer Analysis Response
Team, having worked hundreds of computer and network forensics cases using Red Hat Linux as his primary
platform.
shawn wells, technical editor
Shawn Wells is the Technical Director for the Intelligence Community on the Red Hat Federal Team.
steve gruBB, technical editor
SteveGrubbleadsRedHat’ssecuritytechnologiesteam,whichworksonsecuritycertificationsandguidance
andmaintainsmanyofthesecuritytoolsthatyoufindonLinuxsystems,includingtheAuditSubSystem.He
hasworkedonLinuxsecurityforover10years,mostlyonflawdiscoveryandrepairformanyoftheimpor-
tant programs in use.
PaRt 1: oPERatIoNal USE of tHE RED Hat
ENtERPRISE lINUx 6 aUDIt
loG MaNaGEMENt INfRaStRUCtURE
The Red Hat Enterprise Linux 6 AuditingSubSystem provides kernel-resident logging of system calls and
user space tools to collect and view the logs, allowing for a means to provide both detailed and granular
forensics investigation as well as incident response. RHEL 6 allows for the capability to monitor real-time
occurrences of, or accumulation of, auditable events that may indicate an imminent violation of the security
policy.
Using an open source framework to catch the bad guy
www.redhat.com 5
In fact, the RHEL 6 AuditingSubSystemisconfigurabletoallowcontroloverwhatspecificinformationis
written to the logs. This information is useful in debugging security-related issues. The auditd daemon is
also used to log Security-Enhanced Linux (SELinux)
10
events. SELinux represents the culmination of nearly
40yearsofoperatingsystemsecurityresearchandprovidesapowerful,flexible,mandatoryaccesscontrol
mechanism to RHEL 6. SELinux generates audit messages at system initialization, policy load, and when
Boolean state are changed. These SELinux logs and the log management facility of auditd allow for security
relevanteventstobesecure,reliable,finegrained,andconfigurable,withavarietyofusesincluding:
• postmortem analysis,
• intrusion detection, and
• live system monitoring and debugging.
RHEL 6 audit logs are most useful for identifying or investigating suspicious activity involving a particular
host.Theauditlogscanbeconsultedtogetmoreinformationonaspecificactivityandallowaneventtobe
investigated at a more granular level. Built-in audit utilities such as aureport, ausearch, and aulast enable
an organization to view the audit log information in detail for analysis. These tools are very powerful and in
combination with shell scripting and the cron
11
facility can be very powerful as we’ll show later in this white-
paper. RHEL 6 also comes with the audit-viewer tool. The audit-viewertoolisaGraphicalUserInterface
(GUI)forviewingandsummarizingeventscollectedbytheauditsubsystem(seeFigure 1: The audit-
viewer GUI and Figure 2:The audit-viewer GUI with options shown).
The RHEL 6 Auditing Sub System also has the ability to monitor tty
12
logging that will log all tty sessions
(keystrokes) via the pam_tty_audit PAM
13
module. The pam_tty_audit PAM module is used to enable
or disable tty auditing. When tty auditing is enabled via this PAM module, it is inherited by all processes
startedbythatuser.Daemonsrestartedbyauserwillstillhavethisspecifickeystrokeauditingenabled.In
the session section of the /etc/pam.d/system-authfileyouneedtoaddthefollowinglineasshown
in Table 2: /etc/pam.d/system-auth pam_tty_audit addition to monitor all keystrokes the root
user has entered. It is recommended to use the disable=*asthefirstoptionsothatwillturnoffaudittty
input for all users except for the root user, whose we turn on with the enable=root option. Once set up, the
keystroke monitoring can be audited, and the data that was logged by the kernel can be seen by using the
aureport command (see Table 3:Reviewing keystroke captures with aureport). In the aureport
command we added the -ts today option to print out all keystrokes captured for the current day.
Table 2: /etc/pam.d/system-auth pam_tty_audit addition
/etc/pam.d/system-auth pam_tty_audit addition
Session required pam_tty_audit.so disable=* enable=root
Table 3: Reviewing keystroke captures with aureport
/etc/pam.d/system-auth pam_tty_audit addition
#aureport --tty -ts today
10 SELinux is developed by the National Security Agency (NSA), Red Hat, and the open source community.
11 The cron daemon is used to execute scheduled commands.
12 tty – serial terminal lines.
13 PAM is a system of libraries that handle the security tasks of applications on the system. The library provides a stable application
programminginterfacethatprivilegedgrantingprogramsdefertoforspecificsecuritytasks.
Using an open source framework to catch the bad guy
6 www.redhat.com
fIGURE 1: tHE aUDIt-vIEWER GUI aND PICtURE
fIGURE 2: tHE aUDIt-vIEWER GUI WItH oPtIoNS SHoWN
Using an open source framework to catch the bad guy
www.redhat.com 7
NIST SP800-92 notes that an audit log is a record of the events occurring within a system or network.
Logsarecomposedoflogentries.Eachlogentrycontainsspecificinformationrelatedtoaneventthathas
occurred.Logsshouldbeusedinconjunctionwithothernetwork/computerlogfilestopaintacomplete
story/history of an occurrence. As an example, a Network Intrusion Detection device might detect an attack
signature against a particular RHEL host or even record malicious commands given from a particular server.
Investigate using the audit tools; the RHEL host audit logs may indicate further evidence if a particular user
wasloggedintothehostatthatspecifictime,andifthespecificattackwassuccessful.
TheRHEL6AuditingSubSystemallowsthehosttogranularlylogandtrackusers,accesstofiles,direc-
tories, as well as system resources and system calls. Real-time monitoring can locate occurrences of or
accumulation of these auditable events that may indicate an imminent violation of security policy. Red Hat
hasalsohardenedtheauditlogfilesagainstloginjectionattacksbecauseallentrustedfieldshavebeen
formatted in hex encoded ascii to allow correct parsing. The RHEL 6 audit capabilities enable an organiza-
tion to monitor a system for application misbehavior or code malfunctions. By creating a management policy
consistingofasophisticatedsetofrulesincludingfilewatchesandsystemcallauditing,anorganizationcan
make sure that any violation of its security policies are noted and properly addressed.
With the increasing number of threats and the number and volume of computer security logs ever on the
increase, there is true demand for Computer Security Log Management. Log Management is the process
for generating, transmitting, storing, and analyzing computer security log data. The following sections
cover each of these aspects in depth and map the RHEL 6 audit sub system to being used in an operational
process.
1.1 estaBlishing policies and procedures for log management
To establish and maintain successful log management activities, an organization should develop a standard
process for performing log management. Most security policies state that there should be testing and audit
bytheInformationSystemsSecurityOfficer(ISSO)
14
and/or Information Systems Security Manager (ISSM)
of the security posture of the information system by employing various intrusion/attack detection and
monitoringtools.Theoutputofsuchtoolsmustbeprotectedagainstunauthorizedaccess,modification,or
deletion. These tools must also build upon audit reduction and analysis tools to aid the ISSO/ISSM in the
monitoring and detection of suspicious, intrusive, or attack-like behavior.
Anorganizationshoulddefineitsloggingrequirementsandgoals.Dependingonthethetypeofbusiness
or organization these requirements and goals could be very different. In addition to these requirements
andgoals,anorganizationshouldthendeveloppoliciesthatdefinelogmanagementauditactives.Log
Managementensuresthatcomputersecurityrecordsarestoredinsufficientdetailforanappropriate
period of time. Routine use of RHEL 6 audit tools to review and analyze will identify security incidents, policy
violations, and fraudulent activity in real-time. Table 4: Red Hat Enterprise Linux audit tools
provides an overview of the tools in RHEL 6. This along with the other fundamentals of Log Management
covered in this whitepaper are useful in performing forensic analysis as well as supporting the organization’s
internal investigations.
14 ISSO in this context could also mean the responsibility of the System Administrator as well in some organizations.
Using an open source framework to catch the bad guy
8 www.redhat.com
taBle 4: red hat enterprise linux audit tools
aUDIt tool
DEfINItIoN
auditd
The daemon auditd is the user space component of the Linux Auditing System. It is
responsibleforwritingauditrecordstothedisk.Configuringtheauditrulesisdone
with the auditctl utility; during start-up, the rules in the /etc/audit/audit.rules are
read by the auditctlcommand.Theauditdaemoncanbecustomizedinthefile
/etc/audit/auditd.conf. Viewing the logs is done with the ausearch, aureport,
and aulast facilities.
/etc/audit/audit.rules
The audit.rulesfilecontainsauditrulesthatwillbeloadedbytheauditdaemon’sinit
script any time the daemon is started. The auditctl program is used by the initscripts to
performactionsinthisfile.
/etc/audit/auditd.conf
The auditd.conffileistheconfigurationfilefortheauditdaemon.
auditctl
The auditctl command is used to assist controlling the kernel’s audit system. You
can get status, and add or delete rules into kernel audit system. You can also use this
commandtosetawatchonafile.
ausearch
The ausearch command is used to query the audit daemon logs based for events based
on different search criteria.
aureport
The aureport command will produce a summary reports of the audit system logs.
aulast
The aulast command will print out a listing of the last logged in users similarly to the
program last and lastb. The aulast command searches back through the audit logs
orthegivenauditlogfileanddisplaysalistofallusersloggedinandoutbasedonthe
range of time in the audit logs.
autrace
The autrace audit tool is a program that will add the audit rules to trace a process
similar to strace
15
. The is very useful to see what a program maybe doing.
audispd
The audispd daemon is an audit event multiplexor. It has to be started by the audit
daemon in order to get events. It takes audit events and distributes them to child
programs that want to analyze events in real time.
/etc/audisp/audispd.
conf
The audispd.conffilecontrolstheconfigurationoftheauditeventdispatcher.
1.1.1 rhel 6 log storage and rotation
TheRHEL6AuditSubSystemallowsforthestorageoflogfilesfromboththesystemlevelandinfrastruc-
turelevel.Auditlogfilescanberetainedonthesystemaswellastransmittedtothelogmanagementinfra-
structure host. If either the system or infrastructure logging host fails to log, this allows the other to retain
the log data. In addition, during an incident on a system, the system’s logs might be altered or destroyed by
attackers. Incident response can then use the data from the infrastructure logs to help with the forensics.
Comparing both infrastructure logs to the system logs can also help to determine what data was changed or
removed, helping indicate what the attacker wanted to conceal.
SystemlogfilesinRHEL6arestoredinthe/var/log directory. This directory should have its own parti-
tion or logical volume
16
. The RHEL 6 audit sub system stores its logs in the /var/log/audit directory. This
should also have its own partition or logical volume. We recommended that both /var/log and var/log/
audit have their own separate partitions or logical volumes to keep the log data separate and secure. The
audit trail is so important in a CAPP environment (which supports many Regulations and Standards)
17
that
access to system resources must be denied if an audit trail cannot be created.
15 The strace command will trace system calls and signals. It is shipped in the strace RPM with RHEL 6.
16 ItisrecommendedthatpartitioningrequirementsshouldmatchtheUnitedStatesGovernmentsConfigurationBaseline(USGCB).Formore
information see http://usgcb.nist.gov/
17 PCI, FISMA, HIPAA, SOX, DOD Directive 8500.2, DCID 6/3 as examples.
Using an open source framework to catch the bad guy
www.redhat.com 9
Note: For the examples in this whitepaper, we will assume that CAPP must be met in our SecurityPolicy,
soallsettingshereinwillreflectthisassumption.
Thepartitionsandspecificconfigurationfilescanbesetafterthesystemhasbeeninstalledorwhenitis
provisioned and written within a kickstart
18
file.Akickstartfileallowsforautomation,whichprovides:
• Reliability: settings are performed in the same (correct) way every time.
Adedicatedpartitionpreventstheauditdlogsfromdisruptingsystemfunctionalityiftheyfillandprevents
any other activity in the /varfilesystemfromfillingthepartitionandstoppingtheaudittrail.
The partition size should be larger than the maximum space that auditd will use. The following formula
can help the system administrator determine the partition size. Where MAX_SIZE_OF_LOG_FILE is the
sizeofeachlogfile,andNUMBER_OF_LOG_FILESisthenumberoflogfilesbeingrotated(SeeTable 5:
Formula to determine log space).
taBle 5: log rotation script
foRMUla
MAX_SIZE_OF_LOG_FILE X NUMBER_OF_LOG_FILES
1.1.1.1 non auditdlogfilesin/var/log
Since we have assumed that we are using a CAPP environment, log rotation should be set system-wide. This
includeslogfilesthatauditddoesnotmanage.Logfilesin/var/log should be rotated as well as turning
compression on to save space on the system. This whitepaper is covering auditd in terms of forensics and
incident response, but it would not be complete if we did not take a section to cover all non auditdlogfiles
in /var/log as part of the rotation procedure.
Torotateaswellascompresstheselogfilesin/var/log, run the following script noted in Table 6: Log
rotation script. This script will set the log rotation to 12 weeks (3 months) and compress each
/var/loglogfile.Thefile/etc/logrotate.confisdesignedtoeaselogfileadministrationforthese
logfilesbyallowingautomaticrotationandcompression.Notethatinthe/etc/logrotate.conffileno
packages own wtmp and btmp,sotheyarerotatedinthisfile.RPMpackagesdroplogrotationinformation
into the /etc/logrotate.d directory. Also in RHEL 6 by default the dateext option is now enabled. This
optionarchivesoldversionsoflogfilesbyaddinganextensionrepresentingthedateinYYYYMMDD format.
Previously,anumberwasappendedtofiles.
taBle 6: log rotation script
SCRIPt
for logconf in `ls -1 /etc/logrotate.conf`
do
perl -npe ‘s/rotate\s+4/rotate 12/’ -i $logconf
perl -npe ‘s/\#compress/compress/’ -i $logconf
done
18 Kickstarts allow for an automated installation method where partitions as well as logical volumes can be set. Security settings discussed
inthiswhitepaperforauditcanalsobesetinakickstartfile.
Using an open source framework to catch the bad guy
10 www.redhat.com
1.1.1.2 auditdlogfilesinthedefault/var/log/auditdirectory
The default settings with auditd rotates 4 logs by size (5MB), retaining a maximum of 20MB of data. This
makes it possible to loose audit data with auditd.Justlikerotatingandcompressinglogfilesin /var/
log rotation and compression should also be done for the auditddaemon.Specificallythisisdoneinthe/
etc/logrotate.d/auditfile.ThebelowscriptinTable 7: Log rotation script for auditd sets
compression for audit compress and rotates the logs for 90 days (rotate 90). The log is kept daily (daily),
it will not rotate if it is empty (notifempty), and if the log is missing, go on to the next one without issuing
an error message (missingok). The lines between postrotate and endscript are executed using bash, in
this example restarting the audit daemon.
Itshouldbenotedthatcompressingthelogfileswillmaketheaudittoolsaureport and ausearch unable to
readthem.Ifyouusethesetoolswhenthelogfilesarecompressed,youwillhavetousethezcat or bzcat
19
commandstodecompressthefilestostdout for the audit tools to read into stdin. This will allow the ISSO
toworkwiththelogfileincompressedmode.
taBle 7: log rotation script for auditd
SCRIPt
cat «lOGROT1 > /etc/logrotate.d/audit
compress
/var/log/audit/audit.log
{
rotate 90
daily
notifempty
missingok
postrotate
/sbin/service auditd restart 2> /dev/null > /dev/null || true
endscript
}
lOGROT1
Itisalsoimportanttosettherotationtimetobeasclosetomidnightaspossible,sothatlogfilescanbe
rotated on a near daily basis according to the 24 hour clock. To do this, in the /etc/audit/auditd.conf
filesetthemax_log_file_action to ignore (See Table 8: max_log_file_action Setting and as an
alternative to doing the edit by hand, a script can also be run. See Table 9: max_log_file_action
Script):
taBle 8:max_log_file_actionSetting
SEttING
max_log_file_action=ignore
19 Both zcat and bzcatwilluncompressthelogfilesbyexaminingthecorrectmagicnumberwhethertheyhavethecorrect .gz or .bz2
suffixornot.Forspecificsonmagicnumberspleasereadtheman page for magic (5).
Using an open source framework to catch the bad guy
www.redhat.com 11
Next copy the script auditd.cron that was shipped with the audit RPM (located in /usr/share/doc/
audit-version directory) to the /etc/cron.daily directory, change the permissions to 0770, and make
sure the ownership remains root.root. See Table 10: auditd.cron script. After the rotate the log
will be named audit.log.1.
taBle 9: max_log_file_actionScript
SCRIPt
#perl -npe ‘s/max_log_file_action = ROTATE/max_logfile_action = IGNORE/’ /etc/audit/auditd.
conf
taBle 10: auditd.cronscript
SCRIPt
#!/bin/sh
##########
# This script can be installed to get a daily log rotation
# based on a cron job.
##########
/sbin/service auditd rotate
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
/usr/bin/logger -t auditd “ALERT exited abnormally with [$EXITVALUE]”
fi
exit 0
RHEL 6 includes the cronie package as a replacement for vixie-cron. The main difference between these
packages is how the regular jobs (daily, weekly, and monthly) are done. Cronie uses the /etc/anacrontab
filetostartitsdailycron jobs, which is different from vixie-cron. To ensure that the daily rotation is close
to the 24 hour clock, you will want to edit /etc/anacrontabfilewiththefollowingchanges(SeeTable
11: /etc/anacrontab file). In the /etc/anacrontabfilethesettingsareasfollows:theRANDOM_DELAY
variable to 0 so no random delay is added, as well as setting the START_HOURS_RANGE to 0,whichdefines
the midnight interval when scheduled jobs can run. Lastly, we set the delay in minutes for cron.daily
to 0,sothatitspecifiesthatanacron will not delay and do cron.daily as close to midnight as possible.
Using an open source framework to catch the bad guy
12 www.redhat.com
taBle 11: /etc/anacrontabfile
/etc/anacrontab file
# /etc/anacrontab: configuration file for anacron
# See anacron(8) and anacrontab(5) for details.
SHell=/bin/sh
PaTH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
# the maximal random delay added to the base delay of the jobs
RANDOM_DELAY=0
# the jobs will be started during the following hours only
START_HOURS_RANGE=0
#period in days delay in minutes job-identifier command
1 0 cron.daily nice run-parts /etc/cron.daily
7 25 cron.weekly nice run-parts /etc/cron.weekly
@monthly 45 cron.monthly nice run-parts /etc/cron.monthly
1.1.2 rhel 6 remote host storage
TheRHEL6hostshouldbeconfiguredtosenditslogstoaremotehostaswellashavingalocalrepository.
An intruder or malicious user who has compromised the root accountonamachinemaydeletethelogfiles.
If system logs are to be useful in detecting malicious activities, it is necessary to send them to a remote log
server that is running defense-in-depth security features to protect the logs. Virtual Lans (VLANs) should
alsobeconsidered:havingthelogfilesbedistributedonaseparatenetwork.VLANsallowanorganization
to separate network segments and apply access control based on security rules. It is recommended that the
audit data be segmented on its own VLAN. This will also increase network performance and segment the
audit data over the network.
1.1.2.1 log management with the /etc/audit/auditd.conf file
As discussed and a continuous theme of this whitepaper, the purpose of auditing is being able to do an inves-
tigation periodically or whenever an incident occurs. Logs contain records of system and network security,
thustheyneedtobeprotectedfrombreachesoftheirconfidentialityandintegrity.Logsthataresecured
improperly in storage or in transit might also be susceptible to intentional or unintentional alteration and
destruction. This could cause activities to go unnoticed, and even hide the evidence to conceal the identity of
a malicious party.
Inadditiontotheconfidentialityandintegrityofarchivedlogfiles,organizationsneedtoalsoprotectthe
availabilityofthelogfiles.Forexample,logfilesizelimitandlogrotationplayimportantrolesforLog
Management in terms of data retention requirements. Table 12: The /etc/audit/auditd.conf log
server setup configurationdescribestheconfigurationsettingsfortheauditdaemonforaserverthat
isbeingconfiguredtoaggregateandcollectlogfilesfromnumeroushosts.Itisimportanttomakesurethat
theconfigurationissettoallowlogfilestogrowwithoutbound.
Using an open source framework to catch the bad guy
www.redhat.com 13
taBle 12: The/etc/audit/auditd.conflogserversetupconfiguration
aUDIt toolS
DEfINItIoN
log_file = /var/log/audit/
audit.log
Specifiesthefullpathnametothelogfilewhereauditrecordswillbestored.
Thismustbearegularfile.
Inthisexamplewechosethedefaultlogfile.
log_format = raw
Describes how the information should be stored on disk. There are two options:
raw and nolog. If set to raw, the audit records will be stored in a format exactly
as the kernel sends it. If the option is set to nolog, then all audit information is
discarded and not written to disk. This mode does not affect data sent to the
audit event dispatcher.
log_group = isso
Specifiesthegroupthatisappliedtothelogfile’spermissions.Thedefaultis
root. The group name can be either numeric or spelled out. This is the opportu-
nitytomakeagroupforalltheInformationSystemInformationOfficers.
In this example we assume that there is a group made for the Information
SystemsSecurityOfficerswhowillbelookingatthelogfiles.
priority_boost = 4
This is an non-negative number that tells the audit daemon how much of a
priority boost it should take.
The default is 4, which we are using in this example.
flush = data
Valid values are none, incremental, data, and sync.
none:nospecialeffortismadetoflushtheauditrecordstodisk.
incremental: If set to incremental, then the freq parameter is used to determine
howoftenanexplicitflushtodiskisissued.
data: The data parameter tells the audit daemon to keep the data portion of the
diskfilesync’datalltimes.
sync: The sync option tells the audit daemon to keep both the data and meta-
data fully sync’d with every write to the disk.
In this example, we are having the audit daemon to keep the data portion of the
diskfilesync’datalltimes.
freq =
This is a non-negative number that tells the audit daemon how many records to
writebeforeissuinganexplicitflushtodiskcommand.Theflushkeywordmust
be set to incremental.
num_logs = 90
Specifiesthenumberoflogfilestokeepifrotateisgivenasthemax_log_
file_action.Thisnumbermustbe99orless.Thedefaultis0whichmeansno
rotation.
disp_qos = lossless
Controls blocking/lossless or non-blocking/lossy communication between the
audit daemon and the dispatcher. There is a 128k buffer between the audit
daemon and dispatcher. If lossy is chosen, incoming events going to the
dispatcher are discarded when this queue is full. Lossy is the default value.
dispatcher = /sbin/audispd
The dispatcher program is a program that is started by the audit daemon when
it starts it. It will pass a copy of all audit events to that application’s stdin.
It this example we are using the dispatcher /sbin/audispd this will be set on
theserversandclientstoreceiveandsendlogfilestoanaggregatehost.
(continued on next page)
Using an open source framework to catch the bad guy
14 www.redhat.com
aUDIt toolS
DEfINItIoN
name_format = numeric
Controls how computer node names are inserted into the audit event stream.
none: no computer name is inserted into the audit event.
hostname: name returned by the gethostname syscall.
fqd: means that it takes the hostname and resolves it with dns for a fully quali-
fieddomainnameofthatmachine.
numeric: is similar to fqd, except it resolves the IP address of the machine.
user: userisanadmindefinedstringfromthenameoption.Thedefaultvalue
is none.
In this example we are setting each log to have the IP address of the host where
it originated. The IP address is inserted into the audit stream.
name =
Theadmin-definedstringthatidentifiesthemachineifuserisgivenasthe
name_format option.
max_log_file =
Specifiesthemaxfilesizeinmegabytes.Whenthelimitisreached,itwilltrigger
aconfigurableaction.Mustbeanumericvalue.
In this example we are not setting a max_log_file size.
We are rotating daily.
max_log_file_action = ignore
This parameter tells the system what action to take when the system has
detectedthatmaxfilesizelimithasbeenreached.
ignore: The audit daemon does nothing.
syslog: Issue a warning to syslog.
suspend: will cause the audit daemon to stop writing records to the disk.
rotate: causes the audit daemon to rotate the logs.
keep_logs: similar to rotate except it does not use the num_logs setting. This
prevents the audit logs from being overwritten.
In this example we are not setting a max_log_file size. We are rotating daily.
action_mail_acct = isso_
name@example.com
Contains a valid email address or alias. The default address is root. Requires /
usr/lib/sendmail to exist on the machine.
In this example, we have set the email address to a user named isso_name @
example.com. Of course this would have to be set to a valid user. Perhaps
different ISSOs will be monitoring different machines, so this can get granular.
RedHatNetworkSatelliteisagreatoptiontoprovidespecificfilestoagroupof
hosts. We could version control the /etc/audit/auditd.conffile.
space_left = 500
This is a numeric value in megabytes that tells the audit daemon when to
performaconfigurableactionbecausethesystemisstartingtorunlowon
space.
In this example, we have set a numeric value of 500 megabytes that will tell the
audit daemon to send an email as noted in space_left_action = email.
(continued on next page)
Using an open source framework to catch the bad guy
www.redhat.com 15
aUDIt toolS
DEfINItIoN
space_left_action = email
This parameter tells the system what action to take when the system has
detected that it is starting to get low on disk space.
ignore: nothing happens.
syslog: means it will issue a warning to syslog.
email:sendsaemailwarningtotheemailaccountspecifiedinaction_mail_acct
as well sending the message to syslog.
exec: /path/to/a/script
suspend: will cause the audit daemon to stop writing records to the disk.
single: will put the system in single user mode.
halt: will shutdown the computer system.
In this example we are saying that when the system is running low on disk space
at the 500 megabyte notice, email a warning to isso_name@example.com
admin_space_left = 200
This is a numeric value in megabytes that tells the audit daemon when to
performaconfigurableaction.Thisshouldbeconsideredthelastchancetodo
something before running out of disk space.
In this example, we have set a numeric value of 200 megabytes that will tell
the audit daemon to send an email as noted in admin_space_left_action =
email. note: This should be considered a last chance to do something before
running out of disk space.
admin_space_left_action =
email
This parameter tells the system what action to take when the system has
detected that it is low on disk space.
ignore: nothing happens.
syslog: means it will issue a warning to syslog.
email: sendsaemailwarningtotheemailaccountspecifiedinaction_mail_acct
as well sending the message to syslog.
exec: /path/to/a/script
suspend: will cause the audit daemon to stop writing records to the disk.
single: will put the system in single user mode.
halt: will shutdown the computer system.
In this example we are saying that when the system is running low on disk space
at the 200 megabyte notice, email a warning to isso_name@example.com
disk_full_action = halt
This parameter tells the system what action to take when the system has
detectedthatthepartitiontowhichlogfilesarewrittenhasbecomefull.
ignore: nothing happens.
syslog: means it will issue a warning to syslog.
email:sendsaemailwarningtotheemailaccountspecifiedinaction_mail_acct
as well sending the message to syslog.
exec: /path/to/a/script
suspend: will cause the audit daemon to stop writing records to the disk.
single: will put the system in single user mode.
halt: will shutdown the computer system.
In this example, we are assuming that the system is a system that is critical to
our security policy (as should be an aggregate log server) and when the /var/
log/audit partition is full, the machine will shut down.
(continued on next page)
Using an open source framework to catch the bad guy
16 www.redhat.com
aUDIt toolS
DEfINItIoN
disk_error_action = email
This parameter tells the system what action to take when there is an error
detected while writing audit events to disk or rotating logs.
ignore: nothing happens.
syslog: means it will issue a warning to syslog.
email:sendsaemailwarningtotheemailaccountspecifiedinaction_mail_acct
as well sending the message to syslog.
exec: /path/to/a/script
suspend: will cause the audit daemon to stop writing records to the disk.
single: will put the system in single user mode.
halt: will shutdown the computer system.
In this example we are saying that when the system provides a disk error
notice, email a warning to isso_name@example.com We have elected to just
send an email because the system may still be able to write to disk, and that this
is something a systems administrator should look at.
tcp_listen_port = 60
This s a numeric value in the range of 1 – 65535, which causes auditd to listen
on the corresponding TCP port for audit records from remote systems.
The audit daemon can be linked with tcp_wrappers. Access controls can be set in
the /etc/hosts.allow and /etc/hosts.denyfiles.
In this example, we are having auditd listen on port 60 for incoming audit logs
from client servers. SELinux as well as iptables policy is established for port 60.
tcp_listen_queue = 200
This is a numeric value that indicates how many pending (requested but unac-
cepted) connections are allowed. The default is set to 5.
In this example we have adjusted the value to the number of systems on the
network that will be sending audit log data to the server. This is to ensure that if
we had all servers provisioned and /or started at the same time our connections
would not be rejected.
tcp_max_per_addr = 1
This is a numeric value that indicates how many concurrent con- connections
from one IP address is allowed. The default is 1 and the maximum is 16.
In this example we leave the default to one. The default should be adequate in
most cases unless a custom written recovery script runs to forward unsent
events. In this case you would increase the number only large enough to let it in
too.
use_libwrap= yes
This setting has a value of either yes or no. It determines whether or not to use
TCP wrappers to discern connection attempts that are from allowed machines.
Inthisexampleweareusingtcpwrappersfortheaddedsecuritybenefits.For
specificuseoftcpwrappersman page 5 host_access.
tcp_client_ports = 1-1023
This parameter may be a single numeric value or two values separated by a
dash. It indicates which client ports are allowed for incoming connections. If
notspecified,anyportisallowed.Valuesmaybe1-65535.Specifying1-1023
makes sure that clients send from a privileged port to help prevent log injection
attacks by untrustworthy users.
tcp_client_max_idle = 120
This parameter indicates the number of seconds that a client may be idle before
auditd complains.
In this example we set the number of seconds that a client may be idle to 120
before auditd complains. This is set higher than the client heartbeat_timeout
setting by a factor of two.
(continued on next page)
Using an open source framework to catch the bad guy
www.redhat.com 17
aUDIt toolS
DEfINItIoN
enable_krb5 = yes
RHEL6currentlydoesnotsupportKerberos.SeeSSHPortForwardsection
ofthiswhitepapertosendlogfilesencrypted.
If set to “yes,” Kerberos 5 will be used for authentication and encryption. The
default is “no.”
In this example due to security policy and the sensitivity of the information in
thelogfiles,wedonotwanttosendmessagesinthewireincleartext.Wewill
be using Kerberos 5 for encryption.
krb5_principal = auditd
RHEL6currentlydoesnotsupportKerberos.SeeSSHPortForwardsection
ofthiswhitepapertosendlogfilesencrypted.
This is the principal for this server. The default is “auditd.”
In this example, we are using the default auditd as the Kerberos prin-
ciple.Giventhisdefault,theserverwilllookforakeynamedlikeauditd/
hostname@EXAMPLE.COM stored in /etc/audit/audit.key to authenti-
cate itself, where hostname is the canonical name for the server’s host, as
returned by a DNS look up of its IP address.
krb5_key_file = /etc/audit/
audit.key
RHEL6currentlydoesnotsupportKerberos.SeeSSHPortForwardsection
ofthiswhitepapertosendlogfilesencrypted.
Locationofthekeyforthisclient’sprinciple.Thekeyfilemustbeownedbyroot
and mode set to 0400. The default is /etc/audit/audit.key
Inthisexample,weareusingthedefaultkeyfile/etc/audit/audit.key. It
should be noted that there may be a name schema that is developed for the key
names, which would include a version control system. Red Hat Network Satellite
is an excellent way to safely store, version control, and provision this audit.key.
1.1.2.1.1 encryption over the wire with ssh port forwarding
RHEL 6 currently does not support Kerberos encryption with auditd (noted rows in the Tables with the color
dark gray). To implement this very important feature, this section uses SSH Port Forwarding as an alterna-
tiveexample.Logfileswhentransmittedmusthaveintegritymechanismsadequatetoassuretheintegrity
andconfidentialityofalltransmittedloginformation.Thisincludesthepreventionofhijackingofacommu-
nications session. SSH
20
Port Forwarding allows a port from one host to appear on another, using a connec-
tion through SSHandallowingthetrafficthatisbeingforwardedtobeencryptedwithSSH. The tunnel
provided with SSH Port Forwarding uses the TCP transport method. The supported ciphers are:
• 3des-cbc
• aes128-cbc
• aes192-cbc
• aes256-cbc
• aes128-ctr
• aes192-ctr
• aes256-ctr
• arcfour128
• arcfour256
20 SSH provides a secure encrypted communications between two hosts over an insecure network. With SSH you can specify which cipher to
use.
Using an open source framework to catch the bad guy
18 www.redhat.com
• arcfour
• blowfish-cbc
• cast128-cbc
As long as the remote aggregated log server is running the ssh daemon, it is possible to tunnel the auditd
logfilesovertothehostviassh. Tunneling uses SSH to create connections between the client RHEL 6
hostsendingitslogfilestotheaggregatedlogserver.Theclienthostmustspecifyanon-standardportto
connect. We have also set in both the /etc/auditd/auditd.conffile(seeTable 12: The /etc/audit/
auditd.conf log server setup configuration) and the /etc/audit/audisp-remote.conf (see
Table 16: /etc/audit/audisp-remote.conf log client setup configuration) to send outgoing
clientlogfilesandlistenforincomingclientlogfilesontheaggregatedlogserveronportslowerthanport
1023. Specifying ports 1-1023 makes sure that clients send form a privileged port to help prevent log injec-
tion attacks by untrustworthy users.
On the aggregated log server the ISSO will need to set up a port forwarding channel that listens for connec-
tions on the localhost. See Table 13: Aggregated log server port forwarding script. The script
sets up a SSHtunnelbetweenanaggregateloghostanditsclientsthatwillbesendingtheirlogfilestothe
server.Wefindthatitiseasiertocontrolandmanagethescriptfromtheaggregatelogserverandhaveit
run to the clients rather than having all the clients run a script (especially if the clients are in the hundreds).
Note: The script could also be run from /etc/init.d/add-log-server.sh. The script should handle the
start, stop, and status input commands. The ISSO could then use chkconfig –add add-log-server.
sh to set the script to be launched during booting. The script of course needs to be able to work with
iptables.
Using an open source framework to catch the bad guy
www.redhat.com 19
The script originates the SSH tunnel from the central aggregate log host machine and connects to the client
machinesthatitwillgetthelogfilesfrom.Thescriptshouldbecalledfrom/etc/rc.local so that it is run
every time the aggregate log hosts boots up and after all the network services have been run.
Examining the agg-log-server.sh script, the for command does the bulk of the work pulling from vari-
ables for the host names that the ISSO should set. The -R 61:loghost.example.com:60 initiates the
reverse SSH tunnel from Port 61 on the remote server to loghost.example.com Port 60 on the aggre-
gate log host machine. The -nflagtellsSSH to associate the standard input with /dev/null. There will
not be any command line input with SSH,justatunneltoencryptthelogfilesfromtheclienttotheaggre-
gate log server. The script also sends the standardoutput as well as the standarderror to /dev/null as
well (> /dev/null 2>&1). The -N option tells the SSH client to only set up the tunnel and do not prepare
a command stream for issuing commands on the remote system. The -T argument does not allocate a
pseudo-tty on the remote system. The -x argument disables X11 forwarding, just as a defense-in-depth
option.
Thescriptiseasytoimplement,anditachievesveryimportantsecuritygoalsforbothconfidentialityand
integrityofthelogfilesbeingsenttotheaggregatelogserver.Itshouldalsobenotedthattherearemany
ways to achieve this goal and that this is just an overview example. For example, we recommend using
Red Hat Network Satellite to distribute scripts to client systems (whether virtualized or bare metal),
keepingconfigurationmanagementbestpracticesaswellasmaintainingsecurityonthescriptitself.
taBle 13: aggregated log server port forwarding script
SCRIPt
#!/bin/bash
#Red Hat Federal Senior Solutions Architect Team
#Written By: Norman Mark St. Laurent
#Name: agg-log-server.sh
#Version: 1.1
#Summary:
# This script will setup a SSH tunnel between an aggregate log host
# and its clients that will be sending their log files. The script
# will originate the SSH tunnel from the central aggregate log
# host machine, and connect to the client machines that it will get
# the log files from.
# The Script should be called from /etc/rc.local so that it gets run
# every time it boots up.
#
#Last Modified: 12/06/2011
#Aggregate Log Server Hostname
HOSTNAME1=$(echo $HOSTNAME | awk -F. ‘{ print $1 }’)
#Array of Hostnames for Centralized Logging
HOSTNAMECLIENTS=( $HOSTNAME1 hostname1 hostname2 hostname3 hostname4 )
for CHOSTS in ${HOSTNAMECLENTS[@]}
do
/usr/bin/ssh -nNTx -R 61:loghost.example.com:60 $CHOSTS.example.com >
/dev/null 2>&1
done
Using an open source framework to catch the bad guy
20 www.redhat.com
1.1.2.2 Logmanagementwiththe/etc/audisp/audispd.conffile
The /etc/audisp/audispd.conffileisthefilethatcontrolstheconfigurationoftheauditeventdispatcher.
Thisfileshouldbesetuponboththeaggregatelogserveraswellastheclientsasitwillcontroltheremote
server setup and actions needed by the audit event multiplexer audispd. It takes audit events and distrib-
utes them to child programs that want to analyze events in real time. See Table 14: The /etc/audisp/
audispd.conf filefortheconfigurationsettingforthisfile.
taBle 14: the /etc/audisp/audispd.conffile
aUDIt RUlE
DEfINItIoN
q_depth = 400
This is a numeric value that tells how big to make the internal queue of the
auditeventdispatcher.Abiggerqueueletsthedispatcherhandleafloodof
events better. If syslog indicates that audit events are getting dropped, then
increase this number. The default value is 80.
overflow_action
Thisoptiondetermineshowthedaemonshouldreacttooverflowingits
internal queue. When this happens, it means that more events are being
received than it can get rid of.
ignore: nothing happens.
syslog:means it will issue a warning to syslog.
suspend: will cause the audit daemon to stop writing records to the disk.
single: will put the system in single user mode.
halt: will shutdown the computer system.
priority_boost
This is a non-negative number that tells the audit event dispatcher how much
of a priority boost it should take. The default is 4. No change is 0.
max_restarts
This is a non-negative number that tells the audit event dispatcher how many
times it can try to restart a crashed plug-in. The default is 10.
name_format
This is the option that controls how computer node names are inserted into
the audit event stream.
none: no computer name is inserted into the audit event. This is the default.
hostname: is the name returned by the gethostname syscall.
fqd: means that it takes the hostname and resolves it with dns for a fully quali-
fieddomainnameofthemachine.
numeric: is similar to fqd except it resolves the IP address of the machine.
user: istheadminuserdefinedstringfromthenameoption.
name
Thisistheadmindefinedstringthatidentifiesthemachineifauserisgivenas
the name_format option.
1.1.2.2.1 Logmanagementwiththe /etc/audit/audisp-remote.conffile
ToconfigureaRHEL6clienthostforremoteloggingtoanaggregateloggingserver,youmustusethe
audisp-remote plugin for the audit event dispatcher daemon audispd. The ISSO can tell if the audisp-
remote plugin is installed by running the following RPM command. See Table 15: RPM command for
audisp-remote plugin. If the rpm is not installed, then install it with the yum
21
command.
21 YUM(YellowdogUpdaterModified)isaninteractive,RPM based, package manager written at Duke University. It can automatically perform
system updates, including dependency analysis and obsolete processing based on “repository” metadata.
Using an open source framework to catch the bad guy
www.redhat.com 21
taBle 15: rpm command for audisp-remote plugin
CoMMaND
[root@mstlaure /]# rpm -qa | grep audispd-pugins
audispd-plugins-2.1-5.el6.x86_64
taBle 16: /etc/audit/audisp-remote.conflogclientsetupconfiguration
ExaMPlE SEttING
DESCRIPtIoN
remote_server = 192.168.1.22
This is a one word character string that is the remote server hostname or
IP Address that this plugin will send log information to. This can be the
numeric address or a resolvable hostname.
port = 60
This option indicates what port to connect to on the remote log server.
In this example, we are having auditd on the aggregate log host listen on
port 60 for incoming audit logs from client servers. SELinux as well as
iptables policy is established for port 60.
local_port = 61
This option indicates what local port to connect from on the local machine.
You can use the option any, which will set the port to any available unprivi-
leged port. The port should be set to a used port less than 1024, like we
did in this example. This ensures that only privileged users can bind to that
port.
Ifyousetaspecificport,thenyouwillhavetomatchtheportnumberon
the aggregating auditd.conffiletcp_client_ports directive to match
the ports that the client is sending from.
Inthisexample,wewanttosendthelogfileviaaprivilegedport.Thisis
important to ensure that only privileged users can bind to that port. This
matches the settings in the tcp_client_ports in the aggregating auditd.
conffileontheserver.
transport = tcp
This parameter tells the remote logging plugin how to send the events to
the remote system. The only valid option currently is tcp. If this is set to
tcp, the remote logging plugin will make a normal clear text connection to
the remote system.
This is not used if Kerberos is enabled.
If an ISSO is aggregating multiple machines, that person should enable node information in the audit event
stream. This can be done in one of two places. If computer node names are written to disk as well as sent
to the realtime event stream, then edit the name_format option in the /etc/audit/auditd.conffile
(See Table 12: /etc/audit/auditd.conf log server setup configuration) as an example. If the
security requirements need to have node names only appear in the realtime event stream, then edit the
name_format option in the /etc/audisp/audispd.conf. If you edit in both places then it will put two
nodefieldsintheeventstream.
Table 16: /etc/audit/auditd.conf log client setup configuration describes the needed setting
in the /etc/audit/audisp-remote.conf filethatwillallowaRHEL6hosttobecomealogclient,which
willthensenditslogfilesencryptedoverthewiretoaRHEL6aggregatelogserveronthenetwork.The
/etc/audit/audisp-remote.conffilecontrolstheconfigurationoftheauditremoteloggingsubsystem.
(continued on next page)
Using an open source framework to catch the bad guy
22 www.redhat.com
ExaMPlE SEttING
DESCRIPtIoN
mode = forward
This parameter tells the remote logging plugin what strategy to use getting
records to the remote system. Valid values are immediate and forward.
If set to immediate, the remote logging plugin will attempt to send events
immediately after getting them.
In this example we set to forward. The plugin will then store the events
to disk and then attempt to send the records. If the connection cannot be
made, it will queue the records until it can connect to the remote system.
The depth of the queue is controlled by the queue_depth option.
queue_file = /var/log/audit/
remote.log
Thisisthepathofthefilethatisusedfortheeventqueueifmodeissetto
forward. The default is /var/spool/audit/remote.log.
Inthisexample,wehaveoptedtoputthelogfileinthe/var/log/audit
directory because of it residing on its own partition and for easy mainte-
nance with aide (which we will talk about in Part 2 of this whitepaper).
queue_depth = 400
This option determines how many records can be buffered to disk or in
memory before considering to be a failure sending. The default is 200.
It this example for security purposes we have upped this number to 400.
In RHEL 6.2 the default is 120 (which is low). Each slot eats about 9Kof
memory.
network_retry_time = 5
The time, in seconds between retires, when a network error is detected.
The default is 1 second. This applies after the second attempt to avoid
unneededdelaysinareconnectissufficienttofixtheproblems.
format = managed
This parameter tells the remote logging plugin what data format will be
used for the messages sent over the network. The default is managed,
which adds some overhead to ensure each message is properly handled on
the remote end, and to receive status messages from the remote server.
If ascii is given instead, each message is a simple ASCII text line with no
overhead.
If mode is set to forward, the the format must be managed.
max_tries_per_record = 10
The maximum number of times an attempt is made to deliver each
message. The minimum value is 1; the default is 3. If too many attempts
are made, the network_failure_action is performed.
In this example, we set the max_tries_per_record to 10.
heartbeat_timeout = 60
This parameter determines how often in seconds the client should send a
heartbeat event to the remote server. This is used to let both the client and
server know that each end is alive and has not terminated.
The default value is 0 which disables sending a heartbeat. In this example,
we have asked for a heartbeat every 1 minute.
(continued on next page)
Using an open source framework to catch the bad guy
www.redhat.com 23
ExaMPlE SEttING
DESCRIPtIoN
network_failure_action = suspend
This parameter tells the system what action to take whenever there is an
error detected when sending audit events to the remote system. Values
are:
ignore: the remote logging plugin does nothing.
syslog: means that it will issue a warning to syslog.
exec:/path/to/a/script that will be executed.
suspend: causes the remote logging plugin to stop sending records to the
remote system. The logging plugin will still be alive.
single: this option will cause the remote logging plugin to put the computer
system in single user mode.
stop: this action will cause the remote logging plugin to exit, but leave
other plugins running.
halt: this option will cause the remote logging plugin to shutdown the
computer system.
In this example, we have opted to suspend. This causes the remote logging
plugin to stop sending records to the remote system. The logging plugin
will still be alive.
disk_low_action = suspend
Likewise, this parameter tells the system what action to take if the remote
end signals a disk low error. The default is to ignore it. Values are:
ignore: the remote logging plugin does nothing.
syslog: means that it will issue a warning to syslog.
exec: /path/to/a/script that will be executed.
suspend: causes the remote logging plugin to stop sending records to the
remote system. The logging plugin will still be alive.
single: this option will cause the remote logging plugin to put the computer
system in single user mode.
stop: this action will cause the remote logging plugin to exit, but leave
other plugins running.
halt: this option will cause the remote logging plugin to shut down the
computer system.
In this example, we have opted to suspend. This causes the remote logging
plugin to stop sending records to the remote system. The logging plugin
will still be alive.
disk_full_action = suspend
Likewise, this parameter tells the system what action to take if the remote
end signals a disk full error. The default is to ignore it. Values are:
ignore: the remote logging plugin does nothing.
syslog: means that it will issue a warning to syslog.
exec: /path/to/a/script that will be executed.
suspend: causes the remote logging plugin to stop sending records to the
remote system. The logging plugin will still be alive.
single: this option will cause the remote logging plugin to put the computer
system in single user mode.
stop: this action will cause the remote logging plugin to exit, but leave
other plugins running.
halt: this option will cause the remote logging plugin to shut down the
computer system.
In this example, we have opted to suspend. This causes the remote logging
plugin to stop sending records to the remote system. The logging plugin
will still be alive.
(continued on next page)
Using an open source framework to catch the bad guy
24 www.redhat.com
ExaMPlE SEttING
DESCRIPtIoN
disk_error_action = suspend
Likewise, this parameter tells the system what action to take if the remote
end signals a disk error. The default is to ignore it. Values are:
ignore: the remote logging plugin does nothing.
syslog: means that it will issue a warning to syslog.
exec: /path/to/a/script that will be executed.
suspend: causes the remote logging plugin to stop sending records to the
remote system. The logging plugin will still be alive.
single: this option will cause the remote logging plugin to put the computer
system in single user mode.
stop: this action will cause the remote logging plugin to exit, but leave
other plugins running.
halt: this option will cause the remote logging plugin to shut down the
computer system.
In this example, we have opted to suspend. This causes the remote logging
plugin to stop sending records to the remote system. The logging plugin
will still be alive.
remote_ending_action = suspend
Likewise, this parameter tells the system what action to take if the remote
end signals a disk full error. The default is to ignore it. Values are:
ignore: the remote logging plugin does nothing.
syslog: means that it will issue a warning to syslog.
exec:/path/to/a/script that will be executed.
suspend: causes the remote logging plugin to stop sending records to the
remote system. The logging plugin will still be alive.
single: this option will cause the remote logging plugin to put the computer
system in single user mode.
stop:this action will cause the remote logging plugin to exit, but leave
other plugins running.
halt: this option will cause the remote logging plugin to shut down the
computer system.
reconnect: this option tells the remote plugin to attempt to reconnect to
the server upon receipt of the next audit record. If it is unsuccessful, the
audit record could be lost. The default is to suspend logging.
In this example, we have opted to suspend. This causes the remote logging
plugin to stop sending records to the remote system. The logging plugin
will still be alive.
(continued on next page)
Using an open source framework to catch the bad guy
www.redhat.com 25
ExaMPlE SEttING
DESCRIPtIoN
generic_error_action = syslog
Likewise, this parameter tells the system what action to take if the remote
end signals an error we do not recognize.. The default is to send it. To
syslog. Values are:
ignore: the remote logging plugin does nothing.
syslog: means that it will issue a warning to syslog.
exec: /path/to/a/script that will be executed.
suspend: causes the remote logging plugin to stop sending records to the
remote system. The logging plugin will still be alive.
single: this option will cause the remote logging plugin to put the computer
system in single user mode.
stop: this action will cause the remote logging plugin to exit, but leave
other plugins running.
halt: this option will cause the remote logging plugin to shut down the
computer system.
In this example, we have opted to suspend. This causes the remote logging
plugin to stop sending records to the remote system. The logging plugin
will still be alive.
generic_warning_action = syslog
Likewise, this parameter tells the system what action to take if the remote
end signals a warning we do not recognize.. The default is to send it. To
syslog. Values are:
ignore: the remote logging plugin does nothing.
syslog: means that it will issue a warning to syslog.
exec: /path/to/a/script that will be executed.
suspend: causes the remote logging plugin to stop sending records to the
remote system. The logging plugin will still be alive.
single: this option will cause the remote logging plugin to put the computer
system in single user mode.
stop: this action will cause the remote logging plugin to exit, but leave
other plugins running.
halt: this option will cause the remote logging plugin to shut down the
computer system.
In this example, we have opted to suspend. This causes the remote logging
plugin to stop sending records to the remote system. The logging plugin
will still be alive.
(continued on next page)
Using an open source framework to catch the bad guy
26 www.redhat.com
ExaMPlE SEttING
DESCRIPtIoN
queue_error_action = suspend
Likewise, this parameter tells the system what action to take if there is a
problem working with a local record queue. The default is to send it. To
syslog. Values are:
ignore: the remote logging plugin does nothing.
syslog:means that it will issue a warning to syslog.
exec: /path/to/a/script that will be executed.
suspend: causes the remote logging plugin to stop sending records to the
remote system. The logging plugin will still be alive.
single: this option will cause the remote logging plugin to put the computer
system in single user mode.
stop: this action will cause the remote logging plugin to exit, but leave
other plugins running.
halt: this option will cause the remote logging plugin to shut down the
computer system.
In this example, we have opted to suspend. This causes the remote logging
plugin to stop sending records to the remote system. The logging plugin
will still be alive.
overflow_action = syslog
Likewise, this parameter tells the system what action to take if the internal
eventqueueoverflows.Thedefaultistosendit.tosyslog.Valuesare:
ignore: the remote logging plugin does nothing.
syslog: means that it will issue a warning to syslog.
exec: /path/to/a/script that will be executed.
suspend: causes the remote logging plugin to stop sending records to the
remote system. The logging plugin will still be alive.
single: this option will cause the remote logging plugin to put the computer
system in single user mode.
stop: this action will cause the remote logging plugin to exit, but leave
other plugins running.
halt: this option will cause the remote logging plugin to shut down the
computer system.
In this example, we have opted to suspend. This causes the remote logging
plugin to stop sending records to the remote system. The logging plugin
will still be alive.
enable_krb5 = yes
RHEL6currentlydoesnotsupportKerberos.SeeSSHPortForward
sectionofthiswhitepapertosendlogfilesencrypted.
If set to “yes,” kerberos 5 will be used for authentication and encryption.
The default is “no.”
In this example do to security policy because of the sensitivity of the infor-
mationinthelogfiles,wedonotwanttosendmessagesonthewireinclear
text. We will be using Kerberos 5 for encryption.
krb5_principal =
RHEL6currentlydoesnotsupportKerberos.SeeSSHPortForward
sectionofthiswhitepapertosendlogfilesencrypted.
This is the principal for this server. The client and server will use the speci-
fiedprincipletonegotiatetheencryption.
Inthisexample,wedonotusefiled.Novariableisspecifiedsothekrb5_
client_name and remote_server values are used.
(continued on next page)
Using an open source framework to catch the bad guy
www.redhat.com 27
ExaMPlE SEttING
DESCRIPtIoN
krb5_client_name = auditd
RHEL6currentlydoesnotsupportKerberos.SeeSSHPortForward
sectionofthiswhitepapertosendlogfilesencrypted.
Thisspecifiesthenameportionoftheclient’sownprincipal.Ifunspeci-
fied,thedefaultisauditd. The remainder of the principal will consist of
thehost’sfullyqualifieddomainnameandthedefaultKerberosrealm:
auditd/host14.example.com@EXAMPLE.COM (assuming you gave
“auditd” as the krb_client_name).
Note that the client and server must have the same principal name and
realm.
krb5_key_file = /etc/audit/audit.
key
RHEL6currentlydoesnotsupportKerberos.SeeSSHPortForward
sectionofthiswhitepapertosendlogfilesencrypted.
Locationofthekeyforthisclient’sprincipal.Notethatthekeyfilemust
be owned by root and mode 0400. The default is /etc/audisp/audisp-
remote.key
Inthisexample,weareusingthedefaultkeyfile/etc/audit/audit.key.
It should be noted that there maybe a name schema that is developed for
the key names, which would include a version control system.
Red Hat Network Satellite
22
is an excellent way to safely store, version
control, and provision this audit.key.
1.1.3 specific rhel 6 log generation settings
As we saw in Section 1.1.1 and Section 1.1.2logsourcesneedtobeconfiguredsothattheycapture
the necessary information in the desired format as well as the desired locations, and retain information for
theappropriateperiodoftime.RHEL6allowsforitslogsourcestohaveverygranularconfigurationoptions
and allows an organization to meet CAPP requirements.
By default, the audit daemon auditd only logs SELinux denials, which are helpful for monitoring SELinux
anddiscoveringintrusionattemptsandsecurityeventssuchasmodificationstouseraccountsandcalls
to sudo.Inthissection,theaudit.rulesfilewillbetunedtoprovidegranulardatainthelogfiles.Themost
current data for the auditd logs is stored in the /var/log/audit/audit.logfile.Therotatinglogfiles
are also found in the directory /var/log/audit.Rotationnamesinthisdirectoryfortherotatedfilesare
/var/log/audit/audit.log.{1,2,3,n}.Logfilesarestoreddailyinoursetup.Thus,onedayback
would be audit.log.1, two days back would be audit.log.2 and so on. When using the aureport,
ausearch, and aulast commands remember to use the --input file-nameflagtouseaspecificlogfile
other than the default /var/log/audit/audit.log. This is to aid in analysis where the logs have been
moved or rotated.
The auditd daemon in RHEL 6 is responsible for writing audit records to the disk. During start-up, the rules
in /etc/audit/audit.rules are read by auditd. Editing the /etc/audit/audit.rulesfileallowsforthe
configurationofspecificpolicy.Viewingtheauditlogsisdonewiththeausearch, aureport, and aulast
commands. The ISSO will want to make sure that the auditd service is enabled. This is the default, but
should be checked (See Table 17: Enabling auditd command).
22 Red Hat Network (RHN) Satellite is a systems management platform that makes Red Hat Linux deployable, scaleable, manageable,
and consistent. See http://www.redhat.com/red_hat_network/ for more information on RHN Satellite.
Using an open source framework to catch the bad guy
28 www.redhat.com
If any of the run levels were noted to be “off” the ISSO should start the auditd to run at all run levels (See
Table 18: Enabling auditd run levels).
taBle 17: enaBling auditd command
CoMMaND
[root@mstlaure /]# chkconfig --list | grep auditd
auditd
0:off 1:off 2:on
3:on
4:on
5:on
6:off
taBle 18: enaBling auditd run levels
CoMMaND
[root@mstlaure /]# chkconfig auditd on ; service auditd start
Starting auditd:
To ensure that all processes can be audited, even those which start prior to the audit demon, add the
argument audit=1 to the kernel line in the /etc/grub.conffile.Youcanuseyourfavoriteeditortodoso
(See Table 19: /etc/grub.conf edit to start audit) or an automated shell script (See Table 20:
/etc/grub.conf audit automated script).
taBle 19: /etc/grub.confedittostartaudit
KERNEl lINE: /EtC/GRUb.CoNf ExaMPlE
kernel /vmlinuz-2.6.32-131.17.1.el6.x86_64 ro root=/dev/mapper/HelpDeskRHEL6-Root rd_
LVM_LV=HelpDeskRHEL6/Root rd_LUKS_UUID=luks-82b2a0a6-c7e5-4219-8b4d-2e8fed1418c4 rd_
LVM_LV=HelpDeskRHEL6/Swap rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16
KEYBOARDTYPE=pc KEYTABLE=us crashkernel=auto rhgb quiet vga=0x318 audit=1
Automated one liner to get the job done.
taBle 20: /etc/grub.confauditautomatedscript
SCRIPt
#sed -i ‘/^kernel/s|$| audit=1|’ /boot/grub/grub.conf
1.1.3.1 Logmanagementwiththe/etc/audit/audit.rulesfile
The audit.rulesfilegoeshandinhandwiththeauditctl command. The auditctl command is a utility
used to assist controlling the kernel’s audit sub system. It can be used to control the behavior, get status,
and add or delete rules into the 2.6 kernel’s audit sub system. The rules for maintainability should be kept in
the /etc/audit/audit.rulesfile.Thisfilesisessentiallythesamewhentypinginanauditctl command
at the shell prompt except that the actual command name “auditctl: is not needed as it is implied. The
audit rules come in three varieties (See Table 21: The /etc/audit/audit.rules varieties).
Using an open source framework to catch the bad guy
www.redhat.com 29
taBle 21: the /etc/audit/audit.rules varieties
aUDIt toolS
DEfINItIoN
Control
Controlcommandsgenerallyinvolveconfiguringtheauditsystemrather
than telling it what to watch for. These commands include: deleting all rules
on start-up, setting the size of the kernels backlog queue, setting the failure
mode, setting the event rate limit. These rules are usually given at the top of
the audit.rulesfile.
File System
Thefilesystemrulesarealsocalledwatches.Watchesareusedtoaudit
accesstoparticularfilesanddirectories.Ifthepathgivenintheruleisa
directory, then the rule is used recursively to the bottom of the directory tree
(excluding any directories that maybe mount points).
The syntax is as follows:
RHEL4andRHEL5
-w /path/to/file/to/watch -p permissions -k keyname
RHEL6Recommended
-a exit,always -F path=/path/to/file/to/watch -F key=keyname
Where the permissions are anyone of the following:
r-readofthefile
w–writetothefile
x–executethefile
a-changeinthefile’sattribute
System Call
The system call rules are loaded into a matching engine that intercepts each
syscall that all programs on the system make.
This can effect performance. The more rules, the bigger the performance
hit. You can help performance by combining syscalls into one rule whenever
possible.
Thekernelhasfiefilters:
Task: Only checked the fork or clone syscall.
Entry:Runs through each syscall entry.
Exit:Checkedonthesyscallexit.TheEntryfilterwillbedeprecated.Rules
ontheexitfilteraremuchmorecommonandallfieldsareavailableforuseat
syscall exit.
user:Usedtofiltereventsthatoriginateinuserspace.Fieldsthatarevalid
for use are uid, auid, gid, and the pid.
Exclude: Used to exclude certain events from being emitted.
Using an open source framework to catch the bad guy
30 www.redhat.com
Using keys in both the watches and system call rules to give the rule a meaning helps to reduce the workload
and make the policy granular.
The RHEL 6 audit sub system should collect the execution of privileged commands for all users and the root
user to meet CAPP requirements. This requires adding an audit rule to watch execution of each setuid
23
or
setgid
24
program. The following command will locate all setuid and setgid programs for each local parti-
tion on the system. The lines found would then need to be added to the /etc/audit/audit.rulesfile.
PARTequalsthePartitiontosearchwiththefindcommand(e.g.,“/home” or “/”) (See Table 22: Finding
and setting setuid and setgid watches).
The auditd program can perform comprehensive monitoring of system activity. This section describes
configurationsettingsforcomprehensiveauditingusingtheDepartmentofDefenseDefenseInformation
SystemsAgencySecurity(DISA)TechnicalImplementationGuide(STIG)asthefoundationandbasisforthe
audit rules that are set in the example /etc/audit/audit.rulesfile.Theauditsubsystemsupportsan
amplecollectionofevents,toincludethetracingofarbitrarysystemcalls(identifiedbysystemcallname,or
bysystemcallnumber.TheauditsubsystemcanalsofilterbyPID,UDI,systemcallsuccess,andsystemcall
argument.Perhapstheauditsubsystem’smostgranularaspectistheabilitytomonitoryspecificfilesfor
modificationstothefile’scontentsormetadata.
TheNationalSecurityAgency(NSA)hasalsodevelopedanddistributedconfigurationguidanceforRHEL
that is currently being used throughout the government and by numerous entities as a security baseline for
theirRHELsystems.Youwillnotetheseauditrecommendationsinthe/etc/audit/audit.rulesfileexample
below (See Table 23: The /etc/audit/audit.rules file). Table 23: The /etc/audit/audit.
rules filealsoprovidesadefinitionofeachsettingsothattheISSOhasadeepunderstandingoftherule
aswellasabasistochangeforaspecificrequirementthataspecificorganizationmayhave.
Notes:
Thesettingshereinmaybechangeddependingonyourorganization’sspecificauditpolicy.Thesettingsare
are for use as a best case scenario to get you started and to follow CAPP.
The audit rules and settings in this whitepaper follow the recommended audit rules settings in /usr/share/
doc/audit-version/stig.rules
25
. Red Hat has provided audit.rules templates that meet a number of
standards and regulalations. These can also be found in this directory. We recommend starting with the
stig.rules and modifying as need be.
The system `arch` lines will have to be removed if they do not match the system that the /etc/audit/
audit.rulesfileison.
23 The setuid(ShortforsetuserID)isanaccessrightsflagthatallowuserstorunanexecutablewiththepermissionsoftheexecutables
owner.
24 The setgid(ShortforsetgroupID)isanaccessrightsflagthatallowuserstorunanexecutablewiththepermissionsoftheexecutables
group.
25 TheSTIG(SecurityTechnicalImplementationGuide)isamethodologyforstandardizedsecureinstallationofRedHatEnterpriseLinux.
TheDefenseInformationSystemsAgency(DISA)cratestheconfigurationguidanceinsupportoftheDOD.
Using an open source framework to catch the bad guy
taBle 22: finding and setting setuid and setgid watches
SCRIPt
#find PART -xdev \( -perm 4000 -o -perm -2000 \) \
-type f | awk ‘{ print “-a always,exit -F path=” $1 “ -F perm=x \
-F auid>=500 -F auid!=4294967295 -k ISSO-privileged” }’
www.redhat.com 31
taBle 23: the /etc/audit/audit.rulesfile
aUDIt RUlE
DEfINItIoN
-D
Remove any existing rules.
-b 8192
Increase the buffer size to handle the increased number of
messages.
-f 2
Set the failure mode to panic. This option lets you deter-
mine how you want the kernel to handle critical errors.
-e 2
Thisparameterwilllocktheauditconfigurationsothat
it cannot be changed. This should be the last line in the
audit.rulesfile.
-a exit,always -F path=/var/log/audit -F
key=ISSO-audit
Successful and unsuccessful attempts to read information
fromtheauditrecords;allmodificationstotheaudittrail.
a exit,always -F path=/etc/audit -F perm=wa -F
key=ISSO-cfg-audit
-a exit,always -F path=/etc/sysconfig/auditd -F
perm=wa -F key=ISSO-cfg-audit
-a exit,always -F path=/etc/libaudit.conf -F
perm=wa -F key=ISSO-cfg-audit
-a exit,always -F path=/etc/audisp/ -F perm=wa
-F key=ISSO-cfg-audit
Modificationstoauditconfigurationthatcanoccurwhile
theauditcollectionfunctionareoperating;allmodifica-
tions to the set of audited events.
This portion will allow enough audit information to deter-
mine the date and the time of the action, the locale of the
action, the system entity that initiated or completed the
action, the resources involved, and the action involved.
-a always,exit -F arch=b32 -S settimeofday -S
stime -F key=ISSO-time-change
-a always,exit -F arch=b64 -S settimeofday -F
key=ISSO-time-change
-a exit,always -F path=/etc/localtime -F
perm=wa -F key=ISSO-time-change
Changes for RHEL 6.2
-a always,exit -F arch=b32 -S clock_settime -F
a0=0 -F key=ISSO-time-change
-a always,exit -F arch=b64 -S clock_settime -F
a0=0 -F key=ISSO-time-change
Things that could affect time.
(continued on next page)
Using an open source framework to catch the bad guy
32 www.redhat.com
aUDIt RUlE
DEfINItIoN
-a always,exit -F arch=b32 -S sethostname -S
setdomainname -F key=ISSO-system-locale
-a always,exit -F arch=b64 -S sethostname -S
setdomainname -F key=ISSO-system-locale
-a exit,always -F path=/etc/issue -F perm=wa
-F key=ISSO-system-local
-a exit,always -F path=/etc/issue.net -F
perm=wa -F key=ISSO-system-locale
-a exit,always -F path=/etc/hosts -perm=wa -F
key=ISSO-system-locale
-a exit,always -F path=/etc/sysconfig/network
-F perm=wa -F key=ISSO-system-local
Things that could affect system locale.
-a exit,always -F path=/etc/selinux -F perm=wa
-F key=ISSO-mac-policy
Things that could affect Mandatory Access Control (MAC)
policy.
-a always,exit -F arch=b32 -S chmod -S fchmod
-S fchmodat -F auid>=500 -F auid!=4294967295
-F key=ISSO-dac-policy
-a always,exit -F arch=b64 -S chmod -S fchmod
-S fchmodat -F auid>=500 -F auid!=4294967295
-F key=ISSO-dac-policy
-a always,exit -F arch=b32 -S chown -S
fchown -S fchownat -S lchown -F auid>=500 -F
auid!=4294967295 -F key=ISSO-dac-policy
-a always,exit -F arch=b64 -S chown -S
fchown -S fchownat -S lchown -F auid>=500 -F
auid!=4294967295 -F key=ISSO-dac-policy
-a always,exit -F arch=b32 -S setxattr -S
lsetxattr -S fsetxattr -S removexattr -S
lremovexattr -S fremovexattr -F auid>=500 -F
auid!=4294967295 -F key=ISSO-dac-policy
-a always,exit -F arch=b64 -S setxattr -S
lsetxattr -S fsetxattr -S removexattr -S
lremovexattr -S fremovexattr -F auid>=500
-F auid!=4294967295 -F key=ISSO-dac-policy
-a always,exit -F arch=b32 -S setxattr -S
lsetxattr -S fsetxattr -S removexattr -S
lremovexattr -S fremovexattr -F auid>=500 -F
auid!=4294967295 -F key=ISSO-dac-policy
-a always,exit -F arch=b64 -S setxattr -S
lsetxattr -S fsetxattr -S removexattr -S
lremovexattr -S fremovexattr -F auid>=500 -F
auid!=4294967295 -F key=ISSO-dac-policy
Things that could affect Discretionary Access Control
(DAC) permissions.
-a exit,always -F path=/var/log/faillog -F
perm=wa -F key=ISSO-logins
-a exit,always -F path=/var/log/lastlog -F
perm=wa -F key=ISSO-logins
-a exit,always -F path=/var/log/tallylog -F
perm=wa -F key=ISSO-logins
-a exit,always -F path=/var/log/faillock -F
perm=wa -F key=ISSO-logins
Successful and unsuccessful logins and logoffs. The
Red Hat login, gdm, and openssh programs will log all
relevant information.
The faillock command is an application that can be used
toexamineandmodifythecontentsofthetallyfiles.It
can display the recent failed authentication attempts of
theusernameorclearthetallyfilesforallorindividual
usernames.Itusesthe/var/log/faillockfiletorecord
information.
(continued on next page)
Using an open source framework to catch the bad guy
www.redhat.com 33
aUDIt RUlE
DEfINItIoN
Successful and unsuccessful accesses to security relevant
objects and directories, including creation, open, close,
modification,anddeletion.
-a exit,always -F path=/var/run/utmp -F
perm=wa -F key=ISSO-session
-a exit,always -F path=/var/log/btmp -F
perm=wa -F key=ISSO-session
-a exit,always -F path=/var/log/wtmp -F
perm=wa -F key=ISSO-session
Session initiation is audited by pam without any rules
needed.
The utmpfileletsyoudiscoverinformationaboutwhois
currently using the system.
-a always,exit -F arch=b32 -S creat -S
open -S openat -S truncate -F exit=-
EACCES -F auid>=500 -F auid!=4294967295 -F
key=ISSO-access
-a always,exit -F arch=b32 -S creat -S open -S
openat -S truncate -F exit=-EPERM -F auid>=500
-F auid!=4294967295 -F key=ISSO-access
-a always,exit -F arch=b64 -S creat -S
open -S openat -S truncate -F exit=-
EACCES -F auid>=500 -F auid!=4294967295 -F
key=ISSO-access
-a always,exit -F arch=b64 -S creat -S open -S
openat -S truncate -F exit=-EPERM -F auid>=500
-F auid!=4294967295 -F key=ISSO-access
Unsuccessfulaccessattemptstofiles
-a always,exit -F path=/bin/ping -F perm=x
-F auid>=500 -F auid!=4294967295 -F
key=ISSO-privilege
Use of privileged commands (unsuccessful and successful)
-a always,exit -F arch=b32 -S mount
-F auid>=500 -F auid!=4294967295 -F
key=ISSO-media-export
-a always,exit -F arch=b64 -S mount
-F auid>=500 -F auid!=4294967295 -F
key=ISSO-media-export
Export to media (successful)
-a exit,always -F path=/etc/sudoers -F perm=wa
-F key=ISSO-admin-actions
Record system administration actions
-a exit,always -F path=/etc/group -F perm=wa
-F key=ISSO-auth
-a exit,always -F path=/etc/passwd -F perm=wa
-F key=ISSO-auth
-a exit,always -F path=/etc/gshadow -F perm=wa
-F key=ISSO-auth
-a exit,always -F path=/etc/shadow -F perm=wa
-F key=ISSO-auth
-a exit,always -F path=/etc/security/opasswd
-F perm=wa -F key=ISSO-auth
Changes in user authentication and identity.
-a exit,always -F path=/var/log/audit/audit.
log -F key=ISSO-audit-logs
Audit trail protection. The contents of the audit trail
shouldbeprotectedagainstunauthorizedaccess,modifi-
cation,ordeletion.Thisiscoveredbycorrectfilepermis-
sions, but activity watches are set here.
(continued on next page)
Using an open source framework to catch the bad guy
34 www.redhat.com
aUDIt RUlE
DEfINItIoN
-a always,exit -F arch=b32 -S ptrace -F
key=ISSO-tracing
-a always,exit -F arch=b64 -S ptrace -F
key=ISSO-tracing
Could indicate hacker activity or just a programmer
debugging.
-a always,exit -F arch=b32 -S personality -k
ISSO-bypass
-a always,exit -F arch=b64 -S personality -k
ISSO-bypass
Could be an attempt to bypass audit, or simply a legacy
program.
-a exit,always -F path=/sbin/insmod -F perm=x
-F key=ISSO-modules
-a exit,always -F path=/sbin/rmmod -F perm=x
-F key=ISSO-modules
-a exit,always -F path=/sbin/modprobe -F
perm=x -F key=ISSO-modules
-a always,exit -F arch=b32 -S init_module -S
delete_module -F key=ISSO-modules
-a always,exit -F arch=b64 -S init_module -S
delete_module -F key=ISSO-modules
Module actions (insert,delete, and probes).
-a exit,always -F path=/etc/cron.deny -F
perm=wa -F key=ISSO-cron-at-jobs
-a exit,always -F path=/etc/cron.allow -F
perm=wa -F key=ISSO-cron-at-jobs
-a exit,always -F path=/etc/at.deny -F perm=wa
-F key=ISSO-cron-at-jobs
-a exit,always -F path=/etc/at.allow -F
perm=wa -F key=ISSO-cron-at-jobs
Specificwatchesaresethereforinternalprogramfiles.
1.2 red hat enterprise linux 6 log management operational process
Whatfollowsisthemostimportantsectionofthiswhitepaper.Nowthatwehaveconfiguredauditdto
collectlogsaccordingtoCAPP,theremustbeanoperationalprocessinplacetoreviewthelogfiles.This
processmustberoutine,organized,flexible,andmademandatorywithintheorganization’ssecuritypolicy.
Otherwise,alltheconfigurationandguidanceweappliedintheprevioussectionareirrelevantandthereis
no framework to catch the “BadGuy.” The following sections put forth guidelines on an operational process
thatmonitorslogfilesinrealtime.
1.2.1 defining roles and responsiBilities
NISTSpecialPublication800-92“GuidetoComputerSecurityLogManagement”notesanorganization
shouldperformsignificantplanningandpreparatoryactionsforperforminglogmanagementandtoestab-
lish and maintain successful a log management infrastructure. It is recommended that the ISSO oversee the
log management infrastructure as well as analyzing the logs periodically, reporting on the results of the log
management activities to the ISSM. In addition, system and network administrators need to periodically
analyzethelogfiles.Securityadministratorsifavailableshouldalsoperformloganalysis.
Typically, system, network, and security administrators are responsible for managing logging on their
systems, performing regular analysis of their log data, as well as documenting and reporting the results. For
the purpose of this whitepaper we will call the subset of these individuals ISSOs.
Using an open source framework to catch the bad guy
www.redhat.com 35
1.2.2 rhel 6 forensics and incident response log analysis
When performing log analysis or even working a postmortem investigation, an ISSO should start up front
with the main aureport output to just get an idea about what is happening on the system. This report will
tell you about events that are hard coded by the audit system such as login and logout, uses of authentica-
tion, system anomalies, how many users have been on the machine, and if SELinux has detected any AVCs
26
.
Once a point of interest has been found, the ISSO can look up the event with the ausearch -a event
number (as all reports have the audit event number). Specifying start and stop times will also help narrow
downspecifics.Thereportsproducedbytheaureport command should be used as building blocks for
more complicated analysis. See Table 25: Log analysis commands by shift for a detailed review and
example of forensics and incident response log analysis. This table provides a starting point and routine to
helpfindanomaliesandsituationsthatdonotcomplywiththesecuritypolicy.Useittoevolvebestprac-
ticesinyourdailyauditreviewingactivities.Youwillbesurprisedwhatyoufind,andtheinformationwillput
you one step ahead of the BadGuy!
taBle 25: log analysis commands By shift
SHIft
loG aNalYSIS CoMMaND
DESCRIPtIPoN
1st Shift:
6:00am – 2:00pm
The 1st Shift is the Primary Shift
reviewinglogfilesfromthedaybefore
as well as generating reports.
aureport --summary --start
yesterday
Running this report will allow the
ISSO to get a rough overview of the
current audit statistics (events, logins,
processes, etc.) for the previous day.
The 1st Shift should run reports
summarizing the last days events.
Use ausearch --event audit-
event-id if need be to tunnel down.
aureport --failed --start
yesterday
Running this report will allow the ISSO
to get statistics of failed events.
(continued on next page)
In the next section we setup a routine log analysis policy geared toward using auditd on a RHEL 6 system.
Because we have setup remote host logging, we have chosen in this example to perform log analysis
centrally.CentralizinglogfilesallowstheISSOtogetthebigpicture,asithaseverylogfilefromtheenter-
prisecentrallyontheaggregateloghost.RedHataudittoolscanbeusedtotunneldownandfindspecific
notesofinterest.Remembertousethe–inputfile-nameflagwiththetoolstoviewdatainarotatedlogfile.
Youcouldalsotakeasubsetoflogfilesandconcatenatethemtogetherwiththecat command for analysis
(togathermultiplerotationdaysoflogfiles).See(Table 24: Concatenating compressed log files
in the /var/log/audit directory for analysis). This command actually will concatenate all the log
filesinthe/var/log/audit directory. The ISSO could creative by using the seqcommandtopullaspecific
number of logs out.
26 SELinux AVC (Access Vector Cache) is a new operating system component that provides caching of access decision computations to
minimize the performance overhead of the Flask security mechanisms.
Using an open source framework to catch the bad guy
taBle 24: Concatenatingcompressedlogfilesinthe/var/log/auditdirectoryforanalysis
SCRIPt
for i in `ls -1 /var/log/audit/audit.log.*` ; do cat /var/log/audit/audit.log > reviewlog.audit ; zcat $i >> reviewlog.audit ;
done
36 www.redhat.com
SHIft
loG aNalYSIS CoMMaND
DESCRIPtIPoN
aureport –l --failed --start
--yesterday
This command will allow the ISSO
to get more granular detail of failed
events for login-related events.
aureport –f --failed --start
--yesterday
This command will allow the ISSO
to get more granular detail of failed
eventsforfile-relatedevents.
aureport –p --failed --start
--yesterday
This command will allow the ISSO
to get more granular detail of failed
events for process-related events.
aureport –u --failed --start
--yesterday
This command will allow the ISSO
to get more granular detail of failed
events for user-related events.
aureport –k KeyName --start
--yesterday
This command will provide a high level
report on all the keys we set in the
audit.rulesfile.SpecificKeyName
we can use are all the Keys starting
with our ISSO-KeyName.
ausearch -k ISSO-audit --start
--yesterday
This command will allow the ISSO to
see both successful and unsuccessful
attempts to read information from the
auditrecordsandanymodifications
to the audit trail.
ausearch -k ISSO-cfg-audit
--start --yesterday
This command will allow the ISSO
toseemodificationstoaudit
configuration.
ausearch -k ISSO-time-change
--start --yesterday
This command will allow the ISSO to
see any audit record that could affect
the time of the system.
ausearch -k ISSO-system-local
--start --yesterday
This command will allow the ISSO to
see any audit record that could note a
change in system locale.
ausearch -k ISSO-MAC-policy
--start --yesterday
This command will allow the ISSO to
see any audit record that could note
a change in system the Mandatory
Access Control Policy.
ausearch -k ISSO-MAC-policy
--start --yesterday
This command will allow the ISSO to
see any audit record that could note
a change in system the Mandatory
Access Control Policy.
ausearch -k ISSO-access
--start --yesterday
This command will allow the ISSO to
see any audit record that could note
unsuccessfulaccessattemptstofiles.
ausearch -k ISSO-privilege
--start --yesterday
This command will allow the ISSO to
see any use of privileged commands
both unsuccessful and successful.
(continued on next page)
Using an open source framework to catch the bad guy
www.redhat.com 37
SHIft
loG aNalYSIS CoMMaND
DESCRIPtIPoN
ausearch -k ISSO-media-export
--start --yesterday
This command will allow the ISSO to
see any and all successful exports to
media.
The ISSO should make note of the
user.
ausearch -k ISSO-admin-actions
--start --yesterday
This command will allow the ISSO to
seeallactionwiththesudoersfile.
ausearch -k ISSO-auth --start
--yesterday
This command will allow the ISSO to
see all changes in user authentication
and identity.
ausearch -k ISSO-audit-logs
--start --yesterday
This command will allow the ISSO to
see if any unauthorized access, modi-
fication,ordeletionhastakenplace
on the audit trail.
ausearch -k ISSO-modules
--start --yesterday
This command will allow the ISSO to
see if any unauthorized access, modi-
fication,ordeletionhastakenplace
with kernel modules.
ausearch -k ISSO-cron-at-jobs
--start --yesterday
This command will allow the ISSO
to see if any unauthorized access or
modificationhastakenplacewith
cron and at jobs (cron.deny, cron.
allow, at.deny,and at.allow).
aureport --summary --start
today 00:00:01
Running this report will allow the
ISSO to get a rough overview of the
current audit statistics (events, logins,
processes, etc.) for the days events to
the current time.
Use ausearch --event audit-
event-id if need be to tunnel down.
ausearch – a audit_event_id
Running this search will allow the
ISSO to view all records carrying a
suspicious audit event ID. Each audit
event message has a unique ID. One
application’s system call may have
several events that are logged, and
this will allow a trail of more than one
record to be pieced together to tell a
story.
(continued on next page)
Using an open source framework to catch the bad guy
38 www.redhat.com
SHIft
loG aNalYSIS CoMMaND
DESCRIPtIPoN
2nd Shift:
2:00pm – 10:00pm
aureport --summary --start
today 00:00:01
Running this report will allow the
ISSO to get a rough overview of the
current audit statistics (events, logins,
processes, etc.) for the day’s events
to the current time.
Use ausearch --event audit-
event-id to tunnel down for further
investigation.
This will also check for the morning
shift as well as yesterday, as most of
the organization's programmers work
during the morning shift.
ausearch -k ISSO-ptrace
--start --yesterday
This command will allow the ISSO to
see hacker activity by a user, or just a
programmer debugging.
Should be investigated.
ausearch -k ISSO-bypass
--start --yesterday
This command will allow the ISSO to
see if there was an attempt to bypass
audit, or it could be a legacy program.
Should be investigated. Any legacy
program that has been approved
should be noted as a false positive.
Could | grep -v to elininate false
positives.
3rd Shift:
10:00pm – 6:00am
aureport --summary --start
today 00:00:01
Running this report will allow the
ISSO to get a rough overview of the
current audit statistics (events, logins,
processes, etc.) for the day’s events
to the current time.
Use ausearch --event audit-
event-id to tunnel down for further
investigation..
aulast --bad --start today
00:00:01
Running this report will allow the ISSO
to report on all bad logins for the day.
All users found in this list should be
emailed and asked if they had failed
loginsforthatspecificday.When
they come in for work the next day,
they will see their email. Policy states
that they are to reply back if they did
not have the failed login attempt.
(continued on next page)
Using an open source framework to catch the bad guy
www.redhat.com 39
PaRt 2: HoSt-baSED INtRUSIoN DEtECtIoN SYStEM
A host-based IDS provides the data integrity needed to ensure adequate protection of information and
system data, helping you meet security requirements and compliance. In Red Hat Enterprise Linux 6, the
RPM program and AIDE program delivers continuous and automated monitoring for security compliance as
well as implementing the needed security controls for a true defense-in-depth approach allowing for Built-in
Forensics, Incident Response, and Security to catch the BadGuy.
The RPM Package Manager (RPM) is a program that can be used as a host-based IDS. RPM contains various
optionsforqueryingpackagesandtheircontents.Theseverificationoptionscanbeinvaluabletoaforensics
investigationandcouldleadtocriticalsystemfilesandexecutablesthathavebeenmodified.
AdvancedIntrusionDetectionEnvironment(AIDE)isafileintegritycheckertoolthatisshippedwithRed
Hat Enterprise Linux 6. Using rules read from the /etc/aide.conffile,AIDEcreatesadatabaseoffile
attributes and extended attribute information. It uses several hashing algorithms for integrity checking,
including md5, sha1, rmd10, tiger, haval, sha256, and sha512. Once the database is initialized it
canbeusedtoverifytheintegrityoffiles.
coming soon
SHIft
loG aNalYSIS CoMMaND
DESCRIPtIPoN
Additional investigation command
notes:
These commands may also
tunneldownandprovidespecific
information.
To see all syscalls made by a
specific program:
auditctl -a exit,always -S all
-F pid=1005
To see files opened by a
specific user:
auditctl -a exit,always -S
open -F auid=510
To see unsuccessful open
calls:
auditctl -a exit,always -S
open -F success=0
String based matches
(Hostname, IPADDR, Filename,
SELinux Context)
ausearch --word IPADDRESS
Search for an event with the
given login user ID.
ausearch –loginuid login-id
Using an open source framework to catch the bad guy
red hat sales and inquiries
north america
1–888–REDHAT1
www.redhat.com
sales@redhat.com
europe, middle east
and africa
00800 7334 2835
www.europe.redhat.com
europe@redhat.com
asia pacific
+65 6490 4200
www.apac.redhat.com
apac@redhat.com
latin america
+54 11 4329 7300
www.latam.redhat.com
info-latam@redhat.com
Copyright © 2010 Red Hat, Inc. Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, MetaMatrix,
and RHCE are trademarks of Red Hat, Inc., registered in the U.S. and other countries. Linux
®
is the registered
trademark of Linus Torvalds in the U.S. and other countries.
www.redhat.com
#0000000_0010
BiBliography
1. Kent,Karen,andMurugiahSouppaya.GuidetoComputerSecurityLogManagementRecommendations
oftheNationalInstituteofStandardsandTechnology.Gaithersburg,MD:U.S.Dept.ofCommerce,
Technology Administration, National Institute of Standards and Technology, 2006. Print.
2. Mayer, Frank, Karl MacMillan, and David Caplan. SELinux by Example: Using Security Enhanced Linux.
Upper Saddle River, NJ: Prentice Hall, 2007. Print.
3. Common Criteria : The Common Criteria Portal. Web. 06 Dec. 2011. <http://www.commoncriteriaportal.
org/>.
4. TheUnitedStatesGovernmentConfigurationBaseline(USGCB)-NIST.Web.06Dec.2011.<http://usgcb.
nist.gov/index.html>.
5. Need,Business.“Redhat.com|Government.”Redhat.com|TheWorld’sOpenSourceLeader.Web.06
Dec. 2011. <http://www.redhat.com/solutions/government/>.
6. Frields, By Paul. “Red Hat Magazine | SSH Port Forwarding.” Red Hat Magazine | Now Showing: open-
source.com. Web. 06 Dec. 2011. <http://magazine.redhat.com/2007/11/06/ssh-port-forwarding/>.
7. MS. Http://csrc.nist.gov/groups/STM/cmvp/documents/140-1/140sp/140sp1384.pdf.
8. Gift,ByNoah.“RedHatMagazine|AdvancedSSHConfigurationandTunneling:We
Don’t Need No Stinking VPN software.” Red Hat Magazine | Now Showing: open-
source.com. Web. 06 Dec. 2011. <http://magazine.redhat.com/2007/11/27/
advanced-ssh-configuration-and-tunneling-we-dont-need-no-stinking-vpn-software/>.
9. ”RemoteLoggingwithSSHandSyslog-NG.”ComputerForensicInvestigationsandInformationSecurity
Consulting | Deer Run Associates. Web. 06 Dec. 2011. <http://www.deer-run.com/~hal/sysadmin/
SSH-SyslogNG.html>.
10. “Operating Systems - NSA/CSS.” Welcome to the National Security Agency - NSA/CSS. Web. 07 Dec. 2011.
<http://www.nsa.gov/ia/mitigation_guidance/security_configuration_guides/operating_systems.shtml>.
“GuidetotheSecureConfigurationOfRedHatEnterpriseLinux5”.
11. Information Assurance Support Environment Home Page. Web. 07 Dec. 2011. <http://iase.disa.mil/>.