Linux Troubleshooting Bible [Electronic resources]

Christopher Negusand, Thomas Weeks

نسخه متنی -صفحه : 213/ 35
نمايش فراداده

Locking Down Desktops and Servers

This section covers some example settings you might want to add to your template workstation or server loads to lock them down, secure them, or just customize or standardize them for your enterprise environment.

Anything that you could normally run or do on a client PC or a new server install from the command line can be put into your

ks.cfg 's postinstall script section to be automated. To begin, you'll need to think through the kind of standardized settings your enterprise PC and server installs will require.

Identifying Your Enterprise Desktop Needs

Just pointing some of your client PCs to different DNS servers, giving them all the same internal home page, bookmarks, or company directory can be invaluable. And how about creating standard command aliases or desktop icons for common company tasks, or "back door" administrative accounts, or a help desk VNC remote desktop setup? Wouldn't you like your people to have standardized corporate NFS/SMB drive mappings, printer settings, and antivirus package configurations? Not to mention things like desktop backups and system health warnings. These are just a few of the things a customized installation template can provide.

Most experienced IT administrators already know the importance of building a customized OS install template. Many inexperienced IT guys learn this the hard way: by months of fighting with users over the phone, countless visits to the cube farms, manual matching, reinstalls, and troubleshooting sessions. A smart set-up design can help you avoid wasted hours and frayed tempers.

Identifying Your Enterprise Server Needs

This is an area where you really want to sink some thought: Would you like all of your servers to have nightly cron jobs to download and apply the latest security patches? Does a root-kit or intrusion security check and e-mail you anything out of the norm? Send you an e-mail if the machine is rebooted? Tar up and backup, to a single place on the network, a set of important config files, logs, or database files? What about setting up trusted SSH keys in every new server that goes online so that you can log into any new server, even if the root password has changed? These are all things that can be put into the %postscript area of your

ks.cfg file.

Planning and defining each of your enterprise server issues is really the key. It's also essential to discuss these plans with all the system administrators in your company to make sure everyone understands the benefits of a standardized kickstart file and to generate ideas for useful %postscript options.

Every company's needs are different. You need to sit down, hash out, and plan for your departmental and organizational needs. Now with all that out of the way, let's take a look at what you can do to make a standard install of Fedora Core Linux (or most any Linux), a bit more tight and secure.

Locking Down Your Template Installs

In this section are some good rules of thumb for locking down and securing the common services that you have running on your machines. What you use here depends on your security needs. You might find it useful to refer to the iptable/firewalling configurations, discussed in Chapter 11, to set up and implement some basic iptable automation for all of your server loads. Just follow along and you'll quickly get an idea of what's possible.

Note

If you have not yet done a basic install of your server template machine, you should do one before proceeding. The commands that we are going to use in our %postscripting section will need to be run and tested as if from the actual install machine itself. As such, you will need to have a test machine to run these commands on to verify proper operation before implementing them in your own

ks.cfg file. To assemble a reliable set of %postscripting commands, you will need to run the various test commands, take notes, and copy and paste your good commands into what will end up being the final %postinstall script section within your

ks.cfg file.

The main install changes that you want to examine right off for your servers and perhaps for some desktop systems, include the following:

Shutting off unneeded services

Locking down networking/IPv4 settings

Patching automation

System "Health checks"

Basic intrusion detection or root-kit checks

On-disk hardware testing packages

The following subsections cover each of these items.

Shutting Off Unneeded Services

Before you can lock down services, you should first see what your default runlevel is. On most Red Hat/Fedora Core-based Linux systems it will either be a default runlevel of 3 (for console/text logins) or 5 (for Xll GUI display manager logins). Reboot your template server and type the following:

# cat /etc/inittab | grep ^id:
id:5:initdefault:
#
# runlevel
N 5

If you're on a Red Hat/Fedora machine and both results are 5, you are running a X11 GUI-based login machine. If you prefer a text login only server, edit

/etc/inittab and change the line you see above to an

initdefault of 3, reboot, and check it again. Runlevel 3 should be most servers' default runlevel if a run-time GUI is not desired. In the examples here, I'm operating on a server load installed on my laptop, so runlevel 5 will be the default.

Tip

You want to have a good reason before running a server in runlevel 5 with X11 GUI running, not only is this a security issue but it also eats up system resources, there are serious performance issues with running a server with X windows in place. Just think about some 3-D screen savers and the processing power they eat up. Do you really want your web server or mail server that's under attack from the latest virus having to tread water while competing for system memory and CPU resources with the latest 3-D "The Matrix" screen saver? I think not.

The following

chkconfig listing shows the services I have running in my runlevel 5:

# chkconfig --list|grep 5:on|sort
acpid          0:off   1:off    2:off   3:on   4:on   5:on   6:off
anacron        0:off   1:off    2:on    3:on   4:on   5:on   6:off
apmd           0:off   1:off    2:on    3:on   4:on   5:on   6:off
atd            0:off   1:off    2:off   3:on   4:on   5:on   6:off
autofs         0:off   1:off    2:off   3:on   4:on   5:on   6:off
crond          0:off   1:off    2:on    3:on   4:on   5:on   6:off
cups           0:off   1:off    2:on    3:on   4:on   5:on   6:off
gpm            0:off   1:off    2:on    3:on   4:on   5:on   6:off
httpd          0:off   1:off    2:on    3:on   4:on   5:on   6:off
iptables       0:off   l:off    2:on    3:on   4:on   5:on   6:off
irqbalance     0:off   1:off    2:off   3:on   4:on   5:on   6:off
isdn           0:off   1:off    2:on    3:on   4:on   5:on   6:off
kudzu          0:off   1:off    2:off   3:on   4:on   5:on   6:off
messagebus     0:off   1:off    2:off   3:on   4:on   5:on   6:off
microcode_ctl  0:off   1:off    2:on    3:on   4:on   5:on   6:off
named          0:off   1:off    2:on    3:on   4:on   5:on   6:off
netfs          0:off   1:off    2:off   3:on   4:on   5:on   6:off
network        0:off   1:off    2:on    3:on   4:on   5:on   6:off
nfslock        0:off   1:off    2:off   3:on   4:on   5:on   6:off
pcmcia         0:off   1:off    2:on    3:on   4:on   5:on   6:off
random         0:off   1:off    2:on    3:on   4:on   5:on   6:off
rawdevices     0:off   1:off    2:off   3:on   4:on   5:on   6:off
rhnsd          0:off   1:off    2:off   3:on   4:on   5:on   6:off
sendmail       0:off   1:off    2:on    3:on   4:on   5:on   6:off
smartd         0:off   1:off    2:on    3:on   4:on   5:on   6:off
spamassassin   0:off   1:off    2:on    3:on   4:on   5:on   6:off
sshd           0:off   1:off    2:on    3:on   4:on   5:on   6:off
syslog         0:off   1:off    2:on    3:on   4:on   5:on   6:off
touchpad       0:off   1:off    2:off   3:on   4:off  5:on   6:off
vsftpd         0:off   1:off    2:on    3:on   4:on   5:on   6:off
xfs            0:off   1:off    2:on    3:on   4:on   5:on   6:off
xinetd         0:off   1:off    2:on    3:on   4:on   5:on   6:off

That's a lot of services configured to be running. Also, don't forget about all the xinetd spawned daemons. These subdaemons will either be configured to be on or off at the same runlevel that xinetd is configured to be on:

#  export LC_ALL=C
#  chkconfig --list| grep [a-z]: | grep on$| sort
cups-lpd:               on
finger:                 on
imap:                   on
imaps:                  on
pop3s:                  on
sgi_fam:                on

If any of the aforementioned services are unfamiliar, take note of them. We'll tell you how to look up and determine what unfamiliar services are, so that you can decide on whether to shut them down or not.

You should also check which services are bound to the network IP addresses, with the

netstat command:

# netstat -at
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address        Foreign Address         State
tcp        0   0 *:imaps                 *:*                     LISTEN
tcp        0   0 *:pop3s                 *:*                     LISTEN
tcp        0   0 *:printer               *:*                     LISTEN
tcp        0   0 *:pop3                  *:*                     LISTEN
tcp        0   0 localhost.localdoma:783 *:*                     LISTEN
tcp        0   0 *:imap                  *:*                     LISTEN
tcp        0   0 *:finger                *:*                     LISTEN
tcp        0   0 *:http                  *:*                     LISTEN
tcp        0   0 *:ftp                   *:*                     LISTEN
tcp        0   0 dav.mydomain.com:domain *:*                     LISTEN
tcp        0   0 mydomain.com:domain     *:*                     LISTEN
tcp        0   0 10.10.10.1:domain       *:*                     LISTEN
tcp        0   0 192.168.128.25:domain   *:*                     LISTEN
tcp        0   0 localhost.locald:domain *:*                     LISTEN
tcp        0   0 *:ssh                   *:*                     LISTEN
tcp        0   0 localhost.localdoma:ipp *:*                     LISTEN
tcp        0   0 *:smtp                  *:*                     LISTEN
tcp        0   0 localhost.localdom:rndc *:*                     LISTEN
tcp        0   0 *:https                 *:*                     LISTEN
tcp        0   0 localhost.localdoma:ipp localhost.localdo:32963 FIN_WAIT2
tcp        1   0 192.168.128.25:32926    216.239.39.104:http     CLOSE_WAIT
tcp        1   0 192.168.128.25:32925    216.239.57.99:http      CLOSE_WAIT
tcp        1   0 localhost.localdo:32963 localhost.localdoma:ipp CLOSE_WAIT
tcp        1   0 localhost.localdo:32921 localhost.localdoma:ipp CLOSE_WAIT

In this netstat, you see quite a few protocols listening on IP address 0.0.0.0, or in nonnetworking talk, all IPs. Do you want all of these services exposed to the outside? Again, take notes on these template machine settings.

First, you want to turn off the services that you don't want running in your default runlevel, or for good measure, runlevels 2 to 5. For example, browsing though the first

chkconfig listing above, I know that the services

autofs , cups , isdn , kudzu , and

nfslock don't need to be running. Configure them to be

off for all usable runlevels (2, 3, 4, and 5) with the

chkconfig serviceoff command, and then stop the ones that are currently running by calling the daemon init script directly with the

stop option; do this for each service that you don't want running:

#  chkconfig autofs off; /etc/init.d/autofs stop
Stopping automount:                [ OK ]
#  chkconfig cups off ; /etc/init.d/cups stop
Stopping cups:                     [ OK ]
.
.
#  chkconfig nfslock off; /etc/init.d/nfslock stop
Stopping NFS statd:                [FAILED]

Don't worry about

FAILED messages. This is usually because of an interrelated service dependency or an issue that a reboot will clear up. Do another

netstat -at if you're not sure whether you just locked clown those services.

And for xinetd subdaemons, if you don't want

cups-lpd , finger , or

imap running on your server load by default either, switching xinetd services to

off will stop them immediately:

#  chkconfig cups-lpd off
#  chkconfig finger off
#  chkconfig imap off

Now do a netstat listing again with

netstat -at and see what services are now bound to your IPs. Then, do another

chkconfig-list | grep5:on listing and now look as to what's configured to come up at boot time. Better?

Tip

If you're not sure what certain services do and you want a quick run down of each, Red Hat- and Fedora Core-based systems offer a quick, easy-to-use ncurses TUI (or text-based GUI) interface tool called ntsysv. After running ntsysv, it has a concise F1 help description for every service installed on your system. Invoke it as root by running ntsysv, or use the new, handsome, full GUI version

redhat-config-services . It gives you a very good description of each service on your machine. You can read up and then decide whether to turn off each service or leave it running. WARNING: If you're not sure about a given service, ask someone or leave it on until you learn more.

So the

chkconfig service off ; /etc/init.d/service stop commands that you just ran to lock down the services on this template server did a good job. Now copy/paste them into a text file called

/root/post-script.txt or the like for eventual inclusion in your

ks.cfg file later. Your file should look something like the following:

# Shut off all the services we don't want running....
chkconfig autofs off ; /etc/init.d/autofs stop
chkconfig cups off ; /etc/init.d/cups stop
chkconfig isdn off ; /etc/init.d/isdn stop
chkconfig kudzu off ; /etc/init.d/kudzu stop
chkconfig nfslock off ; /etc/init.d/nfslock stop
chkconfig cups-lpd off
chkconfig finger off
chkconfig imap off

Now that you have a basic starting place for your

ks.cfg file's

%post config, let's continue to see what you can do to tighten your kickstart-based OS install.

Locking Down Networking/Ip Stack Settings

Professionals who know Linux love it when it comes to networking because you have control over the networking and TCP/IP stack more so than any other production operating system. You interact with networking and the IPv4 settings in a number of ways (such as with iptables, as you'll see in Chapter 11), but when dealing with the real-time kernel settings, you're usually reading and writing the settings to some kernel variable under the the Procfs, aka the

/proc virtual filesystem. There are a lot of settings for the IPv4 networking system under

/proc . Most of these

/proc parameters can be controlled very nicely with the

sysctl system via the

/etc/sysctl.conf file . This file is run every time the system boots, and is the more modern and safer way of working with

/proc changes (as compared to the old school method of using post-boot

echo variable>

/proc/sys/... commands in the

rc.local file). If you need any kernel run-time parameters changed, do it here when possible. Many times when people need a kernel recompiled to make some adjustment to the kernel runtime parameters, for things like running Oracle, memory hungry apps, or specialized networking apps, sysctl is what they really need (since it will remain active even after a kernel upgrade or the like).

Note

In the old days, administrators used to control the kernel settings either with a kernel recompile or via

/proc directly by echoing changes out to specific settings within proc from the

rc.local startup file, as in this snippet:

# stops SYN flood attacks
echo 1 > /proc/sys/net/ipv4/tcp_syncookies

This is old school, rather dirty, and not really safe since most administrators did this when the

/etc/rc.d/rc.local file was run, which is very late in the boot process. This creates time-related vulnerabilities, and in some cases (such as Oracle) simply cannot be done. Instead of echoing values like this out to

/proc , simply poke such values into the

/etc/sysctl.conf file. Read the man sysctl.conf page for more information.

To help secure your TCP/IP stack on a standard Linux mahcine, you will want to add the following lines to your

/etc/sysctl.conf file:

##TWW: 2004-02-22
# stop syn-flood attacks
net.ipv4.tcp_syncookies = 1
# Throws off people trying to fingerprint your OS by its TTL (def=64)
net.ipv4.ip_default_ttl = 61
# stops the routing table from being modified by packets non-sourc routed
net.ipv4.conf.all.accept_redirects = 0
# same as above but stops your system doing the same to others
net.ipv4.conf.all.send_redirects = 0

You can add this to your script by storing these settings on a local FTP server (that the kickstart will have access to), and in the

%post config pull them down from your ftp.example.com FTP server (for example) and append it on the end of the existing

/etc/sysctl.conf :

# Get all of our sysctl/networking adjustments
wget /pub/installs/sysctl - append
cat sysctl-append >> /etc/sysctl.conf

So with this in your

%post config script, you will append the contents of your standard

sysctl-append file to the end of your new install's

/etc/sysctl.conf file, thus enabling these settings every time one of these systems is booted-a very nice way to ensure all of your systems are just that much more secure.

Additional System Lock Downs

Here are some common security measures that some system administrators like to see in place on their servers. These are basically a collection of shell-based commands that will help lock down and secure various aspects of the system.

# restricts those not in the root group from being able to su to root
chmod 4750 /bin/su
# restricts the mount/umount commands to root group only users
chmod 4750 /bin/mount
chmod 4750 /bin/umount
#Install "scponly" scp replacement for bash and add to /etc/shells
rpm -ivh /pub/installs/scponly-3.8-1.i586.rpm
echo /usr/bin/scponly >>/etc/shells
#make scponly the default user shell
wget /pub/installs/useradd
mv -f useradd /etc/default/useradd

And finally, here are some details we would like to add on all servers that get put online with our kickstart file:

#Send an email whenever this box is rebooted
echo "echo | mail -s\ "Server \ $HOSTNAME Rebooted\ " pager@mail.example.com">>/etc/rc.d/rc.local
#Document which kickstart file this is:
echo ks-server-install_2004-02-22.crg > /etc/ks-release

Note that the Rebooted line will work with either the trailing \ on the first line or as a single unwrapped line without the trailing \.

These two lines are very nice. The first will send you an e-mail whenever one of the servers that it's installed on is rebooted, and the second will allow you to track which kickstart created a given server.

You may want to add other lock clown measures, such as these:

Adding SSH-key based authentication (Chapter 18)

iptable/firewall service restrictions by network (Chapter 11)

On-disk hardware testers such as memtest86 (www.memtest86.com ), smartctl (http://smartmontools.sourceforge.net ), or cpuburnin (http://users.bigpond.net.au/cpuburn/ )

You can add these in your %postscript and test them out.

The only real limit to what you can do with the

%post config scripts in an enterprise environment is how general you want your

ks.cfg files to be (that is, how many or few you wish to maintain), and how much time you have to spend on fine-tuning them.

Tip

If you want good support for getting great lock-down information from other departments, make their system administrators and professionals a part of the effort. Show them what is possible with package adds, lock downs, security settings, and so on-and let them "own" their own little section of the script that deals with their work (security, FTP/Web admin, DNS admins, backup folks, and so forth). Not only will you get better acceptance of this methodology and be including others in setting de facto policy, but you'll get some really great scripts out of it!

The Final %postscript

Frankly, you can spend years tweaking this type of script. Let's take a look at what we have after all the changes thus far. I have also added some sections that are "works in progress." Some of these sections are commented out but left in so that you can see how they work:

%post
###########Setup###########
exec < /dev/tty6 > /dev/tty6 2>/dev/tty6 #Set up chvt to handle VTtoggles
chvt 6                                   #put this in all post-scripts
mkdir /tmp/install                       #Set up temp file space
#cd /tmp/install
#######iptable-ssh from MY LAN#######
# May have to do this in the "first boot" after install
#/etc/init.d/iptables restart
#iptables -I RH-Firewall-1-INPUT 7 -s 10.1.1.0/24 -p #tcp --dport 22 -j ACCEPT
#iptables-save >/etc/sysconfig/iptables
###########Services Stuff###########
# Shut off all the services we don't want running....
chkconfig autofs off ; /etc/init.d/autofs stop
chkconfig cups off ; /etc/init.d/cups stop
chkconfig isdn off : /etc/init.d/isdn stop
chkconfig kudzu off ; /etc/init.d/kudzu stop
chkconfig nfslock off ; /etc/init.d/nfslock stop
chkconfig cups-lpd off                    #These are xinetd based
chkconfig finger off
chkconfig imap off
###########Networking############
# Get all of our sysctl/networking adjustments
service network start                      #Turn on the network
wget /pub/installs/sysctl-append
cat sysctl-append >> /etc/sysctl.conf
###########System Stuff###########
# Only root group has access to su
chmod 4750 /bin/su
# Only root group users have mount/umount
chmod 4750 /bin/mount
chmod 4750 /bin/umount
# Install "scponly" scp replacement for bash
rpm -ivh /pub/installs/scponly-3.8-1.i586.rpm
# And add it to /etc/shells
echo /usr/bin/scponly >>/etc/shells
# Make scponly the default user shell
wget /pub/installs/useradd
mv -f useradd /etc/default/useradd
###########Intrustion Detection########
## Need to get latest chkrootkit and put here....
## Get from http://www.chkrootkit.org/ and
#wget ftp://ftp.pangeia.com.br/pub/seg/pac/chkrootkit.tar.gz
# Untar it:
#tar xzvf chkrootkit.tar.gz
# Compile it:
#cd chkrootkit-*/
#make sense
#cd .. ; my chkrootkit-* /root/
# Clean up
#/bin/rm -rf chkrootk*
# Can also set up automation scan & updates via cron
# Also see chapter 10 on how to "hide" chkrootkit for
# better post-hack trustworthiness of the binary.
#########AutoPatching/Up2date###########
# uncomment when we're ready to use with registration
#up2date-nox -uf
###########Final Custom Config########
# Send an email whenever this box is rebooted
echo "echo | mail -s\ "Server \ $HOSTNAME Rebooted\ " \
pager@mail.example.com">>/etc/rc.d/rc.local
#Document which kickstart file this is:
echo ks-server-install_2004-02-22.cfg > /etc/ks-release
###########CLEAN UP##############
cd /root
/bin/rm -rf /tmp/install
echo | mail --s"Done With Kickstart on $HOST" admin@example.com
chvt 1                                #Tear down the fancy chvt

Note

Installing things like chkrootkit at install time is rather cool, but ultimately not very useful. This is because once a box is cracked (a.k.a. hacked), you cannot trust anything on the box. Many of the newer rootkits out there can change the way the kernel works and modify system memory (infecting seemingly secure parts of the system) systemically. The chkrootkit install in this kickstart %postsection is just a demonstration to just let you see what's possible from within this environment (on the fly downloading, decompressing, and compiling and installation of a package). For more information on the modern issues of crackers and the threats of rootkits, see this article: http://neworder.box.sk/newsread.php?newsid=4182 .

That's it. Save the changes to the

%post config area of your

ks-server.cfg file to do some testing in the next section.