Contents

Introduction

This document attempts to collect together the most useful management tools available on Linux distributions, with strong emphasis on Red Hat Enterprise Linux (version 4 mainly). It also contains some notes about what are the largest differences between Linux-based systems to HP-UX systems since that was the original target audience for this document, but the document has since grown out of the original scope.

Target audience for this document is system administrators who have some command line shell experience in UNIX-like systems (not including Windows) and want to learn the Linux-specific ways of doing things.

What is Linux?

For pedantic people Linux is the kernel that is used to build UNIX-like systems. For rest of us Linux means some set of software packages on top of this kernel, tested and integrated to form a system that can be administered using same techniques as any UNIX-like system.

There are two main branches of Linux-based systems (distributions): The commercial distributions are meant for production use in organizations and provide long product life and a reduced set of supported software. The free/non-commercial distributions are more targeted towards the home-users or speciality environments (scientific clusters) and there ease of use and customization and modularity are the key attributes. Normally non-commercial distributions are not tested against commercial brand-servers and as such might not contain all the low-level device drivers that are present in commercial distributions. On the other hand, non-free distributions contain a lot of end-user drivers that are useful for home-use which in turn are not present in commercial distributions (NTFS driver for example).

Most notable commercial distributions at the moment are:

Most notable non-commercial distributions:

Most notable Live-CD distributions (run directly off the CD-ROM, no need to install):

You can find a nice portal page that collects all the distributions at http://distrowatch.com/.

What is the best distribution to select? This is as difficult question as "what is the best car?". Obviously no one distribution is the best for everyone. For production using a commercial distribution is highly recommended as they normally are supported from 5 to 7 years after each release. This means that software will be only updated for security fixes and no new features will be introduced to the system. This will require the least amount of work after the system has been installed. Commercial distribution vendors also test their distributions on production servers from brand hardware manufacturers.

For home use where software capability set stability is not so critical, any of the freely available distributions are good. Most distributions feature an easy installer system so that shouldn't pose a problem. Pick one that you can find support for easily (friends, forums and such). Commercial support is also available for some of the free distributions (Ubuntu for example).

In some cases the list of architectures for a distribution will affect your choice. The Linux kernel is very versatile and has been ported to over 15 different architectures (PC being only one such target), but you will need other software as well. Gentoo seems to be a good choice nowadays for more interesting architectures (HP-PA, Alphas, Super-H).

Device names and device file management

In Linux all device files are located under /dev/, so nothing new here compared to other UNIX-like systems. The device major and minor codes are different (as they are between all UNIXes). The modern Linux distributions are using a daemon called udevd that will track kernel device related events and will create device nodes when the devices are attached to system. It will also remove device nodes when the hardware devices leave the system. This is very useful with dynamic bus systems like USB and Firewire, but the same system is also used with hotswap PCI and SCSI. Older Linux distributions (like RHEL3) still had a /dev/ filled with all possible device files, even if you didn't have the hardware for them. Some other older distributions used a kernel facility called devfs which was a dynamic filesystem that was mounted on top of /dev/ and "created" devices which were really present. Linus never really liked this system so once the hotplug-event interface was ready the device naming decisions could me moved into user-space and udevd was born. devfs is no longer part of the 2.6 series kernel.

Notable device files in Linux:

As a first example of device file use in Linux we'll setup the first serial RS-232C port to accept terminal login sessions. This way you can attach a real hardware terminal, or a laptop running terminal software that can be used to login into the system.

You should add these lines into /etc/inittab which is the configuration file for init:

# First column must be unique id for each line. We use
# Sx to signify "serial-lines", although you might want to
# use something else. Doesn't really matter as long as id
# field is unique for each action that init should perform.
# We also enable these two programs for run levels 1-5.
# This means that these will also be available on single
# user mode. Respawning means that init will restart the
# program if the program will terminate for some reason.

# Start getty on first "COM" port and getty will initialize
# the serial line to 8N1 9600 (man getty for other options)
# Also tell getty that it should set TERM to 'vt100' for
# programs spawned on that terminal line.
S0:12345:respawn:/sbin/getty -L ttyS0 9600 vt100

# Call-in via modem is supported via mgetty-program
# Here we tell mgetty to init the modem via AT&T Hayes
# command X0 and init the speed to 57600 baud on second
# serial port
S1:12345:respawn:/sbin/mgetty -x0 -s 57600 ttyS1

[ Enabling login on serial lines ]

As a second example we'll make an "image" file out of storage media and then use the image file instead of the real physical media. Similar approach can be used with any filesystem, but we'll use the ISO 9660 standard here (iso9660 is the filesystem driver name in Linux kernel).

[root@system ~]# cp /dev/cdrom /tmp/cdrom.iso

Connect /dev/loop0 to file /tmp/cdrom.iso
[root@system ~]# losetup /dev/loop0 /tmp/cdrom.iso

Mount the filesystem on /dev/loop0 onto /media/cdrom
[root@system ~]# mount -t iso9660 /dev/loop0 /media/cdrom

Verify that the mount succeeded by listing active mount points
[root@system ~]# df -T
Filesystem    Type   1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
              ext3     3644604   1384840   2074628  41% /
/dev/sda1     ext3      101086      9213     86654  10% /boot
none         tmpfs       81356         0     81356   0% /dev/shm
/dev/loop0 iso9660        4930      4930         0 100% /media/cdrom
[root@system ~]# ls -l /media/cdrom
total 3
drwxr-xr-x  2 root root 2048 Oct  1 19:49 isolinux
-r--r--r--  1 root root  220 Oct  1 19:48 TRANS.TBL
[root@system ~]# umount /media/cdrom

Verify that unmounting succeeded
[root@system ~]# df -T
Filesystem    Type   1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
              ext3     3644604   1384844   2074624  41% /
/dev/sda1     ext3      101086      9213     86654  10% /boot
none         tmpfs       81356         0     81356   0% /dev/shm

Check whether there is anything connected to /dev/loop0
[root@system ~]# losetup /dev/loop0
/dev/loop0: [fd00]:255579 (/tmp/cdrom.iso)

There is, remove the association
[root@system ~]# losetup -d /dev/loop0

The mount command can also do the dirty work for you
[root@system ~]# mount -oloop /tmp/cdrom.iso /media/cdrom
[root@system ~]# df -T
Filesystem    Type   1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
              ext3     3644604   1384844   2074624  41% /
/dev/sda1     ext3      101086      9213     86654  10% /boot
none         tmpfs       81356         0     81356   0% /dev/shm
/tmp/cdrom.iso
           iso9660        4930      4930         0 100% /media/cdrom
[root@system ~]# losetup /dev/loop0
/dev/loop0: [fd00]:255579 (/tmp/cdrom.iso)

If mount allocated the loop device, it will also de-associate it automatically
[root@system ~]# umount /media/cdrom

[ Using loop-devices in Linux ]

Here we first copy the contents of a CD-ROM disc into a file using cp. We then bind /dev/loop0 (first loop device) to be connected to /tmp/cdrom.iso. We can then use the virtual loop device as we would with any block device. We proceed to mount it and play a bit with it. We then unmount and break the association between /dev/loop0 and the underlying file. The easier and more common way is to use -o loop with mount. mount will automatically select the first unused loop device and allocate that for us. So in effect, it is doing all the dirty work. There is also another virtual loop device which will move all reads and writes to the device over the network. This device driver is /dev/nbd0 (network block device). It requires a daemon on some server that will handle the actual storage, but unfortunately nbd is not supported in RHEL-systems.

By the way, if you check the manual page for losetup, you will notice that it can also do encryption and decryption for you (the actual cryptography is done inside the kernel, losetup will only set it up for you).

System inspection tools

Linux distributions come with a variety of useful tools that can give you the necessary information (and also a lot of unnecessary).

Here is a short list of the most useful utilities:

As an example, let's display some SCSI-related information (including partitioning information).

[root@system ~]# ls -l /proc/scsi
total 0
-r--r--r--  1 root root 0 Mar 12 16:34 device_info
dr-xr-xr-x  2 root root 0 Mar 12 16:34 mptscsih
-r--r--r--  1 root root 0 Mar 12 16:34 scsi

Display all SCSI HBAs and their information
[root@system ~]# cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: VMware,  Model: VMware Virtual S Rev: 1.0
  Type:   Direct-Access                    ANSI SCSI revision: 02

Display information from one SCSI HBA driver (mpt)
[root@system ~]# cat /proc/scsi/mptscsih/0
ioc0: LSI53C1030, FwRev=00000000h, Ports=1, MaxQ=128

Grep for SCSI-related kernel messages to find out the assigned device names
[root@system ~]# dmesg | grep scsi
scsi0 : ioc0: LSI53C1030, FwRev=00000000h, Ports=1, MaxQ=128, IRQ=9
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0

Display PC-DOS based partition table from first SCSI disk
[root@system ~]# fdisk -l /dev/sda
Disk /dev/sda: 4294 MB, 4294967296 bytes
255 heads, 63 sectors/track, 522 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14         522     4088542+  8e  Linux LVM

Display kernel's current information about partitions and their sizes
[root@system ~]# cat /proc/partitions
major minor  #blocks  name

   8     0    4194304 sda
   8     1     104391 sda1
   8     2    4088542 sda2
 253     0    3702784 dm-0
 253     1     327680 dm-1

[ Display SCSI and partitioning information ]

And more or less same information from a real system (different architecture this time, using fdisk doesn't work since the partitioning system is different).

[root@system ~]# cat /proc/cpuinfo
processor  : 0
vendor     : GenuineIntel
arch       : IA-64
family     : Itanium 2
model      : 1
revision   : 5
archrev    : 0
features   : branchlong
cpu number : 0
cpu regs   : 4
cpu MHz    : 1000.000000
itc MHz    : 1000.000000
BogoMIPS   : 1497.36
[root@system ~]# cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: HP 36.4G Model: MAP3367NC        Rev: HPC6
  Type:   Direct-Access                    ANSI SCSI revision: 03
Host: scsi0 Channel: 00 Id: 01 Lun: 00
  Vendor: HP 36.4G Model: MAP3367NC        Rev: HPC6
  Type:   Direct-Access                    ANSI SCSI revision: 03
[root@system ~]# cat /proc/scsi/mptscsih/0
ioc0: LSI53C1030, FwRev=01030600h, Ports=1, MaxQ=203
[root@system ~]# cat /proc/scsi/mptscsih/1
ioc1: LSI53C1030, FwRev=01030600h, Ports=1, MaxQ=203
[root@system ~]# dmesg | grep scsi
scsi0 : ioc0: LSI53C1030, FwRev=01030600h, Ports=1, MaxQ=203, IRQ=54
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
Attached scsi disk sdb at scsi0, channel 0, id 1, lun 0
scsi1 : ioc1: LSI53C1030, FwRev=01030600h, Ports=1, MaxQ=203, IRQ=55

Even when Itanium-based systems use a different partitioning scheme (GPT),
Linux still uses the same proc-file to display the current partitioning
information. Use GNU parted to edit GPT-based partition tables
[root@paris ~]# cat /proc/partitions
major minor  #blocks  name

   8     0   35566480 sda
   8     1      51200 sda1
   8     2   10240000 sda2
   8    16   35566480 sdb
   8    17     512000 sdb1
   8    18   34643968 sdb2
   8    19     409600 sdb3

Display SCSI disk identification information for all SCSI disks
[root@system ~]# cat /sys/block/sd*/device/{vendor,model,rev}
HP 36.4G
HP 36.4G
MAP3367NC
MAP3367NC
HPC6
HPC6

Display PCI/AGP/PCI-X bus information
[root@system ~]# lspci
00:01.0 USB Controller: NEC Corporation USB (rev 41)
00:01.1 USB Controller: NEC Corporation USB (rev 41)
00:01.2 USB Controller: NEC Corporation USB 2.0 (rev 02)
00:02.0 IDE interface: Silicon Image, Inc. SiI 0649 Ultra ATA/100 PCI to ATA Host Controller (rev 02)
00:03.0 Ethernet controller: Intel Corporation 82557/8/9 [Ethernet Pro 100] (rev 0d)
20:01.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07)
20:01.1 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07)
20:02.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5701 Gigabit Ethernet (rev 15)
e0:01.0 Communication controller: Hewlett-Packard Company Auxiliary Diva Serial Port (rev 01)
e0:01.1 Serial controller: Hewlett-Packard Company Diva Serial [GSP] Multiport UART (rev 03)
e0:02.0 VGA compatible controller: ATI Technologies Inc Radeon RV100 QY [Radeon 7000/VE]

Same as above but without translating the PCI vendor and device IDs.
These are useful to use with googling for device driver support in Linux
[root@system ~]# lspci -n
00:01.0 Class 0c03: 1033:0035 (rev 41)
00:01.1 Class 0c03: 1033:0035 (rev 41)
00:01.2 Class 0c03: 1033:00e0 (rev 02)
00:02.0 Class 0101: 1095:0649 (rev 02)
00:03.0 Class 0200: 8086:1229 (rev 0d)
20:01.0 Class 0100: 1000:0030 (rev 07)
20:01.1 Class 0100: 1000:0030 (rev 07)
20:02.0 Class 0200: 14e4:1645 (rev 15)
e0:01.0 Class 0780: 103c:1290 (rev 01)
e0:01.1 Class 0700: 103c:1048 (rev 03)
e0:02.0 Class 0300: 1002:5159

List USB-bus connected devices.
[root@system ~]# lsusb
Bus 003 Device 002: ID 413c:2005 Dell Computer Corp.
Bus 003 Device 001: ID 0000:0000
Bus 002 Device 001: ID 0000:0000
Bus 001 Device 001: ID 0000:0000

Get more information about the funny Dell device
[root@system ~]# lsusb -v -s 003:002
Bus 003 Device 002: ID 413c:2005 Dell Computer Corp.
Device Descriptor:
  bLength                18
  bDescriptorType         1
  Device USB version. This device is USB 1.1
  bcdUSB               1.10
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0
  bDeviceProtocol         0
  bMaxPacketSize0         8
  idVendor           0x413c Dell Computer Corp.
  idProduct          0x2005
  bcdDevice            1.04
  iManufacturer           1 DELL
  This is the useful bit of information
  iProduct                2 DELL USB Keyboard
  iSerial                 0
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           34
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0
    bmAttributes         0xa0
      Remote Wakeup
    MaxPower              100mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         3 Human Interface Devices
      bInterfaceSubClass      1 Boot Interface Subclass
      bInterfaceProtocol      1 Keyboard
      iInterface              0
        HID Device Descriptor:
          bLength                 9
          bDescriptorType        33
          bcdHID               1.10
          bCountryCode            0 Not supported
          bNumDescriptors         1
          bDescriptorType        34 Report
          wDescriptorLength      65
cannot get report descriptor
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               none
          Usage Type               Data
        wMaxPacketSize     0x0008  bytes 8 once
        bInterval              10
  Language IDs: (length=4)
     0409 English(US)

[ SCSI information from another architecture ]

Linux kernel (starting from 2.4 series) contains a lot of run-time changeable parameters that can affect the operation of the various sub-systems inside the kernel. These parameters can be found under /proc/sys/ and you can use echo to change the values. The values are changes in real-time and not stored into any configuration file. In order to change something permanently you should check out the sysctl-tool (whose configuration file is /etc/sysctl.conf).

Network-related tools

Linux kernel development has borrowed and reimplemented a lot of networking features from other UNIX-like systems. Some basic tools are the same as on any BSD based UNIX, however there are also some low level commands that are unique to Linux.

Non-complete list of networking related tools in Linux:

As an example we display the base IPv4 configuration on our system, notice that we have a service enabled which we don't really need (CUPS, aka Common UNIX Printing System), disable the network printing service and then verify that the port is not listened anymore. At the end we also check the current firewall rules setting, and they are empty on this system ("firewall disabled").

Start by displaying all configured network interfaces
[root@system ~]# ifconfig
MAC of the NIC is listed on the first line. Useful for DHCP-address
reservations
eth0 Link encap:Ethernet  HWaddr 00:0C:29:92:A2:09
     IPv4 address and netmask
     inet addr:192.168.1.19  Bcast:192.168.1.255  Mask:255.255.255.0
     List of all IPv6 addresses. At least one (automatically generated)
     inet6 addr: fe80::20c:29ff:fe92:a209/64 Scope:Link
     UP = interface configured and "running"
     UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
     Statistics. They overflow at 64-bit
     RX packets:114 errors:0 dropped:0 overruns:0 frame:0
     TX packets:107 errors:0 dropped:0 overruns:0 carrier:0
     collisions:0 txqueuelen:1000
     RX bytes:13278 (12.9 KiB)  TX bytes:15302 (14.9 KiB)
     Interrupt:11 Base address:0x1400

Loopback interface which will receive all traffic for network 127
lo   Link encap:Local Loopback
     inet addr:127.0.0.1  Mask:255.0.0.0
     inet6 addr: ::1/128 Scope:Host
     UP LOOPBACK RUNNING  MTU:16436  Metric:1
     RX packets:14 errors:0 dropped:0 overruns:0 frame:0
     TX packets:14 errors:0 dropped:0 overruns:0 carrier:0
     collisions:0 txqueuelen:0

Display current routing table
[root@system ~]# route -n
Kernel IP routing table
Destination  Gateway        Genmask       Flags Metric Ref Use Iface
192.168.1.0  0.0.0.0        255.255.255.0 U     0      0     0 eth0
169.254.0.0  0.0.0.0        255.255.0.0   U     0      0     0 eth0
This is the default gateway setting
0.0.0.0      192.168.1.254  0.0.0.0       UG    0      0     0 eth0

Display DNS-resolver configuration. It has been generated
by DHCP client
[root@system ~]# cat /etc/resolv.conf
; generated by /sbin/dhclient-script
search intranet
nameserver 192.168.1.254

Display all TCP ports that have a process listening for
incoming connections. You need root privileges to display
the process names.
0.0.0.0 = all IPv4 addresses in system
127.0.0.1 = local loopback only
::: = all IPv4 and IPv6 addresses
[root@system ~]# netstat -tlnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address  Foreign Address  State   PID/Program name
tcp        0      0 0.0.0.0:32768  0.0.0.0:*        LISTEN  1916/rpc.statd
tcp        0      0 0.0.0.0:111    0.0.0.0:*        LISTEN  1897/portmap
tcp        0      0 0.0.0.0:631    0.0.0.0:*        LISTEN  2028/cupsd
tcp        0      0 127.0.0.1:25   0.0.0.0:*        LISTEN  2123/sendmail: acce
tcp        0      0 :::22          :::*             LISTEN  2091/sshd

Do the same for UDP ports. At this point we notice that
we have an unnecessary service running by default (cupsd)
[root@system ~]# netstat -ulnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address  Foreign Address  PID/Program name
udp        0      0 0.0.0.0:32768  0.0.0.0:*        1916/rpc.statd
udp        0      0 0.0.0.0:820    0.0.0.0:*        1916/rpc.statd
udp        0      0 0.0.0.0:68     0.0.0.0:*        3363/dhclient
udp        0      0 0.0.0.0:111    0.0.0.0:*        1897/portmap
udp        0      0 0.0.0.0:631    0.0.0.0:*        2028/cupsd

Check the service runlevel configuration
[root@system ~]# chkconfig --list cups
cups            0:off   1:off   2:on    3:on    4:on    5:on    6:off
Disable and verify
[root@system ~]# chkconfig cups off
[root@system ~]# chkconfig --list cups
cups            0:off   1:off   2:off   3:off   4:off   5:off   6:off
Stop service
[root@system ~]# service cups stop
Stopping cups:                                             [  OK  ]

Verify that network port is not listened to anymore.
You should also verify the TCP port in this case, but we omit that
[root@system ~]# netstat -ulnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address  Foreign Address  PID/Program name
udp        0      0 0.0.0.0:32768  0.0.0.0:*        1916/rpc.statd
udp        0      0 0.0.0.0:820    0.0.0.0:*        1916/rpc.statd
udp        0      0 0.0.0.0:68     0.0.0.0:*        3363/dhclient
udp        0      0 0.0.0.0:111    0.0.0.0:*        1897/portmap

Show current IPv4 kernel firewall rules. All chains are
empty and default policies are at 'ACCEPT' which means that
all traffic is allowed in and out
[root@system ~]# iptables -vnL
ip_tables: (C) 2000-2002 Netfilter core team
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target   prot opt in   out   source     destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target   prot opt in   out   source     destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target   prot opt in   out   source     destination

[ Displaying the networking settings ]

Next we'll study a traffic capture process. We want to see what goes over the link when a browser is using HTTP-proxy and requesting for a page. We assume that the traffic is generated on some other host than we're using for analysis. Running ethereal on servers for capture is not a good idea since it will need root-privileges and since the program is really large and contains holes (most of the time), we decide to do the analysis with normal user rights on a separate graphical workstation.

So, we start by doing the capture first:

Start by setting the HTTP proxy information (for our environment)
[root@system ~]# export http_proxy=http://192.168.1.254:8080
Start capturing traffic on interface eth0 (-i eth0),
without DNS lookups (-n), with full packet length (-s 0) and
storing the packets into a file (instead of screen, -w /tmp/capture.dump)
Lastly we specify the BPF-rules that will be used to limit the
packets that are captured to our HTTP-proxy only.
We start tcpdump on the background with &
[root@system ~]# tcpdump -i eth0 -n -s0 -w /tmp/capture.dump host 192.168.1.254 &
[1] 1050
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

Retrieve starting web page from kernel.org
[root@system ~]# wget http://www.kernel.org/
--18:23:33--  http://www.kernel.org/
           => `index.html'
Connecting to 192.168.1.254:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: unspecified [text/html]

    [  <<=>                                ] 23,098        49.02K/s

18:23:35 (48.99 KB/s) - `index.html' saved [23098]

Move the tcpdump from background to the foreground so that we
can terminate it with Ctrl+c
[root@system ~]# fg
tcpdump -i eth0 -n -s0 -w /tmp/capture.dump host 192.168.1.254
Stop capturing
<Ctrl+C>
44 packets captured
44 packets received by filter
0 packets dropped by kernel

Verify that we have something in the capture file
[root@system ~]# ls -la /tmp/capture.dump
-rw-r--r--  1 root root 27110 Mar 12 18:23 /tmp/capture.dump
[root@system ~]# file /tmp/capture.dump
/tmp/capture.dump: tcpdump capture file (little-endian) - version 2.4
  (Ethernet, capture length 65535)
Compress file so that we don't have to transfer so much
data. In this case this is pretty useless as the file is so small
[root@system ~]# gzip -9 /tmp/capture.dump
[root@system ~]# file /tmp/capture.dump.gz
/tmp/capture.dump.gz: gzip compressed data, was "capture.dump", from Unix, max compression
[root@system ~]# ls -la /tmp/capture.dump.gz
-rw-r--r--  1 root root 8917 Mar 12 18:23 /tmp/capture.dump.gz

[ Capturing and generating traffic ]

We then transfer the capture file (for example with scp) and then start the analysis with ethereal /foo/capture.dump.gz. This should be done without root-privileges. Note that capturing and analyzing network traffic might violate privacy laws . Always ask for permission for the relevant authorities (networking administrators normally).

Main window in ethereal
[ Main window in ethereal ]

We then click on the first packet of HTTP-transaction, press right mouse button and select Follow TCP stream which will reconstruct the client/server discussion. Client data is shown in pink and server in light blue.

Client/server discussion. Proxy name anonymized to protect the guilty.
[ Client/server discussion. Proxy name anonymized to protect the guilty. ]

This feature of ethereal is useful for any network-related problem where the protocol in use is in plaintext. Obviously using ethereal for traffic which has been sent over SSH or SSL is not very useful. Ethereal is also available for windows in precompiled form so that you can ask other people to capture traffic on proprietary systems.

Filesystem Hierarchy

Most Linux distributions implement some version of Filesystem Hierarchy Standard which specifies which directory is meant for which use. The FHS is constantly evolving, but mainly nowadays some new directories are introduced. You can always get a "short" version of the FHS by reading the man-page for "hier" (with command man hier). All FHS versions are available on the internet as are the mailing lists on which discussion takes place about FHS evolution (FHS uses a public process). The current version of FHS can be found at http://www.pathname.com/fhs/. The version that your Linux distribution uses is specified at the end of the hier-manual page. RHEL4 uses FHS 2.2.

One notable difference from HP-UX is the /etc/-directory. In Linux systems it must not contain any binary files. It also should not contain any scripts other than related to service control and system runlevel switching. All system wide configuration files should be stored under /etc/ unless the configuration files are for a 3rd party provided software (ie, from someone else than the Linux distributor). RHEL deviates from this a bit since its /etc/ is quite large (over 60 MiBs).

High-level Management Tools

Some Linux distributors include an integrated graphical tool through which you can do almost all basic management tasks. SUSE/Novell for example includes a program called YaST for this purpose. In Red Hat, there is no one tool that combines everything, instead Red Hat ships with various tools that each concentrate on one thing. In RHEL3 and older releases, the tools can be quickly found by typing redhat-config- and pressing the Tab-key couple of times (asking bash to autocomplete the commands). In RHEL4 and later, these tools have been renamed to system-config-* and some new tools have appeared (for example for LVM management).

Most of the tools require to be run under X, since they're graphical. But some of them (most low-level) will also run a text-mode menu based tools instead if you start them without X.

For service and runlevel management there are many possibilities (these are Red Hat-specific):

Package Management Basics

In all modern Linux distributions software is managed using a package management system. Red Hat developed their own, known as Red Hat Package Manager (RPM), and since this software was open source, other commercial distributions also started to use the same program and process. Debian (which is non-commercial) opted to implement their own system (apt/dpkg), which still obeys the same packaging semantics, only the commands are a bit different (use dpkg, apt-cache, apt-get instead of rpm). Most non-commercial distributions are based on Debian, so you might encounter dpkg as well (for example in Ubuntu and in Knoppix).

A package can be thought to be an compressed archive file (very similar to a .tar.gz-file) but it will also contain installation and removal scripts as well as dependency information about the package. Dependency information is used to tell the system that in order to get some software inside one package to work, one needs to have other packages installed first. Most commonly this is used to make sure that certain shared libraries (and proper versions) are installed before allowing the installation of the software application that uses the libraries. Sometimes the dependency information is also used to tell about conflicts. Conflicts are used when there are two different software packages that use the same command names or implement the same exact network service (use the same ports).

Lastly, there are virtual packages, which don't contain any files themselves, but instead only contain dependencies. Installing a virtual package will pull all the dependencies in and cause a certain larger subsystem to be installed. This is commonly used in the non-commercial distributions. For example installing package kde-desktop would actually install over 30 other packages which contain the common software for the KDE desktop environment.

This structure is a bit different from the concept employed in HP-UX nowadays where you have bundles, products, sub-products and filesets. There are no such things under any Linux distribution. A distribution might group the software packages using some other mechanism, but this mechanism is not part of the packaging system per-se.

Package Manager Database

In order to track installed packages and provide information about installed files, all package management systems use some kind of an database (Berkeley DB-based normally, using some SQL-based system would be an overkill). When you remove an installed package, the package manager will consult this database for the installed files and dependencies and will build knowledge about operations which must be done and also will decide whether dependency rules will still hold.

In RPM-based systems the database also contains install time information about files, such as installed size, install time last-modified timestamp, ownerships and MD5-sums of files. This information can be later be used in order to verify whether the files have been changed after installation. The RPM database contents are critical for proper operation of your system, so you should always backup the database. The database consists of multiple files which are stored in /var/lib/rpm/.

The RPM-database more or less implements the same functions as the IPD (Installed Packages Database) in HP-UX.

RPM commands

The rpm command is used for all packaging needs, including own RPM-package building. For this reason the rpm man-page might seem a bit overwhelming.

RPM has been designed automation in mind and for this reason it will never ask any questions interactively (this is different from dpkg). Newly installed services are off by default (95% of time) so you'll need to enable them specifically aftewards using chkconfig or other program.

Here are some example commands that you might find useful instead of the corresponding sw*-commands on HP-UX.

Common RPM-usages

Most software packages have been built for a specific target architecture. The target architecture is encoded in the .rpm-filename.

Common architecture specifiers:

For a bit more paranoid environments RPM provides digital signatures and verification of distributed packages. For example all RHEL RPM-packages are digitally signed by Red Hat's package distribution key and you can verify that the package is from Red Hat originally.

The public key that can be used to verify signatures is available on first installation CD-ROM. In the next example we trust the key on the NFS-server to be from Red Hat (obviously the first CD-ROM would be much better source for the key) and we first import the key to root's rpm tool and then use that key to verify that the package is signed by an imported key.

First find all the GPG keys present on our system
[root@system /usr/share]# ls -la `find . -name "*GPG*"`
-rw-r--r--  1 root root 1910 Sep 19 20:34 ./doc/redhat-release-4ES/RPM-GPG-KEY
-rw-r--r--  1 root root 1489 Nov 11  2004 ./doc/rpm-4.3.3/BETA-GPG-KEY
-rw-r--r--  1 root root 1913 Nov 11  2004 ./doc/rpm-4.3.3/RPM-GPG-KEY
-rw-r--r--  1 root root 1489 Mar 20  2002 ./rhn/BETA-RPM-GPG-KEY
-rw-r--r--  1 root root 1913 Aug 30  2002 ./rhn/RPM-GPG-KEY
-rw-r--r--  1 root root 1519 Oct 29  2003 ./rhn/RPM-GPG-KEY-fedora
-rw-r--r--  1 root root 1076 Oct 29  2003 ./rhn/RPM-GPG-KEY-fedora-test
As you can see there are many keys that Red Hat uses for
package signing. However for us only one key is the correct one.

Mount RHEL4u2 installation directory over on /mnt/
[root@system share]# mount nfsserver:/data/linux /mnt/

The installation directory (as well as CD#1) contains the
key used for RPM verification. Compare that key to another one
that is present on our system
[root@system share]# ls -la /mnt/RPM-GPG-KEY
-rw-r--r--  1 1000 1000 1910 Oct 31 15:03 /mnt/RPM-GPG-KEY
[root@system share]# diff /mnt/RPM-GPG-KEY ./doc/redhat-release-4ES/RPM-GPG-KEY
diff doesn't print out anything, which means that the files contain
the same data. You can also use program called cmp which is meant
for binary files (since diff is meant for text files, it might not
be correct program to use in all cases. GPG keys are ASCII files though)

Store the key for later use so that rpm can be used to verify
signatures
[root@system share]# rpm --import /mnt/RPM-GPG-KEY

Verify a package file (not installed yet)
[root@system share]# rpm -K /mnt/RedHat/RPMS/gimp-2.0.5-5.i386.rpm
/mnt/RedHat/RPMS/gimp-2.0.5-5.i386.rpm: (sha1) dsa sha1 md5 gpg OK
Verification is ok. rpm also communicates this status via exit codes

Exit code for successful verification is zero (OK)
[root@system share]# echo $?
0

Next we'll create a package file for which the verification will fail
[root@system share]# cp /mnt/RedHat/RPMS/gimp-2.0.5-5.i386.rpm /tmp/foo.rpm
NFS share is not needed beyond this point
[root@system share]# umount /mnt/
[root@system share]# ls -la /tmp/foo.rpm
-rw-r--r--  1 root root 10119963 Mar 12 19:47 /tmp/foo.rpm
Add 4 KiBs of random data to the end of the file
[root@system share]# dd if=/dev/urandom bs=1024 count=4 >> /tmp/foo.rpm
4+0 records in
4+0 records out
[root@system share]# ls -la /tmp/foo.rpm
-rw-r--r--  1 root root 10124059 Mar 12 19:47 /tmp/foo.rpm

Verify. Should fail since package contains random garbage at the end
[root@system share]# rpm -K /tmp/foo.rpm
/tmp/foo.rpm: (sha1) dsa sha1 MD5 GPG NOT OK
This is communicated with a failure exit code as well
[root@system share]# echo $?
1

[ Verifying package file signatures ]

For other examples, please read on about updating software.

Software Updates

Contrary to HP-UX which still uses "software patches", most Linux distributions (Red Hat included) have moved into a system where the whole software package is redistributed when it has security problems fixed. This means that Linux systems do not use patches, but also means that the software update process is less painful from the standpoint of the package management tool.

In order to update an installed software package, use rpm -Uvh filename.rpm . Please note that if there is no older version of the package installed, rpm -U will install the package anyway. So in effect, -U should be understood to mean "update or install if not installed".

In some cases it is beneficial to have a directory (over NFS for example) that contains all updated versions of RPM files available at that time so that the system can be updated with all the newest packages. In these cases it is not possible to use rpm -U since that would also install packages that weren't installed before. This is what rpm -F is for. It is used in the same way as -U, but will only update a package if there is an older version already installed.

With such directory one could use a command like rpm -Fvh *.rpm . Please try to arrange a test system which contains the same software package selection that your production system does, as some times the "update from directory" technique will run into dependency problems (distributor has released updated packages but because of too strict dependencies they cannot be applied). All updates will fail in this case (nothing will be updated).

Kernel Updates

Normally installing multiple versions of the same package is not possible since the packages contain files that conflict (files have the same names and go into the same directories). rpm will complain about this very loudly. The only software package where installing multiple versions makes any sense is the kernel package. In fact, it is beneficial to leave the older kernel versions installed so that we can restart the system with known good kernel. In this case you should use rpm -ivh kernel-blurb.rpm, not -U or -F (since both of these will remove the older kernel).

The kernel package installation will also integrate the new kernel into the boot manager menus, but you should always check /boot/grub/menu.lst for the default-keyword whether it refers to the correct new kernel. In 99% of cases this will be updated automatically to the newest kernel.

When you're sure that the new kernel works properly (99.99% of cases), you may opt to remove the older kernels if you wish to free some space under /boot/ and /lib/modules/. Normally this is not required since the kernels don't take a lot of space.

Last noteable thing about kernel updates is that kernels in Linux distributions normally come in two flavours: Uniprocessor and SMP (Symmetric Multi Processing). The uniprocessor kernel is installed along side the SMP kernel on SMP-systems (when you have more than one logical CPU, like in Hyper-Threading/SMT-case). This is to aid debugging very hard hardware related problems since Uniprocessor kernel does not contain locking code (it doesn't need to) and hence is more reliable on unstable systems (and buggy drivers, but commercial distributors don't distribute buggy drivers normally).

Reboots

Normally only switching to a new version of kernel will require a reboot. 3rd party software might require reboots as well, for example hardware managing agent updates from the hardware manufacturer, but these are not provided by the Linux distributor.

Epilog

All material and code copyright 2006 Aleksandr Koltsoff. Use for personal education permitted, all other use must be negotiated (czr(at)iki(dot)fi). Constructive critisism and feedback welcome via email.