This document attempts to collect together the most useful management tools available on Linux distributions, with strong emphasis on Red Hat Enterprise Linux (version 4 mainly). It also contains some notes about what are the largest differences between Linux-based systems to HP-UX systems since that was the original target audience for this document, but the document has since grown out of the original scope.
Target audience for this document is system administrators who have some command line shell experience in UNIX-like systems (not including Windows) and want to learn the Linux-specific ways of doing things.
For pedantic people Linux is the kernel that is used to build UNIX-like systems. For rest of us Linux means some set of software packages on top of this kernel, tested and integrated to form a system that can be administered using same techniques as any UNIX-like system.
There are two main branches of Linux-based systems (distributions): The commercial distributions are meant for production use in organizations and provide long product life and a reduced set of supported software. The free/non-commercial distributions are more targeted towards the home-users or speciality environments (scientific clusters) and there ease of use and customization and modularity are the key attributes. Normally non-commercial distributions are not tested against commercial brand-servers and as such might not contain all the low-level device drivers that are present in commercial distributions. On the other hand, non-free distributions contain a lot of end-user drivers that are useful for home-use which in turn are not present in commercial distributions (NTFS driver for example).
Most notable commercial distributions at the moment are:
- Red Hat Enterprise Linux: longest history, very popular within companies in Finland.
- SUSE: nowadays owned by Novell and supported as server platform for NetWare.
- Mandriva: popular in France and Brazil, least commercial of the commercial three.
Most notable non-commercial distributions:
- Ubuntu: Debian-based well integrated and modern distribution (sister project which uses KDE instead of GNOME is called Kubuntu).
- Debian: Politically mostly correct distribution. Widest architecture support.
- Fedora: Test-bed for RHEL releases, as such, will contain a lot of "beta"-software.
- Gentoo: Non-binary distribution which you can customize to your heart's content. Be prepared for long update times as all software is built and compiled on your system.
- Slackware: First Linux distribution, for more experienced users.
Most notable Live-CD distributions (run directly off the CD-ROM, no need to install):
- Knoppix: Debian and KDE-based with very good hardware detection.
- Kanotix: Knoppix based with faster development cycle.
- MEPIS: Debian and KDE-based, with emphasis on older systems (maybe).
You can find a nice portal page that collects all the distributions at http://distrowatch.com/.
What is the best distribution to select? This is as difficult question as "what is the best car?". Obviously no one distribution is the best for everyone. For production using a commercial distribution is highly recommended as they normally are supported from 5 to 7 years after each release. This means that software will be only updated for security fixes and no new features will be introduced to the system. This will require the least amount of work after the system has been installed. Commercial distribution vendors also test their distributions on production servers from brand hardware manufacturers.
For home use where software capability set stability is not so critical, any of the freely available distributions are good. Most distributions feature an easy installer system so that shouldn't pose a problem. Pick one that you can find support for easily (friends, forums and such). Commercial support is also available for some of the free distributions (Ubuntu for example).
In some cases the list of architectures for a distribution will affect your choice. The Linux kernel is very versatile and has been ported to over 15 different architectures (PC being only one such target), but you will need other software as well. Gentoo seems to be a good choice nowadays for more interesting architectures (HP-PA, Alphas, Super-H).
In Linux all device files are located under /dev/, so nothing new here compared to other UNIX-like systems. The device major and minor codes are different (as they are between all UNIXes). The modern Linux distributions are using a daemon called udevd that will track kernel device related events and will create device nodes when the devices are attached to system. It will also remove device nodes when the hardware devices leave the system. This is very useful with dynamic bus systems like USB and Firewire, but the same system is also used with hotswap PCI and SCSI. Older Linux distributions (like RHEL3) still had a /dev/ filled with all possible device files, even if you didn't have the hardware for them. Some other older distributions used a kernel facility called devfs which was a dynamic filesystem that was mounted on top of /dev/ and "created" devices which were really present. Linus never really liked this system so once the hotplug-event interface was ready the device naming decisions could me moved into user-space and udevd was born. devfs is no longer part of the 2.6 series kernel.
Notable device files in Linux:
- /dev/hda..hdz: IDE block devices (ATA disks, ATAPI CD-ROMs).
- /dev/hda1..16: Partitions within IDE channel 0 master block device. Note that Linux kernel knows about many different partitioning schemes, (called slices in some other systems), but normally on PC the old DOS-partitioning scheme is used. On Itanium based systems partitioning is handled with a scheme called GPT. Partitions are supported on any block device, but here the /dev/hda serves as an example.
- /dev/sda..sdNNN: SCSI block device, SATA disks, USB mass storage profile devices. Note that it is not possible to tell by the device file name whether the device is a real SCSI disk or an USB flash. This may become a problem if you implement filesystems directly on partitions on such systems. If you use LVM or filesystem labels, this is not really a problem. Also, since device node creation is handled by udevd, you can modify the rules that it uses when it will decide the names for the devices (configuration under /dev/udev/).
- /dev/random: Slow entropy gathering PRNG. In some systems this might even be a real random number generator.
- /dev/urandom: Fast PRNG, might not contain proper amount of true entropy to be suitable for cryptography.
- /dev/zero: When read, returns infinite amount of binary zero, when written, writes will fail.
- /dev/null: When written, nothing happens. When read, EOF will be returned immediately.
- /dev/ttyS0..N: Serial "COM" ports (PC and non-PC)
- /dev/lp0..N: Parallel ports (PCs only)
- /dev/shm/: Directory that has tmpfs mounted on it. This is used to store POSIX shared memory segments, but also acts as a VM-based temporary directory. tmpfs uses physical RAM at first but the may be swapped to disk as needed.
- /dev/mem: Represents translated physical bus addresses. Normally used by low-level software that needs to access some PCI device directly. Examples are X servers and hardware management agents for motherboard/system integrated sensors.
As a first example of device file use in Linux we'll setup the first serial RS-232C port to accept terminal login sessions. This way you can attach a real hardware terminal, or a laptop running terminal software that can be used to login into the system.
You should add these lines into /etc/inittab which is the configuration file for init:
# First column must be unique id for each line. We use # Sx to signify "serial-lines", although you might want to # use something else. Doesn't really matter as long as id # field is unique for each action that init should perform. # We also enable these two programs for run levels 1-5. # This means that these will also be available on single # user mode. Respawning means that init will restart the # program if the program will terminate for some reason. # Start getty on first "COM" port and getty will initialize # the serial line to 8N1 9600 (man getty for other options) # Also tell getty that it should set TERM to 'vt100' for # programs spawned on that terminal line. S0:12345:respawn:/sbin/getty -L ttyS0 9600 vt100 # Call-in via modem is supported via mgetty-program # Here we tell mgetty to init the modem via AT&T Hayes # command X0 and init the speed to 57600 baud on second # serial port S1:12345:respawn:/sbin/mgetty -x0 -s 57600 ttyS1
[ Enabling login on serial lines ]
As a second example we'll make an "image" file out of storage media and then use the image file instead of the real physical media. Similar approach can be used with any filesystem, but we'll use the ISO 9660 standard here (iso9660 is the filesystem driver name in Linux kernel).
[root@system ~]# cp /dev/cdrom /tmp/cdrom.iso Connect /dev/loop0 to file /tmp/cdrom.iso [root@system ~]# losetup /dev/loop0 /tmp/cdrom.iso Mount the filesystem on /dev/loop0 onto /media/cdrom [root@system ~]# mount -t iso9660 /dev/loop0 /media/cdrom Verify that the mount succeeded by listing active mount points [root@system ~]# df -T Filesystem Type 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 ext3 3644604 1384840 2074628 41% / /dev/sda1 ext3 101086 9213 86654 10% /boot none tmpfs 81356 0 81356 0% /dev/shm /dev/loop0 iso9660 4930 4930 0 100% /media/cdrom [root@system ~]# ls -l /media/cdrom total 3 drwxr-xr-x 2 root root 2048 Oct 1 19:49 isolinux -r--r--r-- 1 root root 220 Oct 1 19:48 TRANS.TBL [root@system ~]# umount /media/cdrom Verify that unmounting succeeded [root@system ~]# df -T Filesystem Type 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 ext3 3644604 1384844 2074624 41% / /dev/sda1 ext3 101086 9213 86654 10% /boot none tmpfs 81356 0 81356 0% /dev/shm Check whether there is anything connected to /dev/loop0 [root@system ~]# losetup /dev/loop0 /dev/loop0: [fd00]:255579 (/tmp/cdrom.iso) There is, remove the association [root@system ~]# losetup -d /dev/loop0 The mount command can also do the dirty work for you [root@system ~]# mount -oloop /tmp/cdrom.iso /media/cdrom [root@system ~]# df -T Filesystem Type 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 ext3 3644604 1384844 2074624 41% / /dev/sda1 ext3 101086 9213 86654 10% /boot none tmpfs 81356 0 81356 0% /dev/shm /tmp/cdrom.iso iso9660 4930 4930 0 100% /media/cdrom [root@system ~]# losetup /dev/loop0 /dev/loop0: [fd00]:255579 (/tmp/cdrom.iso) If mount allocated the loop device, it will also de-associate it automatically [root@system ~]# umount /media/cdrom
[ Using loop-devices in Linux ]
Here we first copy the contents of a CD-ROM disc into a file using cp. We then bind /dev/loop0 (first loop device) to be connected to /tmp/cdrom.iso. We can then use the virtual loop device as we would with any block device. We proceed to mount it and play a bit with it. We then unmount and break the association between /dev/loop0 and the underlying file. The easier and more common way is to use -o loop with mount. mount will automatically select the first unused loop device and allocate that for us. So in effect, it is doing all the dirty work. There is also another virtual loop device which will move all reads and writes to the device over the network. This device driver is /dev/nbd0 (network block device). It requires a daemon on some server that will handle the actual storage, but unfortunately nbd is not supported in RHEL-systems.
By the way, if you check the manual page for losetup, you will notice that it can also do encryption and decryption for you (the actual cryptography is done inside the kernel, losetup will only set it up for you).
Linux distributions come with a variety of useful tools that can give you the necessary information (and also a lot of unnecessary).
Here is a short list of the most useful utilities:
- df: Show current mounts and filesystem usage information. Use this instead of bdf (that you might use on HP-UX).
- lspci: List PCI bus information using a text-database to show manufacturer codes and device names. Use -v for more information and -n to get raw PCI device vendor and device IDs (useful when googling for Linux driver support for example in http://pci-ids.ucw.cz/iii/.
- lsusb: Similar tool for USB devices and busses.
- dmesg: Show kernel log. All kernel drivers and kernel subsystems will store their messages in the kernel log, and normally the system logging daemon (either klogd or syslogd, depending on distribution) will take the messages and store them to disk files (by default).
- hdparm: Useful tool for IDE-based devices. Be careful.
- smartctl: Tool to communicate with embedded computers in hard disks using the S.M.A.R.T. protocol. Useful to get predicted failure information from supported disks. All disks support the S.M.A.R.T. command set, but the bus driver might not
pass the commands (SCSI HBAs (host bus adapter) and some SATA drivers).- vmstat, iostat (normally needs to be installed separately), top, ps, as well as the other usual suspects. Note that in Linux 99% of all data displayed by these tools is available from virtual files under /proc/ so it is possible to write your own scripts and tools that collect only the data that you need. Also there is a lot of data there that is not currently displayed by any tool (other than cat).
There is some documentation on the /proc/-files under kernel source (./Documentation/filesystems/proc.txt).- free: Display current system memory utilization. Use the
-/+ line as it is the most correct one. Also shows swap utilization.- /proc/swaps: Display current swap devices. You may activate and deactivate swap online with swapon and swapoff commands.
- /proc/mounts: Display current mounts. All mounts are shown, not just "real filesystems" that keep data on disks or other stable storage.
- /proc/cpuinfo: Display information about all logical processors.
Note that the cpu MHz field displays the current clock of the CPU, not the maximum possible (on PC/x86-64 that is). This is because most modern Linux distributions include a CPU frequency and voltage
control daemons that will lower both when there is no system load or the load is insignificant.- /proc/modules: List all currently loaded dynamic kernel modules. (you can also use lsmod for this).
- /proc/partitions: Show all partitions that the kernel knows about currently. Also shows sizes in kilobytes. Use partprobe to tell the kernel if the partitions are changed online (using fdisk, sfdisk, cfdisk or parted.
partprobe is also necessary when you hotswap disks with different partitions on them or different sizes. This might fail on some systems because of limited support of the HBA driver, but it is not Linux kernel's fault.- /proc/version: Kernel version and build information. Also available with uname -a.
- /etc/redhat-release and /etc/debian-version: Distribution version.
- /proc/scsi/*: Informative files from the SCSI subsystem. Also scsi_info program is sometimes available.
- /sys/: Directory that is new with 2.6 kernels and modern distribution versions. Contains a dynamic filesystem similar to /proc but is meant for driver to daemon communication. Contains a lot of really useful information, but not used yet by tools.
As an example, let's display some SCSI-related information (including partitioning information).
[root@system ~]# ls -l /proc/scsi total 0 -r--r--r-- 1 root root 0 Mar 12 16:34 device_info dr-xr-xr-x 2 root root 0 Mar 12 16:34 mptscsih -r--r--r-- 1 root root 0 Mar 12 16:34 scsi Display all SCSI HBAs and their information [root@system ~]# cat /proc/scsi/scsi Attached devices: Host: scsi0 Channel: 00 Id: 00 Lun: 00 Vendor: VMware, Model: VMware Virtual S Rev: 1.0 Type: Direct-Access ANSI SCSI revision: 02 Display information from one SCSI HBA driver (mpt) [root@system ~]# cat /proc/scsi/mptscsih/0 ioc0: LSI53C1030, FwRev=00000000h, Ports=1, MaxQ=128 Grep for SCSI-related kernel messages to find out the assigned device names [root@system ~]# dmesg | grep scsi scsi0 : ioc0: LSI53C1030, FwRev=00000000h, Ports=1, MaxQ=128, IRQ=9 Attached scsi disk sda at scsi0, channel 0, id 0, lun 0 Display PC-DOS based partition table from first SCSI disk [root@system ~]# fdisk -l /dev/sda Disk /dev/sda: 4294 MB, 4294967296 bytes 255 heads, 63 sectors/track, 522 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 * 1 13 104391 83 Linux /dev/sda2 14 522 4088542+ 8e Linux LVM Display kernel's current information about partitions and their sizes [root@system ~]# cat /proc/partitions major minor #blocks name 8 0 4194304 sda 8 1 104391 sda1 8 2 4088542 sda2 253 0 3702784 dm-0 253 1 327680 dm-1
[ Display SCSI and partitioning information ]
And more or less same information from a real system (different architecture this time, using fdisk doesn't work since the partitioning system is different).
[root@system ~]# cat /proc/cpuinfo processor : 0 vendor : GenuineIntel arch : IA-64 family : Itanium 2 model : 1 revision : 5 archrev : 0 features : branchlong cpu number : 0 cpu regs : 4 cpu MHz : 1000.000000 itc MHz : 1000.000000 BogoMIPS : 1497.36 [root@system ~]# cat /proc/scsi/scsi Attached devices: Host: scsi0 Channel: 00 Id: 00 Lun: 00 Vendor: HP 36.4G Model: MAP3367NC Rev: HPC6 Type: Direct-Access ANSI SCSI revision: 03 Host: scsi0 Channel: 00 Id: 01 Lun: 00 Vendor: HP 36.4G Model: MAP3367NC Rev: HPC6 Type: Direct-Access ANSI SCSI revision: 03 [root@system ~]# cat /proc/scsi/mptscsih/0 ioc0: LSI53C1030, FwRev=01030600h, Ports=1, MaxQ=203 [root@system ~]# cat /proc/scsi/mptscsih/1 ioc1: LSI53C1030, FwRev=01030600h, Ports=1, MaxQ=203 [root@system ~]# dmesg | grep scsi scsi0 : ioc0: LSI53C1030, FwRev=01030600h, Ports=1, MaxQ=203, IRQ=54 Attached scsi disk sda at scsi0, channel 0, id 0, lun 0 Attached scsi disk sdb at scsi0, channel 0, id 1, lun 0 scsi1 : ioc1: LSI53C1030, FwRev=01030600h, Ports=1, MaxQ=203, IRQ=55 Even when Itanium-based systems use a different partitioning scheme (GPT), Linux still uses the same proc-file to display the current partitioning information. Use GNU parted to edit GPT-based partition tables [root@paris ~]# cat /proc/partitions major minor #blocks name 8 0 35566480 sda 8 1 51200 sda1 8 2 10240000 sda2 8 16 35566480 sdb 8 17 512000 sdb1 8 18 34643968 sdb2 8 19 409600 sdb3 Display SCSI disk identification information for all SCSI disks [root@system ~]# cat /sys/block/sd*/device/{vendor,model,rev} HP 36.4G HP 36.4G MAP3367NC MAP3367NC HPC6 HPC6 Display PCI/AGP/PCI-X bus information [root@system ~]# lspci 00:01.0 USB Controller: NEC Corporation USB (rev 41) 00:01.1 USB Controller: NEC Corporation USB (rev 41) 00:01.2 USB Controller: NEC Corporation USB 2.0 (rev 02) 00:02.0 IDE interface: Silicon Image, Inc. SiI 0649 Ultra ATA/100 PCI to ATA Host Controller (rev 02) 00:03.0 Ethernet controller: Intel Corporation 82557/8/9 [Ethernet Pro 100] (rev 0d) 20:01.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07) 20:01.1 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07) 20:02.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5701 Gigabit Ethernet (rev 15) e0:01.0 Communication controller: Hewlett-Packard Company Auxiliary Diva Serial Port (rev 01) e0:01.1 Serial controller: Hewlett-Packard Company Diva Serial [GSP] Multiport UART (rev 03) e0:02.0 VGA compatible controller: ATI Technologies Inc Radeon RV100 QY [Radeon 7000/VE] Same as above but without translating the PCI vendor and device IDs. These are useful to use with googling for device driver support in Linux [root@system ~]# lspci -n 00:01.0 Class 0c03: 1033:0035 (rev 41) 00:01.1 Class 0c03: 1033:0035 (rev 41) 00:01.2 Class 0c03: 1033:00e0 (rev 02) 00:02.0 Class 0101: 1095:0649 (rev 02) 00:03.0 Class 0200: 8086:1229 (rev 0d) 20:01.0 Class 0100: 1000:0030 (rev 07) 20:01.1 Class 0100: 1000:0030 (rev 07) 20:02.0 Class 0200: 14e4:1645 (rev 15) e0:01.0 Class 0780: 103c:1290 (rev 01) e0:01.1 Class 0700: 103c:1048 (rev 03) e0:02.0 Class 0300: 1002:5159 List USB-bus connected devices. [root@system ~]# lsusb Bus 003 Device 002: ID 413c:2005 Dell Computer Corp. Bus 003 Device 001: ID 0000:0000 Bus 002 Device 001: ID 0000:0000 Bus 001 Device 001: ID 0000:0000 Get more information about the funny Dell device [root@system ~]# lsusb -v -s 003:002 Bus 003 Device 002: ID 413c:2005 Dell Computer Corp. Device Descriptor: bLength 18 bDescriptorType 1 Device USB version. This device is USB 1.1 bcdUSB 1.10 bDeviceClass 0 (Defined at Interface level) bDeviceSubClass 0 bDeviceProtocol 0 bMaxPacketSize0 8 idVendor 0x413c Dell Computer Corp. idProduct 0x2005 bcdDevice 1.04 iManufacturer 1 DELL This is the useful bit of information iProduct 2 DELL USB Keyboard iSerial 0 bNumConfigurations 1 Configuration Descriptor: bLength 9 bDescriptorType 2 wTotalLength 34 bNumInterfaces 1 bConfigurationValue 1 iConfiguration 0 bmAttributes 0xa0 Remote Wakeup MaxPower 100mA Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 0 bAlternateSetting 0 bNumEndpoints 1 bInterfaceClass 3 Human Interface Devices bInterfaceSubClass 1 Boot Interface Subclass bInterfaceProtocol 1 Keyboard iInterface 0 HID Device Descriptor: bLength 9 bDescriptorType 33 bcdHID 1.10 bCountryCode 0 Not supported bNumDescriptors 1 bDescriptorType 34 Report wDescriptorLength 65 cannot get report descriptor Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x81 EP 1 IN bmAttributes 3 Transfer Type Interrupt Synch Type none Usage Type Data wMaxPacketSize 0x0008 bytes 8 once bInterval 10 Language IDs: (length=4) 0409 English(US)
[ SCSI information from another architecture ]
Linux kernel (starting from 2.4 series) contains a lot of run-time changeable parameters that can affect the operation of the various sub-systems inside the kernel. These parameters can be found under /proc/sys/ and you can use echo to change the values. The values are changes in real-time and not stored into any configuration file. In order to change something permanently you should check out the sysctl-tool (whose configuration file is /etc/sysctl.conf).
Linux kernel development has borrowed and reimplemented a lot of networking features from other UNIX-like systems. Some basic tools are the same as on any BSD based UNIX, however there are also some low level commands that are unique to Linux.
Non-complete list of networking related tools in Linux:
- ifconfig: tool to control networking interfaces.
- ifconfig -a: display information about all current network interfaces, even ones that haven't been configured.
- ifconfig eth0: display information only about eth0, which is the name given to the first Ethernet-like device (also WLAN interfaces nowadays).
- ifconfig eth0 down: disable eth0. No traffic is routed trough that interface any more.
- ifconfig eth0 192.168.0.1 netmask 255.255.255.0: Set IPv4 address 192.168.0.1/24 to interface eth0 and enable it.
- ifconfig eth0:1 192.168.1.1 netmask 255.255.255.0: Add an IPv4 alias address to eth0.
- ifconfig eth0:1 down: remove IPv4 alias.
- ifconfig eth0 mtu 9000: change the maximum transmission units for eth0 (ie, enable jumbo frames). Not all NIC interface chips support MTU larger than 1522).
- service network stop: deactive all networking in RHEL (start will activate). Preferred method to control networking since takes into account DHCP clients and other higher-level operations.
- ifdown eth0: disable eth0 using service scripts (recommended over using ifconfig). There is also ifup which is used by the network setup script.
- route -n: print routing table. Entry for network 0.0.0.0 is the default gateway.
- /etc/resolv.conf: configuration file for the DNS "client" which is called the DNS resolver on UNIX-style systems. Note that this file will be re-created if your system is running the DHCP client (has the network card set to acquire it's address via DHCP).
- /etc/hosts: local IP-address to/from hostname translation file.
Normally (depending on /etc/nsswitch.conf) this file is consulted first whenever a hostname is used in a program. Useful when you're operating in a network with a broken or unstable DNS-environment.- vconfig: tool to control IEEE 802.1Q (Virtual LANs) on Linux. VLAN-support requires the loading of 8021q software driver into the kernel (with modprobe 8021q for example). No NIC driver support is required in Linux for VLAN-support (for 99% of NIC drivers and interfaces).
- arp: ARP-table manipulation program.
- netstat: network statistics.
- ethtool: low-level NIC manipulation program. Very powerful. Can be used to setup your NIC to start your system via Wake-On-LAN-packets (WoL). These packets can be generated with a program called etherwake, but there are other non-Linux programs for this as well.
- mii-tool: similar to ethtool, but for older NICs.
- iptables: program to show and setup kernel internal firewalling logic rules. Requires networking knowledge in order to use (low-level tool). On RHEL there is a high-level utility to generate iptables-rulesets called lokkit. Firewall rules for IPv6-based traffic are controlled via ip6tables. Normally this utility is not integrated in distributions.
- tc: traffic control/QoS setup program for Linux. Very powerful, not used or integrated in distributions, but very useful if you're building your own routers using Linux distributions. Using RHEL for this might not be the best idea, but there are special purpose (small) distributions for this.
- ip: next generation routing and interface control program. Very powerful. Not used or integrated by distributions. Possible to use to create policy based routing rules (similar to Cisco's route maps).
- tcpdump: program to capture and decode packets on network interfaces (available on most UNIXes).
- ethereal: graphical protocol decoder with powerful decoding capabilities.
- wget and curl: programs that can retrieve files over HTTP and FTP.
- links or elinks: text-mode browsers.
As an example we display the base IPv4 configuration on our system, notice that we have a service enabled which we don't really need (CUPS, aka Common UNIX Printing System), disable the network printing service and then verify that the port is not listened anymore. At the end we also check the current firewall rules setting, and they are empty on this system ("firewall disabled").
Start by displaying all configured network interfaces [root@system ~]# ifconfig MAC of the NIC is listed on the first line. Useful for DHCP-address reservations eth0 Link encap:Ethernet HWaddr 00:0C:29:92:A2:09 IPv4 address and netmask inet addr:192.168.1.19 Bcast:192.168.1.255 Mask:255.255.255.0 List of all IPv6 addresses. At least one (automatically generated) inet6 addr: fe80::20c:29ff:fe92:a209/64 Scope:Link UP = interface configured and "running" UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Statistics. They overflow at 64-bit RX packets:114 errors:0 dropped:0 overruns:0 frame:0 TX packets:107 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:13278 (12.9 KiB) TX bytes:15302 (14.9 KiB) Interrupt:11 Base address:0x1400 Loopback interface which will receive all traffic for network 127 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:14 errors:0 dropped:0 overruns:0 frame:0 TX packets:14 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 Display current routing table [root@system ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 This is the default gateway setting 0.0.0.0 192.168.1.254 0.0.0.0 UG 0 0 0 eth0 Display DNS-resolver configuration. It has been generated by DHCP client [root@system ~]# cat /etc/resolv.conf ; generated by /sbin/dhclient-script search intranet nameserver 192.168.1.254 Display all TCP ports that have a process listening for incoming connections. You need root privileges to display the process names. 0.0.0.0 = all IPv4 addresses in system 127.0.0.1 = local loopback only ::: = all IPv4 and IPv6 addresses [root@system ~]# netstat -tlnp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:32768 0.0.0.0:* LISTEN 1916/rpc.statd tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1897/portmap tcp 0 0 0.0.0.0:631 0.0.0.0:* LISTEN 2028/cupsd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2123/sendmail: acce tcp 0 0 :::22 :::* LISTEN 2091/sshd Do the same for UDP ports. At this point we notice that we have an unnecessary service running by default (cupsd) [root@system ~]# netstat -ulnp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address PID/Program name udp 0 0 0.0.0.0:32768 0.0.0.0:* 1916/rpc.statd udp 0 0 0.0.0.0:820 0.0.0.0:* 1916/rpc.statd udp 0 0 0.0.0.0:68 0.0.0.0:* 3363/dhclient udp 0 0 0.0.0.0:111 0.0.0.0:* 1897/portmap udp 0 0 0.0.0.0:631 0.0.0.0:* 2028/cupsd Check the service runlevel configuration [root@system ~]# chkconfig --list cups cups 0:off 1:off 2:on 3:on 4:on 5:on 6:off Disable and verify [root@system ~]# chkconfig cups off [root@system ~]# chkconfig --list cups cups 0:off 1:off 2:off 3:off 4:off 5:off 6:off Stop service [root@system ~]# service cups stop Stopping cups: [ OK ] Verify that network port is not listened to anymore. You should also verify the TCP port in this case, but we omit that [root@system ~]# netstat -ulnp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address PID/Program name udp 0 0 0.0.0.0:32768 0.0.0.0:* 1916/rpc.statd udp 0 0 0.0.0.0:820 0.0.0.0:* 1916/rpc.statd udp 0 0 0.0.0.0:68 0.0.0.0:* 3363/dhclient udp 0 0 0.0.0.0:111 0.0.0.0:* 1897/portmap Show current IPv4 kernel firewall rules. All chains are empty and default policies are at 'ACCEPT' which means that all traffic is allowed in and out [root@system ~]# iptables -vnL ip_tables: (C) 2000-2002 Netfilter core team Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination
[ Displaying the networking settings ]
Next we'll study a traffic capture process. We want to see what goes over the link when a browser is using HTTP-proxy and requesting for a page. We assume that the traffic is generated on some other host than we're using for analysis. Running ethereal on servers for capture is not a good idea since it will need root-privileges and since the program is really large and contains holes (most of the time), we decide to do the analysis with normal user rights on a separate graphical workstation.
So, we start by doing the capture first:
Start by setting the HTTP proxy information (for our environment) [root@system ~]# export http_proxy=http://192.168.1.254:8080 Start capturing traffic on interface eth0 (-i eth0), without DNS lookups (-n), with full packet length (-s 0) and storing the packets into a file (instead of screen, -w /tmp/capture.dump) Lastly we specify the BPF-rules that will be used to limit the packets that are captured to our HTTP-proxy only. We start tcpdump on the background with & [root@system ~]# tcpdump -i eth0 -n -s0 -w /tmp/capture.dump host 192.168.1.254 & [1] 1050 tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes Retrieve starting web page from kernel.org [root@system ~]# wget http://www.kernel.org/ --18:23:33-- http://www.kernel.org/ => `index.html' Connecting to 192.168.1.254:8080... connected. Proxy request sent, awaiting response... 200 OK Length: unspecified [text/html] [ <<=> ] 23,098 49.02K/s 18:23:35 (48.99 KB/s) - `index.html' saved [23098] Move the tcpdump from background to the foreground so that we can terminate it with Ctrl+c [root@system ~]# fg tcpdump -i eth0 -n -s0 -w /tmp/capture.dump host 192.168.1.254 Stop capturing <Ctrl+C> 44 packets captured 44 packets received by filter 0 packets dropped by kernel Verify that we have something in the capture file [root@system ~]# ls -la /tmp/capture.dump -rw-r--r-- 1 root root 27110 Mar 12 18:23 /tmp/capture.dump [root@system ~]# file /tmp/capture.dump /tmp/capture.dump: tcpdump capture file (little-endian) - version 2.4 (Ethernet, capture length 65535) Compress file so that we don't have to transfer so much data. In this case this is pretty useless as the file is so small [root@system ~]# gzip -9 /tmp/capture.dump [root@system ~]# file /tmp/capture.dump.gz /tmp/capture.dump.gz: gzip compressed data, was "capture.dump", from Unix, max compression [root@system ~]# ls -la /tmp/capture.dump.gz -rw-r--r-- 1 root root 8917 Mar 12 18:23 /tmp/capture.dump.gz
[ Capturing and generating traffic ]
We then transfer the capture file (for example with scp) and then start the analysis with ethereal /foo/capture.dump.gz. This should be done without root-privileges. Note that capturing and analyzing network traffic might violate privacy laws . Always ask for permission for the relevant authorities (networking administrators normally).
[ Main window in ethereal ]We then click on the first packet of HTTP-transaction, press right mouse button and select Follow TCP stream which will reconstruct the client/server discussion. Client data is shown in pink and server in light blue.
[ Client/server discussion. Proxy name anonymized to protect the guilty. ]This feature of ethereal is useful for any network-related problem where the protocol in use is in plaintext. Obviously using ethereal for traffic which has been sent over SSH or SSL is not very useful. Ethereal is also available for windows in precompiled form so that you can ask other people to capture traffic on proprietary systems.
Most Linux distributions implement some version of Filesystem Hierarchy Standard which specifies which directory is meant for which use. The FHS is constantly evolving, but mainly nowadays some new directories are introduced. You can always get a "short" version of the FHS by reading the man-page for "hier" (with command man hier). All FHS versions are available on the internet as are the mailing lists on which discussion takes place about FHS evolution (FHS uses a public process). The current version of FHS can be found at http://www.pathname.com/fhs/. The version that your Linux distribution uses is specified at the end of the hier-manual page. RHEL4 uses FHS 2.2.
One notable difference from HP-UX is the /etc/-directory. In Linux systems it must not contain any binary files. It also should not contain any scripts other than related to service control and system runlevel switching. All system wide configuration files should be stored under /etc/ unless the configuration files are for a 3rd party provided software (ie, from someone else than the Linux distributor). RHEL deviates from this a bit since its /etc/ is quite large (over 60 MiBs).
Some Linux distributors include an integrated graphical tool through which you can do almost all basic management tasks. SUSE/Novell for example includes a program called YaST for this purpose. In Red Hat, there is no one tool that combines everything, instead Red Hat ships with various tools that each concentrate on one thing. In RHEL3 and older releases, the tools can be quickly found by typing redhat-config- and pressing the Tab-key couple of times (asking bash to autocomplete the commands). In RHEL4 and later, these tools have been renamed to system-config-* and some new tools have appeared (for example for LVM management).
Most of the tools require to be run under X, since they're graphical. But some of them (most low-level) will also run a text-mode menu based tools instead if you start them without X.
For service and runlevel management there are many possibilities (these are Red Hat-specific):
- system-config-services: Graphical tools to start, stop, enable and disable services on runlevels 3 to 5.
- system-config-network-tui: Text-mode interface to setup basic networking parameters.
- lokkit: Text-mode interface to enable or disable "firewall" in RHEL.
- ntsysv: high-level menu-based tool to do the same.
- chkconfig: powerful command line tool to control services on all run-levels. Some examples:
- Turn off sshd from all runlevels: chkconfig sshd off .
- Enable sshd on runlevels 3 to 5: chkconfig --level 345 sshd on .
- Disable sshd from runlevel 4: chkconfig --level 4 sshd off .
- List all services and xinetd controlled subsystems: chkconfig --list .
- List only sshd service runlevel status: chkconfig --list sshd .
- service: shortcut so that you don't have to type /etc/init.d/.
For example: service sshd start will start the SSH service immediately. You can achieve the same with /etc/init.d/sshd start and this way is more portable between various Linux distribution versions.
In all modern Linux distributions software is managed using a package management system. Red Hat developed their own, known as Red Hat Package Manager (RPM), and since this software was open source, other commercial distributions also started to use the same program and process. Debian (which is non-commercial) opted to implement their own system (apt/dpkg), which still obeys the same packaging semantics, only the commands are a bit different (use dpkg, apt-cache, apt-get instead of rpm). Most non-commercial distributions are based on Debian, so you might encounter dpkg as well (for example in Ubuntu and in Knoppix).
A package can be thought to be an compressed archive file (very similar to a .tar.gz-file) but it will also contain installation and removal scripts as well as dependency information about the package. Dependency information is used to tell the system that in order to get some software inside one package to work, one needs to have other packages installed first. Most commonly this is used to make sure that certain shared libraries (and proper versions) are installed before allowing the installation of the software application that uses the libraries. Sometimes the dependency information is also used to tell about conflicts. Conflicts are used when there are two different software packages that use the same command names or implement the same exact network service (use the same ports).
Lastly, there are virtual packages, which don't contain any files themselves, but instead only contain dependencies. Installing a virtual package will pull all the dependencies in and cause a certain larger subsystem to be installed. This is commonly used in the non-commercial distributions. For example installing package kde-desktop would actually install over 30 other packages which contain the common software for the KDE desktop environment.
This structure is a bit different from the concept employed in HP-UX nowadays where you have bundles, products, sub-products and filesets. There are no such things under any Linux distribution. A distribution might group the software packages using some other mechanism, but this mechanism is not part of the packaging system per-se.
In order to track installed packages and provide information about installed files, all package management systems use some kind of an database (Berkeley DB-based normally, using some SQL-based system would be an overkill). When you remove an installed package, the package manager will consult this database for the installed files and dependencies and will build knowledge about operations which must be done and also will decide whether dependency rules will still hold.
In RPM-based systems the database also contains install time information about files, such as installed size, install time last-modified timestamp, ownerships and MD5-sums of files. This information can be later be used in order to verify whether the files have been changed after installation. The RPM database contents are critical for proper operation of your system, so you should always backup the database. The database consists of multiple files which are stored in /var/lib/rpm/.
The RPM-database more or less implements the same functions as the IPD (Installed Packages Database) in HP-UX.
The rpm command is used for all packaging needs, including own RPM-package building. For this reason the rpm man-page might seem a bit overwhelming.
RPM has been designed automation in mind and for this reason it will never ask any questions interactively (this is different from dpkg). Newly installed services are off by default (95% of time) so you'll need to enable them specifically aftewards using chkconfig or other program.
Here are some example commands that you might find useful instead of the corresponding sw*-commands on HP-UX.
Common RPM-usages
- Install new software package from file: rpm -ivh filename.rpm
Note that multiple files can be installed at the same time. rpm will deduce the correct installation order in this case. If there are any conflicts because of dependencies or some package is already installed, rpm will bail with an error and won't do anything.- Install new software package from file, but also automatically install any other packages that are required if the other package files are located in the same directory: rpm -ivh --aid filename.rpm
- Remove installed package: rpm -ev package-name
Notice that this time the package internal name is used, not the filename from which it was installed. Infact, once you've installed a package from an .rpm-file, you will not need the file anymore.- List all installed packages and their versions: rpm -qa
Since RPM uses Berkeley DB tables as storage for data, the package list will be printed in "random" order. Piping into sort is highly recommended.- Show version of installed package: rpm -q package-name
- Display a list of all files that came with package: rpm -ql package-name
- Which package introduced file /path/x: rpm -qf /path/x
- Show list of changes to installed files: rpm -V packagename
This will display a line per each changed file if the file has been changed after it was installed. The line will contain a bitmap of changed attributes (please see the man page of rpm for more info).- Show information about installed package: rpm -qi packagename
- Show changelog of installed package: rpm -qi --changelog packagename
- Show information about non-installed package: rpm -qip filename.rpm Note that the option -p filename.rpm can be used with most other queries above. For example rpm -qlp filename.rpm would print a list of filenames that would be introduced if one would install the package.
Also rpm -qip --changelog filename.rpm is quite useful.Most software packages have been built for a specific target architecture. The target architecture is encoded in the .rpm-filename.
Common architecture specifiers:
- i386: 32-bit PC-architecture. Requires at least a 80386-level processor to run.
- i586: Same as i386, but binaries contain instructions that are available starting from Pentium processors.
- i686: Contains instructions present in Pentium Pro/Pentium II microarchitectures (and newer).
- x86-64: 64-bit PC-architecture. Also known as EM64T (intel) or x86-64 (AMD). Same thing. Requires that your whole distribution is built for x86-64.
- ia64: Itanium and Itanium II. Not the same thing at all as x86-64.
- noarch: Package contains only text or binary files that are not code but something else (compressed man-pages for example).
For a bit more paranoid environments RPM provides digital signatures and verification of distributed packages. For example all RHEL RPM-packages are digitally signed by Red Hat's package distribution key and you can verify that the package is from Red Hat originally.
The public key that can be used to verify signatures is available on first installation CD-ROM. In the next example we trust the key on the NFS-server to be from Red Hat (obviously the first CD-ROM would be much better source for the key) and we first import the key to root's rpm tool and then use that key to verify that the package is signed by an imported key.
First find all the GPG keys present on our system [root@system /usr/share]# ls -la `find . -name "*GPG*"` -rw-r--r-- 1 root root 1910 Sep 19 20:34 ./doc/redhat-release-4ES/RPM-GPG-KEY -rw-r--r-- 1 root root 1489 Nov 11 2004 ./doc/rpm-4.3.3/BETA-GPG-KEY -rw-r--r-- 1 root root 1913 Nov 11 2004 ./doc/rpm-4.3.3/RPM-GPG-KEY -rw-r--r-- 1 root root 1489 Mar 20 2002 ./rhn/BETA-RPM-GPG-KEY -rw-r--r-- 1 root root 1913 Aug 30 2002 ./rhn/RPM-GPG-KEY -rw-r--r-- 1 root root 1519 Oct 29 2003 ./rhn/RPM-GPG-KEY-fedora -rw-r--r-- 1 root root 1076 Oct 29 2003 ./rhn/RPM-GPG-KEY-fedora-test As you can see there are many keys that Red Hat uses for package signing. However for us only one key is the correct one. Mount RHEL4u2 installation directory over on /mnt/ [root@system share]# mount nfsserver:/data/linux /mnt/ The installation directory (as well as CD#1) contains the key used for RPM verification. Compare that key to another one that is present on our system [root@system share]# ls -la /mnt/RPM-GPG-KEY -rw-r--r-- 1 1000 1000 1910 Oct 31 15:03 /mnt/RPM-GPG-KEY [root@system share]# diff /mnt/RPM-GPG-KEY ./doc/redhat-release-4ES/RPM-GPG-KEY diff doesn't print out anything, which means that the files contain the same data. You can also use program called cmp which is meant for binary files (since diff is meant for text files, it might not be correct program to use in all cases. GPG keys are ASCII files though) Store the key for later use so that rpm can be used to verify signatures [root@system share]# rpm --import /mnt/RPM-GPG-KEY Verify a package file (not installed yet) [root@system share]# rpm -K /mnt/RedHat/RPMS/gimp-2.0.5-5.i386.rpm /mnt/RedHat/RPMS/gimp-2.0.5-5.i386.rpm: (sha1) dsa sha1 md5 gpg OK Verification is ok. rpm also communicates this status via exit codes Exit code for successful verification is zero (OK) [root@system share]# echo $? 0 Next we'll create a package file for which the verification will fail [root@system share]# cp /mnt/RedHat/RPMS/gimp-2.0.5-5.i386.rpm /tmp/foo.rpm NFS share is not needed beyond this point [root@system share]# umount /mnt/ [root@system share]# ls -la /tmp/foo.rpm -rw-r--r-- 1 root root 10119963 Mar 12 19:47 /tmp/foo.rpm Add 4 KiBs of random data to the end of the file [root@system share]# dd if=/dev/urandom bs=1024 count=4 >> /tmp/foo.rpm 4+0 records in 4+0 records out [root@system share]# ls -la /tmp/foo.rpm -rw-r--r-- 1 root root 10124059 Mar 12 19:47 /tmp/foo.rpm Verify. Should fail since package contains random garbage at the end [root@system share]# rpm -K /tmp/foo.rpm /tmp/foo.rpm: (sha1) dsa sha1 MD5 GPG NOT OK This is communicated with a failure exit code as well [root@system share]# echo $? 1
[ Verifying package file signatures ]
For other examples, please read on about updating software.
Contrary to HP-UX which still uses "software patches", most Linux distributions (Red Hat included) have moved into a system where the whole software package is redistributed when it has security problems fixed. This means that Linux systems do not use patches, but also means that the software update process is less painful from the standpoint of the package management tool.
In order to update an installed software package, use rpm -Uvh filename.rpm . Please note that if there is no older version of the package installed, rpm -U will install the package anyway. So in effect, -U should be understood to mean "update or install if not installed".
In some cases it is beneficial to have a directory (over NFS for example) that contains all updated versions of RPM files available at that time so that the system can be updated with all the newest packages. In these cases it is not possible to use rpm -U since that would also install packages that weren't installed before. This is what rpm -F is for. It is used in the same way as -U, but will only update a package if there is an older version already installed.
With such directory one could use a command like rpm -Fvh *.rpm . Please try to arrange a test system which contains the same software package selection that your production system does, as some times the "update from directory" technique will run into dependency problems (distributor has released updated packages but because of too strict dependencies they cannot be applied). All updates will fail in this case (nothing will be updated).
Normally installing multiple versions of the same package is not possible since the packages contain files that conflict (files have the same names and go into the same directories). rpm will complain about this very loudly. The only software package where installing multiple versions makes any sense is the kernel package. In fact, it is beneficial to leave the older kernel versions installed so that we can restart the system with known good kernel. In this case you should use rpm -ivh kernel-blurb.rpm, not -U or -F (since both of these will remove the older kernel).
The kernel package installation will also integrate the new kernel into the boot manager menus, but you should always check /boot/grub/menu.lst for the default-keyword whether it refers to the correct new kernel. In 99% of cases this will be updated automatically to the newest kernel.
When you're sure that the new kernel works properly (99.99% of cases), you may opt to remove the older kernels if you wish to free some space under /boot/ and /lib/modules/. Normally this is not required since the kernels don't take a lot of space.
Last noteable thing about kernel updates is that kernels in Linux distributions normally come in two flavours: Uniprocessor and SMP (Symmetric Multi Processing). The uniprocessor kernel is installed along side the SMP kernel on SMP-systems (when you have more than one logical CPU, like in Hyper-Threading/SMT-case). This is to aid debugging very hard hardware related problems since Uniprocessor kernel does not contain locking code (it doesn't need to) and hence is more reliable on unstable systems (and buggy drivers, but commercial distributors don't distribute buggy drivers normally).
Normally only switching to a new version of kernel will require a reboot. 3rd party software might require reboots as well, for example hardware managing agent updates from the hardware manufacturer, but these are not provided by the Linux distributor.
All material and code copyright 2006 Aleksandr Koltsoff. Use for personal education permitted, all other use must be negotiated (czr(at)iki(dot)fi). Constructive critisism and feedback welcome via email.