Diskless netboot

Revision as of 15:35, 12 June 2014 by WikiFreak (talk | contribs)



Context and aim

Aim

I want to achieve the following configuration:

Target network diskless configuration


Key points:

  • Each client is a diskless station.
  • I want to use the same distribution everywhere.


For all this tutorial I'll be using a local network 172.16.50.0/24 with 172.16.50.2 as master server.



Diskless station means:

The clients don't need any hard drive to run, they will retrieve the file system from the TFTP server and use a NFS share as hard drive. The system will only run in RAM disk. So:

  • All the clients will share the installation, configuration files and so on.
  • Each client will run a dedicated instance of the operating system in his own RAM disk
  • Logs will be centralized on the common NFS server - so we don't loose data on each reboot.
  • The user will be able to choose the O.S to run on boot - thanks to a PXE menu



Monitoring point of view:

  • Each client will have a local SNMP agent + a dedicated zabbix agent to send back its status. In my case, as I'm using Java, I'll install the Oracle JDK and I will use some JMX beans too.
  • The current server smartcard-gw will also have a SNMP agent + a dedicated zabbix agent so I can monitor it too.


File access:

  • The common NFS share will be exposed by a Samba server as a network file-share. Therefore I'll be able to browse for it and access it easily.



Requirements

To achieve that you need to have:


Optional:


Target folder tree (server side)

This is how we'll setup our files and folders:

# TFTP root
/tftpboot/                                   

###############
# Network bootable image(s) using NFS technology
################       

#### Boot file            
/tftpboot/pxelinux.0                   # Initial boot file - only use to load the PXE NetBoot manager
/tftpboot/{menu.c32 || vesamenu.c32}   # PXE interactive menu managers (text or graphical)
/tftpboot/pxelinux.cfg/                # PXE configuration(s)
/tftpboot/pxelinux.cfg/default         # default PXE configuration

#### Kernel file

/tftpboot/images/      

# Debian 7.x [Wheezy] 
/tftpboot/images/wheezy/   
/tftpboot/images/wheezy/vmlinuz
/tftpboot/images/wheezy/initrd.img

# [X]Ubuntu 14.04 [Trusty] 
/tftpboot/images/trusty/  
/tftpboot/images/trusty/vmlinuz
/tftpboot/images/trusty/initrd.img


#### NFS 
# This is where the runnable will be. Each image will be in a dedicated folder.
/nfs/                    

# Debian 7.x [Wheezy] 
/nfs/wheezy/   

# Ubuntu 14.04 [Trusty] 
/nfs/trusty/


Client overview

Each client must have, at least, 4 Go of RAM.


4 GO RAM configuration

This is how we're gonna populate the client:

NetBoot client RAM overview - 4Go


As you can see, each client will have some space dedicated for swap + some RAMdisk to allow writing in /var, /tmp and /proc.


Configuration of a 4Go RAM disk:

  • No swap
  • Local TMPFS (read/write for /dev, /tmp, ...) : 1 Go
    • /tmp = 512 M
    • /var/tmp = 128 M
    • /var/log = 128 M
    • /var/run = 8 M
    • /var/lock = 8 M
    • /run/shm = 256 M
  • O.S (NFS read only) : all the rest ~ 2.8 Go
  • Common share (NFS read write) : Remote disk


2 Go

Due to budget restriction we might encounter some low memory machines with only 2 Go...


This is how we're gonna populate the client:

NetBoot client RAM overview - 2Go


In case of 2Go RAM then you have to use some tricks:

  • No swap
  • O.S (NFS read only) : ~ 1.2 Go
  • Common share (NFS read write) : Remote disk
  • Local TMPFS (read/write for /dev, /tmp, ...) : all the rest
    • /tmp = 372 M
    • /var/tmp = auto
    • /var/log = 128 M
    • /var/run = auto
    • /var/lock = auto
    • /run/shm = auto


How big is the client image ?

By default the deboostrap Ubuntu 14.04 LTS image is 239 Mo. With the applications we're gonna use that size will increase to about 1 or 1.3 Go depending if you copy (or not) the kernel sources. It may even take 1.6 Go if you're using XFCE frontend.



Installation

NFS support

apt-get install nfs-kernel-server nfs-common

Debootstrap (manage netboot image)

apt-get install debootstrap


Initramfs (to manage "virtual disks")

apt-get install initramfs-tools



Preparation

You have to create a dedicated folder on your server where you will host the distributions kernels + Boot settings.

mkdir -p /tftpboot/pxelinux.cfg
chmod -R 755 /tftpboot/pxelinux.cfg

mkdir -p /tftpboot/images
chmod -R 755 /tftpboot/images


The pxelinux.cfg/ folder is mandatory. Inside you can provide:

  • configuration for a specific IP @ or hostname
  • configuration for a group
  • default configuration (required)



Boot menu and Kernel setup

The first thing to do is to setup a booting kernel. To do so we'll use the "syslinux" files.


Reminder If your client(s) will use some smart-cards driver then you MUST install these drivers on the on the NFS server + reboot the server ; before going through the following steps. See Drivers#Smart-card_drivers



Root file: pxelinux.0

The pxelinux.0 is the root file. That's the file that allows the netboot.

This is the file that is serve by the TFTP server.

cp /usr/lib/syslinux/pxelinux.0 /tftpboot/


Create NetBoot menu | defaults

Now, we have to specify which kernel to use and which distributions are available for NetBoot.


Create the default configuration file:

vim /tftpboot/pxelinux.cfg/default


Put the following:

# Debian 7.x
LABEL wheezy
    kernel images/wheezy/vmlinuz
    initrd images/wheezy/initrd.img

# Ubuntu 14.04
LABEL trusty
    kernel images/trusty/vmlinuz
    initrd images/trusty/initrd.img


# Prompt user for selection
PROMPT 1
# No timeout
TIMEOUT 0
  • Each LABEL is a specific configuration that will displayed on the NetBoot menu.
  • PROMPT 0 = enable user prompt so you can choose the configuration
  • TIMEOUT 0 = timeout (in seconds) before the default option is chosen. 0 == no timeout


Note that I used a reference to "trusty/", that's a folder I need to create later on.


Init Kernel files

Create directories

Create the target kernel folders. You should create 1 folder for each distribution you'd like to provide in NetBoot.

# Debian 7.x
mkdir -p /tftpboot/images/wheezy

# Ubuntu 14.04
mkdir -p /tftpboot/images/trusty


Prepare initramfs to boot over NFS

This step must to be run on the machine that has the kernel you are going to serve to your clients.


>>> In our case it has to be run on the TFTP server


Copy initramfs settings for PXE boot

cp -r /etc/initramfs-tools /etc/initramfs-pxe


Adjust PXE boot configuration

cd /etc/initramfs-pxe/
vim /etc/initramfs-pxe/initramfs.conf


Add / adjust the following options:

BOOT=nfs
MODULE=netboot


Copy and prepare kernel

You have to copy your current kernel files to the boot folder:

# Debian 7.x
cp /boot/vmlinuz-`uname -r` /tftpboot/images/wheezy/vmlinuz
cp /boot/initrd.img-`uname -r` /tftpboot/images/wheezy/initrd.img

# Ubuntu 14.04
cp /boot/vmlinuz-`uname -r` /tftpboot/images/trusty/vmlinuz
cp /boot/initrd.img-`uname -r` /tftpboot/images/trusty/initrd.img


Enable NFS boot on target kernel:

mkinitramfs -d /etc/initramfs-pxe -o /tftpboot/images/trusty/initrd.img


Adjust rights:

chmod -R 755 /tftpboot/images/


Notes:

  • Do NOT use some symlink for "vmlinuz" and "initrd.img" !! It won't work.
  • If you don't want to use `uname -r` [current kernel version and architecture] then adjust the values to target kernel number + architecture
  • You have to run mkinitramfs for each kernel you'll provide
  • Don't forget to adjust the rights to 755 for every distribution



NFS server setup

Configuration

The NFS configuration is done in the /etc/exports file

vim /etc/exports


Add something like that:

### list of available O.S
  /nfs/trusty         172.16.50.0/24(ro,no_root_squash,no_subtree_check,async,insecure)
  /nfs/wheezy         172.16.50.0/24(ro,no_root_squash,no_subtree_check,async,insecure)
### common share
  /nfs/common         172.16.50.0/24(rw,no_root_squash,no_subtree_check,async,insecure)


Adjust "172.16.50.0/24" to your own network address

  • rw : Allow clients to read as well as write access
  • ro : Read only access
  • insecure : Tells the NFS server to use unpriveledged ports (ports > 1024).
  • no_subtree_check : If the entire volume (/users) is exported, disabling this check will speed up transfers.
  • async : async will speed up transfers.
  • no_root_squash: This phrase allows root to connect to the designated directory.


- NOTE -

  • It's always a good idea to use Read-Only if you plan to share this disk. That will avoid user to mess with your image!
  • There must not be any space between network IP and "("
  • If you plan to share a NFS to all users - like my /nfs/common - don't forget to set chmod -R 777 /nfs/common


Security

Like TFTP, this part is insecure !

You must restrict the access to your NFS server by a firewall script and filtering BEFORE reaching the LAN !


NFS is using dynamic ports numbers because it runs over rpcbind. Making NFS using specifics port is a pain in the ass !! :(

So, instead of that you should allow your LAN communication.


    IPTABLES=`which iptables`
    LAN_ADDRESS="172.16.50.0/24"

    # Allow LAN communication
    $IPTABLES -A INPUT -s $LAN_ADDRESS -d $LAN_ADDRESS -m state ! --state INVALID -j ACCEPT
    $IPTABLES -A OUTPUT -s $LAN_ADDRESS -d $LAN_ADDRESS -m state ! --state INVALID -j ACCEPT


Management

service nfs-kernel-server {status|start|stop|restart}


Test the server

Install the NFS v4 client:

apt-get install nfs-common


To mount the default path:

mount -t nfs nfs-server:/ /mnt

You'll see: "/nfs"


It's better to do:

mount -t nfs nfs-server:/nfs /mnt




NFS client image

There are different way to setup a NFS client image.

The main ones are:

  • Manually
    • debootstrap
    • copying the install from your server
    • Manual install on a client, then, when the system is ready, copy everything to the NFS share
  • Using script and software like "Puppet" or "Chef"


Setup client distribution

You have to create one target for each distribution you want to serve:

mkdir -p /nfs/trusty
mkdir -p /nfs/wheezy
mkdir -p /nfs/common


- NOTES -

  • The folder name should match your NetBoot settings. Folder name = a LABEL in the NetBoot config.
  • The folder name should match a Linux (Debian like) distribution name


Configure client distribution


Kernel modules and source

-- This is an optional step ; but it can avoid many bugs --


If you're using a local kernel as the default NetBoot kernel, then you need to do copy the modules + kernel source to every distribution.

# Copy kernel modules
cp -r /lib/modules/`uname -r` /nfs/trusty/lib/modules

# Copy kernel sources
cp -r /usr/src/linux-headers-`uname -r` /nfs/trusty/usr/src


Note that you have to adjust the /nfs/XXX



Backup distribution

You can create an archive of your current distribution for later restore / re-use.


Compression

cd /nfs
tar cvpjf trusty.tar.bz2 ./trusty


Restoration

cd /nfs
tar -xvjf trusty.tar.bz2

PXE interactive menu

You can create interactive NetBoot menus, see:



Local server monitoring

Install the following services:



File sharing

If you want to expose the NFS common folder as a file-share, you have to install and configure Samba. See: Samba server



References

Ubuntu diskless how-to: https://help.ubuntu.com/community/DisklessUbuntuHowto


Mind reference: http://mindref.blogspot.se/2011/03/debian-diskless.html


Super video tutorials:


Nice explanation of PXE process: http://www.linux.com/learn/docs/ldp/497-Diskless-root-NFS-HOWTO