Dell Deployment Toolkit (DTK) and Cobbler

Hello everybody, it is been so long since my last post here, but now I'm back :) and today I'm going to describe my last assignment for provisioning system.

So let's get it started straight with the problem. I got a few racks filled with Dell servers and they should be configured and provisioned with Linux as fast as I can do it. After wondering which provisioning solution to choose I pick cobbler.

For anyone, who configured Dell servers, knows that the basic configuration of BIOS, iDRAC and RAID can costs you almost half of the time of installing OS. With so much hardware I'll need 2-3 days doing the same task like a monkey, only to prepare the servers for OS deployment. So I decided to go with some sort of auto configuration solution, that will help me, deliver them times faster. As I mentioned I have already decided to use provisioning services for OS deployment and start digging this direction. After some googling I found that Dell provides automatic deployment (DTK) tool for configuring the system (BIOS, iDRAC, RAID) and even installing an OS. Unfortunately, it is highly limited to RHEL, SUSE Enterprise and Windows. But it is superb for configuring a bare metal server and you got working iDRAC for no time.

You can get the Dell Deployment Toolkit from here. Unfortunately after a while, the link for downloading OpenManage Deployment Toolkit for Linux x64 became broken, but after some googling I found the most recent (as the date of these writing) DTK version from 11 November 2014, you can get it from here. You can mount it and see what we got in the iso file:

# mount -t iso9660 -o loop dtk_4.4_1294_Linux64_A01.iso /mnt

Two of the directories are interesting to us. The first one is RPMs/ and the second one is isolinux/. Before we continue let's clear out what are our possibilities and how we can achieve our goal with Dell DTK. First we need some tools to export the existing configuration from already installed and configured server. Then there should be some environment in which we can upload already exported files and import them on a new system. We need some sort of script that runs in minimal OS environment and changes the BIOS configuration (I'll use for short BIOS, but I mean all of them BIOS, iDRAC and RAID) and after a one or two restarts we got working iDRAC and fortunately RAID configuration. This is our goal and on the other hand Dell DTK provide us with exactly this functionality. The three tools that we will do the job are: SYSCFG, RAIDCFG and RACADM. These tools along with others can be found in the RPMs/ directory. There is a very nice video tutorial how to install them on RHEL6/ CentOS 6 in the Dell DTK Wiki. We can save the current configuration to a file, edit it and installed it directly to a new server with the same tools.

The first one, which I'm going to use will be SYSCFG. It can be used for configuring primarily BIOS. You can dump your whole current configuration with the command:

# syscfg -o bios-settings.ini

The output file is in ini format and it is quite long. I'm not going to discuss different parts of the file. You can see what every variable sets from the Dell OpenManage Deployment Toolkit Command Line Interface Reference Guide (I got it from here). If you considering using the DTK seriously have a look at its user's guide.

Once you got the BIOS configuration it is good to get the RAID one as well with:

# raidcfg -o=raidcfg.ini

The different configuration parts can be found in the Reference guide as well. The last thing we need is iDRAC configuration and we can get it with the most powerful tool, which is capable of configuring some of the BIOS settings as well, RACADM. RACADM is actually a tool for cli connection to iDRAC module mostly through SSH. You can read and write different configuration variables, export and import a configuration, create users. It is quite flexible and most of the information can be found here. Actually, there is more recent version of iDRAC – v. 8, but my environment consist of version 7 modules. So to export your current iDRAC configuration type:

# racadm getconfig -f idrac-config.ini

As the others files this one has INI syntax, but it is quite large. Almost all of the iDRAC configuration options can be found there. However there are few, that are not present. Complete list of options can be found in the reference guide. 

Finally we got a configured server, that is gonna be used as a template and we took all of its configuration in files we will use in future deployments.

Now the fun part starts. We got the configuration files ready for deployment. It is time for the second interesting directory from the DTK iso – isolinux/. In the isolinux/ directory we got a whole custom distribution which can be used for booting and setting minimal environment and includes the same tools we used in the exporting process. Actually, this distro provides even more tools for replication the configuration, but we are not going to use them here. For more info please refer to the DTK user's guide (the link is provided above).

Let's put cobbler into the action. I installed from packages (Epel repository) and there aren't any big modification. I'm using it with TFTP, DHCP and it provides me a HTTP service. DHCP and TFTP are not managed by cobbler.

And now here is the question, how can I add the DTK distro inside the isolinux/ folder to cobbler. Thanks to google, again, I found this. The blog post from the link contains all the details we need to make it work. There is kernel options in the cobbler distro command that make the magic and it is working. However, when I started to dig inside the isolinux/ folder, I find a file 'isolinux.cfg', which has all of the needed kernel options for different environments – NFS, SMB, TFTP, CDROM and so. Better look there for kernel options if you are trying more current version of DTK. The commands I used for adding the DTK to cobbler system looks like:

1. Create a distro:

# cobbler distro add --name=dtk-4.4_A01 --arch=x86_64 --breed=debian --kernel=/opt/isolinux/SA.1 --initrd=/opt/isolinux/SA.2

Before continue just a few words. The /opt/isolinux directory is where I copied the entire ioslinux/ directory. I did this, because after I added the distro to cobbler and umounting the DTK iso, cobbler fired an error of missing the SA.1 and SA.2 file at their mount location. I did not investigate further

2. Create a profile:

# cobbler profile add --name="dtk-basic" --distro="dtk-4.4_A01" --comment="This is includes basic configuration. No RAID or special disk configuration - just iDRAC and BIOS configurations" --kopts="ramdisk_size=262144 Stage3_type=cdrom DEBUG=0 BUILD=1294 vmalloc=256M share_type=tftp share_location=dtk share_script=basic-config.sh tftp_ip=192.168.0.1 selinux=0"

3. And the last one:

# cobbler system add --name=”new-server” --profile=”basic-dtk” --mac=AA:BB:CC:DD:EE:A0 --interface=eth0

Now let's talk about the commands above :). In the first one there is nothing special, the one thing, that is different from the blog post I mentioned, is that the kernel option (kopts) argument. I moved it to the profile stage, because as you can see in the kernel option argument, there is a share_script option. This options can be used, when the DTK is booted, to automatically execute the script. In other words, this is the place where you put all the configuration options and settings. If you need different script, in my case, I'll configure the RAID controller later, you use the same ditro and just add new profile. If you do everything in just one step, you can put the kopts argument directly in the cobbler distro command (first stage).

The other thing it worth mention is the share_location option in the kopts argument. This option looks in the TFTP root directory (/var/lib/tftboot by default) for directory with the given name. You can use one of the existing there, but after:

# cobbler sync

your script disappears. That's why I created a new directory called dtk and put there the basic-config.sh script. If you are using SELinux you can add:

# semanage fcontext -a -t cobbler_var_lib_t "/var/lib/tftpboot/dtk(/.*)?"

The last command assigns the profile to a new server with MAC address AA:BB:CC:DD:EE:A0. After execution you can check, if it is properly added to cobbler with:

# cobbler system list

I hope you ain't fall asleep from the whole procedure :), the last part of the puzzle is the script itself. I won't give all of minel, because it has sections, that are strictly for my case. The only limitation here are your knowledge of BASH and shell scripting. I'll give some of the options I used:

#!/bin/bash

echo '**********************************************'

echo 'WARNING THIS IS A SPECIAL SCRIPT!'

echo '**********************************************'

INPUT=details.txt

SRV=http://192.168.0.1/cobbler

RACADM=/opt/dell/srvadmin/sbin/racadm

SYSCFG=/opt/dell/toolkit/bin/syscfg

cd /srv

wget -q $SRV/$INPUT

RACKSLOT=`grep "$hostname" $INPUT | cut -d , -f 2`

$RACADM set System.LCD.Configuration 512

$RACADM set System.ThermalSettings.BaseAlgorithm 1

$RACADM set System.ThermalSettings.FanSpeedOffset 1

$SYSCFG --bootmode=Bios

$SYSCFG --asset=$hostname

$SYSCFG --embsataraid=AhciMode

$SYSCFG --SysProfile=PerfOptimized

$SYSCFG pefcfgparams --filter=tempfail --filteraction=powerdown

$SYSCFG --powerbutton=Enabled

#Change default boot order

$SYSCFG --bootseq=NIC.Embedded.1-1-1,HardDisk.List.1-1

echo '**********************************************'

echo '*** END OF SCRIPT ***'

echo '**********************************************'

A few words about my script. Do you see the $hostname variable, it isn't defined anywhere, because it is a cobbler system var. The associated variables with a system can be seen with the command:

# cobbler system dumpvars --name=”new-server”

So the variable $hostname is very valuable and it contains in this case “new-server”. I created a file on the cobbler server and populate it with details. Using this hostname I had a key to search within the file and get more info about the server. There you can specify iDRAC IP, rack slot, location and so on. The other options in my sample script are all documented in the reference guides and I won't discuss them here.

The one thing that made me crazy is that whatever I try I can't power off the server. The shutdown command in DTK distro just reboots it and if you do not have a solution it can enter a loop.

Finally we are ready to switch on the server and see if everything is working as expected. My first several attempts did not end very well, but on the third reboot everything works as expected. In the end I managed to cut the configuration time for a whole rack in a little more than 2 hours, which is very good for me. How much time do you save