Dell Deployment Toolkit (DTK) and Cobbler

Hello everybody, it is been so long since my last post here, but now I'm back :) and today I'm going to describe my last assignment for provisioning system.

So let's get it started straight with the problem. I got a few racks filled with Dell servers and they should be configured and provisioned with Linux as fast as I can do it. After wondering which provisioning solution to choose I pick cobbler.

For anyone, who configured Dell servers, knows that the basic configuration of BIOS, iDRAC and RAID can costs you almost half of the time of installing OS. With so much hardware I'll need 2-3 days doing the same task like a monkey, only to prepare the servers for OS deployment. So I decided to go with some sort of auto configuration solution, that will help me, deliver them times faster. As I mentioned I have already decided to use provisioning services for OS deployment and start digging this direction. After some googling I found that Dell provides automatic deployment (DTK) tool for configuring the system (BIOS, iDRAC, RAID) and even installing an OS. Unfortunately, it is highly limited to RHEL, SUSE Enterprise and Windows. But it is superb for configuring a bare metal server and you got working iDRAC for no time.

You can get the Dell Deployment Toolkit from here. Unfortunately after a while, the link for downloading OpenManage Deployment Toolkit for Linux x64 became broken, but after some googling I found the most recent (as the date of these writing) DTK version from 11 November 2014, you can get it from here. You can mount it and see what we got in the iso file:

# mount -t iso9660 -o loop dtk_4.4_1294_Linux64_A01.iso /mnt

Two of the directories are interesting to us. The first one is RPMs/ and the second one is isolinux/. Before we continue let's clear out what are our possibilities and how we can achieve our goal with Dell DTK. First we need some tools to export the existing configuration from already installed and configured server. Then there should be some environment in which we can upload already exported files and import them on a new system. We need some sort of script that runs in minimal OS environment and changes the BIOS configuration (I'll use for short BIOS, but I mean all of them BIOS, iDRAC and RAID) and after a one or two restarts we got working iDRAC and fortunately RAID configuration. This is our goal and on the other hand Dell DTK provide us with exactly this functionality. The three tools that we will do the job are: SYSCFG, RAIDCFG and RACADM. These tools along with others can be found in the RPMs/ directory. There is a very nice video tutorial how to install them on RHEL6/ CentOS 6 in the Dell DTK Wiki. We can save the current configuration to a file, edit it and installed it directly to a new server with the same tools.

The first one, which I'm going to use will be SYSCFG. It can be used for configuring primarily BIOS. You can dump your whole current configuration with the command:

# syscfg -o bios-settings.ini

The output file is in ini format and it is quite long. I'm not going to discuss different parts of the file. You can see what every variable sets from the Dell OpenManage Deployment Toolkit Command Line Interface Reference Guide (I got it from here). If you considering using the DTK seriously have a look at its user's guide.

Once you got the BIOS configuration it is good to get the RAID one as well with:

# raidcfg -o=raidcfg.ini

The different configuration parts can be found in the Reference guide as well. The last thing we need is iDRAC configuration and we can get it with the most powerful tool, which is capable of configuring some of the BIOS settings as well, RACADM. RACADM is actually a tool for cli connection to iDRAC module mostly through SSH. You can read and write different configuration variables, export and import a configuration, create users. It is quite flexible and most of the information can be found here. Actually, there is more recent version of iDRAC – v. 8, but my environment consist of version 7 modules. So to export your current iDRAC configuration type:

# racadm getconfig -f idrac-config.ini

As the others files this one has INI syntax, but it is quite large. Almost all of the iDRAC configuration options can be found there. However there are few, that are not present. Complete list of options can be found in the reference guide. 

Finally we got a configured server, that is gonna be used as a template and we took all of its configuration in files we will use in future deployments.

OSSEC issues connecting to MySQL

Hello again, today I'm going to do some troubleshooting with OSSEC and MySQL. OSSEC is very powerful host-based intrusion detection system, which can help you monitor in real time attacks that target your infrastructure. Main features are monitoring local and remote logs, rootkit detection, file integrity checking, it can be configured even for active response. More about OSSEC features can be found here.

The OSSEC system, by default, put all of its logs in a plain text inside /var/ossec/logs/. However there is an option all of the alerts and logs activity to be put inside a database. The two options are – PostgreSQL and MySQL. The biggest advantage of using a database is to monitor in real time attacks against your servers using some graphic tool. There are a few web-based tools, that I found, which can be connected to MySQL server, used by OSSEC. The first one is OSSEC Web UI (OSWUI) which can be downloaded from here. The second one, that I found, is Splunk add-on and it can be downloaded from here. The last web-based GUI, which I chose was Analogi.

The problem – connecting OSSEC with MySQL

The installation went very well, I just followed the installation instructions. I added new repository for OSSEC in CentOS, it appears I need EPEL repository too, because of missing dependencies. I installed OSSEC and MySQL through yum and it reports that the MySQL was installed from ossec repository (atomic), that I added earlier. The MySQL version is 5.5.30 and OSSEC version is 2.7.

I followed the instructions for connecting and configuring OSSEC with MySQL from the OSSEC documentation and everything seemed to be ok. Just two things to mention – if the mysql server is on the localhost, inside /var/ossec/etc/ossec.conf in the <hostname> tags put 127.0.0.1, I tried with 'localhost', but it gave me error inside the /var/ossec/logs/ossec.log. The other thing to watch for is the mysql.schema file. On RedHat/CentOS systems the file could be found under /var/ossec/etc/mysql/ there is no need for downloading the source.

After I installed everything I started watching the ossec.log file for errors. I connected several times with a wrong username/password through ssh to generate some alerts and then in the log appear the following error message:

ERROR: Error executing query 'INSERT INTO server(last_contact, version, hostname, information) VALUES ('1362937495', 'v2.7', 'monitor.example.com', 'Linux monitor.example.com 2.6.32-279.22.1.el6.x86_64 #1 SMP Wed Feb 6 03:10:46 UTC 2013 x86_64 - OSSEC HIDS v2.7')'. Error: 'Lock wait timeout exceeded; try restarting transaction'.

After digging in the database I found this lock:

TRANSACTIONS

------------

Trx id counter E402

Purge done for trx's n:o < 9B28 undo n:o < 0

History list length 303

LIST OF TRANSACTIONS FOR EACH SESSION:

---TRANSACTION 0, not started

MySQL thread id 5, OS thread handle 0x7f28f821f700, query id 8354 localhost root

SHOW ENGINE INNODB Status

---TRANSACTION E401, ACTIVE 22 sec inserting

mysql tables in use 1, locked 1

LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s), undo log entries 1

MySQL thread id 4, OS thread handle 0x7f28f8260700, query id 8352 localhost 127.0.0.1 ossecuser update

INSERT INTO server(last_contact, version, hostname, information) VALUES ('1362991366', 'v2.7', 'monitor.example.com', 'Linux monitor.example.com 2.6.32-279.22.1.el6.x86_64 #1 SMP Wed Feb 6 03:10:46 UTC 2013 x86_64 - OSSEC HIDS v2.7')

Trx read view will not see trx with id >= E402, sees < E400

------- TRX HAS BEEN WAITING 22 SEC FOR THIS LOCK TO BE GRANTED:

RECORD LOCKS space id 0 page no 597 n bits 72 index `hostname` of table `ossec`.`server` trx id E401 lock mode S waiting

Record lock, heap no 2 PHYSICAL RECORD: n_fields 2; compact format; info bits 0

0: len 27; hex 6d7973716c2d73727630312e7365637572657368616b652e636f6d; asc monitor.example.com;;

1: len 2; hex 0001; asc ;;

------------------

---TRANSACTION E400, ACTIVE 972 sec

8 lock struct(s), heap size 1248, 1 row lock(s), undo log entries 3146

MySQL thread id 2, OS thread handle 0x7f28f82a1700, query id 8349 localhost 127.0.0.1 ossecuser

Trx read view will not see trx with id >= E401, sees < E401

My concerns confirmed after watching the process list inside mysql console. After version 5.5 MySQL default storage engine is InnoDB, before it was MyISAM. I thought that inside the schema file the storage engine would be marked, but unfortunately there were only table definition. Next I decided to convert the engine to MyISAM for all the tables used by OSSEC for reporting. First connect to mysql with root credentials:

# mysql -h localhost -u root -p

Once connected, enter ossec database and change the storage engine:

mysql > USE ossec;

mysql > SHOW TABLES;

mysql > ALTER TABLE agent ENGINE=MYISAM;

Repeat the last command for all tables inside ossec database. The result can be seen with:

mysql > SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = 'ossec' AND ENGINE = 'MYISAM';

After confirmation that all of the tables use MyISAM engine restart the ossec server. Check again the ossec.log and you should have no issues regarding database connection.

If you read this before installation of OSSEC, It will be better to change the mysql.schema file. Just put the default storage engine into table definitions before applying the schema and they will be created with the correct configuration. The definition should look like:

CREATE TABLE category

(

cat_id SMALLINT UNSIGNED NOT NULL AUTO_INCREMENT,

cat_name VARCHAR(32) NOT NULL UNIQUE,

PRIMARY KEY (cat_id),

INDEX (cat_name)

) ENGINE = MYISAM;

This should be added to all table definitions. 

I hope someone will find this useful. I want to share a few additional resources about setting OSSEC and Analogi on CentOS:

http://linuxdrops.com/install-the-coolest-of-all-analogi-ossec-web-user-interface-on-centos-rhel-debian-ubuntu/

http://scottlinux.com/2012/07/08/how-to-install-ossec-on-red-hat-or-centos-6/

And, of course if you consider OSSEC deployment in your infrastructure I highly recommend this book.