As promised earlier, here is the first installment of the how-to.  Before we get going to far, this series of how-to’s has the following disclaimers

  • A basic knowledge of Linux is assumed.  I’ll provide the commands to perform certain activities, but I’ll assume you know how to get a basic Linux install going
  • This how-to is written with CentOS 5.4 in mind – any distro will do, but you may need to modify some commands to make it happen.
  • This how-to uses virtual machines and is the result of proofing out the concept.  I’m sure that when I build the final servers – I’ll fine tune it a bit more (which will result in an addendum to the how-to :) )
  • This how-to will be done in a progressive fashion, each part will layer another level functionality of onto the configuration.

In Part 1 of the how-to, we will complete the create of two Virtual Machines, install Linux, VMtools, needed storage space, and required software.  At the end of this part, you will have two functional virtual machines that will replicate storage between themselves in a Primary/Secondary fashion, and you will be able to share out the LUN from the primary node via iSCSI.

0.  Most commands below need to be execute with root privilege

  1. Create two virtual machines.  I created them on my desktop using Virtual Workstation.  I created the machines with two Network Cards, one of which was bridged over the desktops NIC, the other on vmnet0 (or any other non-bridge network) - to provide a private network connection for the two storage nodes.  Additional – I removed all fluff hardware from the VM (USB, Floppy, etc.)  I attached an 2.5 GB thin provisioned HDD to each machine.
  2. Install Linux – I did this by doing a netinstall, and selecting all packages from the install process.  This means I will need to add everything to the system.  The output of df -h on a completed system is below
    • /dev/sda2             1.5G  1.1G  325M  78% /
      /dev/sda4             396M   34M  342M   9% /var
      /dev/sda1              99M   17M   78M  18% /boot
      tmpfs                 189M     0  189M   0% /dev/shm
    • Configure eth0 to communicate on your public network
    • Configure eth1 to communicate on the private VM only network
  3. I disabled selinux, I realize this may be a debatable action, however I just don’t understand it enough, and I want to eliminate it as a potential problem.
    • vi /etc/selinux/config
    • change enabled  to disabled  
  4. optional - Next we’re going to remove a few pieces of software that I don’t want on the system -
    • echo y | yum remove iptables
    • echo y | yum remove cups-libs
  5. Next we are going to install perl and than upgrade the system
    • yum install perl
    • echo y | yum upgrade
    • reboot
  6. Install VMtools via your preferred choice – once you have installed it besure to clean up the install files as they take up a lot of hard drive space
  7. Now we are going to install all of the software for the core functionality of the system
    • echo y | yum install mdadm
    • echo y | yum install kmod-drbd83
    • echo y | yum install wget
    • echo y | yum install make
    • echo y | yum install gcc
    • echo y | yum install openssl-devel
    • echo y | yum install kernel-devel
    • echo y | yum install patch
  8. Download and install ietd
  9. At this point – add an additional two disks two your VM (reboot or rescan the scsi bus to find them)
  10. Let’s create a single raid 0 array on the two disks we just added
    • mdadm –create /dev/md/d0 –auto=mdp –level=0 –raid-devices=2 /dev/sdb /dev/sdc
    • mdadm –detail –scan >> /etc/mdadm.conf
    • add DEVICE /dev/sd*to mdadm.conf as the first line of the file
  11. Step 10 creates a software raid array that we can partition.  Making the array is very important for future expansion of the array, without disrupting the drbd resource that is on the array.
    • This will be evident down the road when I walk us through dynamically adding additional space to our storage system.
  12. Now let’s partition the new array
    1. fdisk /dev/md/d0
    2. n – command to create a new partition
    3. p – create it as a primary partition
    4. 1 – partition number
    5. <enter> – begin at the beginning of the disk
    6. <enter> – end at the end of the disk
    7. w – write the partition table and exit
  13. It’s time to create your drbd.conf file.  You can scour the internet and make your own, or you can download mineand modify it to match your hostnames/IP config
  14. Create the drbd resource meta data
    • drbdadm create-md r0
  15. At this point you should either clone this VM and modify it appropriately for the second node, or repeat all of the above steps on the second VM.  Do as you feel comfortable

At this point we have two identically configured VMs, that have a raid 0 array, drbd installed and configured to write to that array.  The next step is to fire up drbdfor the first time and perform the initial sync between the nodes

  1. Before we go any further, let’s set both drbd and iscsi-target to manual startup – this is annoying for a reboot, and makes it unusable in a real sense – but it allows us total control of our test scenarios
    • chkconfig –level 0123456 drbd off
    • chkconfig –level 0123456 iscsi-target off
  2. In step 13 above, you should have created your drbd.conf file.  Going forward I will refer to the nodes as I have them named in my example file.  kroker01 is the primary node with kroker02 the secondary.
  3. On kroker01 start drbd up and tell it to perform a sync to kroker02 (when it comes online)
    • service drbd start
    • drbdadm — –overwrite-data-of-peer primary r0
  4. Check out /proc/drbd to verify that kroker01 is in a WFConnection state, and is the primary
    •  1: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r—-
          ns:10372472 nr:0 dw:0 dr:10380564 al:0 bm:633 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
  5. Go ahead and start drbd on the second node; at this point if you cat/proc/drbd you should see connection status as Connected and ro: as Primary/Secondary, with ds:UpToDate/Inconsistent.  You should also see a line in /proc/drbd detailing the status of the resource.  At this point we are safe to direct disk traffic to /dev/drbd1!
    • Please note
  6. Setup ietd.conf
    • Again, google is your friend or download my example, making any needed modifications for hostname and the like.
    • service iscsi-target start
  7. From any box that has an iscsi-initiator, point it towards the IP/hostname of the primary box (kroker01).
  8. After performing a disk rescan on the client host, you should now have a new disk to work with and format.

Play around with stopping and restart drbd on the secondary node, and observe what you see in /var/log/messages, and /proc/drbd.  As the secondary node comes up and down – drbd tells you various information about what it is doing and the status of the secondary disk.

What can I do with this?

Honestly – at this point we have an expensive, highly redundant, but manual raid 1 disk array for you to play with.  Should your primary node fail, you will have a block for block copy of your data on a second node, that you could pretty easily bring up as the primary node to access your data.