Each power supply powers one of the mainboards, and two of the fans (1 in/1 out).  Prior to putting the door on the case I’m going to add a shroud onto each of the IN fans, to direct air onto the motherboards.

I mounted the boards so that all of the connectors are facing forward for ease of access.

The systems will be direct connected by a Fiber connection and a copper connection.  The fiber connection will be through the Qlogic HBA that will be running a driver with TCP support for the DRBD replication.  The copper connection will be an additional ring for the cluster stack to communicate.

Each system will also have a fiber and copper connection that leaves the box and plugs into the appropriate switch.

I haven’t decided how I’m going to install the base storage of 4 disks (with plans to expand to 8 soon).  I’d like to have them in the case, but I’m concerned things might get to crowded – so I may build a small box and run them external.  I just have to make sure the SATA runs stay less than 1m.

Here’s a picture of the case I’ve built to house the computing end of my home SAN.  The lower two fans pull air into the case, the upper fans pull air out.

The next installment will cover configuring DRBD in an dual-primary configuration, monitored by pacemaker.

ETA: Thursday

Currently my environment is pretty simple to consists of a generic system running CentOS 5.4 farming out a few LUNS to my ESX server.

• Pentium 4 2.8 ghz
• Abit IS7-e w/ 1.5 GB RAM
• Dual Port Intel E1000 Nic
• LSILogic MegaRaid 150-4 Sata controller
• 3×250 GB SATA disks in a Raid 5
• A couple misc IDE drives

I have centOS installed onto a 2.5 GB partition, the remainder of the disk (approximately 800 GB usable total) is presented as iSCSI targets using ietd 1.4.19.

 As a side note, until just a few days ago this box was running Openfiler 2.3 with a P3 700 + Soyo SY-7VCA2, but i started having some stability problems with the motherboard/cpu and a MB/CPU swap seems to have made things better.

Said ESX box is currently running 3.5 with some decent hardware. 

• Rioworks/Accelertech/Arima HDAMA Rev. G
• 2xOpteron 248 ( Rev E.)
• 8×1 GB PC1600R Dimms

The ESX box and CentOS system are connected via a crossover cable for iSCSI traffic.

Prior to the split setup between my ESX server and CentOS system, my storage (via the megaraid 150-4) card existed in my ESX box, and due to the need for more storage but not wanting to buy another ESX HCL listed SATA/SCSI card and the needed drives.  By moving it all to an iSCSI system, I removed the need for having to use only certain types of drives.

So what’s next?  I’m glad you asked – the Bowe ESX farm v3.0 will be a 100% availability infrastructure.  Well not truly 100% – I will have some limitations due to internal house electricity, but the [eventual] purchase of appropriately sized UPS capacity will solve that problem.

How do I intend to accomplish this – lots of used hardware.  Some of the hardware I have sitting around from previous spending binges, others have been acquired or will be acquired over the next few weeks via careful ebay shopping.  I will list the price I paid, plan to pay, or would expect to pay as appropriate for my purchase situation (got, getting, had).

ESXi (2x)

• Tyan K8SR (paid 27.50 ea. shipped)
• Dual Opteron 270 (budget 40 per pair shipped – ebay)
• 8x 1GB PC1600R (would pay $10-$15/dimm – bought a bunch of hese a LONG time ago)
• 1x Emulex 9802 HBA (paid $5 shipped – ebay)
• 1x Tyan OOB mangement card (freebie with the K8SR)

Storage (2x)

• Rioworks/Accelertech/Arima HDAMA (rev prior to G)  (paid $20 ea. on ebay)
• Dual Opteron 246HE (freebie CPUs that came with the K8SRs in my host setup.)
• 2x 1 GB PC1600R (see above for pricing)
• 1x Qlogic 2340 HBA (paid 11.50 ea. – ebay)
• 1x Emulex 9802 HBA (paid $5 shipped – ebay)
• Generic SATA Controller ($20 ea.)
• 2×250 GB Sata drives (bought a while ago, but market rate is ~$35 ea.)

All systems will be booting off a 2.5 GB Compact flash microdrive in an IDE adapter – ~50 dollars for 4 drives and adapters on ebay.  All systems also have power 350/400 watt power supplies.  Two of which came with the HDAMA MBs in the storage boxes, the other three of which I have on hand, but I may replace with a Sparkle FSP350-601u – which can be had on ebay for less than 20 bucks

Infrastructure

• 16+ port Gigabit managed switch, that supports VLAN tagging (I’ve seen some of these on ebay for ~$50 in the past few weeks.  I am budgeting approximately $75 + shipping)
• 8+ port 2 GB Fibrechannel switch (budget is $50 + shipping on ebay)
• GBics (market seems to be $5-$10 ea. on eBay)

So the total infrastructure cost is less than $800 – less if you already own some of the hardware.

Software stack

Storage

• SCST will be installed on each node to load an Emulex FC target driver to share out disk resources on the SAN Fabric
• drbd 8.3 (in dual primary configuration) will be installed to perform replication of the disk between the two storage nodes
  • drbd will be using the Qlogic HBAs using an older driver with TCP/IP support for replication
• pacemaker will be installed (most likely) to help control drbd and to control split-brain and act as STONITH
• The 250 GB drives will be configured in a software raid 0.
  • Choosing to do a software raid removes dependencies on hardware raid controllers and it also will allow me to effectively scale the arrays outward by simply adding more drives.

ESX – I will be using ESXi 4.

Misc

• My current ESX hardware will be repurposed as a physical Forefront Security Gateway 2010 (yeah Technet subscription) system.  One NIC into the cable modem, the other NIC into the Gig switch with all VLANS trunked to it

So what does this give me?  Once this is built out I will have a fully redundant ESX farm.  I will be able to power down either ESX server or either storage server for patching, maintenance, etc without taking down my virtual machines.  The only box at “risk” will be the ISA system.

At some point I’ll drop in appropriately sized UPS system(s) to provide 5-10 minutes+ of backup, although this looks like a pretty sweet solution.

Of the above environment described the only pieces I am missing are the Opteron 270s, the Fibre Switch (and Gbics, although I’m going to try to get them both in one auction if possible), and the gig network switch.  Depending on ebay availiblity I’m looking to have all of the hardware acquired in the next month or so and the entire environment built out shortly after that – although once a few packages arrive this week I should be able to start the actual build out

What do I know about this setup – DRBD works.  I’ve having a blast playing with various failure scenarios and split brain detection.  I feel like I have a setup that is very reliable at picking the right “master” to start sync from in a failure state, but I am going to start looking at pacemaker a bit to see if it makes building some logic into the setup easier.  Otherwise I will probably just code some rough bash scripts to control start-up of DRBD/SCST.

I have a fair bit to learn about the details Fibre Channel, but I’m looking forward to the challenge.

As I said earlier – once I start the build out, or have finalized my configuration – I’ll post a detailed how-to and possible sanitized VMs to work with.

I’m in the process of aquiring all of the hardware I need and will do the write up as I’m installing and configuring that environment.

ETA is mid January.  If you would like a copy of my notes in the mean time please visit the contact page and drop me a note (or leave a comment).

If so, please comment or visit the contact page.

If you notice anything out of place, please drop me a note.

I don’t care if that information is about Triathlons, Computers, Home Theater, or narwhal’s – I feel strongly that when you know something better then someone else you shouldn’t hold it above them, or hoard the knowledge, but share it.  Karma is a cold hearted mofo if you ask me.

Maybe that makes me a bit of an idealist, and/or a bit naieve, but I think Jesus summed it up best when he said “Give a man a fish and feed him for a day; teach a man to fish and feed him for a lifetime.”

Anyways, what was I getting at – yes, my blog.  I will continue to do my best to post often, and with quality – but the posts will be a collection of my thoughts about real situations I encounter in my day-to-day computer shenannigans - a sudo cat /var/log/brain > blog if you will.  I can’t promise it will be easy to understand, and I definitely don’t promise anything I post will be a turn key solution, but I will promise to answer any questions I generate (publicly of course), and admit when I don’t know something, and point you at somebody I think might.