sentania.net

Have you ever been at work or at home working on a computer, entered a command (or otherwise take a critical action) – and as your finger is releasing pressure off of the <enter> key, or the mouse button you suddenly enter a cold sweat, realize what you’ve just done – and suddenly have a horrendous urge to reach for a garbage can to vomit?

I just had one of those moments.  I rebuilt my ESX server, upgrading it from 3.5 to ESXi 4.0 U1.  That went smooth, I presented my iSCSI LUNS – all smooth.

I than upgraded the virtual hardware on my domain controller.  Then I realized that it’s Server 2008 Enterprise edition, then I recalled that I have AD installed on the D: drive, and finally I remembered this KB article.  Essentially, ESX 4 presents VMDKs to systems as a SAN device, which by default Enterprise edition set Offline, until instructed otherwise.

Welcome to a BSOD loop on my only domain controller.

Fortunately, the system successfully booted to Directory Services Restore Mode – which allowed me to online the D: disk, and successfully reboot my system.

As promised earlier, here is the first installment of the how-to.  Before we get going to far, this series of how-to’s has the following disclaimers

• A basic knowledge of Linux is assumed.  I’ll provide the commands to perform certain activities, but I’ll assume you know how to get a basic Linux install going
• This how-to is written with CentOS 5.4 in mind – any distro will do, but you may need to modify some commands to make it happen.
• This how-to uses virtual machines and is the result of proofing out the concept.  I’m sure that when I build the final servers – I’ll fine tune it a bit more (which will result in an addendum to the how-to )
• This how-to will be done in a progressive fashion, each part will layer another level functionality of onto the configuration.

In Part 1 of the how-to, we will complete the create of two Virtual Machines, install Linux, VMtools, needed storage space, and required software.  At the end of this part, you will have two functional virtual machines that will replicate storage between themselves in a Primary/Secondary fashion, and you will be able to share out the LUN from the primary node via iSCSI.

0.  Most commands below need to be execute with root privilege

1. Create two virtual machines.  I created them on my desktop using Virtual Workstation.  I created the machines with two Network Cards, one of which was bridged over the desktops NIC, the other on vmnet0 (or any other non-bridge network) - to provide a private network connection for the two storage nodes.  Additional – I removed all fluff hardware from the VM (USB, Floppy, etc.)  I attached an 2.5 GB thin provisioned HDD to each machine.
2. Install Linux – I did this by doing a netinstall, and selecting all packages from the install process.  This means I will need to add everything to the system.  The output of df -h on a completed system is below
   • /dev/sda2             1.5G  1.1G  325M  78% /
     /dev/sda4             396M   34M  342M   9% /var
     /dev/sda1              99M   17M   78M  18% /boot
     tmpfs                 189M     0  189M   0% /dev/shm
   • Configure eth0 to communicate on your public network
   • Configure eth1 to communicate on the private VM only network
3. I disabled selinux, I realize this may be a debatable action, however I just don’t understand it enough, and I want to eliminate it as a potential problem.
   • vi /etc/selinux/config
   • change enabled  to disabled  
4. optional - Next we’re going to remove a few pieces of software that I don’t want on the system -
   • echo y | yum remove iptables
   • echo y | yum remove cups-libs
5. Next we are going to install perl and than upgrade the system
   • yum install perl
   • echo y | yum upgrade
   • reboot
6. Install VMtools via your preferred choice – once you have installed it besure to clean up the install files as they take up a lot of hard drive space
7. Now we are going to install all of the software for the core functionality of the system
   • echo y | yum install mdadm
   • echo y | yum install kmod-drbd83
   • echo y | yum install wget
   • echo y | yum install make
   • echo y | yum install gcc
   • echo y | yum install openssl-devel
   • echo y | yum install kernel-devel
   • echo y | yum install patch
8. Download and install ietd
   • wget http://iweb.dl.sourceforge.net/project/iscsitarget/iscsitarget/1.4.19/iscsitarget-1.4.19.tar.gz
   • tar zxvf iscsitarget-1.4.19.tar.gz
   • cd iscsitarget-1.4.19
   • make
   • make install
9. At this point – add an additional two disks two your VM (reboot or rescan the scsi bus to find them)
10. Let’s create a single raid 0 array on the two disks we just added
    • mdadm –create /dev/md/d0 –auto=mdp –level=0 –raid-devices=2 /dev/sdb /dev/sdc
    • mdadm –detail –scan >> /etc/mdadm.conf
    • add DEVICE /dev/sd*to mdadm.conf as the first line of the file
11. Step 10 creates a software raid array that we can partition.  Making the array is very important for future expansion of the array, without disrupting the drbd resource that is on the array.
    • This will be evident down the road when I walk us through dynamically adding additional space to our storage system.
12. Now let’s partition the new array
    1. fdisk /dev/md/d0
    2. n – command to create a new partition
    3. p – create it as a primary partition
    4. 1 – partition number
    5. <enter> – begin at the beginning of the disk
    6. <enter> – end at the end of the disk
    7. w – write the partition table and exit
13. It’s time to create your drbd.conf file.  You can scour the internet and make your own, or you can download mineand modify it to match your hostnames/IP config
14. Create the drbd resource meta data
    • drbdadm create-md r0
15. At this point you should either clone this VM and modify it appropriately for the second node, or repeat all of the above steps on the second VM.  Do as you feel comfortable

At this point we have two identically configured VMs, that have a raid 0 array, drbd installed and configured to write to that array.  The next step is to fire up drbdfor the first time and perform the initial sync between the nodes

1. Before we go any further, let’s set both drbd and iscsi-target to manual startup – this is annoying for a reboot, and makes it unusable in a real sense – but it allows us total control of our test scenarios
   • chkconfig –level 0123456 drbd off
   • chkconfig –level 0123456 iscsi-target off
2. In step 13 above, you should have created your drbd.conf file.  Going forward I will refer to the nodes as I have them named in my example file.  kroker01 is the primary node with kroker02 the secondary.
3. On kroker01 start drbd up and tell it to perform a sync to kroker02 (when it comes online)
   • service drbd start
   • drbdadm — –overwrite-data-of-peer primary r0
4. Check out /proc/drbd to verify that kroker01 is in a WFConnection state, and is the primary
   •  1: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r—-
         ns:10372472 nr:0 dw:0 dr:10380564 al:0 bm:633 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
5. Go ahead and start drbd on the second node; at this point if you cat/proc/drbd you should see connection status as Connected and ro: as Primary/Secondary, with ds:UpToDate/Inconsistent.  You should also see a line in /proc/drbd detailing the status of the resource.  At this point we are safe to direct disk traffic to /dev/drbd1!
   • Please note
6. Setup ietd.conf
   • Again, google is your friend or download my example, making any needed modifications for hostname and the like.
   • service iscsi-target start
7. From any box that has an iscsi-initiator, point it towards the IP/hostname of the primary box (kroker01).
8. After performing a disk rescan on the client host, you should now have a new disk to work with and format.

Play around with stopping and restart drbd on the secondary node, and observe what you see in /var/log/messages, and /proc/drbd.  As the secondary node comes up and down – drbd tells you various information about what it is doing and the status of the secondary disk.

What can I do with this?

Honestly – at this point we have an expensive, highly redundant, but manual raid 1 disk array for you to play with.  Should your primary node fail, you will have a block for block copy of your data on a second node, that you could pretty easily bring up as the primary node to access your data.

So I’ve been talking bragging to my co-workers and friends about my under-construction ESX environment at home.

Currently my environment is pretty simple to consists of a generic system running CentOS 5.4 farming out a few LUNS to my ESX server.

• Pentium 4 2.8 ghz
• Abit IS7-e w/ 1.5 GB RAM
• Dual Port Intel E1000 Nic
• LSILogic MegaRaid 150-4 Sata controller
• 3×250 GB SATA disks in a Raid 5
• A couple misc IDE drives

I have centOS installed onto a 2.5 GB partition, the remainder of the disk (approximately 800 GB usable total) is presented as iSCSI targets using ietd 1.4.19.

 As a side note, until just a few days ago this box was running Openfiler 2.3 with a P3 700 + Soyo SY-7VCA2, but i started having some stability problems with the motherboard/cpu and a MB/CPU swap seems to have made things better.

Said ESX box is currently running 3.5 with some decent hardware. 

• Rioworks/Accelertech/Arima HDAMA Rev. G
• 2xOpteron 248 ( Rev E.)
• 8×1 GB PC1600R Dimms

The ESX box and CentOS system are connected via a crossover cable for iSCSI traffic.

Prior to the split setup between my ESX server and CentOS system, my storage (via the megaraid 150-4) card existed in my ESX box, and due to the need for more storage but not wanting to buy another ESX HCL listed SATA/SCSI card and the needed drives.  By moving it all to an iSCSI system, I removed the need for having to use only certain types of drives.

So what’s next?  I’m glad you asked – the Bowe ESX farm v3.0 will be a 100% availability infrastructure.  Well not truly 100% – I will have some limitations due to internal house electricity, but the [eventual] purchase of appropriately sized UPS capacity will solve that problem.

How do I intend to accomplish this – lots of used hardware.  Some of the hardware I have sitting around from previous spending binges, others have been acquired or will be acquired over the next few weeks via careful ebay shopping.  I will list the price I paid, plan to pay, or would expect to pay as appropriate for my purchase situation (got, getting, had).

ESXi (2x)

• Tyan K8SR (paid 27.50 ea. shipped)
• Dual Opteron 270 (budget 40 per pair shipped – ebay)
• 8x 1GB PC1600R (would pay $10-$15/dimm – bought a bunch of hese a LONG time ago)
• 1x Emulex 9802 HBA (paid $5 shipped – ebay)
• 1x Tyan OOB mangement card (freebie with the K8SR)

Storage (2x)

• Rioworks/Accelertech/Arima HDAMA (rev prior to G)  (paid $20 ea. on ebay)
• Dual Opteron 246HE (freebie CPUs that came with the K8SRs in my host setup.)
• 2x 1 GB PC1600R (see above for pricing)
• 1x Qlogic 2340 HBA (paid 11.50 ea. – ebay)
• 1x Emulex 9802 HBA (paid $5 shipped – ebay)
• Generic SATA Controller ($20 ea.)
• 2×250 GB Sata drives (bought a while ago, but market rate is ~$35 ea.)

All systems will be booting off a 2.5 GB Compact flash microdrive in an IDE adapter – ~50 dollars for 4 drives and adapters on ebay.  All systems also have power 350/400 watt power supplies.  Two of which came with the HDAMA MBs in the storage boxes, the other three of which I have on hand, but I may replace with a Sparkle FSP350-601u – which can be had on ebay for less than 20 bucks

Infrastructure

• 16+ port Gigabit managed switch, that supports VLAN tagging (I’ve seen some of these on ebay for ~$50 in the past few weeks.  I am budgeting approximately $75 + shipping)
• 8+ port 2 GB Fibrechannel switch (budget is $50 + shipping on ebay)
• GBics (market seems to be $5-$10 ea. on eBay)

So the total infrastructure cost is less than $800 – less if you already own some of the hardware.

Software stack

Storage

• SCST will be installed on each node to load an Emulex FC target driver to share out disk resources on the SAN Fabric
• drbd 8.3 (in dual primary configuration) will be installed to perform replication of the disk between the two storage nodes
  • drbd will be using the Qlogic HBAs using an older driver with TCP/IP support for replication
• pacemaker will be installed (most likely) to help control drbd and to control split-brain and act as STONITH
• The 250 GB drives will be configured in a software raid 0.
  • Choosing to do a software raid removes dependencies on hardware raid controllers and it also will allow me to effectively scale the arrays outward by simply adding more drives.

ESX – I will be using ESXi 4.

Misc

• My current ESX hardware will be repurposed as a physical Forefront Security Gateway 2010 (yeah Technet subscription) system.  One NIC into the cable modem, the other NIC into the Gig switch with all VLANS trunked to it

So what does this give me?  Once this is built out I will have a fully redundant ESX farm.  I will be able to power down either ESX server or either storage server for patching, maintenance, etc without taking down my virtual machines.  The only box at “risk” will be the ISA system.

At some point I’ll drop in appropriately sized UPS system(s) to provide 5-10 minutes+ of backup, although this looks like a pretty sweet solution.

Of the above environment described the only pieces I am missing are the Opteron 270s, the Fibre Switch (and Gbics, although I’m going to try to get them both in one auction if possible), and the gig network switch.  Depending on ebay availiblity I’m looking to have all of the hardware acquired in the next month or so and the entire environment built out shortly after that – although once a few packages arrive this week I should be able to start the actual build out

What do I know about this setup – DRBD works.  I’ve having a blast playing with various failure scenarios and split brain detection.  I feel like I have a setup that is very reliable at picking the right “master” to start sync from in a failure state, but I am going to start looking at pacemaker a bit to see if it makes building some logic into the setup easier.  Otherwise I will probably just code some rough bash scripts to control start-up of DRBD/SCST.

I have a fair bit to learn about the details Fibre Channel, but I’m looking forward to the challenge.

As I said earlier – once I start the build out, or have finalized my configuration – I’ll post a detailed how-to and possible sanitized VMs to work with.

Over the next few weeks I’ll be writing a how-to for setting up an Active/Active iSCSI storage solution for VMware ESX.

My solution will entail:

2xiSCSI servers running CentOS 5.x, DRBD 8.3,  ietd 1.4.19

2xVMware ESXi Boxes

vCenter

The goal is to setup a infrastructure that will survive everything except a simeloutanous power ouage at my house.  In theory I should be able to take either any of the boxes down for maintenance without having to shut my VMs down.  That is probably my biggest annoyance about my setup at home – the need to shutdown everything down to patch.

I’m in the process of constructing the environment with VMs to proof it out.

I am pretty confident that this will work – I just want to proof it out before dropping $$$ on storage.

Inspiration for this:

http://blog.core-it.com.au/?p=62

Here is a good follow up on my post on snapshots and disk expansions

The post references the following VMware KB Articles

VMware KB 1007849

 VMware KB 1004232

It just so happened that today I was browsing the VMware blog for the Windows VI Toolkit, and came across a post the described in a fair amount of detail why get-harddisk was slow for me.

Looks like the culprit is that I typically use get-view to pull my information out of the API, and as such I had to pass a VM name into get-harddisk, which results in some extremely slow object name translation voodoo.

This:

get-harddisk -vm (get-vm -name s092464).name

was over a minute slower than this:

get-harddisk -vm (get-vm -name s092464).

So imagine that times 1200.

That said, I think my script will still stay with the method I wrote about last week, simply because get-view suits my data needs better, plus it’s WAY faster:

get-vm -name s092464 = 2 minutes

get-view -viewtype virtualmachine -filter @{“Name” = “s092464″} = 5 seconds

I have been working a lot with the Windows VI Toolkit lately, it is a very straight forward way of working with your Virtual Infrastructure programmatically.

One of the things I have noticed is that there are a few cmdlets, that are exceedingly slow.  These cmdlets aren’t so bad to use if you are working with a single VM, but if you start doing some scripting against hundreds or thousands of machines, whoa boy – I mean hit enter, go make a cup of coffee, drink it, drop the kids off at the pool, and you’ll still have time to spare.

Thus far the worst cmdlets I’ve come across are get-vm and get-harddisk.  get-vmhost, and get-cluster are pretty pokey as well, but fortunately even in a large environment, there aren’t that many hosts and there are even fewer clusters so they don’t annoy me with their pokeyness much.  Yet.

Here’s an example of how slow these cmdlets are, I have a script that I wrote that looks at my VI, and grabs every single VM that fits a certain naming pattern, which business rules dictate are VDI instances.  Next, I take the VM collection, and pull some interesting data: name, memory allocation, guestOS, IP, VLAN (portgroup), and VMDK size.  I knew from the get go that get-vm was slow, so I was using get-view.  get-view doesn’t return quite as friendly an object, but if you are willing to explore, it has everything you need.

So my collection of ~1200 VMs, comes back, and I do a little dance with it, then I get the portgroup:

$tempvm.vlan = (Get-View -Id $vm.network).name fairly speedy return

Then I get the VMDK size:

get-harddisk -VM $vm.name | foreach-object -process {$tempvm.diskGB += $_.CapacityKB}
    $tempvm.diskGB = [math]::round(($tempvm.diskGB / 1048576),0)
    if ($tempvm.diskGB -eq 0)
    {
        $tempvm.diskGB = “N-A”
    }

Let’s jsut say you can go take a nice nap, but it returns the total size of the VMDKs attached to the system.

I then scheduled this script to run via AutoSys, and it happily generated a CSV of all my VDI instances, and the info I wanted – in 35 hours!

I was a bit perturbed by the run time, but figured it was a batch job, and I could adjust the start times, and I’d just live with it.

Not any longer!  Today I had an epiphany of sorts while exploring the get-view object returned for a VM – it was if the skies had parted, God spoke to me, while a chorus of Angels sang.  Or I could have seen a Narwhal, I’m not really sure, but it was that cool.

The epiphony caused me to modify two lines of code – the temp.vlan line, and the get-harddisk line – replacing them with:

foreach ($device in ($myvm.config.hardware.device)) {if ($device.backing.devicename -like “VLAN*”) {$tempvm.vlan=$device.backing.devicename}}

and

foreach ($device in ($vm.config.hardware.device)) {if ($device.backing.filename -like “*vmdk”){$tempvm.diskGB += $device.capacityinKB}}

After the changes, I force started my job and waited eagerly for it to complete – 2 minutes and 15 seconds later.  Same result, only 35 hours less of waiting.

And if you don’t believe me, here is the run log from AutoSys

Original Code:

Revised Code:

Not long ago, I posted a powershell script, that would set a standard memory and CPU reservation across an entire virtual center instance.

As cool as that is, let’s make it even cooler, let’s do it as a scheduled task and so you don’t have to enter credentials!

The first thing you need to do is create a text file that contains an encrypted password.

read-host -assecurestring | convertfrom-securestring | out-file C:\securestring.txt

This will store the password you type into a text file, that is only decryptable by the user that ran the script.

Next we make a little modification of the script I posted earlier (the modfied script is below, with updated parts in italics)

add-pssnapin VMware.VimAutomation.Core -erroraction Silentlycontinue
$vcServer=”virtualcenter”
$user=”Cooluser”
$credentialFile=”C:\securestring.txt”
$pass = cat $credentialFile | convertto-securestring
$credentials = new-object -typename System.Management.Automation.PSCredential -argumentlist $user,$pass
$vcConnection=connect-viserver -server $vcServer -credential $credentials -warningaction Silentlycontinue
$vms=get-view -viewtype VirtualMachine -server $vcConnection

$vms | % {$spec = new-object VMware.Vim.VirtualMachineConfigSpec;
        $spec.memoryAllocation = New-Object VMware.Vim.ResourceAllocationInfo;
        $spec.memoryAllocation.Shares = New-Object VMware.Vim.SharesInfo;
        $spec.memoryAllocation.Shares.Level = “normal”;
        $spec.memoryAllocation.Limit = -1;
        $spec.memoryAllocation.Reservation = 512;
        $spec.cpuAllocation = New-Object VMware.Vim.ResourceAllocationInfo;
        $spec.cpuAllocation.reservation = 0
        $spec.cpuallocation.Shares = New-Object VMware.Vim.SharesInfo;
        $spec.cpuAllocation.Shares.Level = “normal”;
        $spec.cpuAllocation.Limit= -1;
        Get-View($_.ReconfigVM_Task($spec))}
       
disconnect-viserver -Confirm:$false

The best part of this script is, if you do run it interactively, it won’t display a certificate warning message, and it will disconnect your VC session – kind of like “Be kind, rewind”

Me, I’m a big fan of standards.  Standard processes, standard this, standard that – standards are a great way to make your life easier when dealing with hundreds or thousands of objects.

Inside of my Virtual Infrastructure, I’m a huge fan of ignoring the micro level of things, and paying attention to things at a macro level.  One of the ways I do that, is by ensuring that every virtual machine is configured with identical reservations – regardless of how much resources they are allocated.  If a given virtual machine (or group of virtual machines) requires a guaranteed amount of resources, I create a resource pool and toss them in there.

Not only does this prevent a “rouge” VM from stealing resources it may not need at a moment in time, it calls out the big hitters and puts them out in the open.

Our virtual machine creation process defines all new virtual machines with our standard reservation, but over the course of weeks and months, these drift.  Machines change, an admin “tests” something and forget’s to change it back, or we might even change the standard.  Here is a script that will locate all virtual machines in your infratructure, and update them to whatever makes you happy.

add-pssnapin VMware.VimAutomation.Core -erroraction Silentlycontinue
$vcServer=”virtualcenter”
$credentials = get-credential
$vcConnection=connect-viserver -server $vcServer -credential $credentials
$vms=get-view -viewtype VirtualMachine -server $vcConnection

$vms | % {$spec = new-object VMware.Vim.VirtualMachineConfigSpec;
        $spec.memoryAllocation = New-Object VMware.Vim.ResourceAllocationInfo;
        $spec.memoryAllocation.Shares = New-Object VMware.Vim.SharesInfo;
        $spec.memoryAllocation.Shares.Level = “normal”;
        $spec.memoryAllocation.Limit = -1;
        $spec.memoryAllocation.Reservation = 512;
        $spec.cpuAllocation = New-Object VMware.Vim.ResourceAllocationInfo;
        $spec.cpuAllocation.reservation = 0
        $spec.cpuallocation.Shares = New-Object VMware.Vim.SharesInfo;
        $spec.cpuAllocation.Shares.Level = “normal”;
        $spec.cpuAllocation.Limit= -1;
        Get-View($_.ReconfigVM_Task($spec))}

A former co-worker of mine recently blogged about trying to do a disk expansion on a server with a snapshot.  As a general rule – this is something you want to avoid doing.  Fortunately, the VI Client is smart enough to know this is a bad thing and will prevent you from doing it.

Unfortunately, the service console is not smart enough, and follows the general Unix rule of, well shucks if you have root level permissions, you must  know what you are doing.

So let’s say you try to expand a VMDK of a Virtual machine with a Snapshot in the VI Client, you’ll get an error message.  There are two different error messages you’ll see (as of VC Build: 119598 – VC 2.5 U3).

If the machine is powered off you’ll get:

If the machine is powered on you’ll get:

Hopefully at this point, you will do the intelligent thing and do what you should have done initially, and Commit (Delete All) or Revert (Rollback) any snapshots this machine has.  If you happen to be less intelligent, and log on to the Service Console, and execute a
vmkfstools -x #gb /vmfs/volumes/datastore1/OMGIRDUMB/OMGIRDUMB.vmdk.  You’ll be pleased to know that your vmdk is now expanded.

What you won’t be so pleased to know is that your virtual machine won’t power on anymore.  Your snapshot for that drive is invalid.  The specific error message you’ll get will be along the lines of “Cannot open the disk /vmfs/volumes/datastore1/OMGIRDUMB/OMGIRDUMB-000001.vmdk or one of the snapshot disk it depends on.”  ROFLMAO, you are officially screwed now!

Haha, okay I am done laughing at your expense and am ready to help you keep your job.

You have two ways to resolve this issue, manually rehome your disks using the VI Client to the base disks and not the deltas; and then use vmware-cmd X removesnapshot and revert the VM to the state when the snapshot was taken, with the added bonus of an enlarged disk.

The better, albiet more complicated option is to fix your retardedness and save the server.  First things first, using the VI client, Delete All (Commit) snapshots.  This task will progress, but it will generate an error (it is likely to be the “Internal error” viewed above).

At this point you need to verify that VC believes that this machine has no snapshots, and that all VMDKs except the one you attempted to expand, point to the base disk.  This ensures that all the other drives in the system are up to the current state, and the snapshots state of the VM reflects this.

Next, you need to log back on to the Service console.  cd to the home directory of the virtual machine.  Again, verify that the -delta files for all VMDKs have been commited, except for your naughty vmdk.

A little background.  When you have a chain of vmdk’s, you can use vmkfstools -i child.vmdk target.vmdk, to consolidate the chain of vmdks into one disk.  We will be using this command to fix our problem.  If you attempt this straight-away, you will get an error message that ranges from “cannot read beyond end of disk”, to “parent VMDK has been modified”

In order to fix this problem we need to make sure that the parent VMDK and all the children involved are correctly tied together.  If you recall, last month, I wrote an entry that described the contents of a VMDK.  The entries we care about today is the parentCID value, and the Extent Description.

Start off by doing a cat /vmfs/volumes/OMGIRDUMB/OMGIRDUMB-000001.vmdk, and make note of the parentCID value, and the number that comes after RW in the Extent Description section.  Now, do vi /vmfs/volumes/OMGIRDUMB/OMGIRDUMB.vmdk, and ensure that the CID matches the CID of the child.  Also, update the value in this extent description to match the value in the -000001.vmdk.

At this point you should be able to successfully execute vmkfstools -i, and generate a new base disk.  You’ll now want to execute a vmkfstools -x #gb vmdk, to update the extent description field.  Finally, you will want to add this new disk to your VM, and remove the disk that points to the old delta chain.

Power your VM on, and verify that you have your data.  You may encoutner some issues with the filesystem inside of this disk (i.e. the disk will be 100 GB, but the file system will report it’s former size, and not be able to be modified); so I strongly recommend that you do a backup/restore to a new vmdk, or robocopy or something.

Once you have successfully recovered the data, clean up your vmdk mess so you don’t get confused.

Hopefully this helps you out some day, if you ever need help in a situation like this, or have questions about this process, feel free to contact me.