Tag: repair

The redo log is corrupted. If the problem persists, discard the redo log.


Yesterday about an hour before the end of my work day one of our critical servers fell over and was displaying the following message in the vSphere client.

The redo log of VisualSVNServer_1-000001.vmdk is corrupted. If the problem persists, discard the redo log.

The error message refers to a redo log, but this is legacy VMware terminology. VMware have from ESXi 3.1 started to use the term snapshot to mean the same thing but for some reason the error messages still use the old term.

The server was named Subversion and was a VisualSVN Server.

There was a snapshot dated from 15th December 2013 in the Snapshot manager for the Subversion VM so returning to this snapshot would have meant returning to a point several weeks ago and then trying to import the backup of the repository that was made the night of 29th January.

The underlying cause of the corruption cannot be definitively determined but I think was due to the amount of disk activity on the physical disk that constitutes datastore 3_2 on the host server S003-ESXi. This caused the system to fail to write to the log and to create updated delta disks which contain all the changes to the disks since the point of the snapshot.

I believe that if there had not been a snapshot the data corruption probably wouldn’t have happened. I have since educated staff that taking snapshots in vSphere is really not the same as backing up the server and they shouldn’t be doing it on the Subversion server at all.

I resolved the issue with Subversion by carrying out the following steps.

I clicked OK to the error message in the slim hope that the VM could overcome the glitch itself upon a simple reboot.

This didn’t work. So I started the process of backing up the VM by forcing a shutdown of the machine by virtually cutting off the power and then making a copy of the virtual machine folder on the datastore.

Whilst the copy process was going I checked Virtual Machine Logs, vmware-3.log was completely corrupt and the vmware.log was showing some corruption.

The copy process took over an hour as it was 150GB in total size. Mostly due to the two virtual disks the first VisualSVNServer.vmdk which constitutes the C: drive of the server is 40GB and the second VisualSVNServer_1.vmdk which is the E: drive is 100GB.

Having made a copy of everything I attempted to fix the snapshots. I made sure that there was sufficient space on the datastore and then using Snapshot Manager in vSphere created a new snapshot of the Subversion VM.

This operation was successful, so I then tried to commit the changes and to consolidate the disks. This worked for VisualSVNServer.vmdk merging all the changes, but not entirely for VisualSVNServer_1.vmdk, however it did reduce the size of the delta disks significantly meaning that there was likely to be only minimal data lost.

Nothing more could be done through the vSphere client so I then started a process of trying to manually consolidate the following disks into a single disk.

Enabled SSH on the host server s003-esxi.

Using PuTTY I logged into the command line of the host and changed the directory to the relevant directory that contained the virtual machine files for Subversion /vmfs/volumes/Datastore3_2/VisualSVNServer

Then ran the command ls *.vmdk –lrt to display all virtual disk components.

Then starting with the highest number snapshot ran the following command to clone the disk in a way that would merge the delta disks into a copy of the main disk.

vmkfstools –i VisualSVNServer_1-000002.vmdk VisualSVNServer-Recovered_1.vmdk

This process took another hour or so as it was trying to create a 100GB file.

This failed with the following error message displayed:

Failed to clone disk: Bad File descriptor (589833)

Then starting with the next highest number snapshot I ran command to clone the disk without the most recent changes.

vmkfstools –i VisualSVNServer_1-000001.vmdk VisualSVNServer-Recovered_1.vmdk

This process again took about hour as again it was trying to create a 100GB file.

Again this failed with the following error message displayed:
Failed to clone disk: Bad File descriptor (589833)

Abandoned the idea of merging the disks I removed the VM from the inventory in vSphere and then moved all but the following files into a separate folder.

I could then recreate the VM from these files. I downloaded the file VisualSVNServer.vmx which is the virtual machine’s configuration file and stores the settings regarding the virtual devices that make up a virtual machine. I edited the file to change all references to VisualSVNServer_1-000002.vmdk to VisualSVNServer_1.vmdk so that the machine could be booted up ignoring the delta disks and any data they might contain.

Added the VM back into the inventory and then booted up the machine. It booted up fine, checked the E: drive and there appeared to be data written to the disk all the way up to the time that the server fell over so it appeared that there was minimal if any data lost.

Thanks to XtraVirt for the necessary steps.

Bringing a laptop back from death

My responsibilities in this job are exceptionally broad so although I’m the systems administrator with overall responsibility for the company’s IT infrastructure I’m also on occasion called upon to do things such as repair a dead laptop.

I could have delegated it to another staff member but in this case it was important that the issue should be resolved quickly. I thrive under pressure, have a real knack for troubleshooting weird computer problems and generally just enjoy getting my hands dirty when it is something that I’d never encountered before.

It was mid-afternoon when I took the call from one of our consultants that works out of the office with customers on their site and he told me that very weird things were happening with his laptop computer. It was reportedly dead, completely unresponsive and would have a strange effect on any laptop charger that was plugged into it, the LED on the charger would turn off whenever plugged into the laptop and would then not work in any other laptop until the mains power had been turned off and on.

This sounded to me like the laptop was shorting out the laptop power supplies, fortunately not permanently as it might have killed off a swath of power supplies of one of our major customers. I asked my colleague to bring it into the office so I could take a look but I wasn’t holding out much hope that it could be fixed.

I ran through some basic logical checks to see if the laptop was behaving as described and that it wasn’t something stupid like the wrong laptop charger was being used.
The charger was indeed the correct one for that model and hadn’t been swapped or mixed up with someone else’s.
Plugged it into the mains and then the laptop and the LED was the extinguished.
Did the charger work with a different laptop, yes but only after cycling the mains electricity.
Did swapping the battery help. No and the battery appeared good in another laptop.

The problem then must lie with the laptop itself and with the power input socket. If we had a spare laptop I’d have pulled the hard disk and installed it in the replacement laptop and given that to my colleague and then stuck the dead laptop on the junk pile. Unfortunately we have no spares and he needed a working laptop for Monday and couldn’t come via the office so needed me to fix then and there if possible.

No choice then but to completely dismantle the laptop so that’s what I did. About 20 screws later I finally had the case apart and I could see the wiring of the power input and theorising it was a short of some sort I examined the wiring. The wiring from the power input socket appeared to be good to me. However there was a metal bracket that kept the power input socket connected to the chassis and by removing that I could see that there was a metal contact on the socket that would then form an electrical connection to the chassis.

The chassis looked dull and therefore might be corroded and preventing the electrical contact that was required (I’d had a similar issue with the starter motor on my car last year). A bit of abrasion on the chassis at the right point to make it nice and shiny I then reassembled the power input assembly and tested it by plugging in the charger before I completely reassembled the laptop. Moment of truth. I switched the mains electricity on and the LED on the charger stayed lit so it wasn’t being shorted out by the laptop any longer. Unplugged the power. Put the laptop back together again. Plugged the power in once more and hit the on button. Lights up. Dell logo shows on screen briefly and then Windows starts its boot up.

An hour and a bit after being given a mysteriously dead laptop I gave back a working machine and all was right with the world once more.

VMDK reconstruction

The office move meant shutting down the VMware hosts and one of the side effects of this was that a couple of the virtual machines that had been happily running didn’t come back up.

One didn’t matter at all so I could safely ignore it, but the other was a demo server used by one of the company directors to show off our software to prospective clients and therefore needed to be working asap.

I was faced with quite a mystery as it seemed to have disappeared completely, it wasn’t listed in the inventory of the host in Vsphere and the datastores connected to the host didn’t contain the expected files either.

However I wasn’t completely at a loss as there was a message in the hosts inventory indicating that something had gone wrong and a virtual machine it had been hosting was missing and I knew that some of our VMs have been renamed and so the corresponding files in the datastores do not have matching names.

So I went looking for some orphan files. I found the files in question but it was just the VMDK files and they seemed to have been missing the VMDK metadata files also.

So in order to rebuild the virtual machine I needed to reconstruct the VMDK files so that I had usable virtual hard disk to the attach a new virtual machine to and given that we are using the free vSphere ESXi 4.0 version this meant using the unsupported Tech Support Mode and instructions from here http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1002511

To use Tech Support Mode:

Log in to your ESXi host at the console.
Press Alt+F1 to switch to the console window.
Enter unsupported to start the Tech Support Mode login process. Note that no text will appear on the console window.
Enter the password for the root user. Tech Support Mode is now active.
Complete tasks in Tech Support Mode.
Enter the command clear to clear the screen of any residual data from step 5. This may be required by your local security policies.
Enter the command exit to exit Tech Support Mode.
Press Alt+F2 to return the server to DCUI mode.

So I logged into the terminal of the ESXi host.

Then to recreate the virtual machine disks I navigated to the directory that contained the virtual machine disks with the missing descriptor file using the following command (having previously found the relevant volume from browsing Storage in the vSphere client on my PC:

cd "/vmfs/volumes/4bfd0ee1-48e6535e-7d30-0026b97ee7d2/CRJ test/"

The instructions then asked me to identify the type of SCSI controller the virtual disk is using by examining the virtual machine configuration file (.vmx). But I didn’t have the .vmx file so I took a look at similar VMs and they used the SCSI controller type lsilogic.

I identified and recorded the exact size of the -flat file using the command:

# ls -l VS030-Srv08Tmpl-flat.vmdk
-rw——- 1 root root 32212254720 May 29 12:30 VS030-Srv08Tmpl-flat.vmdk

Then used the vmkfstools command to create a new virtual disk:

# vmkfstools -c 32212254720 -a lsilogic -d thin temp.vmdk

This command uses these flags:
-c (This is the size of the virtual disk).
-a (Whether the virtual disk was configured to work with BusLogic or LSILogic).
-d thin (This creates the disk in a thin-provisioned format).

Note: To save disk space, the disk was created in a thin-provisioned format using the type thin. The resulting flat file then consumes minimal amounts of space (1MB) instead of immediately assuming the capacity specified with the -c switch. The only consequence, however, is the descriptor file contains an extra line that must be removed manually in a later step.

The files temp.vmdk and temp-flat.vmdk were created as a result.

I deleted the unneeded temp-flat.vmdk using the command:

# rm temp-flat.vmdk

And renamed temp.vmdk to the name that match the orphaned .flat file (or VS030-Srv08Tmpl-flat.vmdk, in my case):

# mv temp.vmdk VS030-Srv08Tmpl-flat.vmdk

Then the descriptor file needed to be edited to match the .flat file:

Under the Extent Description section, change the name of the .flat file to match the orphaned .flat file you have.

Find and remove the line ddb.thinProvisioned = “1″ if the original .vmdk was not a thin disk. If it was, retain this line.

This completed the reconstruction of the first virtual hard disk, I did the same for the second disk and then I was in a position to rebuild the machine.

Building the VM was straightforward and not worth writing in any great detail as it is simply a case of using the wizard in the vSphere client and then selecting “Use Existing” option instead of “Create New” when it gets to the section about virtual hard disks.