Restoring Windows 2003 Servers via Image based backups

We have several customers on our Managed Backup product that have Windows 2003 servers that they do not intend to upgrade or replace with newer operating systems.  

We always help our customers, so when this customer had a virtual host fail, we sprang into action.  All guest servers were restored into the recovery environment via StorageCraft quickly – with the exception of the Windows 2003 server.  It was fully patched, of course, but would not boot.  The error looked like so:

STOP: c0000135 {Unable To Locate Component}This application has failed to start because CSRSRV.dll was not found. Re-installing the application may fix this problem.

The error is documented in the Microsoft Knowledgebase here:https://support.storagecraft.com/s/article/c0000135-CSRSRV-dll-was-not-found-when-restoring-Server-2003?language=en_US

To successfully boot the server, you must uninstall KB3045999.  But how do you uninstall a patch when you can’t boot the server?

I had 100% success with the below procedure, which replaces the patched files with the originals and will enable the server to boot..

  1. From any server that can mount Storagecraft images as volumes, do a StorageCraft mount of the Target server’s boot disk image on Z:
  2. Open an Administrative Cmd Shell.  It has to be an administrative shell to perform the copy.  
  3. Move to z:\Windows\$NTUninstallKB3045999\spuninst . This is the uninstall information and original files for the machine you are restoring.
  4. Open spuninst.txt in Notepad.
    1. Use Notepad’s find/replace function to replace all instances of “C:” with “Z:”.
    2. Save as spuninst.cmd.  
    3. Close Notepad.
    4. Run the spuninst.cmd script.  This should replace the patched files with their originals.
    5. Dismount the Z: drive, saving the changes to a new patch file.  DO NOT USE QUICK DISMOUNT.  A quick dismount will not give you the option to save the disk as a patch file.
  5. Move to the Patch file – right click on it and select “Virtual Boot.”
  6. From the Wizard, add any other drives that need to be added to the Virtual Machine.  This step is not necessary when using the regular Storagecraft chain, but is necessary when using the “patch file.”
  7. Before you boot the virtual machine – make sure you heed VirtualBox’s warning about the minimum of 64 Mbytes of Video RAM.  Otherwise, the machine will crash when booting.
  8. Boot the Virtual Machine. The VM will perform a chkdisk on all disks.  Probably good to allow Windows to finish the chkdisk.

The server should now boot normally.

A Quick Windows DNS Configuration Script

During a domain migration I had to update the DNS servers for all computers on the domain.  Our domain has hundreds of computers, o on different subnets – many subnets have a DHCP server that supplies addresses for machines created dynamically.  

The new DNS servers were on a special subnet dedicated to authentication  protocols Kerberos, LDAP, ADFS and DNS.   We could not just maintain the existing IP addresses for the DNS servers.

We ran it using our management tool on every machine in our data center.

  1. Changes the DNS settings given out by the DHCP server (this line fails silently if the server is not the DHCP server), 
  2. Changes the Ethernet interface’s DNS settings,
  3. It should run on servers Windows 2003 and above, so could be applied elsewhere.
  4. Disables IPv6 tunnel interfaces.  Ok, while this was not strictly necessary to the project, it made the ipconfig command much cleaner.
echo off
:: New DNS Servers
set DNS1=10.13.100.101
set DNS2=10.13.100.102
 
:: Disable IPv6 Tunnels we don't use:
netsh interface teredo set state disabled
netsh int ipv6 isatap set state disabled
netsh int ipv6 6to4 set state disabled
 
:: Change the DNS Servers that DHCP Gives out.:
netsh Dhcp Server \\localhost set optionvalue 6 IPADDRESS %DNS1% %DNS2%
 
:: Change Primary and Secondary DNS for all Enabled Interfaces
:: Found this on the Internet!
for /f "tokens=1,2,3*" %%i in ('netsh int show interface') do (
    if %%i equ Enabled (
        echo Changing "%%l" : %DNS1% + %DNS2%
        netsh int ipv4 set dns name="%%l" static %DNS1% primary validate=no
        netsh int ipv4 add dns name="%%l" %DNS2% index=2 validate=no
    )
)
ipconfig /flushdns
 
:EOF

Change Monitoring on the Network…

I have experience with quite a few change monitoring systems on the network.  I am distinguishing change monitoring vs change management in that “management” always implies some measure of “control.”

Some quick Pros and Cons of various Systems:

For Active Directory, I have used:

  • ManageEngine AD Audit+: This is a web based application that can monitor and alert on changes to Active Directory objects, including Group Policy Objects.  It can send regular reports that will list the principle who made the change, the object that was changed, and,both the before and after values. The AD specific application is priced by the number of Domain Controllers in your environment.  This can be cheaper for environments with thousands of objects, but fewdomain controllers.
  • Lepide AD Auditor:  Lepide’s offering is a thick client application that connects to each Domain Controller.  It has all the same features as AD Audit + for monitoring Active Directory objects.  It is priced based on the number of objects in your environment, so may be cheaper for smaller environments with under 100-ish bbjects.
  • Netwrx:  Poor User Interface, scalable, and costs as if it were made of gold.  But an add-on module includes vSphere monitoring, which we found very useful.

Message Analyzer Note

How to get a packet capture on a Windows Machine that does not have Message Analyser, Network Monitor, or Wireshark installed on it:

netsh trace start capture=yes Ethernet.Type=IPv4 IPv4.DestinationAddress=<local ip address>

This will create a trace file nettrace.etl by default.

ETL files can be read with Network Monitor 3.4 (My favorite for its simplicity, and the fact that I have been using it so long), Event Viewer, PerfView, and the most comprehensive application: Windows Performance Analyzer.