Hyper-V Could Not Initialize – Could Not Create or Access Saved State File

As part of my relaxing holiday, I spent a fair bit of time upgrading the hardware in my lab and installing Windows Server 2012 R2 onto all of my Hyper-V hosts.  I then went through and pruned out some old test VMs and made sure the ones I had left were still relevant.

After I did the upgrade to 2012 R2 and powered on some machines that had been dormant for a few months, actually quite a few months (years in some cases), I got some errors when powering some machines on.

Update 30-4-2014: Added clarification that icacls.exe should be executed in cmd prompt session and not PowerShell.

The KB has article Hyper-V virtual machine may not start, and you receive a “‘General access denied error’ (0x80070005)” error message covers the scenario of missing permissions to .vhd files.  I saw a similar thing but with the .vsv and .bin files.  On a side note you may or may not see the pre-created .bin file with newer Hyper-V versions.  Back to the issue, what was going on?

The symptoms that I observed were that:

• The VM would import successfully
• Powering on would result in an error  - could not create or access saved state file
• Error 3080 was logged into the Hyper-V Worker event log

An example of the error is shown below:

PS C:> Start-VM Typhoon

Start-VM : 'Typhoon' could not initialize. (Virtual machine ID 5BEF5A39-069D-4887-8688-8D80A505A88C)
'Typhoon' could not create or access saved state file E:ConfigsTyphoonTyphoonVirtual
Machines5BEF5A39-069D-4887-8688-8D80A505A88C5BEF5A39-069D-4887-8688-8D80A505A88C.vsv. (Virtual machine ID 5BEF5A39-069D-4887-8688-8D80A505A88C)
You do not have permission to perform the operation. Contact your administrator if you believe you should have permission to perform this operation.
At line:1 char:1
+ Start-VM Typhoon
+ ~~~~~~~~~~~~~~~~
+ CategoryInfo          : PermissionDenied: (Microsoft.HyperV.PowerShell.VMTask:VMTask) [Start-VM], Virtualization
OperationFailedException
+ FullyQualifiedErrorId : AccessDenied,Microsoft.HyperV.PowerShell.Commands.StartVMCommand

Fortunately this was quick to fix, along the same line as the aforementioned KB.

Service SID

Before we dive in and correct the issue one thing worth mentioning is around the underlying Windows feature that Hyper-V uses –  per service  security identifier (SID).  Windows Server 2008 introduced the concept of the service SID to further strengthen windows services and to provide even more granularity when applying permissions.  You can read more about them on the askperf blog.   The service SID for a Hyper-V VM is made up of two parts.  The identifier  NT VIRTUAL MACHINE and then the GUID of the VM.   For example:

NT VIRTUAL MACHINE5BEF5A39-069D-4887-8688-8D80A505A88C

This is the security context that is used to access the various files that make up the VM.  The VM Worker Process will leverage this to work with the files.  To see this we can open up task manager and on the details tab see the GUID listed in the user name field:

Granting Permissions To The Service SID

We will use ICacls.exe to add the service SID ACE entry to the .bin and .vsv files.

We need to know the service SID, so take the GUID of the VM and add that to “NT VIRTUAL MACHINE”  -- note that there is a back slashbetween the two.  This in essence becomes the user name that will be granted the permissions:

Example:  NT VIRTUAL MACHINE5BEF5A39-069D-4887-8688-8D80A505A88C

Based off the error message we know that we need to add permissions to the .bin and .vsv files.   The syntax used was:

ICacls.exe 5BEF5A39-069D-4887-8688-8D80A505A88C.bin /grant "NT VIRTUAL MACHINE5BEF5A39-069D-4887-8688-8D80A505A88C":(F)

ICacls.exe 5BEF5A39-069D-4887-8688-8D80A505A88C.vsv /grant "NT VIRTUAL MACHINE5BEF5A39-069D-4887-8688-8D80A505A88C":(F)

Note that the above lines will wrap, and that they are a sample.  You will need to adjust to match your GUID, it will be different!  That’s the whole point of a GUID!

If the permissions are correctly set then it will say that each file was successfully processed as per the below:

One the NTFS permissions have been changed, power on the VM and you should be good to go!

On a parting note, Ben Armstrong has a post detailing the layout of a VM and the purpose of each file.  Well worth subscribing to his RSS feed!

ICACLS Error

Added 30-4-2014.  Added this section to point out that the above screen captures are in essence cmd prompt windows.

If you try and run icacls.exe in a PowerShell session then you will probably get this error:

F : The term 'F' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:146
+ ... AC970011E8F5":(F)
+                    ~
+ CategoryInfo          : ObjectNotFound: (F:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException

To illustrate this, let’s fire up a standard PowerShell window, and then change it to a cmd prompt.  “This is PowerShell” is written to the screen to prove that initially this is PowerShell.  We then switch to cmd prompt and use ECHO to write the next message to the screen.

Note that the prompt indicator changes from  “PS C:Users>” to “C:users>”.  To illustrate this the above screenshot has the two prompts highlighted, note the difference….

Cheers,

Rhoderick