Fail to detect VMEM, but OK if rename to RAW. Why?

[Bridgey theGeek]

Hi Rekallites,

I feel like I'm missing something.

I have a suspended Win7SP1x86 VMWare machine. The VMSS file is alongside the VMEM file in the VM's folder.

When I try and run the pslist plugin against the VMEM it fails to find a profile:
$ rekal -v -f Win7SP1x86/Win7SP1x86-9928d7af.vmem pslist
<snip>
INFO:rekall.1:Autodetected physical address space VMemAddressSpace
WARNING:rekall.1:Cache directory inaccessible. Disabling.
DEBUG:rekall.1:Will detect profile using these Detectors: nt_index,osx,pe,windows_kernel_file,rsds,ntfs,linux
DEBUG:rekall.1:Opened url https://raw.githubusercontent.com/google/rekall-profiles/master/v1.0/inventory.gz
DEBUG:rekall.1:Opened url http://profiles.rekall-forensic.com/v1.0/inventory.gz
WARNING:rekall.1:Inventory for repository "/home/btg/.rekall_cache" seems malformed. Are you behind a captive portal or proxy? If this is a custom repository, did you forget to create an inventory? You must use the tools/profiles/build_profile_repo.py tool with the --inventory flag.
DEBUG:rekall.1:Opened url https://raw.githubusercontent.com/google/rekall-profiles/master/v1.0/nt/eprocess_index.gz
DEBUG:rekall.1:Adding nt/eprocess_index to local cache.
INFO:rekall.1:Loaded profile nt/eprocess_index from Local Cache Directory:/home/btg/.rekall_cache
DEBUG:rekall.1:Opened url https://raw.githubusercontent.com/google/rekall-profiles/master/v1.0/nt/index.gz
DEBUG:rekall.1:Adding nt/index to local cache.
INFO:rekall.1:Loaded profile nt/index from Local Cache Directory:/home/btg/.rekall_cache
DEBUG:rekall.1:Opened url https://raw.githubusercontent.com/google/rekall-profiles/master/v1.0/OSX/index.gz
DEBUG:rekall.1:Adding OSX/index to local cache.
INFO:rekall.1:Loaded profile OSX/index from Local Cache Directory:/home/btg/.rekall_cache
DEBUG:rekall.1:Opened url https://raw.githubusercontent.com/google/rekall-profiles/master/v1.0/pe.gz
DEBUG:rekall.1:Adding pe to local cache.
INFO:rekall.1:Loaded profile pe from Local Cache Directory:/home/btg/.rekall_cache
DEBUG:rekall.1:Opened url https://raw.githubusercontent.com/google/rekall-profiles/master/v1.0/ntfs.gz
DEBUG:rekall.1:Adding ntfs to local cache.
INFO:rekall.1:Loaded profile ntfs from Local Cache Directory:/home/btg/.rekall_cache
ERROR:rekall.1:No profiles match this image. Try specifying manually.
<snip>

However, if I rename the VMEM file to a RAW file, Rekall is happy:
$ rekal -v -f Win7SP1x86/Win7SP1x86-9928d7af.raw pslist
<snip>
INFO:rekall.1:Autodetected physical address space FileAddressSpace
WARNING:rekall.1:Cache directory inaccessible. Disabling.
DEBUG:rekall.1:Will detect profile using these Detectors: nt_index,osx,pe,windows_kernel_file,rsds,ntfs,linux
DEBUG:rekall.1:Opened url https://raw.githubusercontent.com/google/rekall-profiles/master/v1.0/inventory.gz
DEBUG:rekall.1:Opened url http://profiles.rekall-forensic.com/v1.0/inventory.gz
WARNING:rekall.1:Inventory for repository "/home/btg/.rekall_cache" seems malformed. Are you behind a captive portal or proxy? If this is a custom repository, did you forget to create an inventory? You must use the tools/profiles/build_profile_repo.py tool with the --inventory flag.
DEBUG:rekall.1:Opened url https://raw.githubusercontent.com/google/rekall-profiles/master/v1.0/nt/eprocess_index.gz
DEBUG:rekall.1:Adding nt/eprocess_index to local cache.
INFO:rekall.1:Loaded profile nt/eprocess_index from Local Cache Directory:/home/btg/.rekall_cache
DEBUG:rekall.1:Opened url https://raw.githubusercontent.com/google/rekall-profiles/master/v1.0/nt/index.gz
DEBUG:rekall.1:Adding nt/index to local cache.
INFO:rekall.1:Loaded profile nt/index from Local Cache Directory:/home/btg/.rekall_cache
DEBUG:rekall.1:Opened url https://raw.githubusercontent.com/google/rekall-profiles/master/v1.0/OSX/index.gz
DEBUG:rekall.1:Adding OSX/index to local cache.
INFO:rekall.1:Loaded profile OSX/index from Local Cache Directory:/home/btg/.rekall_cache
DEBUG:rekall.1:Opened url https://raw.githubusercontent.com/google/rekall-profiles/master/v1.0/pe.gz
DEBUG:rekall.1:Adding pe to local cache.
INFO:rekall.1:Loaded profile pe from Local Cache Directory:/home/btg/.rekall_cache
DEBUG:rekall.1:Opened url https://raw.githubusercontent.com/google/rekall-profiles/master/v1.0/ntfs.gz
DEBUG:rekall.1:Adding ntfs to local cache.
INFO:rekall.1:Loaded profile ntfs from Local Cache Directory:/home/btg/.rekall_cache
DEBUG:rekall.1:Opened url https://raw.githubusercontent.com/google/rekall-profiles/master/v1.0/nt/GUID/684DA42A30CC450F81C535B4D18944B12.gz
DEBUG:rekall.1:Adding nt/GUID/684DA42A30CC450F81C535B4D18944B12 to local cache.
INFO:rekall.1:Loaded profile nt/GUID/684DA42A30CC450F81C535B4D18944B12 from Local Cache Directory:/home/btg/.rekall_cache
DEBUG:rekall.1:Found _EPROCESS @ 0x2980640 (DTB: 0x185000)
INFO:rekall.1:Detected ntkrpamp.pdb with GUID 684DA42A30CC450F81C535B4D18944B12
DEBUG:rekall.1:Detection method rsds worked at offset 0x296903c
<snip>

In the two runs, Rekall downloads a different nt profile. In the second run, it downloads a specific GUID which works.

How come it works as a RAW, but not as a VMEM?
Why is Rekall deciding to behave differently?

Thanks!

[Michael Cohen]

Rekall's support for vmem files is based on the reverse engineered
address space from volatiltiy. Vmware seems to have two files vmem and
vmss which contains the metadata. So in the case where the filename
ends with vmem, Rekall tries to parse the vmss. It is possible that we
dont parse it quite right. It is difficult to tell without the actual
file (can you share it?).

Parsing vmem files is probably not a strong focus for us because we
dont use vmware that much. Happy to fix bugs and include patches
though :-)

Thanks
Michael.

> --
> You received this message because you are subscribed to the Google Groups
> "rekall-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to rekall-discus...@googlegroups.com.
> To post to this group, send email to rekall-...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Bridgey theGeek]

Hi Michael,

Thanks for the speedy reply.

Further reading shows that a VMEM might be a collection of runs (or one giant run) or might just be the raw memory.
(See https://www.vmware.com/support/ws55/doc/ws_learning_files_in_a_vm.html)

On comparison with Volatility, it looks like the VMemAddressSpace class was missing a sanity check on the 'regionsCount' tag.
Even if regionsCount was 0, it still created a list of runs - they just happened to be empty, hence everything ended up falling over.

I've updated my local copy so that it now does this:
IF has_regions THEN:
  populate_runs
ELSE:
  IF has_one_giant_memory_run THEN:
    populate_one_giant_run
  ELSE:
    reject_profile
  END IF
END IF

Now, the vmem is correctly rejected as a VmemAddress space and is picked up by FileAddressSpace later on.

I'm not going to do a pull request until I've had a chance to test it with some "proper" VMEM/VMSS files :)
However, I have attached my diff file in case anybody wants to play.

Thanks again.

[Bridgey theGeek]

Ok, so embarrassingly I had the files the wrong way around when I created the diff.
Correct diff attached.

[Michael Cohen]

Thanks for this. We will incorporate that soon. (On vacation at the moment)

Am 04.10.2015 10:52 nachm. schrieb "Bridgey theGeek" <bridgey...@gmail.com>:

Ok, so embarrassingly I had the files the wrong way around when I created the diff.
Correct diff attached.

--

[Sebastien Bourdon-Richard]

Hey Guys,

Just a quick comment about the adaptation in Rekall (I don't have the time to submit a patch, sorry). VMSS files contains metadata when Vmware workstation is suspended. However, if you create a snapshot with vmware, the metadata will be saved in a VMSN file. 

I suggest there could be a modification here to support also .vmsn extension:

https://github.com/google/rekall/blob/master/rekall-core/rekall/plugins/addrspaces/vmem.py#L46

Regards,

Sébastien