Write APIs are not working on KVM

Skip to first unread message

Sofia Belikovetsky

Oct 4, 2017, 2:32:43 AM10/4/17
to vmitools
Hi guys, 

I wrote a test program to try out the write APIs and they do not work even though I get VMI_SUCCESS after the execution. 
I have two issues: 
First, when I try to extract EIP and ESP (on a 32 machine) I always get the same values - which is weird.
This is the code I'm using: 

    reg_t esp, eip, cr3;
    status_t retVal = vmi_get_vcpureg(vmi, &esp, RSP, 0);
    printf("After esp %8lx %d\n", (unsigned long) esp, retVal);
    retVal = vmi_get_vcpureg(vmi, &eip, RIP, 0);
    printf("After eip %8lx %d\n", (unsigned long) eip, retVal);
    vmi_pid_t current_pid = -1;
    retVal = vmi_get_vcpureg(vmi, &cr3, CR3, 0);
    printf("After cr3 %8lx %d\n", (unsigned long) cr3, retVal);
    current_pid = vmi_dtb_to_pid(vmi, cr3);

Second, when I try to write to memory, it does not change the memory. 
Code Sample: 

uint32_t value; 
addr_t test_addr = esp - 4; 
status_t result = vmi_read_32_va(vmi, test_addr, current_pid, &value);
printf("Reading the value: %8x with result %d\n", value, (int) result);
value +=1; 
result = vmi_write_32_va(vmi, test_addr, current_pid, &value);
printf("Value that was written %8x with result %d\n", value, (int) result); 
value = 0; 
result = vmi_read_32_va(vmi, test_addr, current_pid, &value);
printf("Read the value again: %8x with result %d\n", value, (int) result);

The value of both reads is the same. 
What am I doing wrong? (I have initialized with VMI_INIT_COMPLETE). 
Is it because I'm trying to write to the stack? 

Here is the output without debug: 

Process listing for VM ubuntu32 (id=5)
Beginning of the execute process
After esp c1a75f70 0
After eip c10588f5 0
After cr3 33ba1280 0
Reading the value: f71b34c0 with result 0
Value that was written f71b34c1 with result 0
Read the value again: f71b34c0 with result 0
finished vm4

(I'm not sure why I get the nonsense in the middle)

The full output with the debug prints is attached. 

Thank you!


Tamas K Lengyel

Oct 4, 2017, 10:50:13 AM10/4/17
to vmit...@googlegroups.com
Is the VM paused while you are doing the writing?
> --
> You received this message because you are subscribed to the Google Groups
> "vmitools" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vmitools+u...@googlegroups.com.
> To post to this group, send email to vmit...@googlegroups.com.
> Visit this group at https://groups.google.com/group/vmitools.
> For more options, visit https://groups.google.com/d/optout.

Sofia Belikovetsky

Oct 4, 2017, 12:08:01 PM10/4/17
to vmitools
Hi Tamas, 

Yes, of course. I'm using the initialization sequence from the examples: 

  /* initialize the libvmi library */
    if (vmi_init(&vmi, VMI_AUTO | VMI_INIT_COMPLETE, name) == VMI_FAILURE) {
        printf("Failed to init LibVMI library.\n");
        return 1;

    /* pause the vm for consistent memory access */
    if (vmi_pause_vm(vmi) != VMI_SUCCESS) {
        printf("Failed to pause VM\n");
        goto error_exit;
    } // if


Tamas K Lengyel

Oct 4, 2017, 12:22:49 PM10/4/17
to vmit...@googlegroups.com
So this is a bug of the KVM driver. When you read from a memory page
using the current KVM driver the whole page is getting cached by
LibVMI. When you do the write, it probably does get written into
memory, but when you are reading the value again you are reading it
from the cached version, not directly from memory. You will have to do
a cacheflush after the write to see the value written.

In the log you see
--MEMORY cache set 0x1a75000
and then when you do the read again:
--MEMORY cache hit 0x1a75000

A quick fix for it is to do a memory_cache_remove of the page in kvm_put_memory

On Wed, Oct 4, 2017 at 10:08 AM, Sofia Belikovetsky

Sofia Belikovetsky

Oct 4, 2017, 12:25:09 PM10/4/17
to vmitools
Hi Tamas, 

I wouldn't have thought of this. 
I will try it! 

Thank you so much!!!

Tamas K Lengyel

Oct 4, 2017, 12:29:09 PM10/4/17
to vmit...@googlegroups.com

Sofia Belikovetsky

Oct 4, 2017, 12:41:17 PM10/4/17
to vmitools
Thanks! I will

Sofia Belikovetsky

Oct 8, 2017, 4:45:59 AM10/8/17
to vmitools
Hi Tamas, 

Finally tried it out.
First, I got an ERROR that the virtual address I'm trying to remove from the cache is not aligned. 
So just to check this on a 32bit machine, I've performed an "AND" with 0x0ffff000 (and saw that it worked, the page was removed from the cache). 

However, I still get the same result... the value after the write is not changed (the same as it was before). 
In the debug output, you can see that the read after the write doesn't get a cache hit, so the data is fresh. 
Is there a chance that it is not writing? 

I'm attaching the new output file. 

Thank you! 

Sofia Belikovetsky

Nov 13, 2017, 2:20:28 AM11/13/17
to vmitools
Hi All, 

Just wanted to update that the write function did not work because I was using the slow GDB access. I've started using the patch and added the changes that Tamas has suggested and it worked. 

Thank you for your help! 

Steven Maresca

Nov 13, 2017, 10:18:16 AM11/13/17
to vmit...@googlegroups.com
Excellent, thank you for reporting back with the success Sofia!


To unsubscribe from this group and stop receiving emails from it, send an email to vmitools+unsubscribe@googlegroups.com.

Tamas K Lengyel

Nov 13, 2017, 10:27:53 AM11/13/17
to vmit...@googlegroups.com
Good, the fix is already merged to master so you might want to do a pull.



Dec 4, 2017, 9:53:51 AM12/4/17
to vmitools
Hi Sofia,
i just meet the same problem as you .However ,i did not use the GDB access ,i use the QEMU 2.8 which has been patched .Could you tell me how you deal with this problem and what the changes that Tamas has suggested.
Thanks !

在 2017年11月13日星期一 UTC+8下午3:20:28,Sofia Belikovetsky写道:

Sofia Belikovetsky

Dec 10, 2017, 11:24:58 AM12/10/17
to vmitools
Using the QEMU 2.8 is the right way :) 
The solution that Tamas has suggested and added to the main branch is adding: 
memory_cache_remove(vmi, paddr);
to the function kvm_put_memory in kvm.c. Then, it will flush the cache and you'll see your change.

However, this change has not worked for me, I've added the following line instead:
memory_cache_remove(vmi, (phys_address >> vmi->page_shift) << vmi->page_shift);
because the cache stores pages and you need to find the right page to remove. 
I hope i'm not mistaken...

Hope it will help you! 
Reply all
Reply to author
0 new messages