[dm-cache] environment configuration

[Dulcardo Arteaga Clavijo]

Hi Lenny and Pim,

We are working on solving the problem that you have when booting domU using the dm-cache.

In the description of your environment you mention that you are using xen4 or 4.1 and kernel 2.6.18 or kernel 3.0.

Which specific version of XEN and which specific kernel causes the problem?  the thing is the dm-cache can not be applied to kernel 3.0 and for kernel 2.6.18 the dm-cache is an older version.

I've tried this configurations:

Xen: 3.4.2

Kernel: 2.6.18

dm-cache: 2.6.19

I found some problems that were fixed in later version of dm-cache.

Xen: 4.0

kernel: 2.6.32

dm-cache: 2.6.39:

This works fine for me, putting the VM image on top of the cache

disk        = [
                 'file:/mnt/cache/node01/disk.img,xvda1,w',
]

where /dev/dm-0 is the cache device mapped, mounted to "/mnt/cache" 

Do you need to use the cache directly as a physical device (as opposed to using it like a file) handled by the VM? as your configuration

 disk = [ 'phy:/dev/mapper/tvps054cache,xvda1,w', 
'phy:/dev/mapper/ivpsvg--001-tvps054swap,xvda2,w' ] 

Thanks and Regards,

Dulcardo 

[Pim van Riezen]

Hi Dulcardo,

We source the hypervisor and xend from the gitco.de repos. Our production systems run the gitco-sourced 4.0.3, I mostly checked out 4.1 to rule out xen version as relevant.

The patches needed some coaching for 3.0, I will take a look tomorrow morning to see if I can reconstruct what I did there. To my recollection, it only needed some minor tweaks for internal kernel API calls.

Applying against the RHEL 2.6.18 kernels was more of an adventure. I moved to 3.0 because I couldn't be sure there if my changes to get it to work were sufficient and that the mismatch wasn't the source of the problem. I'll fish for those patches regardless.

Cheers,

Pim

Sent from a mobile device.

[Pim van Riezen]

For 3.0, I took the 2.6.29 copy of dm-cache.c out of git, manually fixed Kconfig and performed these changes to keep up with the kernel API as it is in 3.0:

*** dmpatch/dm-cache.c 2011-09-14 13:02:27.000000000 +0200

--- drivers/md/dm-cache.c 2011-08-11 14:22:56.000000000 +0200

***************

*** 41,47 ****

  

  #include "dm.h"

  #include <linux/dm-io.h>

! #include "dm-bio-list.h"

  #include <linux/dm-kcopyd.h>

  

  #define DMC_DEBUG 0

--- 41,47 ----

  

  #include "dm.h"

  #include <linux/dm-io.h>

! /*#include "dm-bio-list.h"*/

  #include <linux/dm-kcopyd.h>

  

  #define DMC_DEBUG 0

***************

*** 169,175 ****

  struct cache_c *dmc = job->dmc;

  struct dm_io_request iorq;

  

! iorq.bi_rw = (rw | (1 << BIO_RW_SYNCIO));

  iorq.mem.type = DM_IO_BVEC;

  iorq.mem.ptr.bvec = bvec;

  iorq.notify.fn = fn;

--- 169,175 ----

  struct cache_c *dmc = job->dmc;

  struct dm_io_request iorq;

  

! iorq.bi_rw = (rw | REQ_SYNC);

  iorq.mem.type = DM_IO_BVEC;

  iorq.mem.ptr.bvec = bvec;

  iorq.notify.fn = fn;

***************

*** 1477,1505 ****

  goto bad;

  }

  

! r = dm_get_device(ti, argv[0], 0, ti->len,

   dm_table_get_mode(ti->table), &dmc->src_dev);

  if (r) {

  ti->error = "dm-cache: Source device lookup failed";

  goto bad1;

  }

  

! r = dm_get_device(ti, argv[1], 0, 0,

   dm_table_get_mode(ti->table), &dmc->cache_dev);

  if (r) {

  ti->error = "dm-cache: Cache device lookup failed";

  goto bad2;

  }

  

! dmc->io_client = dm_io_client_create(DMCACHE_COPY_PAGES);

  if (IS_ERR(dmc->io_client)) {

  r = PTR_ERR(dmc->io_client);

  ti->error = "Failed to create io client\n";

  goto bad3;

  }

  

! r = dm_kcopyd_client_create(DMCACHE_COPY_PAGES, &dmc->kcp_client);

! if (r) {

  ti->error = "Failed to initialize kcopyd client\n";

  goto bad4;

  }

--- 1477,1505 ----

  goto bad;

  }

  

! r = dm_get_device(ti, argv[0],

   dm_table_get_mode(ti->table), &dmc->src_dev);

  if (r) {

  ti->error = "dm-cache: Source device lookup failed";

  goto bad1;

  }

  

! r = dm_get_device(ti, argv[1],

   dm_table_get_mode(ti->table), &dmc->cache_dev);

  if (r) {

  ti->error = "dm-cache: Cache device lookup failed";

  goto bad2;

  }

  

! dmc->io_client = dm_io_client_create();

  if (IS_ERR(dmc->io_client)) {

  r = PTR_ERR(dmc->io_client);

  ti->error = "Failed to create io client\n";

  goto bad3;

  }

  

! dmc->kcp_client = dm_kcopyd_client_create();

! if (! dmc->kcp_client) {

  ti->error = "Failed to initialize kcopyd client\n";

  goto bad4;

  }

On Sep 14, 2011, at 13:00 , Pim van Riezen wrote:

The "2.6.16-2.6.19" patch that is on github actually appies fine to RHEL 2.6.18, but I did make one minor change, because the kernel complained about kmalloc behavior and it had one include wrong to compile succesfully. A patch of the patch:

*** patch-2.6.19.1 2011-09-14 12:56:37.000000000 +0200

--- kernel-dmcache.patch 2011-08-10 12:16:45.000000000 +0200

***************

*** 38,44 ****

  +#include <linux/pagemap.h>

  +

  +#include "dm.h"

! +#include "dm-io.h"

  +#include "dm-bio-list.h"

  +#include "kcopyd.h"

  +

--- 38,44 ----

  +#include <linux/pagemap.h>

  +

  +#include "dm.h"

! +#include <linux/dm-io.h>

  +#include "dm-bio-list.h"

  +#include "kcopyd.h"

  +

***************

*** 369,375 ****

  +    insertion is completed.

  +  */

  + if (bio_data_dir(bio) == READ) {

! + clone = bio_clone(bio, GFP_NOIO);

  + for (i=bio->bi_idx; i<bio->bi_vcnt; i++) {

  + get_page(bio->bi_io_vec[i].bv_page);

  + }

--- 369,375 ----

  +    insertion is completed.

  +  */

  + if (bio_data_dir(bio) == READ) {

! + clone = bio_clone(bio, GFP_ATOMIC);

  + for (i=bio->bi_idx; i<bio->bi_vcnt; i++) {

  + get_page(bio->bi_io_vec[i].bv_page);

  + }

***************

*** 419,425 ****

  + }

  +

  + nr_vecs = bio->bi_vcnt - bio->bi_idx + job->nr_pages;

! + bvec = kmalloc(nr_vecs * sizeof(*bvec), GFP_NOIO);

  + if (!bvec) {

  + DMERR("do_fetch: No memory");

  + return 1;

--- 419,425 ----

  + }

  +

  + nr_vecs = bio->bi_vcnt - bio->bi_idx + job->nr_pages;

! + bvec = kmalloc(nr_vecs * sizeof(*bvec), GFP_ATOMIC);

  + if (!bvec) {

  + DMERR("do_fetch: No memory");

  + return 1;

***************

*** 1065,1071 ****

  + dest.sector = cache_block << dmc->block_shift;

  + dest.count = src.count;

  +

! + job = mempool_alloc(_job_pool, GFP_NOIO);

  + job->dmc = dmc;

  + job->bio = bio;

  + job->src = src;

--- 1065,1071 ----

  + dest.sector = cache_block << dmc->block_shift;

  + dest.count = src.count;

  +

! + job = mempool_alloc(_job_pool, GFP_ATOMIC);

  + job->dmc = dmc;

  + job->bio = bio;

  + job->src = src;

Next up: 3.0.

Cheers,

Pi

[Pim van Riezen]

[Pim van Riezen]

The RHEL kernel I tested with this one, specifically, was: kernel-xen-2.6.18-194.32.1.el5

Cheers,

Pi