Usage of ext-storage-interface

[Rudolph Bott]

Hi Everyone,

out of couriosity - is anyone of you using the Ganeti ext storage interface [1]? And if so, what are you doing with it? Do you use it to provide blockdevices or storage URIs to Ganeti?

I am currently working on this[2] PR and stumbled across an issue regarding this interface and would like to know how common it is.

Cheers,

Rudi

[1] https://docs.ganeti.org/docs/ganeti/3.0/html/man-ganeti-extstorage-interface.html

[2] https://github.com/ganeti/ganeti/pull/1667#issuecomment-1512141687
--

 Rudolph Bott - bo...@sipgate.de

 Telefon: +49 (0)211-63 55 55-55

 Telefax: +49 (0)211-63 55 55-22

 sipgate GmbH - Gladbacher Str. 74 - 40219 Düsseldorf

 HRB Düsseldorf 39841 - Geschäftsführer: Thilo Salmon, Tim Mois

 Steuernummer: 106/5724/7147, Umsatzsteuer-ID: DE219349391

 www.sipgate.de - www.sipgate.co.uk

[John McNally]

Rudi,

I am not using the ext-storage interface, but it might be nice in our environment as a consistent location for gnt-backup, which uses the default path: /var/lib/ganeti/export.

Currently, I have my own mount/unmount scripts that attach/detach from an iSCSI volume on the local network, with /var/lib/ganeti/export as the mount point. I keep instance backups from all my clusters there. It works fine, but is clunky and brittle: I must remember to unmount the volume when one cluster is done backing up or restoring. Since there is no file locking, filesystem corruption could occur if the volume is mounted by multiple ganeti nodes.

Currently, I have no use for primary storage on an external volume, but it might be nice to streamline things by making my iSCSI backup volume visible through ext-storage. Does that sound doable?

  

_________________
John McNally
jmcn...@acm.org

--
You received this message because you are subscribed to the Google Groups "ganeti" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ganeti+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ganeti/CAPG4N%3DayHh%2BbLp1-dmCvuOXapmukfXS_Kscz5GJJdxcq_BqKHw%40mail.gmail.com.

[Joerg Jaspert]

Am 18.04.2023 15:55, schrieb Rudolph Bott:

> out of couriosity - is anyone of you using the Ganeti ext storage
> interface [1]? And if so, what are you doing with it? Do you use it to
> provide blockdevices or storage URIs to Ganeti?

Yes, using it extensively. Have an interface to HPE 3pars based on it,
maintaining all the volumes needed for the VMs.



--
bye Joerg

[Rudolph Bott]

Hi Joerg,

thanks for your reply.

Am Do., 20. Apr. 2023 um 16:48 Uhr schrieb Joerg Jaspert <joerg+...@nsb-software.de>:

Yes, using it extensively. Have an interface to HPE 3pars based on it,
maintaining all the volumes needed for the VMs.

I suppose the implementation returns blockdevices (or files) rather than userspace URIs to ganeti? Would it be possible for you to test it with this PR?

https://github.com/ganeti/ganeti/pull/1667

Regards,

Rudi

[Thomas Schneider]

Hi Rudi,

> On 18. Apr 2023, at 15:55, Rudolph Bott <bo...@sipgate.de> wrote:
>
> out of couriosity - is anyone of you using the Ganeti ext storage interface [1]?

Yes, we‘re using it for Ceph RBD images. Ganeti supports this natively, but the pool is configured per-cluster, and we have instances on different pools (namely, some with performance requirements reside on an SSD pool, while the bulk is on spinning rust).

We‘re using a slightly patched version[0] of gnt-extstorage-rbd – ported to Python 3 and a few bug fixes.

> And if so, what are you doing with it? Do you use it to provide blockdevices or storage URIs to Ganeti?

Storage URIs, for direct librbd user space access from QEMU. Block device only for instance creation (ganeti-instance-debootstrap).

> I am currently working on this[2] PR and stumbled across an issue regarding this interface and would like to know how common it is.

Before you ask, testing would be not quite trivial for us, but doable if necessary (there are some old nodes that I could create a Ganeti cluster on).

Best,
Thomas

[0]: https://git.fsmpi.rwth-aachen.de/infra/gnt-extstorage-rbd

[Joerg Jaspert]

Am 24.04.2023 08:52, schrieb Rudolph Bott:

>> Yes, using it extensively. Have an interface to HPE 3pars based on it,
>> maintaining all the volumes needed for the VMs.
> I suppose the implementation returns blockdevices (or files) rather
> than userspace URIs to ganeti? Would it be possible for you to test it
> with this PR?

Blockdevs, yes.
(It's doing a bit more even. Accesses 2 3pars here, then uses kernel
raid, so data gets send to both. It then hands ganeti the
/dev/md/whateveritsname)

> https://github.com/ganeti/ganeti/pull/1667

I can try, but I'm not sure exactly when. Can take *a while*, sorry.

--
bye Joerg

[Rudolph Bott]

Hi thomas,

Am Mi., 26. Apr. 2023 um 12:52 Uhr schrieb Thomas Schneider <q...@chaotikum.eu>:

Storage URIs, for direct librbd user space access from QEMU.  Block device only for instance creation (ganeti-instance-debootstrap).

From looking at the code (thanks for the link!) I would assume that my modifications will work with this storage provider because it generates the same type of RBD URI as Ganeti already knows/uses.

 

> I am currently working on this[2] PR and stumbled across an issue regarding this interface and would like to know how common it is.

Before you ask, testing would be not quite trivial for us, but doable if necessary (there are some old nodes that I could create a Ganeti cluster on).

Ha :-) Thanks for the offer. Actually seeing the sourcecode of a storage provider that is used "in the wild" is already quite helpful to me.

 

Best,
Thomas

[0]: https://git.fsmpi.rwth-aachen.de/infra/gnt-extstorage-rbd

Cheers,

Rudi

[Rudolph Bott]

Hi Joerg,

Am Mi., 26. Apr. 2023 um 14:02 Uhr schrieb Joerg Jaspert <joerg+...@nsb-software.de>:

Am 24.04.2023 08:52, schrieb Rudolph Bott:

Blockdevs, yes.
(It's doing a bit more even. Accesses 2 3pars here, then uses kernel
raid, so data gets send to both. It then hands ganeti the
/dev/md/whateveritsname)

That information actually helps. I can build my own quick-and-dirty storage provider that returns a blockdevice and do some testing with that. That should actually be sufficient to cover your case. I am more worried about custom storage providers which leverage the option to return an URI as that can't be simply forwarded to QEMU any more.

 

 

I can try, but I'm not sure exactly when. Can take *a while*, sorry.

No worries. As long as we get things properly tested before there is a new stable release we are probably good. Thanks!

 

--
bye Joerg

Cheers,