exposing target functionality as API

[Kristina Hanicova]

Hello,

I would like to ask if there is any plan or if it would be possible to discuss exposing the target functionality as part of the API. From my understanding, if I want to integrate any existing target (e.g. loop or nbd) I would have to copy the entire code of the target plus some of the functionality handled by the ublksrv_tgt.cpp and somehow glue it together (which is not trivial).

A similar MR was opened last month (https://github.com/ublk-org/ublksrv/pull/154)

Please let me know your thoughts and plans on this.

Have a nice day,

Kristina Hanicova

[Ming Lei]

One big issue of PR154 is that command line handling becomes part of the
API, which can be broken easily.

If you want to reuse the built target, another way is to add/remove the
target via shell command line interface, which should be supported in
most programming language.

Please let me know if calling via shell command line works for you.


Thanks,
Ming

[Kristina Hanicova]

Perhaps I should have explained the context more before.

Basically what I aim to do is create a new target for encryption. We need to use it as an API in a different program. However, it would be easier to just add a new target in /targets and have it exposed as part of the API rather than "merging" it together with ublksrv_tgt.cpp. (And to have it maintained upstream would of course be preferable as well.)

Thanks for your reply,

Kristina

[Ming Lei]

OK, got it.

IMO, there are at least two ways for supporting encryption:

1) add new encryption target

- this way requires to duplicate some target code from loop or nbd or
others, however, we still can refactor target code to abstract APIs
for adding new target easily

2) add encryption target APIs

- so existing target can support encryption over encryption APIs by
passing command line, even command line handling can be encryption APIs
too

The latter looks a bit complicated to go, but the former should be easier to
start.

Maybe we can begin with new encryption target and evolute to 2) finally,
what do you think of the above ways?



Thanks,
Ming

[ronnie sahlberg]

On Sat, 26 Jul 2025 at 09:11, Ming Lei <tom.l...@gmail.com> wrote:
>
> On Wed, Jul 23, 2025 at 08:49:23AM +0200, 'Kristina Hanicova' via ublk wrote:
> > Hello,
> >
> > I would like to ask if there is any plan or if it would be possible to
> > discuss exposing the target functionality as part of the API. From my
> > understanding, if I want to integrate any existing target (e.g. loop or
> > nbd) I would have to copy the entire code of the target plus some of the
> > functionality handled by the ublksrv_tgt.cpp and somehow glue it together
> > (which is not trivial).
> >
> > A similar MR was opened last month (
> > https://github.com/ublk-org/ublksrv/pull/154)
> >
> > Please let me know your thoughts and plans on this.
>
> One big issue of PR154 is that command line handling becomes part of the
> API, which can be broken easily.

I tried to do that in the reworked and re-pushed PR154.
It now unifies the command line parsing so that all parsing happens in
ublk._protocol_.cpp and none happens in ublksrv_tgt.cpp.

These changes are the first 5 commits in PR154 leaving the 6th commit
being where ublksrv_main() is turned into a public API function.

Do you want me to create a separate PR that only changes the parsing
like this and avoiding "some options are parsed in ublksrv_tgt.cpp and
some other options are parsed in ublk._protocol_.cpp" ?
I can create a PR in the next few days that does only that.
Then we can leave the question about ublksrv_main() for later once
the option parsing is sorted out.

--
ronnie s

>
> If you want to reuse the built target, another way is to add/remove the
> target via shell command line interface, which should be supported in
> most programming language.
>
> Please let me know if calling via shell command line works for you.
>
>
> Thanks,
> Ming
>

> --
> You received this message because you are subscribed to the Google Groups "ublk" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ublk+uns...@googlegroups.com.
> To view this discussion, visit https://groups.google.com/d/msgid/ublk/aIQPCoi39KjQtpO6%40fedora.

[Ming Lei]

On Tue, Jul 29, 2025 at 09:53:58PM +1000, ronnie sahlberg wrote:
> On Sat, 26 Jul 2025 at 09:11, Ming Lei <tom.l...@gmail.com> wrote:
> >
> > On Wed, Jul 23, 2025 at 08:49:23AM +0200, 'Kristina Hanicova' via ublk wrote:
> > > Hello,
> > >
> > > I would like to ask if there is any plan or if it would be possible to
> > > discuss exposing the target functionality as part of the API. From my
> > > understanding, if I want to integrate any existing target (e.g. loop or
> > > nbd) I would have to copy the entire code of the target plus some of the
> > > functionality handled by the ublksrv_tgt.cpp and somehow glue it together
> > > (which is not trivial).
> > >
> > > A similar MR was opened last month (
> > > https://github.com/ublk-org/ublksrv/pull/154)
> > >
> > > Please let me know your thoughts and plans on this.
> >
> > One big issue of PR154 is that command line handling becomes part of the
> > API, which can be broken easily.
>
> I tried to do that in the reworked and re-pushed PR154.
> It now unifies the command line parsing so that all parsing happens in
> ublk._protocol_.cpp and none happens in ublksrv_tgt.cpp.

OK, but one big problem is that all targets have to provide one whole `struct option`
table, and the 1st part is for defining generic target command line, the 2nd part is
target-specific.

Meantime the generic command line handling code(ublksrv_main()) is provided as one
generic library API.

This way is fragile:

1) little change in target code may cause command line parsing failure

2) ublksrv_main() API has to be maintained carefully to not break target
code(in-tree and out-of-tree).

>
> These changes are the first 5 commits in PR154 leaving the 6th commit
> being where ublksrv_main() is turned into a public API function.
>
> Do you want me to create a separate PR that only changes the parsing
> like this and avoiding "some options are parsed in ublksrv_tgt.cpp and
> some other options are parsed in ublk._protocol_.cpp" ?
> I can create a PR in the next few days that does only that.
> Then we can leave the question about ublksrv_main() for later once
> the option parsing is sorted out.

I understand ublksrv_main() is convenient for out-of-tree target code,
what do you think about the following way?

1) add ublksrv_main() API, and export it via include/ublksrv.h, meantime
document clearly the potential compatibility issue of the API, so
out-of-tree user may understand the potential risk

2) define generic command line option table centrally and do not duplicate
it for every targets, and make target code to use it easily


Thanks,
Ming

[Kristina Hanicova]

I have already done a prototype (more like a proof of concept) of crypt target and tried to somehow create an API by duplicating the code from the target and ublksrv_tgt.cpp.

It can be found here (be warned, it needs to be - and will be - reworked):
https://github.com/KristinaHa26/ubdsrv/tree/devel

What I meant was in a much more general direction -> to create an API that could somehow work with targets / expose their functionality regardless of the specific target code (whenever the target is part of the library or just implemented locally - in case someone has an external target and does not want to upstream it).

Basically expose what the functionality of ublksrv_tgt does now just internally.

For example, there would be a function "ublk_add()" whose input args would be: structure with general options (the target specific options could be passed as private_data) and struct ublksrv_tgt_type with target-specific callbacks.

The main goal would be to use API with just a few functions (like ublk command-line utility) and to not worry about writing ublksrv_start_daemon() or  target_device_handler() when integrating ublk (as existing demos do right now).

I am not sure if such a thing is doable, but I think it would make the ublk integration into projects much easier and would make sense in the long run.

Thank you,

Kristina

[Ming Lei]

Just take a quick look, and the crypt feature can be supported in generic
& easier way:

1) add crypt feature in libublksrv

- define `struct crypt_tgt_data`

- define generic enc/dec wrapper in libublksrv

2) if crypt feature is enabled:

- call enc wrapper for WRITE IO before running q->tgt_ops->handle_io_async()

- call dec wrapper for READ IO in ublksrv_complete_io()

3) add crypt command line definition & handling in ublksrv_parse_add_opts()

Then if crypt feature is enabled from command line, any target can support
it in generic way.

What do you think about the above way?


Thanks,
Ming

[ronnie sahlberg]

ACK.
I will redo it like that coming weekend.
Thanks!

>
>
> Thanks,
> Ming

[Kristina Hanicova]

I think the crypt feature would be much more complicated - we need to be able to reset some options (like key) and do other operations along the way. A lot of stuff is missing from my initial prototype.

We are months away from a true functional target.

Also it would not make sense for all targets (right now just loop would be applicable?).

What I meant was to widen the ublksrv API that would expose the functionality ublk command-line tool provides right now - ublk add, list, delete etc.

So that the user would not have to write and duplicate the code around that, anytime someone wants to integrate an already existing target (or private one) into their application.

Thank you,

Kristina

[Ming Lei]

IMO the model isn't complicated, we can freeze queue easily before updating key.

The IO fast path interface looks simple, just enc for write and dec after
reading is done.

> We are months away from a true functional target.
> Also it would not make sense for all targets (right now just loop would be
> applicable?).

crypt can be one generic feature, which is controlled by command line.
User can just enable it for the interested target.

If you think it is only usable for loop, another way is to handle it in
loop target directly, just like your prototype, then crypt command
line can be added for loop only.

>
> What I meant was to widen the ublksrv API that would expose the
> functionality ublk command-line tool provides right now - ublk add, list,
> delete etc.
> So that the user would not have to write and duplicate the code around
> that, anytime someone wants to integrate an already existing target (or
> private one) into their application.

OK, sahlberg is working to add ublksrv_main() for simplifying command line
handling, could you raise concrete suggestions wrt. widening APIs?



Thanks,
Ming

[Milan Broz]

Hi,

just for the context (as cryptsetup maintainer), this encryption work
started as a master's thesis as an experiment, and now it is slightly
moving to real use.

On 7/30/25 12:41 PM, Ming Lei wrote:
>> I think the crypt feature would be much more complicated - we need to be
>> able to reset some options (like key) and do other operations along the
>> way. A lot of stuff is missing from my initial prototype.
>
> IMO the model isn't complicated, we can freeze queue easily before updating key.

The goal is that we can optionally replace backend for cryptsetup
(now dm-crypt) with ublk userspace encryption.

There is a lot of various settings dm-crypt currently supports.
Cryptsetup needs a crypt target to provide most of them, but others should
not use these settings. (Many settings are obscure and no longer secure,
but we need them for compatibility.)

For example, we need to implement all IV generators.
We already have this in userspace, as Kristina uses the cryptsetup backend wrapper.
But other targets definitely should not be linked to that code.

Later we would like to add support for authenticated encryption (AEAD) that has many
more parameters like treating IV (otr sector offset) as an authenticated-only data.
Also this will require per-sector metadata tags supported by ublk, not sure
it is there yet (integrity profile metadata).

That's why I think it is better to have a separate target, at least until
everything works as expected. But also we would like to avoid duplication
of the ublk daemon code.

> The IO fast path interface looks simple, just enc for write and dec after
> reading is done.
>
>> We are months away from a true functional target.
>> Also it would not make sense for all targets (right now just loop would be
>> applicable?).
>
> crypt can be one generic feature, which is controlled by command line.
> User can just enable it for the interested target.

Please, better no. I have seen so many wrongly implemented sector encryption.
This will simply invite people to repeat the same mistakes.

That said, did you consider something like stacking target
functionality (similar to device-mapper targets)?

> If you think it is only usable for loop, another way is to handle it in
> loop target directly, just like your prototype, then crypt command
> line can be added for loop only.

I do not think the loop target should be directly combined with encryption.
IMO one target should provide one function, maybe supporting chaining some
callbacks, dunno.
(I am so happy encryption was removed from the kernel loop target -- btw
we can map old cryptoloop in cryptsetup, one of the obscure compatible
settings mentioned above :-)

Thanks,
Milan

[ronnie sahlberg]

Interesting project and it sounds very useful.

For good performance I assume this chain of targets would still be a
single device, a single ring and a single server process.
The top target would be the crypt target which owns the device and
talks to the kernel and also calls into the target below it.

Intuitively it feels like ublk.nfs would be a good candidate for the
first prototype of the interface between targets.
Mostly because ublk.nfs (and .iscsi) does not have very deep
integration with the internals of ublksrv. It basically only accesses
a small subset of the public API and does not access the device
innards or the ring directly.
I think this provides the smallest surface, especially since you
wouldn't have to worry about the two targets stepping on eachother
peeking and poking private internals in the device or ring.

All entrypoints into ublk.nfs are just 6 function pointers in the
nfs_tgt_type structure that could accessed by dlopen and friends.

>
> > If you think it is only usable for loop, another way is to handle it in
> > loop target directly, just like your prototype, then crypt command
> > line can be added for loop only.
>
> I do not think the loop target should be directly combined with encryption.
> IMO one target should provide one function, maybe supporting chaining some
> callbacks, dunno.
> (I am so happy encryption was removed from the kernel loop target -- btw
> we can map old cryptoloop in cryptsetup, one of the obscure compatible
> settings mentioned above :-)
>
> Thanks,
> Milan
>

> --
> You received this message because you are subscribed to the Google Groups "ublk" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ublk+uns...@googlegroups.com.

> To view this discussion, visit https://groups.google.com/d/msgid/ublk/daca074f-105f-4058-827b-07c55604be2b%40gmail.com.

[Ming Lei]

On Wed, Jul 30, 2025 at 01:32:32PM +0200, Milan Broz wrote:
> Hi,
>
> just for the context (as cryptsetup maintainer), this encryption work
> started as a master's thesis as an experiment, and now it is slightly
> moving to real use.
>
> On 7/30/25 12:41 PM, Ming Lei wrote:
> > > I think the crypt feature would be much more complicated - we need to be
> > > able to reset some options (like key) and do other operations along the
> > > way. A lot of stuff is missing from my initial prototype.
> >
> > IMO the model isn't complicated, we can freeze queue easily before updating key.
>
> The goal is that we can optionally replace backend for cryptsetup
> (now dm-crypt) with ublk userspace encryption.
>
> There is a lot of various settings dm-crypt currently supports.
> Cryptsetup needs a crypt target to provide most of them, but others should
> not use these settings. (Many settings are obscure and no longer secure,
> but we need them for compatibility.)
>
> For example, we need to implement all IV generators.
> We already have this in userspace, as Kristina uses the cryptsetup backend wrapper.
> But other targets definitely should not be linked to that code.
>
> Later we would like to add support for authenticated encryption (AEAD) that has many
> more parameters like treating IV (otr sector offset) as an authenticated-only data.
> Also this will require per-sector metadata tags supported by ublk, not sure
> it is there yet (integrity profile metadata).

ublk doesn't support integrity yet.

>
> That's why I think it is better to have a separate target, at least until
> everything works as expected. But also we would like to avoid duplication
> of the ublk daemon code.
>
> > The IO fast path interface looks simple, just enc for write and dec after
> > reading is done.
> >
> > > We are months away from a true functional target.
> > > Also it would not make sense for all targets (right now just loop would be
> > > applicable?).
> >
> > crypt can be one generic feature, which is controlled by command line.
> > User can just enable it for the interested target.
>
> Please, better no. I have seen so many wrongly implemented sector encryption.
> This will simply invite people to repeat the same mistakes.
>
> That said, did you consider something like stacking target
> functionality (similar to device-mapper targets)?

It has performance problem to implement stacking target, such as
ublk over ublk, communication between kernel and userspace increases
for ublk over ublk.

ebpf may avoid the extra communication cost for stacking ublk:

https://lore.kernel.org/linux-block/20250107120417.123...@gmail.com/

but it has ebpf's constraint too.

>
> > If you think it is only usable for loop, another way is to handle it in
> > loop target directly, just like your prototype, then crypt command
> > line can be added for loop only.
>
> I do not think the loop target should be directly combined with encryption.
> IMO one target should provide one function, maybe supporting chaining some
> callbacks, dunno.
> (I am so happy encryption was removed from the kernel loop target -- btw
> we can map old cryptoloop in cryptsetup, one of the obscure compatible
> settings mentioned above :-)

I am fine to add one standalone crypt target, the only code duplication
should be just loop IO handling, and the duplication can be avoided by abstracting
file backed target APIs, please see similar approach used in kernel
selftest code for reusing file backed code between loop and stripe.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/testing/selftests/ublk?h=v6.16


thanks,
Ming