Re: Looking for blktap2 ancient history, and project history before xenclient-oe.git

25 views

Skip to first unread message

Jason Andryuk

unread,

Nov 26, 2024, 7:40:14 AM11/26/24

to Yann Dirson, openxt, Rich Persaud

Hi Yann,

Hopefully this makes its way out to OpenXT google groups.

My involvement in OpenXT was well after Citrix open sourced Xenclient.
Hopefully Rich or someone else from the list can provide you more
information.

Regards,
Jason

> -------- Forwarded Message --------
> Subject: Looking for blktap2 ancient history, and project history before
> xenclient-oe.git
> Date: Thu, 21 Nov 2024 13:27:10 +0100
> From: Yann Dirson <yann....@vates.tech>
> Organization: Vates
> To: ope...@googlegroups.com
>
> Hi there,
>
> I'm trying to map the history of the blktap2 kernel module across
> projects, to understand how the openxt and xenserver versions diverged.
>
> I'm collecting reconstructed history for now at
> https://github.com/ydirson/blktap-module (this may move into xcp-ng
> project at some point).
>
> From what I've gathered up till now, those histories diverged before
> xenclient-oe.git was originally imported, and I was not able to identify
> where development was happening at that time. Could someone please shed
> some additional light, and help plugging the holes?
>
>
>
>
> Yann Dirson | Vates Platform Developer
>
> XCP-ng & Xen Orchestra - Vates solutions
>
> web: https://vates.tech

Rich Persaud

unread,

Nov 26, 2024, 2:56:14 PM11/26/24

to Yann Dirson, Jason Andryuk, openxt, Mark Syms, Christopher Clark

On Nov 26, 2024, at 07:40, Jason Andryuk <jand...@gmail.com> wrote:

Hi Yann,

Hopefully this makes its way out to OpenXT google groups.

Thanks Jason.

-------- Forwarded Message --------
Subject: Looking for blktap2 ancient history, and project history before
xenclient-oe.git
Date: Thu, 21 Nov 2024 13:27:10 +0100
From: Yann Dirson <yann....@vates.tech>
Organization: Vates
To: ope...@googlegroups.com

Hi there,

I'm trying to map the history of the blktap2 kernel module across
projects, to understand how the openxt and xenserver versions diverged.

I'm collecting reconstructed history for now at
https://github.com/ydirson/blktap-module (this may move into xcp-ng
project at some point).

From what I've gathered up till now, those histories diverged before
xenclient-oe.git was originally imported, and I was not able to identify
where development was happening at that time. Could someone please shed
some additional light, and help plugging the holes?

Hi Yann,

If you're attempting to construct a Grand Unified Theory of the Xen blktap storage layer, you may have more success with a single blktap repo that combines blktap1 from upstream Xen, blktap2 (upstream Xen, XenServer, XenClient, OpenXT) and blktap3 (XAPI/XenServer/XCP-ng and OpenXT). That would help git to identify points of divergence in code history.

From 2015-ish post-XenClient, OpenXT blktap2,

https://github.com/OpenXT/blktap

OpenXT has an old version (fork really) of the original blktap2 project from
XenServer circa 2010. It was updated numerous times with back ports from the
XenServer project. In addition to the base blktap2 project OpenXT (by way of
work done on XenClient) extends the base project with the following:

1. The current VHD encryption feature based on OpenSSL and the keyhash support
found in the VHD meta structures.
2. ICBINN supprt for VHD management by Synchronizer XT backend.
3. Miscellaneous fixes and attempts at fixes (many of which are not upstream).
4. VHD management support for the defunct Transmitter backend.

This repository is considered deprecated and will be replaced by rebasing 1,
2 and some of 3 on an upstream version of blktap2. This will then be replaced
by rebasing on blktap3.

In 2018, OpenXT worked with Citrix XenServer to consolidate on a single blktap tree, including a forward port of Citrix XenClient 2010 blktap2 disk encryption to blktap3.

https://groups.google.com/g/openxt/c/yN_cX9wUsF4

https://github.com/OpenXT/blktap3

https://github.com/xapi-project/blktap/commit/fa9fa3566dd5c2e4b5c7e84f88c42d4b9fdff062

The Github xapi blktap repo has history for blktap1 (Xen), blktap2 (Xen, XenServer), blktap3 (XenServer, XenClient-OpenXT ported).

https://github.com/xapi-project/blktap

2005-2019 upstream Xen blktap: https://xenbits.xen.org/gitweb/?p=xen.git;a=search;h=HEAD;pg=1;s=blktap;st=commit

If you want more visibility into XenClient history, you could try the source images for XenClient, which requires a Citrix account, https://www.citrix.com/downloads/xenclient/source.html. If you do obtain those, it would be helpful to host the GPL sources on a public repo or directory.

There should also be source ISOs for the Citrix XenServer 5.x Dell/HP OEM product, which was a fork of XenServer (e.g. read-only dom0, local graphical display) that would later influence Citrix XenClient. The XS OEM variant would later be unforked via the XS Supplemental Pack.

If you want to know more about early (pre-2011) blktap2 history, your best bet is someone who worked on the blktap code, via xapi commit history and Xen mailing list, e.g. one of the blktap developers from the Citrix XenClient Vancouver storage team who left to join CoHo and later, AWS S3. They may be reachable via LinkedIn.

2008, https://xen-devel.narkive.com/KPJivIz4/5-patches-synchronize-blktap-with-citrix-blktap2

2010, https://xen-devel.narkive.com/gUHoBSHo/merging-xen-dom0-backend-blktap2

https://www.usenix.org/system/files/conference/fast14/fast14-paper_cully.pd

Please keep us posted on your progress with reconstruction of blktap source history. If Vates is investing resources into support of the blktap storage layer, we (OpenXT, Vates, Citrix) should talk about potentially leveraging Azure or other ecosystem tooling for VHDs on Linux.

Regards,

Rich

Yann Dirson

unread,

Nov 28, 2024, 9:55:19 AM11/28/24

to Rich Persaud, Jason Andryuk, openxt, Mark Syms, Christopher Clark

Hi Rich,

Thanks much for all those info and links, it will take some time to
digest all of this :)

> https://github.com/OpenXT/blktap <https://github.com/OpenXT/blktap>

>
>> OpenXT has an old version (fork really) of the original blktap2
>> project from
>> XenServer circa 2010. It was updated numerous times with back ports
>> from the
>> XenServer project. In addition to the base blktap2 project OpenXT (by
>> way of
>> work done on XenClient) extends the base project with the following:
>>
>> 1. The current VHD encryption feature based on OpenSSL and the
>> keyhash support
>> found in the VHD meta structures.
>> 2. ICBINN supprt for VHD management by Synchronizer XT backend.
>> 3. Miscellaneous fixes and attempts at fixes (many of which are not
>> upstream).
>> 4. VHD management support for the defunct Transmitter backend.
>>
>> This repository is considered deprecated and will be replaced by
>> rebasing 1,
>> 2 and some of 3 on an upstream version of blktap2. This will then be
>> replaced
>> by rebasing on blktap3.
>
> In 2018, OpenXT worked with Citrix XenServer to consolidate on a single
> blktap tree, including a forward port of Citrix XenClient 2010 blktap2
> disk encryption to blktap3.
> https://groups.google.com/g/openxt/c/yN_cX9wUsF4 <https://

> groups.google.com/g/openxt/c/yN_cX9wUsF4>
> https://github.com/OpenXT/blktap3 <https://github.com/OpenXT/blktap3>
> https://github.com/xapi-project/blktap/commit/
> fa9fa3566dd5c2e4b5c7e84f88c42d4b9fdff062 <https://github.com/xapi-
> project/blktap/commit/fa9fa3566dd5c2e4b5c7e84f88c42d4b9fdff062>

>
> The Github xapi blktap repo has history for blktap1 (Xen), blktap2 (Xen,
> XenServer), blktap3 (XenServer, XenClient-OpenXT ported).

> https://github.com/xapi-project/blktap <https://github.com/xapi-project/
> blktap>

>
> 2005-2019 upstream Xen blktap: https://xenbits.xen.org/gitweb/?

> p=xen.git;a=search;h=HEAD;pg=1;s=blktap;st=commit <https://

> xenbits.xen.org/gitweb/?p=xen.git;a=search;h=HEAD;pg=1;s=blktap;st=commit>
>
> If you want more visibility into XenClient history, you could try the
> source images for XenClient, which requires a Citrix account, https://

> www.citrix.com/downloads/xenclient/source.html <https://www.citrix.com/
> downloads/xenclient/source.html>. If you do obtain those, it would be

> helpful to host the GPL sources on a public repo or directory.
>
> There should also be source ISOs for the Citrix XenServer 5.x Dell/HP
> OEM product, which was a fork of XenServer (e.g. read-only dom0, local
> graphical display) that would later influence Citrix XenClient. The XS
> OEM variant would later be unforked via the XS Supplemental Pack.
>
> If you want to know more about early (pre-2011) blktap2 history, your
> best bet is someone who worked on the blktap code, via xapi commit
> history and Xen mailing list, e.g. one of the blktap developers from the
> Citrix XenClient Vancouver storage team who left to join CoHo and later,
> AWS S3. They may be reachable via LinkedIn.
> 2008, https://xen-devel.narkive.com/KPJivIz4/5-patches-synchronize-

> blktap-with-citrix-blktap2 <https://xen-devel.narkive.com/KPJivIz4/5-
> patches-synchronize-blktap-with-citrix-blktap2>
> 2010, https://xen-devel.narkive.com/gUHoBSHo/merging-xen-dom0-backend-
> blktap2 <https://xen-devel.narkive.com/gUHoBSHo/merging-xen-dom0-
> backend-blktap2>
> https://www.usenix.org/system/files/conference/fast14/fast14-
> paper_cully.pd <https://www.usenix.org/system/files/conference/fast14/
> fast14-paper_cully.pd>

>
> Please keep us posted on your progress with reconstruction of blktap
> source history. If Vates is investing resources into support of the
> blktap storage layer, we (OpenXT, Vates, Citrix) should talk about
> potentially leveraging Azure or other ecosystem tooling for VHDs on Linux.
>
> Regards,
> Rich

Yann Dirson

unread,

Dec 3, 2024, 6:14:07 AM12/3/24

to Rich Persaud, Jason Andryuk, openxt, Mark Syms, Christopher Clark, Jan Beulich

Hi all,

I have for now been able to get the following history lines for the
kernel driver:

* D.Stoden's "next-2.6.*" branches -> xenclient 4.1 -> openxt current,
with only coarse grain for the XenClient period
* XenServer coarse grain (release ISOs only) starting from 6.1, with a
rough fork point from D.Stodden's work (quite imperfect representation,
as there were apparently some cherry-picking between those branches),
with linux-3.x.pg providing some finer-grained insight into the 6.5-7.1 era
* 2 branches starting off D.Meyer's initial blktap2 work: one
essentially from I.Campbell, J.Beulich, and D.Stodden, and the other in
xenclient/linux-2.6.27-pq.git

There is a gap between this last group around "the original blkatp2" and
the first two one which I've been unable to find any detail for now.

On 11/26/24 20:55, Rich Persaud wrote:

> https://github.com/OpenXT/blktap <https://github.com/OpenXT/blktap>

> https://groups.google.com/g/openxt/c/yN_cX9wUsF4 <https://
> groups.google.com/g/openxt/c/yN_cX9wUsF4>
> https://github.com/OpenXT/blktap3 <https://github.com/OpenXT/blktap3>
> https://github.com/xapi-project/blktap/commit/
> fa9fa3566dd5c2e4b5c7e84f88c42d4b9fdff062 <https://github.com/xapi-
> project/blktap/commit/fa9fa3566dd5c2e4b5c7e84f88c42d4b9fdff062>
>

> The Github xapi blktap repo has history for blktap1 (Xen), blktap2 (Xen,
> XenServer), blktap3 (XenServer, XenClient-OpenXT ported).

> https://github.com/xapi-project/blktap <https://github.com/xapi-project/
> blktap>

>
> 2005-2019 upstream Xen blktap: https://xenbits.xen.org/gitweb/?

> p=xen.git;a=search;h=HEAD;pg=1;s=blktap;st=commit <https://

> xenbits.xen.org/gitweb/?p=xen.git;a=search;h=HEAD;pg=1;s=blktap;st=commit>
>
> If you want more visibility into XenClient history, you could try the

> source images for XenClient, which requires a Citrix account, https://
> www.citrix.com/downloads/xenclient/source.html <https://www.citrix.com/

> downloads/xenclient/source.html>. If you do obtain those, it would be

> 2008, https://xen-devel.narkive.com/KPJivIz4/5-patches-synchronize-
> blktap-with-citrix-blktap2 <https://xen-devel.narkive.com/KPJivIz4/5-
> patches-synchronize-blktap-with-citrix-blktap2>
> 2010, https://xen-devel.narkive.com/gUHoBSHo/merging-xen-dom0-backend-
> blktap2 <https://xen-devel.narkive.com/gUHoBSHo/merging-xen-dom0-
> backend-blktap2>
> https://www.usenix.org/system/files/conference/fast14/fast14-
> paper_cully.pd <https://www.usenix.org/system/files/conference/fast14/
> fast14-paper_cully.pd>
>

> Please keep us posted on your progress with reconstruction of blktap
> source history. If Vates is investing resources into support of the
> blktap storage layer, we (OpenXT, Vates, Citrix) should talk about
> potentially leveraging Azure or other ecosystem tooling for VHDs on Linux.
>
> Regards,
> Rich

Reply all

Reply to author

Forward

0 new messages