Enabling ADX/AVX2/BMI2 instructions

[Sven Anderson]

Hi all,

I noticed that in my Linux subject some CPU flags from the host CPU are missing and therefore instructions like ADX/AVX2/BMI2/... are not being used for accelerating crypto operations. If I‘m not mistaken that is not something I have to explicitly enable in the Linux config. Do I need to add something to the vCPU profile to forward these flags to the Linux Subjects?

Here are the CPU flags that the Go code in the Linux subject sees (* marks the flags that differ from a bare-metal Linux on the same hardware.)

  "HasAES": true,
* "HasADX": false,
* "HasAVX": false,
  "HasAVX2": false, *
  "HasAVX512": false,
  "HasAVX512F": false,
  "HasAVX512CD": false,
  "HasAVX512ER": false,
  "HasAVX512PF": false,
  "HasAVX512VL": false,
  "HasAVX512BW": false,
  "HasAVX512DQ": false,
  "HasAVX512IFMA": false,
  "HasAVX512VBMI": false,
  "HasAVX5124VNNIW": false,
  "HasAVX5124FMAPS": false,
  "HasAVX512VPOPCNTDQ": false,
  "HasAVX512VPCLMULQDQ": false,
  "HasAVX512VNNI": false,
  "HasAVX512GFNI": false,
  "HasAVX512VAES": false,
  "HasAVX512VBMI2": false,
  "HasAVX512BITALG": false,
  "HasAVX512BF16": false,
* "HasAMXTile": false,
* "HasAMXInt8": false,
* "HasAMXBF16": false,
* "HasBMI1": false,
* "HasBMI2": false,
  "HasCX16": true,
  "HasERMS": true,
* "HasFMA": false,
* "HasOSXSAVE": false,
  "HasPCLMULQDQ": true,
  "HasPOPCNT": true,
  "HasRDRAND": true,
* "HasRDSEED": false,
  "HasSSE2": true,
  "HasSSE3": true,
  "HasSSSE3": true,
  "HasSSE41": true,
  "HasSSE42": true

Thanks and best regards

Sven

[Adrian-Ken Rueegsegger]

Hello Sven,

On 12/14/23 20:28, Sven Anderson wrote:
> I noticed that in my Linux subject some CPU flags from the host CPU are
> missing and therefore instructions like ADX/AVX2/BMI2/... are not being
> used for accelerating crypto operations. If I‘m not mistaken that is not
> something I have to explicitly enable in the Linux config. Do I need to add
> something to the vCPU profile to forward these flags to the Linux Subjects?

Linux discovers the available CPU features via the CPUID instruction,
which is emulated by the Subject Monitor (SM). We added the announcement
of these features to SM some time ago [1][2][3] after adding support for
XSAVE for subjects [4]. Thus, I suspect the version of Muen you are
using does not yet include these changes. Is it an option for you to
update Muen? If so, note that an upgrade would very likely also require
regeneration of the hardware spec (using the latest version of
mugenhwcfg(-live)).

Hope this helps.

Best regards,
Adrian

[1] - https://git.codelabs.ch/?p=muen.git;a=commit;h=08ba41936d2b
[2] - https://git.codelabs.ch/?p=muen.git;a=commit;h=f8c9ef54697a
[3] - https://git.codelabs.ch/?p=muen.git;a=commit;h=3d5653e85fb3
[4] - https://git.codelabs.ch/?p=muen.git;a=commit;h=6ba81af73b15

[Sven Anderson]

Adrian-Ken Rueegsegger <k...@codelabs.ch> schrieb am Fr. 15. Dez. 2023 um 12:51:

Hello Sven,

On 12/14/23 20:28, Sven Anderson wrote:
> I noticed that in my Linux subject some CPU flags from the host CPU are
> missing and therefore instructions like ADX/AVX2/BMI2/... are not being
> used for accelerating crypto operations. If I‘m not mistaken that is not
> something I have to explicitly enable in the Linux config. Do I need to add
> something to the vCPU profile to forward these flags to the Linux Subjects?

Linux discovers the available CPU features via the CPUID instruction,
which is emulated by the Subject Monitor (SM). We added the announcement
of these features to SM some time ago [1][2][3] after adding support for
XSAVE for subjects [4]. Thus, I suspect the version of Muen you are
using does not yet include these changes. Is it an option for you to
update Muen? If so, note that an upgrade would very likely also require
regeneration of the hardware spec (using the latest version of
mugenhwcfg(-live)).

Oh, that is great to know!

Yeah, we are using the latest release, which is still 1.0.0, right?

So, of course I could update Muen, if it doesn’t break anything. :-) Which version/branch/commit should I upgrade to, if I care about stability, since there is no later release tag?

Thanks!

Sven

[Adrian-Ken Rueegsegger]

Hello Sven,

On 12/15/23 13:03, Sven Anderson wrote:
[snip]

> Yeah, we are using the latest release, which is still 1.0.0, right?
>
> So, of course I could update Muen, if it doesn’t break anything. :-) Which
> version/branch/commit should I upgrade to, if I care about stability, since
> there is no later release tag?

The current devel branch [1] would be my recommendation. We are planning
to do a release fairly soon/early next year, although I cannot promise
an exact date etc ;)

Best regards,
Adrian

[1] - commit 2514298f6402a8d5a0aa6fea3e4670f24ebbf1ae

[Sven Anderson]

Adrian-Ken Rueegsegger <k...@codelabs.ch> schrieb am Fr. 15. Dez. 2023 um 14:38:

Hello Sven,

On 12/15/23 13:03, Sven Anderson wrote:
[snip]

> Yeah, we are using the latest release, which is still 1.0.0, right?
>
> So, of course I could update Muen, if it doesn’t break anything. :-) Which
> version/branch/commit should I upgrade to, if I care about stability, since
> there is no later release tag?

The current devel branch [1] would be my recommendation. We are planning
to do a release fairly soon/early next year, although I cannot promise
an exact date etc ;)

I see, thanks a lot!

I have two more questions: is this only a detection issue, so could I still run code with these instructions? Or is the XSAVE support necessary for the instructions to work / be enabled? (From my rough understanding I would guess the second is the case, but I’m not sure.) And is there any difference to mirage-os subjects?

Thanks

Sven

[Adrian-Ken Rueegsegger]

Hello Sven,

On 12/15/23 15:41, Sven Anderson wrote:

[snip]

> I have two more questions: is this only a detection issue, so could I still
> run code with these instructions? Or is the XSAVE support necessary for the
> instructions to work / be enabled? (From my rough understanding I would

Strictly speaking, you can use the instructions without XSAVE. XSAVE
comes into play when you have multiple FPU contexts which need to be
saved/restored. This is the case with multiple processes/tasks and that
is what the Linux kernel does for its userspace.
However, if you e.g. have a (simple) native subject without tasking etc.
you should be able to just use the instructions and not run into issues,
as the SK keeps the per-subject FPU state separate and there is no other
code within the same subject which could corrupt your FPU state.

> guess the second is the case, but I’m not sure.) And is there any
> difference to mirage-os subjects?

I do not know in detail whether and/or how mirage-os subjects do
discovery of supported FPU features. It might even be the case that used
features are used are assumed to be present. If the discovery happens
via CPUID, then the SM changes related to FPU feature announcement are
also necessary. Alternatively, maybe there is a simpler (build-time) way
to configure the enabled FPU features of MirageOS unikernels. It is
worth checking someone knowledgeable from the MirageOS community, though.

[Sven Anderson]

Helps a lot!

Thanks and happy holidays,

Sven

[Sven Anderson]

Am Fr., 15. Dez. 2023 um 14:38 Uhr schrieb Adrian-Ken Rueegsegger <k...@codelabs.ch>:

Hello Sven,

On 12/15/23 13:03, Sven Anderson wrote:
[snip]

> Yeah, we are using the latest release, which is still 1.0.0, right?
>
> So, of course I could update Muen, if it doesn’t break anything. :-) Which
> version/branch/commit should I upgrade to, if I care about stability, since
> there is no later release tag?

The current devel branch [1] would be my recommendation. We are planning
to do a release fairly soon/early next year, although I cannot promise
an exact date etc ;)

[1] - commit 2514298f6402a8d5a0aa6fea3e4670f24ebbf1ae

Unfortunately this requires the latest solo5, because there was an ABI change, but that again needs MirageOS 4, which we can't use for now, so we need something that works with  solo5 0.6.5. I tried the last commit before the ABI update (511d6dfb6ef188c80327898493fe44c57104b108), but solo5 crashes with:

16#0005#|Solo5: trap: type=#MF ec=0x0 rip=0x54dbce rsp=0x414c5ee8 rflags=0x10006

16#0005#|Solo5: ABORT: cpu_x86_64.c:181: Fatal trap
16#0005#|Solo5: Halted

 

Is there a commit that you can recommend before the ABI change? Or is the XSAVE stuff backportable?

Thanks

Sven

[Adrian-Ken Rueegsegger]

Hello Sven,

I hope you started well into the New Year. Sorry for the late reply, I
have been keeping away from computers over the holidays.

See my reply below directly inline.

On 12/30/23 13:29, Sven Anderson wrote:
>
> Unfortunately this requires the latest solo5, because there was an ABI
> change, but that again needs MirageOS 4, which we can't use for now, so we
> need something that works with solo5 0.6.5. I tried the last commit before

I see, thank you for the elaboration.

> the ABI update (511d6dfb6ef188c80327898493fe44c57104b108), but solo5
> crashes with:
>
> 16#0005#|Solo5: trap: type=#MF ec=0x0 rip=0x54dbce rsp=0x414c5ee8
> rflags=0x10006
> 16#0005#|Solo5: ABORT: cpu_x86_64.c:181: Fatal trap
> 16#0005#|Solo5: Halted

This looks like a x87 FPU Floating-Point exception. Can you share, what
opcode the RIP (0x54dbce) points to? Is there a way for me to look at
your system policy (format B)?

> Is there a commit that you can recommend before the ABI change? Or is the
> XSAVE stuff backportable?

I think it is preferable to use the latest commit which does not require
a Solo5 upgrade. A backport is possible but it might not be
straightforward, I would need to take a closer look which changes are
actually needed at a minimum. Before going down that route though, I
think it is better to determine the cause of the trap you are seeing in
your unikernel to see if there is a (simple) fix for it.

Best regards,
Adrian

[Sven Anderson]

Hi Adrian,

Adrian-Ken Rueegsegger <k...@codelabs.ch> schrieb am Mo. 8. Jan. 2024 um 17:07:

I hope you started well into the New Year. Sorry for the late reply, I

Yes, thanks, I hope you too!

have been keeping away from computers over the holidays.

Good call! :-)

On 12/30/23 13:29, Sven Anderson wrote:
>
> the ABI update (511d6dfb6ef188c80327898493fe44c57104b108), but solo5
> crashes with:
>
> 16#0005#|Solo5: trap: type=#MF ec=0x0 rip=0x54dbce rsp=0x414c5ee8
> rflags=0x10006
> 16#0005#|Solo5: ABORT: cpu_x86_64.c:181: Fatal trap
> 16#0005#|Solo5: Halted

This looks like a x87 FPU Floating-Point exception. Can you share, what
opcode the RIP (0x54dbce) points to? Is there a way for me to look at
your system policy (format B)?

Uh, I guess I will need some help from you in order to do that. :-) Where can I see „what opcode the RIP (0x54dbce) points to“? Is that included in the format B system policy? I can try to extract that from out build process. Can we see it otherwise, like from the serial log?

> Is there a commit that you can recommend before the ABI change? Or is the
> XSAVE stuff backportable?

I think it is preferable to use the latest commit which does not require
a Solo5 upgrade.

I see. Is 511d6dfb6ef188c80327898493fe44c57104b108 the right spot then? I wasn’t really sure.

Thank and cheers

Sven

[Adrian-Ken Rueegsegger]

Hello Sven,

On 1/10/24 18:55, Sven Anderson wrote:
>
> Adrian-Ken Rueegsegger <k...@codelabs.ch> schrieb am Mo. 8. Jan. 2024 um
> 17:07:

[snip]

>>> 16#0005#|Solo5: trap: type=#MF ec=0x0 rip=0x54dbce rsp=0x414c5ee8
>>> rflags=0x10006
>>> 16#0005#|Solo5: ABORT: cpu_x86_64.c:181: Fatal trap
>>> 16#0005#|Solo5: Halted
>>
>> This looks like a x87 FPU Floating-Point exception. Can you share, what
>> opcode the RIP (0x54dbce) points to? Is there a way for me to look at
>> your system policy (format B)?
>
> Uh, I guess I will need some help from you in order to do that. :-) Where
> can I see „what opcode the RIP (0x54dbce) points to“? Is that included in
> the format B system policy? I can try to extract that from out build
> process. Can we see it otherwise, like from the serial log?

To see what the RIP points to, you can decompile the component binary,
e.g. using objdump:

$ objdump -S path/to/component/elf/binary > component-asm

The open component-asm in an editor and look for the address 0x54dbce.

[snip]

> I see. Is 511d6dfb6ef188c80327898493fe44c57104b108 the right spot then? I
> wasn’t really sure.

Yes, that commit should be fine.

Regards,
Adrian