Enable SEV-SNP in BIOS

[江立中]

Hi Cloudlab Team,

My current experiment is https://www.cloudlab.us/status.php?uuid=b3ba8b6a-7ed4-11ed-b318-e4434b2381fc.

I want to launch SEV-SNP on my node, but I'm encountering some problems.

First, SEV is disabled in BIOS by default, so I would like to enable it.

In addition, according to this link https://github.com/AMDESE/AMDSEV/tree/sev-snp-devel#prepare-host, I need to modify BIOS further to get SEV-SNP's functionality.

Specifically, it mentions two things.

One is the BIOS upgrade as follows:

----------------------------------------------------------------------------

*NOTE: If your SEV-SNP firmware is older than 1.51, see the "Upgrade SEV firmware" section to upgrade the firmware. *

----------------------------------------------------------------------------

Another one is the BIOS configuration as follows:

----------------------------------------------------------------------------

Verify that the following BIOS settings are enabled. The setting may vary based on the vendor BIOS. The menu option below are from AMD BIOS.

  

CBS -> CPU Common ->
                SEV-ES ASID space Limit Control -> Manual
                SEV-ES ASID space limit -> 100
                SNP Memory Coverage -> Enabled
                SMEE -> Enabled

        -> NBIO common ->
                SEV-SNP -> Enabled

----------------------------------------------------------------------------

Since I cannot modify the BIOS setting by myself, I wonder if you could help me enable SEV and set the BIOS setting as mentioned above?

Thank you in advance!

Sincerely,

Michael

[Mike Hibler]

Working on this now. Sorry for the delay, but it is quite an involved process
as it take multiple reboot cycles.

> --
> You received this message because you are subscribed to the Google Groups
> "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to cloudlab-user...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/
> cloudlab-users/
> CAKJOkPgNG3zO6XqaTsW24PoL4U5_6-ec7q5Zo15eC%2B0XMZ-bqg%40mail.gmail.com.

[Mike Hibler]

This got complicated. Things are not booting correctly once I turned on
UEFI mode (which is necessary to get all the SEV features enabled correctly).
Still looking at it.

On Thu, Dec 22, 2022 at 07:52:57AM -0700, Mike Hibler wrote:
> Working on this now. Sorry for the delay, but it is quite an involved process
> as it take multiple reboot cycles.
>
> On Wed, Dec 21, 2022 at 10:54:19PM +0800, ????????? wrote:
> > Hi Cloudlab Team,
> >

> > My current experiment is??https://www.cloudlab.us/status.php?uuid=

> > b3ba8b6a-7ed4-11ed-b318-e4434b2381fc.
> >
> > I want to launch SEV-SNP on my node, but I'm encountering some problems.

> > First, SEV??is disabled in??BIOS by default,??so I??would like to enable??it.
> > In addition, according??to this link??https://github.com/AMDESE/AMDSEV/tree/
> > sev-snp-devel#prepare-host, I need to modify BIOS further to??get SEV-SNP's

> > functionality.
> >
> > Specifically, it mentions two things.
> >
> > One is the BIOS upgrade as follows:
> > ----------------------------------------------------------------------------
> > *NOTE: If your SEV-SNP firmware is older than 1.51, see the "Upgrade SEV
> > firmware" section to upgrade the firmware. *
> > ----------------------------------------------------------------------------
> >
> > Another one is the BIOS configuration as follows:
> > ----------------------------------------------------------------------------
> > Verify that the following BIOS settings are enabled. The setting may vary based
> > on the vendor BIOS. The menu option below are from AMD BIOS.

> > ????

> > CBS -> CPU Common ->

> > ?? ?? ?? ?? ?? ?? ?? ?? SEV-ES ASID space Limit Control -> Manual
> > ?? ?? ?? ?? ?? ?? ?? ?? SEV-ES ASID space limit -> 100
> > ?? ?? ?? ?? ?? ?? ?? ?? SNP Memory Coverage -> Enabled
> > ?? ?? ?? ?? ?? ?? ?? ?? SMEE -> Enabled
> > ?? ?? ?? ?? -> NBIO common ->
> > ?? ?? ?? ?? ?? ?? ?? ?? SEV-SNP -> Enabled
> >
> > ----------------------------------------------------------------------------
> > Since I cannot modify the BIOS setting by myself, I wonder if you could help??me
> > enable??SEV and set the BIOS setting as mentioned above?

> >
> > Thank you in advance!
> >
> > Sincerely,
> > Michael
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "cloudlab-users" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email
> > to cloudlab-user...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/
> > cloudlab-users/
> > CAKJOkPgNG3zO6XqaTsW24PoL4U5_6-ec7q5Zo15eC%2B0XMZ-bqg%40mail.gmail.com.
>
> --
> You received this message because you are subscribed to the Google Groups "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to cloudlab-user...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/cloudlab-users/20221222145257.GD23970%40flux.utah.edu.

[江立中]

Hi Mike,

I have tried installing Linux kernels of other versions on this node since your first reply until now, but the node always got stuck during the reboot after I updated grub with my newly installed kernel.
I looked into the state shown on the experiment website during the reboot.
The state first went into the "BOOTING" state and ended up with the "SHUTDOWN" state.
It was finally stuck here, and the only thing I could do was reloading the node again.

I wonder if the failure mentioned in your second reply was correlated with my issue here.
To prevent any unexpected issues, I have reloaded this node and will not use it until it is fixed correctly.

Thanks for your help!

Sincerely,
Michael

[Mike Hibler]

The problem (which is a known problem) appears to be the size of the
initramfs associated with the kernel. When the BIOS is in UEFI mode there
appears to be a rather small limit in grub on the amount of memory that
can be allocated and this is causing problems when trying to allocate space
for the ram disk. The only way I got your node to boot was to switch to an
older kernel with a < 50MB initramfs. The default Ubuntu 20 kernel we use
has a > 600MB initramfs.

We will try to work on this, but likely not til after the holidays.

> cloudlab-users/19aed4de-5bea-4a8e-8493-c41eed0c7a38n%40googlegroups.com.

[江立中]

No problem. 

I'll extend this experiment for more days to solve it.

Thanks.

[Mike Hibler]

I gather you also reloaded the OS on the node? Now it has our standard
Ubuntu20 kernel and boots fine in UEFI mode.

> cloudlab-users/31c94176-fe10-43aa-a62d-c1e3447fea86n%40googlegroups.com.

[江立中]

Sorry for that.
I was trying to verify the size of initramfs of the Linux kernel I compiled.
Because I accidentally rebooted the machine with that out-of-size kernel, I have to reload this node to change it back.

By the way, "Reload" the node and then "Reboot" the machine is the only way I can use the standard kernel in UEFI mode.
However, the problem still existed when I was installing the kernel of a newer version. (>600MB)

I'm considering decreasing the size of the initramfs I create to meet the size limits, but I'm not sure about the precise limit.

[Mike Hibler]

We have continued to look at this. Now it appears to be a combination of
UEFI and SEV that limits the amount of memory grub thinks it has on these
machines. We will likely integrate a new version of grub that fixes this,
but that might only be for Ubuntu 22, and maybe Ubuntu 20. The work around
for now (and in the future for older Ubuntu releases) is to keep your
initramfs small. How small is not clear, but certainly under 100MB.

> > cloudlab-users/31c94176-fe10-43aa-a62d-c1e3447fea86n%40googlegroups.com.

>
>
> --
> You received this message because you are subscribed to the Google Groups
> "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to cloudlab-user...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/

> cloudlab-users/12cebbd3-0ac0-49b2-a392-f9faad07db59n%40googlegroups.com.

[江立中]

Sorry for the delayed reply.
I've tried some ways to meet the limitation.
Finally, I decided not to use SNP until it allows me to use my kernel (around 180MB) without the current limitation.

Thank you for helping me address the problem. Very appreciate!

[Mike Hibler]

We have created a new Ubuntu22 image that includes an updated grub that
fixes the problem. If you can use the UBUNTU22-64-BETA image, things
should work for you.

> > cloudlab-users/12cebbd3-0ac0-49b2-a392-f9faad07db59n%40googlegroups.com.

>
>
> --
> You received this message because you are subscribed to the Google Groups
> "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to cloudlab-user...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/

> cloudlab-users/df7e7c20-4ddf-46ca-86b2-36031db765b3n%40googlegroups.com.

[江立中]

I just started a new experiment (https://www.cloudlab.us/status.php?uuid=7876966b-9213-11ed-b318-e4434b2381fc#) with UBUNTU22-64-BETA image to test it.

The default BIOS setting on this node enables "Secure Memory Encryption", "Secure Nested Paging", and "SNP Memory Coverage".

However, "Minimum SEV non-ES ASID" is 32 now, and it should be 100 according to AMD's tutorial.

Could you help me with this?

I have successfully installed my kernel (around 128MB) but cannot launch SEV nor SEV-SNP.

I will not try it until the BIOS is set up correctly.

Thank you very much!

[江立中]

Hi Mike,

The experiment mentioned in my last reply has expired due to another’s reservation, so I made a two-week reservation and started another experiment https://www.cloudlab.us/status.php?uuid=52322970-963b-11ed-b318-e4434b2381fc# with the UBUNTU-22.04 OS image.

On the machine I got, SEV is disabled in BIOS now. I want to set up the BIOS as this https://github.com/AMDESE/AMDSEV/tree/sev-snp-devel#prepare-host says.

Sorry for the inconvenience.

In addition to modifications about SEV-SNP, I want to modify the memory setting and cache prefetches in BIOS, as mentioned in my other post (https://groups.google.com/g/cloudlab-users/c/PkwMbOS6E-M).

Thank you in advance.

Sincerely,

Michael