Question for university project

Pietro Di Maro

unread,

Jan 13, 2023, 2:45:20 AM1/13/23

to muen-dev

Hello everyone,
my university partner and I have chosen to do a project on Muen Separation Kernel. We followed the guide on the site to create a bootable iso image for compatible hardware (Lenovo ThinkPad). By default the policy it runs is "demo_system_vtd.xml" which has 3 subjects: dbgserver, nic_linux, storage_linux. But as soon as we boot the pc all we see is the nic_linux terminal (with its muenfs folder) with no possibility to see anything else. Should we see the storage_linux terminal as well? In the policy it is defined as a console as well.

We, for the project, would simply like to do an isolation test where a real time task (in a VM maybe) prints maybe just the activation times and another subject (a native application or another VM) consuming cpu cycles, showing that the real time task is not impacted by the presence of this last subject. (But we don't know how to procede in the writing of the policy because all we see on screen is the "nic_linux" terminal).

I apologize in advance if this issue seems unclear and I am ready to answer and clarify any doubts about it.

Reto Buerki

unread,

Jan 13, 2023, 4:06:59 AM1/13/23

to Pietro Di Maro, muen-dev

Hi Pietro

Welcome to the list!

On 1/13/23 08:45, Pietro Di Maro wrote:
> Hello everyone,
> my university partner and I have chosen to do a project on Muen Separation
> Kernel. We followed the guide on the site to create a bootable iso image
> for compatible hardware (Lenovo ThinkPad). By default the policy it runs is
> "demo_system_vtd.xml" which has 3 subjects: dbgserver, nic_linux,
> storage_linux. But as soon as we boot the pc all we see is the nic_linux
> terminal (with its muenfs folder) with no possibility to see anything else.
> Should we see the storage_linux terminal as well? In the policy it is
> defined as a console as well.

Pressing F2 should give you the console of the storage Linux.

You can also ssh into storage Linux from NIC Linux:

$ ssh lnx2

> We, for the project, would simply like to do an isolation test where a real
> time task (in a VM maybe) prints maybe just the activation times and
> another subject (a native application or another VM) consuming cpu cycles,
> showing that the real time task is not impacted by the presence of this
> last subject. (But we don't know how to procede in the writing of the
> policy because all we see on screen is the "nic_linux" terminal).

Keep in mind that our VMs only have access to coarse-grained time, so it
might be better to implement this with native subjects (and allowing
them access to RDTSC).

Hope that helps.

Kind regards
- reto

Pietro Di Maro

unread,

Jan 13, 2023, 4:25:27 AM1/13/23

to muen-dev

Thanks for the quick response Reto.

Our idea was just to develop that with native subjects but we don't know where to start to do that.

Is there any similar examples of native subjects that we can look at to better understand?

Reto Buerki

unread,

Jan 13, 2023, 4:32:25 AM1/13/23

to Pietro Di Maro, muen-dev

On 1/13/23 10:25, Pietro Di Maro wrote:
> Thanks for the quick response Reto.
>
> Our idea was just to develop that with native subjects but we don't know
> where to start to do that.

There is the example component in components/example, which is also used
in the Component Specification [1] to explain how components/subjects
work and how their environment looks like.

Allowing access to TSC and calling RDTSC is illustrated in the time
component:

- components/time/spec/time.xml
- time/src/tm-main.adb:52

Kind regards
- reto

[1] - https://muen.sk/muen-component-spec.pdf

Alessandro Monaco

unread,

Jan 13, 2023, 7:42:22 AM1/13/23

to muen-dev

Hello Reto,

I'm Pietro's teammate on this university project on Muen SK.
Thanks for your fast and precious responses, we are now more deeply analyzing the component specification in order to reach our goal.

Can the debuglog channel be accessed/read from within nic_linux's terminal? If so, what's the command/file we should use?

Our idea is to write a simple system policy with only two subjects:
- subject 1: a VM like nic_linux on which we're able to run a cpu stress test and to read a communication channel shared with subject 2 (where subject 2 writes and subject 1reads)
- subject 2: a native subject that runs a simple periodic task (with fixed period) which only writes timing information on every activation (like "iteration n: activation time xxxxxx") on that channel
Our goal is to prove that subject 2's latency is not affected by extensive cpu utilization by subject 1

Best regards,

Alessandro Monaco.

Reto Buerki

unread,

Jan 13, 2023, 9:28:35 AM1/13/23

to Alessandro Monaco, muen-dev

Hi Alessandro

On 1/13/23 13:42, Alessandro Monaco wrote:
> I'm Pietro's teammate on this university project on Muen SK.

Welcome!

> Thanks for your fast and precious responses, we are now more deeply
> analyzing the component specification in order to reach our goal.
>

> Can the *debuglog* channel be accessed/read from within nic_linux's

> terminal? If so, what's the command/file we should use?

yes, the subject monitor (SM) of NIC Linux uses a debuglog channel to
emulate COM1. so you can just write to syslog, or directly to /dev/ttyS0:

$ echo test > /dev/ttyS0

> Our idea is to write a simple system policy with only two subjects:
> - subject 1: a VM like nic_linux on which we're able to run a cpu stress
> test and to read a communication channel shared with subject 2 (where
> subject 2 writes and subject 1reads)

The demo initramfs already contains `sysbench` if this helps.

> - subject 2: a native subject that runs a simple periodic task (with fixed
> period) which only writes timing information on every activation (like
> "iteration n: activation time xxxxxx") on that channel
> Our goal is to prove that subject 2's latency is not affected by extensive
> cpu utilization by subject 1

Sounds interesting, we are looking forward to read your findings.

Kind regards
- reto

Alessandro Monaco

unread,

Jan 13, 2023, 10:38:53 AM1/13/23

to muen-dev

I am not able to read from /dev/ttyS0

I have tried these commands:
$ cat /dev/ttyS0
$ more /dev/ttyS0

It always hangs on the command with blinking cursor and no output, until I exit with Ctrl+Z.

I have tried doing these commands on NIC Linux before, leaving the terminal on blinking cursor and switching to Storage Linux (by pressing F2) where I executed:

$ echo test > /dev/ttyS0

to see if I could achieve some kind of communication between the two vm subject (read the "test" echoed by Storage Linux on /dev/ttyS0 on NIC Linux) with no success unfortunately.

Best regards,

Alessandro.

Reto Buerki

unread,

Jan 13, 2023, 11:00:36 AM1/13/23

to Alessandro Monaco, muen-dev

On 1/13/23 16:38, Alessandro Monaco wrote:
>
> I am not able to read from /dev/ttyS0

You cannot read from ttyS0 as we do not emulate a full-blown serial
controller. /dev/ttyS0 is for NIC_Linux (debug) output only.

> I have tried these commands:
> $ cat /dev/ttyS0
> $ more /dev/ttyS0
>
> It always hangs on the command with blinking cursor and no output, until I
> exit with Ctrl+Z.
>
> I have tried doing these commands on NIC Linux before, leaving the terminal
> on blinking cursor and switching to Storage Linux (by pressing F2) where I
> executed:
> $ echo test > /dev/ttyS0
> to see if I could achieve some kind of communication between the two vm
> subject (read the "test" echoed by Storage Linux on /dev/ttyS0 on NIC
> Linux) with no success unfortunately.

The endpoint of a debuglog channel is the dbgserver subject, which
writes the data to the configured log sink. On your device this is
probably Intel AMT.

If you want to communicate from lnx1 to lnx2 use a network protocol of
your choice over net0, or add another custom muchannel.

Alessandro Monaco

unread,

Jan 14, 2023, 9:02:03 AM1/14/23

to Reto Buerki, muen-dev

Thanks again Reto for your help.

I have successfully configured Intel AMT on the Lenovo laptop and I accessed it through MeshCommander software (https://www.meshcommander.com/meshcommander) on another computer on the same network.

I have tried the Serial-over-LAN feature of AMT but it doesn't show anything on screen. (serial-over-lan attached screenshot).

I have then tried the command

$ echo test > /dev/ttyS0

on both lnx1 and lnx2, to see if "test" showed up in the serial-over-lan window on the other computer, but nothing happened, same empty window.

I have also checked the "Event Log" (log1 screenshot) and "Audit Log" (log2 screenshot) sections, to see if "test" showed up as a log entry there, but it doesn't.

In these two sections there seems to be nothing Muen-related, only log entries of hardware events like boot, bios accesses, etc.

Among all the .xml system policies already provided with Muen, which one do you think suits best the needs of our project, with the least/no modifications?

Is there any native subject component example that simply runs a periodic task, which we can easily modify in order to suit our needs?

Which route do you think is best for us to follow in order to reach the goal of our isolation test?

Best regards,

Alessandro Monaco.

--

Distinti saluti,

Alessandro Monaco

log1.png

log2.png

serial-over-lan.png

Alessandro Monaco

unread,

Jan 14, 2023, 9:05:41 AM1/14/23

to muen-dev

The serial-over-lan.png screenshot shows "Disconnected" status, but ofc it was "Connected" when I did those tests.

Sven Anderson

unread,

Jan 14, 2023, 10:04:18 AM1/14/23

to muen...@googlegroups.com

Reto Buerki <re...@codelabs.ch> schrieb am Fr. 13. Jan. 2023 um 17:00:

The endpoint of a debuglog channel is the dbgserver subject, which
writes the data to the configured log sink. On your device this is
probably Intel AMT.

Speaking of which, it would be great if the log sink could also be a virtual serial device that can be assigned to a Subject as an additional serial device (like you can assign real serial devices to subjects) in order to process it there in a dynamic fashion. (Basically what is possible already now if I would connect two serial ports with a null modem cable.)

Or is that actually possible and I’m not aware of it?

Cheers

Sven

--
You received this message because you are subscribed to the Google Groups "muen-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to muen-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/muen-dev/c94a925a-9a26-53fc-52e3-89982898a003%40codelabs.ch.

Reto Buerki

unread,

Jan 14, 2023, 10:49:53 AM1/14/23

to Alessandro Monaco, muen-dev

On 1/14/23 15:01, Alessandro Monaco wrote:
> Thanks again Reto for your help.
>
> I have successfully configured Intel AMT on the Lenovo laptop and I
> accessed it through MeshCommander software (
> https://www.meshcommander.com/meshcommander) on another computer on the
> same network.
> I have tried the Serial-over-LAN feature of AMT but it doesn't show
> anything on screen. (serial-over-lan attached screenshot).
>
> I have then tried the command
> $ echo test > /dev/ttyS0
> on both lnx1 and lnx2, to see if "test" showed up in the serial-over-lan
> window on the other computer, but nothing happened, same empty window.
>
> I have also checked the "Event Log" (log1 screenshot) and "Audit Log" (log2
> screenshot) sections, to see if "test" showed up as a log entry there, but
> it doesn't.
> In these two sections there seems to be nothing Muen-related, only log
> entries of hardware events like boot, bios accesses, etc.
>
> Among all the .xml system policies already provided with Muen, which one do
> you think suits best the needs of our project, with the least/no
> modifications?

I guess you are using hardware/lenovo-t440s.xml at the moment? While
this hardware configuration is quite similar, it might actually differ
quite a bit when it comes to specific resource locations etc.

Please generate a hardware description for your exact notebook model
using the mugenhwcfg-live image [1], then retry.

I haven't used MeshCommander a while, we prefer the more lightweight
amtterm and amtc tools. See our Dockerfile [2] on how to configure and
build/install these tools. Also, make sure you enable legacy sol
redirection on the T440 via MeshCommander or amtc:

$ amtc -d -M sol=on IP

Test your setup using a Linux distribution of your choice first.

> Is there any native subject component example that simply runs a periodic
> task, which we can easily modify in order to suit our needs?

I suggest you modify the example component for your isolation test
(remove most of the existing code, add the timing output and perform
some regular task). Make sure it runs on the same core as your
stresstest VM, in a distinct scheduling partition.

Kind regards
- reto

[1] - https://www.codelabs.ch/download/mugenhwcfg-live-amd64.hybrid.img.gz
[2] - tools/docker/Dockerfile.muen-dev-env

Reto Buerki

unread,

Jan 14, 2023, 11:34:12 AM1/14/23

to Sven Anderson, muen...@googlegroups.com

On 1/14/23 16:04, Sven Anderson wrote:
> Reto Buerki <re...@codelabs.ch> schrieb am Fr. 13. Jan. 2023 um 17:00:
>
>>
>> The endpoint of a debuglog channel is the dbgserver subject, which
>> writes the data to the configured log sink. On your device this is
>> probably Intel AMT.
>
>
> Speaking of which, it would be great if the log sink could also be a
> virtual serial device that can be assigned to a Subject as an additional
> serial device (like you can assign real serial devices to subjects) in
> order to process it there in a dynamic fashion. (Basically what is possible
> already now if I would connect two serial ports with a null modem cable.)

Not sure I understand the use-case. Can you please elaborate? Especially
what you mean by "process it there in a dynamic fashion."

Sven Anderson

unread,

Jan 14, 2023, 12:24:54 PM1/14/23

to Reto Buerki, muen...@googlegroups.com

Reto Buerki <re...@codelabs.ch> schrieb am Sa. 14. Jan. 2023 um 17:34:

On 1/14/23 16:04, Sven Anderson wrote:
> Speaking of which, it would be great if the log sink could also be a
> virtual serial device that can be assigned to a Subject as an additional
> serial device (like you can assign real serial devices to subjects) in
> order to process it there in a dynamic fashion. (Basically what is possible
> already now if I would connect two serial ports with a null modem cable.)

Not sure I understand the use-case. Can you please elaborate? Especially
what you mean by "process it there in a dynamic fashion."

Sorry, I thought that’s obvious, so I didn’t explain further. The use-case would be in general to make the log available without physical access to the serial interface, like over network, or to a filesystem. Our main application for example can log to a syslog server, but for all the log messages of the other auxiliary subjects you need access to the serial interface. Would be nice to be able to send it somewhere else that is defined at runtime, for example also a syslog-server, or a database.

Hope that makes it more clear.

Cheers

Sven

Reto Buerki

unread,

Jan 14, 2023, 12:50:41 PM1/14/23

to Sven Anderson, muen...@googlegroups.com

On 1/14/23 18:24, Sven Anderson wrote:
> Reto Buerki <re...@codelabs.ch> schrieb am Sa. 14. Jan. 2023 um 17:34:
>
>> On 1/14/23 16:04, Sven Anderson wrote:
>>> Speaking of which, it would be great if the log sink could also be a
>>> virtual serial device that can be assigned to a Subject as an additional
>>> serial device (like you can assign real serial devices to subjects) in
>>> order to process it there in a dynamic fashion. (Basically what is
>> possible
>>> already now if I would connect two serial ports with a null modem cable.)
>>
>> Not sure I understand the use-case. Can you please elaborate? Especially
>> what you mean by "process it there in a dynamic fashion."
>
>
> Sorry, I thought that’s obvious, so I didn’t explain further. The use-case
> would be in general to make the log available without physical access to
> the serial interface, like over network, or to a filesystem. Our main
> application for example can log to a syslog server, but for all the log
> messages of the other auxiliary subjects you need access to the serial
> interface. Would be nice to be able to send it somewhere else that is
> defined at runtime, for example also a syslog-server, or a database.

Thanks for clearly specifying what you actually want to achieve, makes
formulating a precise answer easier.

You could activate the "shmem" dbgserver logsink, assign it to a Linux
subject and process the data there in any way you please. See the
dbgserver_sink_shmem setting in the demo system policy.

There is also an example tool in the devel-desktop branch to read the
log sink channel from Linux.

Hope that helps
- reto

Sven Anderson

unread,

Jan 14, 2023, 7:05:06 PM1/14/23

to muen...@googlegroups.com

Am Sa., 14. Jan. 2023 um 18:50 Uhr schrieb Reto Buerki <re...@codelabs.ch>:

On 1/14/23 18:24, Sven Anderson wrote:
> Sorry, I thought that’s obvious, so I didn’t explain further. The use-case
> would be in general to make the log available without physical access to
> the serial interface, like over network, or to a filesystem. Our main
> application for example can log to a syslog server, but for all the log
> messages of the other auxiliary subjects you need access to the serial
> interface. Would be nice to be able to send it somewhere else that is
> defined at runtime, for example also a syslog-server, or a database.

You could activate the "shmem" dbgserver logsink, assign it to a Linux
subject and process the data there in any way you please. See the
dbgserver_sink_shmem setting in the demo system policy.

There is also an example tool in the devel-desktop branch to read the
log sink channel from Linux.

Oh, although that sounds more complicated than using a serial device, that is very good to know, thanks a lot! Indeed I saw the dbgserver_sink_shmem in the example profile in the system spec PDF before, but I couldn't find any documentation of it, so I wasn't sure what this is about. Actually I thought this is for logging _from_ subjects _to_ dbgserver. I will check out the demo system policy and the tool you mentioned, but is there any documentation written about that feature?

Thanks!

Sven

Reto Buerki

unread,

Jan 16, 2023, 4:10:01 AM1/16/23

to Sven Anderson, muen...@googlegroups.com

While only slightly more complex to setup in a Linux VM, it is easier to
implement in a native Ada(/SPARK) subject. That is always our priority.

There is no extra documentation for the demo system, as it uses the
mechanisms described in the spec documents. The shmstream channels are
specified in doc/shmstream.

Best regards
- reto

Message has been deleted

Sven Anderson

unread,

Jan 16, 2023, 8:51:10 AM1/16/23

to Reto Buerki, muen...@googlegroups.com

That was the important link, that I needed and was not obvious to me. Thanks a lot! I was not aware that shmem refers to shmstream and that there’s more documentation.

I only saw an Ada implementation in the devel-desktop branch in the latest commit, which is not so easy to read for me. There is no C implementation or something like that, right?

Cheers

Sven

Alessandro Monaco

unread,

Jan 16, 2023, 9:01:32 AM1/16/23

to muen-dev

Hello Reto,

we followed your instructions and used the muenhwcfg-live image to generate the output.xml file for our hardware. We placed it as hardware/lenovo-l440.xml and tried

$ make HARDWARE=hardware/lenovo-l440.xml SYSTEM=xml/demo_system_vtd.xml iso

but it gave as an error as it didn't find the corresponding platform/lenovo-l440.xml file.

As we didn't know how to generate that file, we simply duplicated the platform file of lenovo-t440s:

$ cp platform/lenovo-t440s.xml platform/lenovo-l440.xml

We then run the make iso command again but had another error, as he only found 2 cores instead of the 4 required, which is very strange, as both the Lenovo T440s and our L440 have different CPUs, but both with the same 2 physical cores with hyperthreading configuration (4 total logical cores). We were able to solve this problem by replacing policy/xml/demo_system_vtd.xml line 234:

with the entire content of the file policy/scheduling/static/demo_system_vtd-2-cpus.xml

At this stage, we were able to build the image and to boot it, and the Serial-over-Lan worked!

We are still using MeshCommander (with the legacy sol redirection enabled) and we're able to see both the logs of the 2 Linux VM's boot and also the output of the command:

$ echo test > /dev/ttyS0

(if we run this command both in lnx1 and lnx2 we see "test" output in the SoL window in MeshCommander, with different prefix based on which VM we execute it on).

Our problem now is that we haven't been able to have a native subject print on the serial-over-LAN.

We took a look at the example component code, we are not really expert at Ada programming but it seems that, even if we don't modify it, it still should print something (like the greeter message) on the log by default, but we are not able to see that in the SoL.

We also analyzed the default scheduling of demo_system_vtd-2-cpus.xml, which we're using for our policy, and, as the example component is part of the storage_linux scheduling group, which gets scheduled, the example subject should be scheduled too. We're not sure why we don't see any output from it on the serial:

- is it not producing any output?

- is it producing some output on a log channel which doesn't get redirected on the serial?

- is it not running at all?

We also tried removing the example subject from the storage_linux scheduling group and creating its own scheduling group example, which we then inserted in the scheduling's major frame replacing some of the slots which were assigned to the storage_linux group:

demo_system_vtd.xml

...

...

Best regards,

Alessandro.

Reto Buerki

unread,

Jan 16, 2023, 10:41:19 AM1/16/23

to Alessandro Monaco, muen-dev

Hi Alessandro

On 1/16/23 15:01, Alessandro Monaco wrote:
> Hello Reto,

> we followed your instructions and used the *muenhwcfg-live *image to
> generate the *output.xml *file for our hardware. We placed it as

> hardware/lenovo-l440.xml and tried
> $ make HARDWARE=hardware/lenovo-l440.xml SYSTEM=xml/demo_system_vtd.xml iso
> but it gave as an error as it didn't find the corresponding
> platform/lenovo-l440.xml file.
>
> As we didn't know how to generate that file, we simply duplicated the
> platform file of lenovo-t440s:
> $ cp platform/lenovo-t440s.xml platform/lenovo-l440.xml

Correct. The platform abstraction cannot be automatically generated, as
it defines common 'application layer' names defined by the system policy
author.

> We then run the make iso command again but had another error, as he only
> found 2 cores instead of the 4 required, which is very strange, as both the
> Lenovo T440s and our L440 have different CPUs, but both with the same 2
> physical cores with hyperthreading configuration (4 total logical cores).
> We were able to solve this problem by replacing
> policy/xml/demo_system_vtd.xml line 234:
>
> <include href="scheduling_plans.xml"/>
>
> with the entire content of the file
> policy/scheduling/static/demo_system_vtd-2-cpus.xml

You could also create a symlink from demo_system_vtd-lenovo-t440s.xml to
demo_system_vtd-2-cpus.xml.

> At this stage, we were able to build the image and to boot it, and the
> Serial-over-Lan worked!

Great!

> We are still using MeshCommander (with the legacy sol redirection enabled)
> and we're able to see both the logs of the 2 Linux VM's boot and also the
> output of the command:
> $ echo test > /dev/ttyS0
> (if we run this command both in lnx1 and lnx2 we see "test" output in the
> SoL window in MeshCommander, with different prefix based on which VM we
> execute it on).

So far so good.

> Our problem now is that we haven't been able to have a native subject print
> on the serial-over-LAN.

The log of the Linux VMs is written by a native subject monitor (SM) via
emulation of COM1. So log output from native subjects seem to work fine.
Can you please send the log?

Kind regards
- reto

Reto Buerki

unread,

Jan 16, 2023, 11:00:45 AM1/16/23

to Sven Anderson, muen...@googlegroups.com

You could use the Ada tool and pipe the output as needed. If this is not
feasible, there is a C implementation in the Linux kernel:

- components/linux/src/include/muen/reader.h
- components/linux/src/drivers/muen/channel/

Kind regards
- reto

Alessandro Monaco

unread,

Jan 16, 2023, 11:07:04 AM1/16/23

to muen-dev

Sure,

I'm attaching to this massage both the captured SoL log txt file and our current system policy (we've rolled it back to original demo_system_vtd-2-cpus.xml scheduling, without our modifications to scheduling groups and cycles), the log refers to this scheduling.

Anyway, while at the beginning of our tests the SoL log was nearly instantaneous on printing the Linux bootup log, now it has become very slow. Do you know what could've caused this?

Best regards,

Alessandro.

policy.xml

TerminalCapture-Lenovo-L440-Muen-SK-2023-01-16-16-54.txt

Reto Buerki

unread,

Jan 16, 2023, 11:46:14 AM1/16/23

to Alessandro Monaco, muen-dev

On 1/16/23 17:07, Alessandro Monaco wrote:
> Sure,
> I'm attaching to this massage both the captured SoL log txt file and our
> current system policy (we've rolled it back to original
> demo_system_vtd-2-cpus.xml scheduling, without our modifications to
> scheduling groups and cycles), the log refers to this scheduling.

Strange, the example subject output is indeed missing, all other output
looks fine. We are unable to reproduce this behavior using a Lenovo T440s.

Do you have any diff in components/example?

Can you please also send the file policy/obj/demo_system_vtd_b.xml?

Furthermore, please try the attached patch. It should produce a system
panic if the example subject is started.

> Anyway, while at the beginning of our tests the SoL log was nearly
> instantaneous on printing the Linux bootup log, now it has become very
> slow. Do you know what could've caused this?

Is your notebook connected to power or running on battery?

I would suggest you switch to amtc/amtterm, and for automatic deployment
use:

$ make deploy HARDWARE=hardware/lenovo-t440.xml -j`nproc`

Best regards
- reto

ud2.patch

Sven Anderson

unread,

Jan 16, 2023, 12:19:58 PM1/16/23

to Reto Buerki, muen...@googlegroups.com

Reto Buerki <re...@codelabs.ch> schrieb am Mo. 16. Jan. 2023 um 17:00:

> I only saw an Ada implementation in the devel-desktop branch in the latest
> commit, which is not so easy to read for me. There is no C implementation
> or something like that, right?

You could use the Ada tool and pipe the output as needed. If this is not
feasible, there is a C implementation in the Linux kernel:

- components/linux/src/include/muen/reader.h
- components/linux/src/drivers/muen/channel/

Ah, that’s the same. Ok, great. Our Linux subjects only run a monolithic Go executable, so I would prefer to integrate it there without forking into another tool. But good to know that is an option, if I will fail with a Go implementation.

But wait a second, can’t I just poll something in the muenfs then, like I can do for muen events? Adrian pointed me to this[1] before.

BTW: before I build the docs by myself, do you maybe have rendered versions of the shmstream2.tex file available somewhere?

Thanks a lot!

Cheers

Sven

[1] https://git.codelabs.ch/?p=muen/linux/muenfs.git;a=blob;f=test/muenfs-example.c;h=faab6fd0ca1f956ca24cd717846f16cbffcd9912;hb=a554aa0109d8c372e67f4b4d0a38fdcc5610743d

Adrian-Ken Rueegsegger

unread,

Jan 16, 2023, 2:04:52 PM1/16/23

to Sven Anderson, muen...@googlegroups.com, Reto Buerki

Hello Sven,

On 1/16/23 18:19, Sven Anderson wrote:
> Reto Buerki <re...@codelabs.ch> schrieb am Mo. 16. Jan. 2023 um 17:00:
>
>>> I only saw an Ada implementation in the devel-desktop branch in the
>> latest
>>> commit, which is not so easy to read for me. There is no C implementation
>>> or something like that, right?
>>
>> You could use the Ada tool and pipe the output as needed. If this is not
>> feasible, there is a C implementation in the Linux kernel:
>>
>> - components/linux/src/include/muen/reader.h
>> - components/linux/src/drivers/muen/channel/
>>
>
> Ah, that’s the same. Ok, great. Our Linux subjects only run a monolithic Go
> executable, so I would prefer to integrate it there without forking into
> another tool. But good to know that is an option, if I will fail with a Go
> implementation.

Not that I want to discourage you of implementing shmstreamv2 in Go ;)
but if I am not mistaken, u-root allows to package additional files via
the -files parameter. Instead of doing a reimplementation, would it not
be easier to build the mureadshmemlog tool statically and package it
into the u-root initramfs?

> But wait a second, can’t I just poll something in the muenfs then, like I
> can do for muen events? Adrian pointed me to this[1] before.

The shmem debug sink is mapped as a channel and exported via muenfs to
userspace. The mureadshmemlog tool reads this shmstream channel and
prints the log output present in the channel. Currently, the log sink
channels do not have an associated event for notification so some form
of polling is necessary.

> BTW: before I build the docs by myself, do you maybe have rendered versions
> of the shmstream2.tex file available somewhere?

See attached PDF.

Cheers,
Adrian

shmstream2.pdf

Alessandro Monaco

unread,

Jan 17, 2023, 6:42:45 AM1/17/23

to muen-dev

Il giorno lunedì 16 gennaio 2023 alle 17:46:14 UTC+1 Reto Buerki ha scritto:

On 1/16/23 17:07, Alessandro Monaco wrote:
> Sure,
> I'm attaching to this massage both the captured SoL log txt file and our
> current system policy (we've rolled it back to original
> demo_system_vtd-2-cpus.xml scheduling, without our modifications to
> scheduling groups and cycles), the log refers to this scheduling.

Strange, the example subject output is indeed missing, all other output
looks fine. We are unable to reproduce this behavior using a Lenovo T440s.

Do you have any diff in components/example?

We did git clone from scratch and only kept these 3 modified files:

- policy/xml/progetto_rtis.xml (our version of demo_system_vtd.xml, attached to my previous email named as policy.xml)

- policy/hardware/lenovo-l440.xml

- policy/platform/lenovo-l440.xml

Can you please also send the file policy/obj/demo_system_vtd_b.xml?

I attached it to this email (it is named progetto_rtis_b.xml in our environment, as the policy is named progetto_rtis.xml).

Furthermore, please try the attached patch. It should produce a system
panic if the example subject is started.

We applied the patch, make iso, dd on usb drive, booted it but nothing strange happened (no system panic or crash), it goes to NIC Linux's terminal, you can switch with F1/F2 to Storage Linux normally and execute commands on both, everything seems the same as before the patch. Do you need the log captured on SoL for this?

> Anyway, while at the beginning of our tests the SoL log was nearly
> instantaneous on printing the Linux bootup log, now it has become very
> slow. Do you know what could've caused this?

Is your notebook connected to power or running on battery?

All our tests were made with the power adapter connected, including this one.

I would suggest you switch to amtc/amtterm, and for automatic deployment
use:

We will try to learn how to use them and replace MeshCommander with those soon.

$ make deploy HARDWARE=hardware/lenovo-t440.xml -j`nproc`

We usually do:

$ make HARDWARE=hardware/lenovo-l440.xml SYSTEM=xml/progetto_rtis.xml iso

and then burn to usb drive with

$ sudo dd bs=1M if=/home/alessandro/muen/emulate/muen.iso of=/dev/sdb

what does this command do differently?

We tried the modified command:

$ make deploy HARDWARE=hardware/lenovo-l440.xml SYSTEM=xml/progetto_rtis.xml -j 8

as we're using progetto_rtis.xml as our policy and our Linux machine has 8 cores.

But the outcome was the following:

* Successfully created bootable Muen ISO '/home/alessandro/muen/emulate/muen.iso'
Either burn it on a CD-ROM or dump it on an USB stick:
$ sudo dd bs=1M if=/home/alessandro/muen/emulate/muen.iso of=/dev/sdX

HTTP start failed, check /home/alessandro/muen/deploy/http/error.log
make[1]: *** [Makefile:55: .server] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: *** [Makefile:65: target_start_amt] Error 1
make: *** [Makefile:53: deploy] Error 2

(we should always use -j 8 for faster compiling times I think, thanks)

Best regards,

Alessandro.

progetto_rtis_b.xml

Reto Buerki

unread,

Jan 17, 2023, 12:39:38 PM1/17/23

to Alessandro Monaco, muen-dev

On 1/17/23 12:42, Alessandro Monaco wrote:
> Il giorno lunedì 16 gennaio 2023 alle 17:46:14 UTC+1 Reto Buerki ha scritto:
> On 1/16/23 17:07, Alessandro Monaco wrote:
>> Sure,
>> I'm attaching to this massage both the captured SoL log txt file and our
>> current system policy (we've rolled it back to original
>> demo_system_vtd-2-cpus.xml scheduling, without our modifications to
>> scheduling groups and cycles), the log refers to this scheduling.
>
> Strange, the example subject output is indeed missing, all other output
> looks fine. We are unable to reproduce this behavior using a Lenovo T440s.
>
> Do you have any diff in components/example?
>
> We did git clone from scratch and only kept these 3 modified files:
> - policy/xml/progetto_rtis.xml (our version of demo_system_vtd.xml,
> attached to my previous email named as policy.xml)
> - policy/hardware/lenovo-l440.xml
> - policy/platform/lenovo-l440.xml
>
> Can you please also send the file policy/obj/demo_system_vtd_b.xml?
>
> I attached it to this email (it is named progetto_rtis_b.xml in our
> environment, as the policy is named progetto_rtis.xml).

Thanks for the file, nothing unexpected IMO.

Can you please try the attached patch? It looks like the example subject
remains in the initial syncing state.

Until we nailed this issue, you could use `make emulate` [1] to continue
the development of your project. Just keep in mind that the timing
values will be off due to KVM/QEMU.

Please see [2] on how to use deploy with network boot.

Kind regards,
- reto

[1] - https://muen.sk/#emulation
[2] - https://muen.sk/#network-boot

0001-Make-Example-component-self-governed.patch

Alessandro Monaco

unread,

Jan 18, 2023, 6:27:26 AM1/18/23

to muen-dev

We tried the new patch, now they are both applied, tried it but nothing changed.

In the meantime, we are trying to successfully deploy in qemu.

Best regards,

Alessandro.

Alessandro Monaco

unread,

Jan 18, 2023, 7:44:35 AM1/18/23

to muen-dev

We are having some problems with the emulation, we installed qemu

$ sudo apt install qemu-system-x86_64

then executed

$ make SYSTEM=xml/progetto_rtis.xml emulate -j 8

but we were having some strange errors.

As apt installed qemu 6.2.0, which is not the latest version, we uninstalled it and downloaded qemu 7.2.0 from official website, compiled it and added it to PATH.

Then we run again

$ make SYSTEM=xml/progetto_rtis.xml emulate -j 8

but we get this error (emulate.out):

qemu-system-x86_64: -curses: invalid option

We found out that "-curses" is a deprecated option, and we tried replacing it with "-display curses", manually running the following command in /muen/emulate directory:

$ qemu-system-x86_64 -display curses -drive file=muen.iso,index=0,media=disk,format=raw -serial file:serial.out -machine q35,accel=kvm,kernel-irqchip=split -cpu IvyBridge-IBRS,+invtsc,+vmx -m 5120 -smp cores=2,threads=2,sockets=1 -device intel-iommu,intremap=on,device-iotlb=on -device ioh3420,id=pcie.1,chassis=1 -device virtio-net-pci,bus=pcie.1,addr=0.0,netdev=net0,disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on -netdev user,id=net0,net=192.168.254.0/24,dhcpstart=192.168.254.100,hostfwd=tcp::5555-:22 -device qemu-xhci,id=xhci,bus=pcie.0,addr=3.0 -device usb-tablet,bus=xhci.0 -device rtl8139,bus=pcie.0,addr=4.0,netdev=net1 -netdev user,id=net1,net=192.168.253.0/24,dhcpstart=192.168.253.100

But we now get this error, which we haven't been able to solve:

qemu-system-x86_64: -netdev user,id=net0,net=192.168.254.0/24,dhcpstart=192.168.254.100,hostfwd=tcp::5555-:22: network backend 'user' is not compiled into this binary

Best regards,

Alessandro.

Alessandro Monaco

unread,

Jan 18, 2023, 7:59:05 AM1/18/23

to muen-dev

We are running QEMU on a Ubuntu virtual machine, so maybe that's the problem, even if we have enabled "Enable hypervisor applications in this virtual machine" and "Enable IOMMU in this virtual machine" in VMWare Fusion (we are running it on an Intel x86 MacBook Pro)

Reto Buerki

unread,

Jan 18, 2023, 8:17:26 AM1/18/23

to Alessandro Monaco, muen-dev

On 1/18/23 13:59, Alessandro Monaco wrote:
> We are running QEMU on a Ubuntu virtual machine, so maybe that's the
> problem, even if we have enabled "Enable hypervisor applications in this
> virtual machine" and "Enable IOMMU in this virtual machine" in VMWare
> Fusion (we are running it on an Intel x86 MacBook Pro)

I don't think this will work in practice, as running Muen with subjects
on KVM/QEMU is already nested, you are adding another virtualization
layer with VMWare. While theoretically/technically possible, I would
advise to use a Debian/Ubuntu on hardware as development host.

We are currently using QEMU 5.2 from Debian Bullseye, this should work
out of the box.

- reto

> Il giorno mercoledì 18 gennaio 2023 alle 13:44:35 UTC+1 Alessandro Monaco
> ha scritto:
>
>> We are having some problems with the emulation, we installed qemu
>> $ sudo apt install qemu-system-x86_64
>> then executed
>> $ make SYSTEM=xml/progetto_rtis.xml emulate -j 8
>> but we were having some strange errors.
>>
>> As apt installed qemu 6.2.0, which is not the latest version, we
>> uninstalled it and downloaded qemu 7.2.0 from official website, compiled it
>> and added it to PATH.
>> Then we run again
>> $ make SYSTEM=xml/progetto_rtis.xml emulate -j 8
>> but we get this error (emulate.out):
>> qemu-system-x86_64: -curses: invalid option
>>
>> We found out that "-curses" is a deprecated option, and we tried replacing
>> it with "-display curses", manually running the following command in
>> /muen/emulate directory:

>> $ qemu-system-x86_64 -*display curses* -drive

Alessandro Monaco

unread,

Jan 18, 2023, 8:41:50 AM1/18/23

to muen-dev

For the KVM GUEST_CR3 fix, is it enough to have a Linux kernel (>= 5.2.0) or is a manual patch + kernel recompilation needed?

Reto Buerki

unread,

Jan 18, 2023, 8:43:13 AM1/18/23

to Alessandro Monaco, muen-dev

On 1/18/23 14:41, Alessandro Monaco wrote:
> For the KVM GUEST_CR3 fix, is it enough to have a Linux kernel (>= 5.2.0)
> or is a manual patch + kernel recompilation needed?

The Linux kernel of a recent Linux distro is enough.

- reto

Pietro Di Maro

unread,

Jan 18, 2023, 10:04:18 AM1/18/23

to muen-dev

We tried the new patch, now they are both applied, tried it but nothing changed.

What do you suggest to do for bootable usb stick method??? Do we have to change our focus to QEMU?

Reto Buerki

unread,

Jan 18, 2023, 10:45:51 AM1/18/23

to Pietro Di Maro, muen-dev

On 1/18/23 16:04, Pietro Di Maro wrote:
>
> We tried the new patch, now they are both applied, tried it but nothing
> changed.

yes, Alessandro already mentioned that. Since we cannot reproduce the
issue here it is hard to debug from our side. Do you have access to some
other Muen-capable hardware?

An interesting information would be whether example output is visible in
emulate/serial.out once you got the emulate setup running.

> What do you suggest to do for bootable usb stick method??? Do we have to
> change our focus to QEMU?

QEMU is just useful during system development, to get the policy and
component code ready. The actual timing measurements must be performed
on real hardware to be meaningful in this context.

- reto

Pietro Di Maro

unread,

Jan 18, 2023, 5:52:58 PM1/18/23

to muen-dev

Hi Reto,

we want to update you about our progress.

we succeded running muen (policy "demo_system_vtd.xml") with QEMU on a Debian 11 distro and we are able to see the example subject running in the "Serial.out" file.

Now we want to change the "example.adb" file in order to make the subject a simple periodic task.

I've modified the file but when i tried to recompile all with:

$ make emulate -j 4

we are facing a series of error/warning in the building resulting in a failure of the make command.

I send to you the example.adb file that i've modified (i'm not sure if some language errors are present) and the log of the make command.

best regards

Pietro

components.log.build_spark_debug

example.adb

Reto Buerki

unread,

Jan 19, 2023, 3:41:34 AM1/19/23

to Pietro Di Maro, muen-dev

Hi Pietro

On 1/18/23 23:52, Pietro Di Maro wrote:
> we want to update you about our progress.
>
> we succeded running muen (policy "demo_system_vtd.xml") with QEMU on a
> Debian 11 distro and we are able to see the example subject running in the
> "Serial.out" file.

Ok, thanks for the info.

> Now we want to change the "example.adb" file in order to make the subject a
> simple periodic task.
>
> I've modified the file but when i tried to recompile all with:
>
> $ make emulate -j 4
>
> we are facing a series of error/warning in the building resulting in a
> failure of the make command.
>
> I send to you the example.adb file that i've modified (i'm not sure if some
> language errors are present) and the log of the make command.

You need to fix the warnings and style issues shown in the log, as
components are compiled with 'treat all warnings as errors'. During
development, you could also disable this by removing -gnate from
compiler switches (config/shared.gpr).

- reto

Message has been deleted

Pietro Di Maro

unread,

Jan 19, 2023, 4:52:37 AM1/19/23

to muen-dev

I don't know if the "bad indentation" is causing this build to fail, but I don't think so.

However I don't know how to fix the other warnings starting from row 439 of the log.

I've seen the "config/shared.gpr" file but there is no -gnate but instead -gnatef -gnatd and so on, Do I have to remove them all?

components.log.build_spark_debug

Pietro Di Maro

unread,

Jan 19, 2023, 5:48:12 AM1/19/23

to muen-dev

I've fixed the indentation style issue, now the build is giving me some link failures.

I'm attaching to this message the log and the example.adb files to make it more clear.

components.log.build_spark_debug

example.adb

Alessandro Monaco

unread,

Jan 22, 2023, 11:47:09 AM1/22/23

to muen-dev

Hello,

we would like to update you with the current progress with our project.

We were able to:

- modify example component's Ada/Spark code to print useful (for our goal) timing information on the syslog (serial)

- modify the demo_system_vtd-qemu-kvm.xml scheduling plan, inserting example and storage_linux in two different scheduling groups, in two different partitions, but scheduled on the same CPU core

- compile and successfully emulate this system on QEMU

We are still unable to replicate this behavior on our hardware unfortunately, the example component subject doesn't execute like in QEMU, might it be related to the platform xml file? (we are using the Lenovo-T440S one, yet on a different hardware configuration, while the hardware xml is the correct one generated by mugenhwconfig live iso).

We can't figure out why the same policy, with the same scheduling, same components and subject is working in QEMU but not running bare metal on our Lenovo L440 notebook.

Best regards,

Alessandro.

Reto Buerki

unread,

Jan 23, 2023, 3:28:07 AM1/23/23

to Alessandro Monaco, muen-dev

Hi,

On 1/22/23 17:47, Alessandro Monaco wrote:
> we would like to update you with the current progress with our project.
> We were able to:
> - modify example component's Ada/Spark code to print useful (for our goal)
> timing information on the syslog (serial)

> - modify the *demo_system_vtd-qemu-kvm.xml* scheduling plan, inserting

> example and storage_linux in two different scheduling groups, in two
> different partitions, but scheduled on the same CPU core
> - compile and successfully emulate this system on QEMU

That is great progress, thanks for the update!

> We are still unable to replicate this behavior on our hardware
> unfortunately, the example component subject doesn't execute like in QEMU,
> might it be related to the platform xml file? (we are using the
> Lenovo-T440S one, yet on a different hardware configuration, while the
> hardware xml is the correct one generated by mugenhwconfig live iso).
> We can't figure out why the same policy, with the same scheduling, same
> components and subject is working in QEMU but not running bare metal on our
> Lenovo L440 notebook.

As we do not have a L440 notebook here, it is hard for us to reproduce
the behavior. Do you have access to another notebook?

Also, did you port your scheduling plan changes from
demo_system_vtd-qemu-kvm.xml to demo_system_vtd-l440.xml?

Best regards
- reto

Alessandro Monaco

unread,

Jan 24, 2023, 7:35:03 AM1/24/23

to muen-dev

Good morning,

unfortunately we don't have any other notebooks to run Muen (they are either MacBooks, have AMD cpu or don't have Intel AMT to read the serial).

Anyway, we have tried running the same system (policy, scheduling, example component code) we were emulating on Qemu until now.

In the first attempt we were using the following:

- HARDWARE = hardware/lenovo-l440.xml (generated with mugenhwcfg-live on our notebook)

- platform/lenovo-l440.xml (renamed copy of your platform/lenovo-t440s.xml)

- SYSTEM = xml/demo_system_vtd.xml (unmodified from the git cloned one)

- scheduling is scheduling/static/demo_system_vtd-lenovo-l440.xml (created and modified by us) and we've uploaded it here https://pastebin.com/nAD192cU

- example component source folder is completely unmodified

This attempt was very successful, as Muen boots and we can read example subject's messages on the serial log, which we've never been able to achieve before.

So we've tried the same exact configuration but with our edited example component.

The only two files we've modifified are:

- components/example/src/example.adb (to remove anything unnecessary and to print useful information for our tests on the serial) https://pastebin.com/kUcthdBd

- components/example/spec/example.xml (to allow RDTSC access) https://pastebin.com/x7UCKU05 (what does the print_serial boolean here stand for? It was set to false in the previous attempt too, but the example subject was printing on the sol).

This attempt was unsuccessful: Muen boots, but has difficulty with printing on the serial log (we can't read almost anything) and the system is overall slower.

Our guess is that the example task is printing too much lines on the log, on every of its executions, probably overloading the separation kernel for their management.

Best regards,

Alessandro.

Reto Buerki

unread,

Jan 24, 2023, 9:17:43 AM1/24/23

to Alessandro Monaco, muen-dev

This setting is used in the generated documentation [1] to illustrate
the use of integer config variables in the component code. Serial is
"serial number" in this context (see code and doc).

> This attempt was unsuccessful: Muen boots, but has difficulty with printing
> on the serial log (we can't read almost anything) and the system is overall
> slower.
>
> Our guess is that the example task is printing too much lines on the log,
> on every of its executions, probably overloading the separation kernel for
> their management.

This is very improbable, since the Muen SK has nothing to do with
dbgserver / serial processing.

Can you please send us the following:
- Muen repository git revision your work is based on
- git-format'ed patches of your changes
- the exact command used to build/deploy the system image

Thanks
- reto

[1] - https://muen.sk/muen-component-spec.pdf, chapter 4

Pietro Di Maro

unread,

Jan 26, 2023, 4:38:55 AM1/26/23

to muen-dev

Hi Reto,

we're encountering problems with timed events in the example subject.

If we have a tick rate of 1000 in our scheduling plan and example subject has a minor frame of 100 ticks (so 0.1 second if i'm correct),

how can we make the example subject periodic with a period of 2 seconds (for instance) ? So it executes then goes to sleep for the remaining time and it is woke up (at least) 2 seconds after its previous activation.

Our guess was to have a code like this.

declare
use type SK.Word64;

Next_Activation : SK.Word64 := Musinfo.Instance.TSC_Schedule_Start;
--Minor_End : SK.Word64 := Musinfo.Instance.TSC_Schedule_End;

Timed_Events.Timed_Evt.Event_Nr := Example_Component.Events.Timer_ID;
begin

loop

-- does job

Next_Activation := Next_Activation + 2000; -- (is it correct?)

Timed_Events.Timed_Evt.TSC_Trigger_Value := Next_Activation;

SK.Hypercall.Trigger_Event (Number => Example_Component.Events.Sleep_ID);

end loop;
end;

is it correct doing it like this? is that constant +2000 indicating 2 seconds period ?

Message has been deleted

Alessandro Monaco

unread,

Jan 26, 2023, 9:07:36 AM1/26/23

to muen-dev

The example subject's behavior we would like to achieve is the following:
- example is periodic task with a fixed period (es. 2000 milliseconds)

first start:

- example starts the first time when the system is booted

- as soon as it starts the first time, it saves the timestamp of this first activation (activation = current_time)

each period/activation/loop iteration:

- the task does its work (in our case some prints of these timings on the syslog)

- the last activation time will be used as a base for all the following ones, activation variable is increased by a period (activation = activation + 2000 milliseconds)

- a subject sleep is called, with a timed event for its wakeup on "activation"

- when it wakes up it will run this loop again

Best regards,

Alessandro.

Reto Buerki

unread,

Jan 27, 2023, 3:13:58 AM1/27/23

to Pietro Di Maro, muen-dev

On 1/26/23 10:38, Pietro Di Maro wrote:
> we're encountering problems with timed events in the example subject.
>
> If we have a tick rate of 1000 in our scheduling plan and example subject
> has a minor frame of 100 ticks (so 0.1 second if i'm correct),
>
> how can we make the example subject periodic with a period of 2 seconds
> (for instance) ? So it executes then goes to sleep for the remaining time
> and it is woke up (at least) 2 seconds after its previous activation.
>
> Our guess was to have a code like this.
>
> declare
> use type SK.Word64;
> Next_Activation : SK.Word64 := Musinfo.Instance.TSC_Schedule_Start;
> --Minor_End : SK.Word64 := Musinfo.Instance.TSC_Schedule_End;
> Timed_Events.Timed_Evt.Event_Nr := Example_Component.Events.Timer_ID;
> begin
>
> loop
>
> -- does job
>
>
> Next_Activation := Next_Activation + 2000; -- (is it correct?)
> Timed_Events.Timed_Evt.TSC_Trigger_Value := Next_Activation;

As the variable name indicates, the trigger value is in TSC ticks (see
also [1], 8.7). You need to calculate the desired activation time from
TSC_Khz e.g. from Musinfo.

Best regards

Alessandro Monaco

unread,

Jan 30, 2023, 10:26:14 AM1/30/23

to muen-dev

Good afternoon,
me and Pietro finished our project, thanks for all the support and advice you gave us.

We would like to share our little presentation which summarizes all our findings on the isolation tests, both on Qemu and on real hardware.

Best regards,

Alessandro & Pietro.

slides_pdf_v2.pdf

Reto Buerki

unread,

Jan 31, 2023, 4:22:38 AM1/31/23

to muen...@googlegroups.com

On 1/30/23 16:26, Alessandro Monaco wrote:
> me and Pietro finished our project, thanks for all the support and advice
> you gave us.

Great that you were able to resolve all your issues.

> We would like to share our little presentation which summarizes all our
> findings on the isolation tests, both on Qemu and on real hardware.

Thanks for sharing your findings! Now that this project is complete, do
you have other projects with Muen on your roadmap?

Kind regards
- reto

Reply all

Reply to author

Forward