5.5 fails to boot
73 views
Skip to first unread message
Ilpo Järvinen
Jan 27, 2020, 6:29:47 AM1/27/20
to Qubes devel
Hi,
I've noticed that 5.5-rcs and now 5.5 fail to boot in VM (at least when
booted as in VM kernel using pvgrub2-pvh). 5.4 based kernels worked fine
and there seems to have been some changes in drivers/xen post-5.4. Should
I escalate this to xen-devel?
Loading Linux 5.5.0-accecn30 ...
.[5;22H [ initrd.img-5.5.0-acc 16.52MiB 100% 10.23MiB/s ].[5;1HSetting up swapspace version 1, size = 1073737728 bytes
/dev/xvda3: clean, 852118/1294896 files, 3076785/5190907 blocks
[ 2.730931] BUG: kernel NULL pointer dereference, address: 00000000000003b0
[ 2.730959] #PF: supervisor read access in kernel mode
[ 2.730966] #PF: error_code(0x0000) - not-present page
[ 2.730973] PGD 0 P4D 0
[ 2.730978] Oops: 0000 [#1] SMP PTI
[ 2.730985] CPU: 1 PID: 402 Comm: qubesdb-daemon Tainted: G O 5.5.0-accecn30 #31
[ 2.731000] RIP: 0010:mmu_interval_read_begin+0x24/0xc0
[ 2.731008] Code: e9 51 66 e1 ff 90 0f 1f 44 00 00 41 54 49 89 fc 55 53 48 83 ec 30 65 48 8b 04 25 28 00 00 00 48 89 44 24 28 31 c0 48 8b 47 38 <48> 8b a8 b0 03 00 00 48 8d 5d 0c 48 89 df e8 49 27 6f 00 4d 8b 64
[ 2.731030] RSP: 0018:ffff9873001e7d20 EFLAGS: 00010246
[ 2.731037] RAX: 0000000000000000 RBX: ffff8a4e94712500 RCX: 0000000000000000
[ 2.731047] RDX: ffff8a4ef53add00 RSI: 0000000000000000 RDI: ffff8a4e94712500
[ 2.731057] RBP: ffff8a4e0bf7a640 R08: 00007bc5c0573000 R09: 0000000000000008
[ 2.731066] R10: ffff8a4ec756c190 R11: 00007bc5c05a2000 R12: ffff8a4e94712500
[ 2.731076] R13: ffff8a4ed3ab9d50 R14: 0000000000000000 R15: 0000000000000001
[ 2.731086] FS: 00007bc5c00dc7c0(0000) GS:ffff8a4ef5d00000(0000) knlGS:0000000000000000
[ 2.731097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2.731105] CR2: 00000000000003b0 CR3: 000000008148e004 CR4: 00000000003606e0
[ 2.731116] Call Trace:
[ 2.731123] ? vma_merge+0xef/0x370
[ 2.731132] gntdev_mmap+0x153/0x30e [xen_gntdev]
[ 2.731139] mmap_region+0x3d9/0x660
[ 2.731146] do_mmap+0x372/0x520
[ 2.731153] vm_mmap_pgoff+0xd2/0x120
[ 2.731160] ksys_mmap_pgoff+0x1b8/0x270
[ 2.731167] ? ksys_ioctl+0x60/0x90
[ 2.731174] do_syscall_64+0x5b/0x180
[ 2.731182] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 2.731191] RIP: 0033:0x7bc5c03e8133
[ 2.731196] Code: 54 41 89 d4 55 48 89 fd 53 4c 89 cb 48 85 ff 74 56 49 89 d9 45 89 f8 45 89 f2 44 89 e2 4c 89 ee 48 89 ef b8 09 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 7d 5b 5d 41 5c 41 5d 41 5e 41 5f c3 66 2e 0f
[ 2.731219] RSP: 002b:00007ffcbccc89b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000009
[ 2.731230] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007bc5c03e8133
[ 2.731243] RDX: 0000000000000003 RSI: 0000000000001000 RDI: 0000000000000000
[ 2.731252] RBP: 0000000000000000 R08: 0000000000000007 R09: 0000000000000000
[ 2.731263] R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000003
[ 2.731273] R13: 0000000000001000 R14: 0000000000000001 R15: 0000000000000007
[ 2.731284] Modules linked in: xen_netback u2mfn(O) xen_gntdev xen_gntalloc xen_blkback xen_evtchn parport_pc ppdev xenfs xen_privcmd lp parport ip_tables xen_netfront xen_blkfront crc32c_intel
[ 2.731309] CR2: 00000000000003b0
[ 2.731315] fbcon: Taking over console
[ 2.731321] ---[ end trace 5ec57aa3f3a40247 ]---
[ 2.731329] RIP: 0010:mmu_interval_read_begin+0x24/0xc0
[ 2.731336] Code: e9 51 66 e1 ff 90 0f 1f 44 00 00 41 54 49 89 fc 55 53 48 83 ec 30 65 48 8b 04 25 28 00 00 00 48 89 44 24 28 31 c0 48 8b 47 38 <48> 8b a8 b0 03 00 00 48 8d 5d 0c 48 89 df e8 49 27 6f 00 4d 8b 64
[ 2.731358] RSP: 0018:ffff9873001e7d20 EFLAGS: 00010246
[ 2.731365] RAX: 0000000000000000 RBX: ffff8a4e94712500 RCX: 0000000000000000
[ 2.731375] RDX: ffff8a4ef53add00 RSI: 0000000000000000 RDI: ffff8a4e94712500
[ 2.731385] RBP: ffff8a4e0bf7a640 R08: 00007bc5c0573000 R09: 0000000000000008
[ 2.731395] R10: ffff8a4ec756c190 R11: 00007bc5c05a2000 R12: ffff8a4e94712500
[ 2.731405] R13: ffff8a4ed3ab9d50 R14: 0000000000000000 R15: 0000000000000001
[ 2.731415] FS: 00007bc5c00dc7c0(0000) GS:ffff8a4ef5d00000(0000) knlGS:0000000000000000
[ 2.731427] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2.731436] CR2: 00000000000003b0 CR3: 000000008148e004 CR4: 00000000003606e0
[ 2.731446] Kernel panic - not syncing: Fatal exception
[ 2.731527] Kernel Offset: 0x2a000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
--
i.
I've noticed that 5.5-rcs and now 5.5 fail to boot in VM (at least when
booted as in VM kernel using pvgrub2-pvh). 5.4 based kernels worked fine
and there seems to have been some changes in drivers/xen post-5.4. Should
I escalate this to xen-devel?
Loading Linux 5.5.0-accecn30 ...
.[5;22H [ initrd.img-5.5.0-acc 16.52MiB 100% 10.23MiB/s ].[5;1HSetting up swapspace version 1, size = 1073737728 bytes
/dev/xvda3: clean, 852118/1294896 files, 3076785/5190907 blocks
[ 2.730931] BUG: kernel NULL pointer dereference, address: 00000000000003b0
[ 2.730959] #PF: supervisor read access in kernel mode
[ 2.730966] #PF: error_code(0x0000) - not-present page
[ 2.730973] PGD 0 P4D 0
[ 2.730978] Oops: 0000 [#1] SMP PTI
[ 2.730985] CPU: 1 PID: 402 Comm: qubesdb-daemon Tainted: G O 5.5.0-accecn30 #31
[ 2.731000] RIP: 0010:mmu_interval_read_begin+0x24/0xc0
[ 2.731008] Code: e9 51 66 e1 ff 90 0f 1f 44 00 00 41 54 49 89 fc 55 53 48 83 ec 30 65 48 8b 04 25 28 00 00 00 48 89 44 24 28 31 c0 48 8b 47 38 <48> 8b a8 b0 03 00 00 48 8d 5d 0c 48 89 df e8 49 27 6f 00 4d 8b 64
[ 2.731030] RSP: 0018:ffff9873001e7d20 EFLAGS: 00010246
[ 2.731037] RAX: 0000000000000000 RBX: ffff8a4e94712500 RCX: 0000000000000000
[ 2.731047] RDX: ffff8a4ef53add00 RSI: 0000000000000000 RDI: ffff8a4e94712500
[ 2.731057] RBP: ffff8a4e0bf7a640 R08: 00007bc5c0573000 R09: 0000000000000008
[ 2.731066] R10: ffff8a4ec756c190 R11: 00007bc5c05a2000 R12: ffff8a4e94712500
[ 2.731076] R13: ffff8a4ed3ab9d50 R14: 0000000000000000 R15: 0000000000000001
[ 2.731086] FS: 00007bc5c00dc7c0(0000) GS:ffff8a4ef5d00000(0000) knlGS:0000000000000000
[ 2.731097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2.731105] CR2: 00000000000003b0 CR3: 000000008148e004 CR4: 00000000003606e0
[ 2.731116] Call Trace:
[ 2.731123] ? vma_merge+0xef/0x370
[ 2.731132] gntdev_mmap+0x153/0x30e [xen_gntdev]
[ 2.731139] mmap_region+0x3d9/0x660
[ 2.731146] do_mmap+0x372/0x520
[ 2.731153] vm_mmap_pgoff+0xd2/0x120
[ 2.731160] ksys_mmap_pgoff+0x1b8/0x270
[ 2.731167] ? ksys_ioctl+0x60/0x90
[ 2.731174] do_syscall_64+0x5b/0x180
[ 2.731182] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 2.731191] RIP: 0033:0x7bc5c03e8133
[ 2.731196] Code: 54 41 89 d4 55 48 89 fd 53 4c 89 cb 48 85 ff 74 56 49 89 d9 45 89 f8 45 89 f2 44 89 e2 4c 89 ee 48 89 ef b8 09 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 7d 5b 5d 41 5c 41 5d 41 5e 41 5f c3 66 2e 0f
[ 2.731219] RSP: 002b:00007ffcbccc89b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000009
[ 2.731230] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007bc5c03e8133
[ 2.731243] RDX: 0000000000000003 RSI: 0000000000001000 RDI: 0000000000000000
[ 2.731252] RBP: 0000000000000000 R08: 0000000000000007 R09: 0000000000000000
[ 2.731263] R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000003
[ 2.731273] R13: 0000000000001000 R14: 0000000000000001 R15: 0000000000000007
[ 2.731284] Modules linked in: xen_netback u2mfn(O) xen_gntdev xen_gntalloc xen_blkback xen_evtchn parport_pc ppdev xenfs xen_privcmd lp parport ip_tables xen_netfront xen_blkfront crc32c_intel
[ 2.731309] CR2: 00000000000003b0
[ 2.731315] fbcon: Taking over console
[ 2.731321] ---[ end trace 5ec57aa3f3a40247 ]---
[ 2.731329] RIP: 0010:mmu_interval_read_begin+0x24/0xc0
[ 2.731336] Code: e9 51 66 e1 ff 90 0f 1f 44 00 00 41 54 49 89 fc 55 53 48 83 ec 30 65 48 8b 04 25 28 00 00 00 48 89 44 24 28 31 c0 48 8b 47 38 <48> 8b a8 b0 03 00 00 48 8d 5d 0c 48 89 df e8 49 27 6f 00 4d 8b 64
[ 2.731358] RSP: 0018:ffff9873001e7d20 EFLAGS: 00010246
[ 2.731365] RAX: 0000000000000000 RBX: ffff8a4e94712500 RCX: 0000000000000000
[ 2.731375] RDX: ffff8a4ef53add00 RSI: 0000000000000000 RDI: ffff8a4e94712500
[ 2.731385] RBP: ffff8a4e0bf7a640 R08: 00007bc5c0573000 R09: 0000000000000008
[ 2.731395] R10: ffff8a4ec756c190 R11: 00007bc5c05a2000 R12: ffff8a4e94712500
[ 2.731405] R13: ffff8a4ed3ab9d50 R14: 0000000000000000 R15: 0000000000000001
[ 2.731415] FS: 00007bc5c00dc7c0(0000) GS:ffff8a4ef5d00000(0000) knlGS:0000000000000000
[ 2.731427] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2.731436] CR2: 00000000000003b0 CR3: 000000008148e004 CR4: 00000000003606e0
[ 2.731446] Kernel panic - not syncing: Fatal exception
[ 2.731527] Kernel Offset: 0x2a000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
--
i.
Marek Marczykowski-Górecki
Jan 27, 2020, 7:44:34 AM1/27/20
to Ilpo Järvinen, Qubes devel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
On Mon, Jan 27, 2020 at 01:29:43PM +0200, 'Ilpo Järvinen' via qubes-devel wrote:
> Hi,
>
> I've noticed that 5.5-rcs and now 5.5 fail to boot in VM (at least when
> booted as in VM kernel using pvgrub2-pvh). 5.4 based kernels worked fine
> and there seems to have been some changes in drivers/xen post-5.4. Should
> I escalate this to xen-devel?
Yes, sounds like a good idea.
Maybe we should setup some testing of rc kernels? This is the second
time when gntdev driver got broken, in relatively short time (last time
was in July, Linux 5.2).
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAl4u2ysACgkQ24/THMrX
1yypdAf/Sor06saSXxkTfmeruWysWwsj2XOvW1GOSdA2YDV2SBbYD8mU/1jEvoU5
CXqbPMXgHk0mU07/RkgQYSTHujhiOIxKkDQ5ztJ+EnrkQRPh2e4VARD7TqkbsuFo
UKZkiFZ3mO71wf/f9w6ZocyXAeY5hzlDWMaiR8Wd4ETzdoXaR0suBaJFu09vpv89
HgpjUTxLjfokA9SY2B81i6o2vll9RTem0lQ0rKbSIJYMWhPnpW/6oHGKZx8XZc0l
9VUEhBXLkZ2JSlTO0up1FxIVVghF3/KIoe8el1YKZruLe0VKz+KHc5gMIq4oZcj4
A2wBkIzY4ARyvrkSIUH/iV1z2C/gqQ==
=EHsP
-----END PGP SIGNATURE-----
Hash: SHA256
On Mon, Jan 27, 2020 at 01:29:43PM +0200, 'Ilpo Järvinen' via qubes-devel wrote:
> Hi,
>
> I've noticed that 5.5-rcs and now 5.5 fail to boot in VM (at least when
> booted as in VM kernel using pvgrub2-pvh). 5.4 based kernels worked fine
> and there seems to have been some changes in drivers/xen post-5.4. Should
> I escalate this to xen-devel?
Maybe we should setup some testing of rc kernels? This is the second
time when gntdev driver got broken, in relatively short time (last time
was in July, Linux 5.2).
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAl4u2ysACgkQ24/THMrX
1yypdAf/Sor06saSXxkTfmeruWysWwsj2XOvW1GOSdA2YDV2SBbYD8mU/1jEvoU5
CXqbPMXgHk0mU07/RkgQYSTHujhiOIxKkDQ5ztJ+EnrkQRPh2e4VARD7TqkbsuFo
UKZkiFZ3mO71wf/f9w6ZocyXAeY5hzlDWMaiR8Wd4ETzdoXaR0suBaJFu09vpv89
HgpjUTxLjfokA9SY2B81i6o2vll9RTem0lQ0rKbSIJYMWhPnpW/6oHGKZx8XZc0l
9VUEhBXLkZ2JSlTO0up1FxIVVghF3/KIoe8el1YKZruLe0VKz+KHc5gMIq4oZcj4
A2wBkIzY4ARyvrkSIUH/iV1z2C/gqQ==
=EHsP
-----END PGP SIGNATURE-----
Frédéric Pierret
Jan 27, 2020, 9:33:45 AM1/27/20
to Marek Marczykowski-Górecki, Ilpo Järvinen, Qubes devel
Hi,
I think there is an issue about the fact of testing newer kernels. I
think such tests of booting kernel can 'easily' be done. I can easily
create Qubes installations in KVM (R4.0 and R4.1) and each time a kernel
is built, I can snapshot and update the KVM Qubes and check if it's working.
What do you think Marek?
Frédéric
I think there is an issue about the fact of testing newer kernels. I
think such tests of booting kernel can 'easily' be done. I can easily
create Qubes installations in KVM (R4.0 and R4.1) and each time a kernel
is built, I can snapshot and update the KVM Qubes and check if it's working.
What do you think Marek?
Frédéric
Marek Marczykowski-Górecki
Jan 27, 2020, 9:42:29 AM1/27/20
to Frédéric Pierret, Ilpo Järvinen, Qubes devel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hash: SHA256
On Mon, Jan 27, 2020 at 03:33:20PM +0100, Frédéric Pierret wrote:
> Hi,
>
> I think there is an issue about the fact of testing newer kernels. I
> think such tests of booting kernel can 'easily' be done. I can easily
> create Qubes installations in KVM (R4.0 and R4.1) and each time a kernel
> is built, I can snapshot and update the KVM Qubes and check if it's working.
>
> What do you think Marek?
We already have setup to run tests, it's named openQA. Since your bot
> Hi,
>
> I think there is an issue about the fact of testing newer kernels. I
> think such tests of booting kernel can 'easily' be done. I can easily
> create Qubes installations in KVM (R4.0 and R4.1) and each time a kernel
> is built, I can snapshot and update the KVM Qubes and check if it's working.
>
> What do you think Marek?
builds kernel before opening PR, can you also make it upload to a
temporary repository? Then, I can point openQA at this repository.
But I think we'd need to somehow extend it to rc kernels, not only
released ones. Separate branch (that won't land in current-testing)?
BTW, do you build it with qubes-kernel-vm-support from R4.0 or R4.1?
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
1yyC3Af+MlaVkAvpcgbSKxCZfGL1m8UZL6weEpQpNEP+AIOS/tDtAtrH+Nv0lxxz
Vcj+oETF5MWafWevLXcr/l7dwZ+5ORot6WXPrE4H1GQ48YSvSRjjkfnQ0KxbFW2p
jjIyaDvon3aUn6QNLIsNKNy8wU+x6YqzAqvr3j/zNID2juZzaWjjHtFQZr9dZWlC
b/OEYEKLJANgG5hvhkg7xFD3HBM3M4ETM9ZwcHFwjLUyBaHT1bI/oHWsO9XITGjN
dJWmndBkqt0fPR9J183qaBy2RmQJ0FuierD/SQUinpjFLLQhZFtBvpS80BHfRPGR
UoRP2P8DIfutXHACFt3+WfPVIu4WwA==
=36R6
-----END PGP SIGNATURE-----
Frédéric Pierret
Jan 27, 2020, 9:55:00 AM1/27/20
to Marek Marczykowski-Górecki, Ilpo Järvinen, Qubes devel
On 2020-01-27 15:42, Marek Marczykowski-Górecki wrote:
> On Mon, Jan 27, 2020 at 03:33:20PM +0100, Frédéric Pierret wrote:
> > Hi,
>
> > I think there is an issue about the fact of testing newer kernels. I
> > think such tests of booting kernel can 'easily' be done. I can easily
> > create Qubes installations in KVM (R4.0 and R4.1) and each time a kernel
> > is built, I can snapshot and update the KVM Qubes and check if it's
> working.
>
> > What do you think Marek?
>
> We already have setup to run tests, it's named openQA. Since your bot
> builds kernel before opening PR, can you also make it upload to a
> temporary repository? Then, I can point openQA at this repository.
>
uploading builds to my mirror. As It was not used, I disabled it but can
be re-enable.
> But I think we'd need to somehow extend it to rc kernels, not only
> released ones. Separate branch (that won't land in current-testing)?
Yes, maybe we don't want to build every rc kernels.
> released ones. Separate branch (that won't land in current-testing)?
>
> BTW, do you build it with qubes-kernel-vm-support from R4.0 or R4.1?
>
Do you mean if I build all the packages included in linux-kernel for
> BTW, do you build it with qubes-kernel-vm-support from R4.0 or R4.1?
>
R4.0 and R4.1 ? Currently, I'm only building all R4.0 RPMs.
Marek Marczykowski-Górecki
Jan 27, 2020, 10:37:04 AM1/27/20
to Frédéric Pierret, Ilpo Järvinen, Qubes devel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hash: SHA256
On Mon, Jan 27, 2020 at 03:54:40PM +0100, Frédéric Pierret wrote:
>
> On 2020-01-27 15:42, Marek Marczykowski-Górecki wrote:
> > On Mon, Jan 27, 2020 at 03:33:20PM +0100, Frédéric Pierret wrote:
> > > Hi,
> >
> > > I think there is an issue about the fact of testing newer kernels. I
> > > think such tests of booting kernel can 'easily' be done. I can easily
> > > create Qubes installations in KVM (R4.0 and R4.1) and each time a kernel
> > > is built, I can snapshot and update the KVM Qubes and check if it's
> > working.
> >
> > > What do you think Marek?
> >
> > We already have setup to run tests, it's named openQA. Since your bot
> > builds kernel before opening PR, can you also make it upload to a
> > temporary repository? Then, I can point openQA at this repository.
> >
> In case of remote fetching yes very good idea. Previously I was
> uploading builds to my mirror. As It was not used, I disabled it but can
> be re-enable.
That should be enough :)
>
> On 2020-01-27 15:42, Marek Marczykowski-Górecki wrote:
> > On Mon, Jan 27, 2020 at 03:33:20PM +0100, Frédéric Pierret wrote:
> > > Hi,
> >
> > > I think there is an issue about the fact of testing newer kernels. I
> > > think such tests of booting kernel can 'easily' be done. I can easily
> > > create Qubes installations in KVM (R4.0 and R4.1) and each time a kernel
> > > is built, I can snapshot and update the KVM Qubes and check if it's
> > working.
> >
> > > What do you think Marek?
> >
> > We already have setup to run tests, it's named openQA. Since your bot
> > builds kernel before opening PR, can you also make it upload to a
> > temporary repository? Then, I can point openQA at this repository.
> >
> In case of remote fetching yes very good idea. Previously I was
> uploading builds to my mirror. As It was not used, I disabled it but can
> be re-enable.
> > But I think we'd need to somehow extend it to rc kernels, not only
> > released ones. Separate branch (that won't land in current-testing)?
> Yes, maybe we don't want to build every rc kernels.
> >
> > BTW, do you build it with qubes-kernel-vm-support from R4.0 or R4.1?
> >
> Do you mean if I build all the packages included in linux-kernel for
> R4.0 and R4.1 ? Currently, I'm only building all R4.0 RPMs.
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
1yywggf/ZX2qp9Yf8pfPKMA1QnLSsOmqy7LLetDeKRZukUvilKdDRBWaiaw7xmKK
Ya7Cfg2DN20BNwcJFGNRZWxfTBv0IvrpUhUYt7Fy5NiuvHka6MCP0lsA7w7XG4ZP
jF36OnQ0liXmeK7ukY2Skp2mx7dFIyLUFvAJCFEnweI1VieGtIiaycQw7iK3YyZ2
q5vByp/xwLabPg815jZ0R6GONmo2xnxeKeTa5ooG9VEVQpjLFNcF8MImU95D8hCk
mWpbWK05632oZ3+ljbhI3JtQq9riRP3TtPBpJweXL5oNcuRr9fJc2qcAxivxs1u1
kwuK+qusBNc/G82hHbXb7wlYVQX6Ow==
=zm8H
-----END PGP SIGNATURE-----
Reply all
Reply to author
Forward
0 new messages