PR2 router reset
22 views
Skip to first unread message
Muhammad Rihabe
Sep 1, 2022, 1:02:30 AM9/1/22
to PR2_Users
Hi,
Our PR2 has been abandoned for several years now, I'm trying to get it up and running again. I was once able to ssh into one of the computers (several months ago), but now I can no longer ssh.
The router on the back has been reset, not sure if this was before or after the ssh problem. I ended up resetting the router again and configuring it as a DHCP server and set the DHCP IP range to 10.69.0.100-254
I set the router's IP to 10.68.0.5
I expected the two computers to give themselves static addresses. But this was not the case. They were assigned an IP within the 100-254 range.
Another thing that happened was the ethernet cables going from each computer to the network switch in between them have been shuffled. All connections are still there, however, the order may have been changed. I'm not sure if this will affect anything as i don't know how the network switch has been set up.
After all of that, I end up seeing 4 devices on the network. 3 of them are from c2 (which has 3 cables plugged in, all going to the network switch) and the other is from c1. but c1 has 7 ethernet cables plugged into it (one of them is etherCat) shouldn't I expect 6 entries from c1?
Using nmap on these IP addresses I found out that all ports are closed, including port 22 for ssh.
Any advice on what I should do to restore access to the pr2? Thanks in advance!
Michael Görner
Sep 1, 2022, 6:37:28 AM9/1/22
to Muhammad Rihabe, PR2_Users
Hi there,
good luck and a lot of enthusiasm! :)
We have a heavily modified system based on Clearpath's "indigo upgrade"
servers here so our setup definitely is quite different, but some
insights from my side nonetheless.
On Wed, 31 Aug 2022 22:02:29 -0700 (PDT)
Muhammad Rihabe <muhamma...@student.uts.edu.au> wrote:
> Our PR2 has been abandoned for several years now, I'm trying to get
> it up and running again. I was once able to ssh into one of the
> computers (several months ago), but now I can no longer ssh.
What base system is running on C1/C2? ROS indigo? fuerte/hydro?
Do you try to setup a new system? It probably makes sense, but adds a
lot of complication. Whether you can do it depends mostly on whether
you have sufficient experience in Unix administration and ROS
maintenance.
> The router on the back has been reset, not sure if this was before or
> after the ssh problem. I ended up resetting the router again and
> configuring it as a DHCP server and set the DHCP IP range to
> 10.69.0.100-254
If you are talking about the wifi router, that doesn't really need to
be setup at all for DHCP for the base systems to work. They should have
set fixed IPs for multiple interfaces. Prominently the wan port of C1
should have been configured by your group a long time ago to hook into
your local network and get dhcp from there (or use an allocated fixed
IP) and C1's main interface with ip 10.68.0.1. C2 should be statically
setup to 10.68.0.2.
If you are stumbling over 10.69.0.0/24 IPs, afaik these are/were
allocated for communication with the cameras.
The wifi router is mostly relevant as a wifi gateway if you unpluck the
wan cable from the back of the PR2 and should not do much else. Because
C2 netboots from C1 after startup, C1 has a dnsmasq setup with a PXE
netboot image and we also use it to serve dhcp in the 10.68.0.100-...
range.
> I set the router's IP to 10.68.0.5
That is correct.
> I expected the two computers to give themselves static addresses. But
> this was not the case. They were assigned an IP within the 100-254
> range.
Afaik these should be given static IPs (better anyway...) so I would
suggest you disable dhcp clients for both machines.
> Another thing that happened was the ethernet cables going from each
> computer to the network switch in between them have been shuffled.
> All connections are still there, however, the order may have been
> changed. I'm not sure if this will affect anything as i don't know
> how the network switch has been set up.
To the best of my knowledge the switch is unmanaged and plugging order
does not matter. That being said it can be a downside in itself to
deviate from the canonical documentation.
> After all of that, I end up seeing 4 devices on the network. 3 of
> them are from c2 (which has 3 cables plugged in, all going to the
> network switch)
One should be 10.68.0.2 for regular connections and one 10.69.0.12 for
camera communication. On the old servers the third one might be an
remote administration port, but I've never seen this used myself.
> and the other is from c1. but c1 has 7 ethernet
> cables plugged into it (one of them is etherCat) shouldn't I expect 6
> entries from c1?
1. ethercat
2. wan interface that directly connects to the WAN port in the back of
the robot
3-6. lan0 - lan3 that were canonically set to
10.68.0.1
10.69.0.11
10.68.0.21
10.68.0.31
7. a remote administration port for C1
That being said we currently do not use lan2-3 at all and route all
usual traffic through lan0. That works sufficiently well for us.
> Using nmap on these IP addresses I found out that all ports are
> closed, including port 22 for ssh.
Ouch.
> Any advice on what I should do to restore access to the pr2? Thanks
> in advance!
You can try to use the esms remote administration interfaces if you
find instructions/ software for it (I believe it originally came on a
netbook together with the robot?)
Also you can of course try to connect directly with screen/keyboard to
the servers. This might require you taking the servers out of the base
to get to the connectors. For Clearpath indigo upgrade it even means
that you have to exchange the ethernet card on C1 with a graphics card
to even get *any* video output. Yes, it's a hassle.
The fact that C2 booted altogether sounds like a lot of things are
working, so maybe you can figure out why ssh is blocked if you connect
a monitor directly.
Honestly, I do think it's a good idea to get a more current system on
the servers (we use Ubuntu 20.04 at the moment, but it's running on a
Ubuntu 14.04 in a docker container).
If you upgrade the base system I would strongly recommend to get rid of
the c1/c2 netboot setup with unionfs overlay and setup c1 and c2
independently with an NFS share for /home. In my experience the
downsides of having this overcomplicated boot/filesystem setup outweigh
the benefits of 'single system to upgrade' by far these days.
One last random fact: Please be aware that the robot's power board also
has an IP assigned:
https://github.com/PR2/pr2_robot/blob/kinetic-devel/pr2_bringup/pr2.launch#L43
But it does not show up through nmap.
Hope this helps a bit.
Greetings from Hamburg,
Michael
--
Michael 'v4hn' Görner, M.Sc. Cognitive Science, PhD Student
Universität Hamburg
Faculty of Mathematics, Informatics and Natural Sciences
Department of Informatics
Group Technical Aspects of Multimodal Systems
Vogt-Kölln-Straße 30
D-22527 Hamburg
Room: F-315
Phone: +49 40 42883-2432
Website: https://tams.informatik.uni-hamburg.de/people/goerner/
good luck and a lot of enthusiasm! :)
We have a heavily modified system based on Clearpath's "indigo upgrade"
servers here so our setup definitely is quite different, but some
insights from my side nonetheless.
On Wed, 31 Aug 2022 22:02:29 -0700 (PDT)
Muhammad Rihabe <muhamma...@student.uts.edu.au> wrote:
> Our PR2 has been abandoned for several years now, I'm trying to get
> it up and running again. I was once able to ssh into one of the
> computers (several months ago), but now I can no longer ssh.
Do you try to setup a new system? It probably makes sense, but adds a
lot of complication. Whether you can do it depends mostly on whether
you have sufficient experience in Unix administration and ROS
maintenance.
> The router on the back has been reset, not sure if this was before or
> after the ssh problem. I ended up resetting the router again and
> configuring it as a DHCP server and set the DHCP IP range to
> 10.69.0.100-254
be setup at all for DHCP for the base systems to work. They should have
set fixed IPs for multiple interfaces. Prominently the wan port of C1
should have been configured by your group a long time ago to hook into
your local network and get dhcp from there (or use an allocated fixed
IP) and C1's main interface with ip 10.68.0.1. C2 should be statically
setup to 10.68.0.2.
If you are stumbling over 10.69.0.0/24 IPs, afaik these are/were
allocated for communication with the cameras.
The wifi router is mostly relevant as a wifi gateway if you unpluck the
wan cable from the back of the PR2 and should not do much else. Because
C2 netboots from C1 after startup, C1 has a dnsmasq setup with a PXE
netboot image and we also use it to serve dhcp in the 10.68.0.100-...
range.
> I set the router's IP to 10.68.0.5
> I expected the two computers to give themselves static addresses. But
> this was not the case. They were assigned an IP within the 100-254
> range.
suggest you disable dhcp clients for both machines.
> Another thing that happened was the ethernet cables going from each
> computer to the network switch in between them have been shuffled.
> All connections are still there, however, the order may have been
> changed. I'm not sure if this will affect anything as i don't know
> how the network switch has been set up.
does not matter. That being said it can be a downside in itself to
deviate from the canonical documentation.
> After all of that, I end up seeing 4 devices on the network. 3 of
> them are from c2 (which has 3 cables plugged in, all going to the
> network switch)
camera communication. On the old servers the third one might be an
remote administration port, but I've never seen this used myself.
> and the other is from c1. but c1 has 7 ethernet
> cables plugged into it (one of them is etherCat) shouldn't I expect 6
> entries from c1?
2. wan interface that directly connects to the WAN port in the back of
the robot
3-6. lan0 - lan3 that were canonically set to
10.68.0.1
10.69.0.11
10.68.0.21
10.68.0.31
7. a remote administration port for C1
That being said we currently do not use lan2-3 at all and route all
usual traffic through lan0. That works sufficiently well for us.
> Using nmap on these IP addresses I found out that all ports are
> closed, including port 22 for ssh.
> Any advice on what I should do to restore access to the pr2? Thanks
> in advance!
find instructions/ software for it (I believe it originally came on a
netbook together with the robot?)
Also you can of course try to connect directly with screen/keyboard to
the servers. This might require you taking the servers out of the base
to get to the connectors. For Clearpath indigo upgrade it even means
that you have to exchange the ethernet card on C1 with a graphics card
to even get *any* video output. Yes, it's a hassle.
The fact that C2 booted altogether sounds like a lot of things are
working, so maybe you can figure out why ssh is blocked if you connect
a monitor directly.
Honestly, I do think it's a good idea to get a more current system on
the servers (we use Ubuntu 20.04 at the moment, but it's running on a
Ubuntu 14.04 in a docker container).
If you upgrade the base system I would strongly recommend to get rid of
the c1/c2 netboot setup with unionfs overlay and setup c1 and c2
independently with an NFS share for /home. In my experience the
downsides of having this overcomplicated boot/filesystem setup outweigh
the benefits of 'single system to upgrade' by far these days.
One last random fact: Please be aware that the robot's power board also
has an IP assigned:
https://github.com/PR2/pr2_robot/blob/kinetic-devel/pr2_bringup/pr2.launch#L43
But it does not show up through nmap.
Hope this helps a bit.
Greetings from Hamburg,
Michael
--
Michael 'v4hn' Görner, M.Sc. Cognitive Science, PhD Student
Universität Hamburg
Faculty of Mathematics, Informatics and Natural Sciences
Department of Informatics
Group Technical Aspects of Multimodal Systems
Vogt-Kölln-Straße 30
D-22527 Hamburg
Room: F-315
Phone: +49 40 42883-2432
Website: https://tams.informatik.uni-hamburg.de/people/goerner/
Shingo Kitagawa
Sep 1, 2022, 7:24:45 AM9/1/22
to Michael Görner, Muhammad Rihabe, PR2_Users
Hi,
We also modified our PR2 system based on the indigo update, and Michael gave us a lot of good advice to solve the issue.
If you still have the problem, I can help you, too.
I hope the advice works well!
P.S.
(we use Ubuntu 20.04 at the moment, but it's running on a Ubuntu 14.04 in a docker container).
If you upgrade the base system I would strongly recommend to get rid of
the c1/c2 netboot setup with unionfs overlay and setup c1 and c2
independently with an NFS share for /home.
Great solutions!
We have some problems during C2 shutdown in our PR2 noetic upgrade,
which may be caused by unionfs.
This mail thread is also helpful for every PR2 users, I think.
2022年9月1日(木) 19:37 Michael Görner <goe...@informatik.uni-hamburg.de>:
--
You received this message because you are subscribed to the Google Groups "PR2_Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pr2_users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pr2_users/20220901123725.774da2f1%40schubal.
Muhammad Rihabe
Sep 1, 2022, 9:08:32 PM9/1/22
to PR2_Users
This is very, very helpful. Thank you, Michael!
A bit more information about our PR2:
We do have the ROS Indigo upgrade. I actually had to take both servers apart to replace the cmos batteries last week, and I've left them open for easier access for now.
I love the idea of using docker, It's an elegant, modern solution! Instead of trying to make what I currently have work, I'd rather upgrade both systems to use newer ubuntu. I've seen some PR2 packages for ROS melodic, but I'm not sure if it's fully featured. In your opinion, is it better to use a later OS (22.04), run docker and use Ubuntu 14.04? Or would I be better off installing 18.04 with melodic and trying to get all packages working from there? I want to try my best to get everything working on a more modern system.
On that note, it would be very helpful if you can provide a docker image of what you have running currently. I'll also do my own research again with docker in mind.
One thing I realized while working on the robot is that the manual says the c2 netboots off of c1, and c2 does not have its own OS. But with the current setup, I see that there is an OS on both machines separately. I don't think any netbooting is happening, they also don't share a filesystem and c2 doesn't even have ROS installed. Maybe I'm confused as to what netbooting actually means or maybe the Indigo upgrade did away with netbooting altogether?
I have gotten access to c1 by removing the lan card and placing a small graphics card in its place, it's only temporary though. I don't understand why c1 needs 7 lan ports. Is it a bandwidth limitation?
If we do upgrade the OS on both machines, wouldn't it be best to have each one be an independent computer on the same network? Where one will be the ROS Master and the other simply runs nodes to interface which whatever is allocated to it (usb and ethernet devices). Maybe even upgrade the computers and run everything on one machine?
I'm doing this whole project of getting the PR2 running as my capstone (thesis) at my university. So I will be documenting everything. hopefully, I can share it here or someplace else when I'm done to help anyone else that may run into an old PR2 someday.
Muhammad Rihabe
Sep 1, 2022, 9:24:04 PM9/1/22
to PR2_Users
One more thing. Assuming I decide to start from fresh and reinstall the OS. Will following the ROS wiki instructions here bring me back to a functional robot? Of course, I'd need to sort out the network problems first.
Michael Görner
Sep 2, 2022, 5:55:32 AM9/2/22
to Muhammad Rihabe, PR2_Users
Hi again,
One thing ahead: The PR2 ecosystem has a lot of dead/rarely used code.
Some packages that can help with robot maintenance are still not ported
to catkin (and anyone who would use them would want to do that), not to
mention python3. Also, even if it looks like a feature might have
worked in ros fuerte it might have been broken even back then with
nobody noticing. It's just a lot of features that come together here.
So of course it's a good idea to check all the documentation you can
get your hands on (and documentation videos) but don't rely on anything
unless you explicitly tested it in full operation. Sorry for the
pessimistic view, I just find it an healthy attitude with ROS
altogether. :-)
> One more thing. Assuming I decide to start from fresh and reinstall
> the OS. Will following the ROS wiki instructions here
> <http://wiki.ros.org/Robots/PR2/indigo> bring me back to a functional
The "indigo upgrade" by clearpath we received already had various
system issues that took time to fix.
Also Ubuntu 14.04 is out of LTS by now (after all it's been 8 years),
so you can't easily pull from the ubuntu deb repositories anymore.
> I love the idea of using docker, It's an elegant, modern solution!
Yes, but setting it up on top of the indigo servers with netboot is
quite some overhead and comes with disadvantages. No systemd, openvpn
in the Ubuntu 14.04 version (which is a version that does not include
WG's modifications yet to allow for floating connections), ...
If we would do it again, we definitely would setup c1 and c2 separately
at least with Ubuntu 20.04 first, but for now what we have works
sufficiently well.
> In your opinion, is it better to use a later OS (22.04), run docker
> and use Ubuntu 14.04?
O.O" Never. Again, already on indigo many things were somewhat broken
so you want to use current versions and fix issues there. I don't see
any upside to try to run anything older than ROS noetic by now unless
you have it and don't want to change. And yes, you might also find
python3 migration issues there that have not been pushed upstream by
anyone. Please contribute to the repositories at https://github.com/PR2
if you fix something.
Considering whether to setup Ubuntu 20.04 or 22.04 as base system I'm
undecided. Open Robotics's ROS(1) distributions do not support Ubuntu
22.04 anymore, so you would probably want to run your ROS noetic on
20.04 anyway. If you directly setup a 20.04 you will not need any
docker at all for the near future (which I consider a good thing), but
of course a 22.04 base system might give you some benefits simply
because it's newer. (I know at least with 14.04 base images we have
problems with PS4 controller connections, there is always something
that works better with newer versions and always a lot that breaks).
Of course you can also compile your ROS from scratch on 22.04
(the keyword would be ROS-O) or use RoboStack's noetic stack, but you
might not want to start out with it if you don't have a working PR2
right now.
> On that note, it would be very helpful if you can provide a docker
> image of what you have running currently. I'll also do my own
> research again with docker in mind.
We run with a simple Ubuntu 20.04 image with a long list of packages
with whatever we need. I don't think there is much sense in sharing
this with you.
> One thing I realized while working on the robot is that the manual
> says the c2 netboots off of c1, and c2 does not have its own OS. But
> with the current setup, I see that there is an OS on both machines
> separately. I don't think any netbooting is happening, they also
> don't share a filesystem and c2 doesn't even have ROS installed.
> Maybe I'm confused as to what netbooting actually means or maybe the
> Indigo upgrade did away with netbooting altogether?
For our server upgrade C2 had a basic ubuntu 14.04 installed without
any modifications. For the canonical PR2 setup this system should never
boot and they might as well have swiped the disk.
However, if C1 does not start its dnsmasq to offer the pxe boot on
the correct interface, or if the bios on c2 is not configured properly
for netboot only, C2 might eventually boot into that system.
> I have gotten access to c1 by removing the lan card and placing a
> small graphics card in its place, it's only temporary though. I don't
> understand why c1 needs 7 lan ports. Is it a bandwidth limitation?
As I said, it needs at least ethercat, wan, lan with 10.68.0.0/24 and
lan with 10.69.0.0/24 interfaces. you could even map both 10.68 and
10.69 to the same lan interface but you would have to monitor (iftop et
al) the interface to see whether it overloads. I believe the cameras
were split off because of load issues. On the other hand the original
hardware design is over 12 years old now and the new network interfaces
might be able to handle more. Also the stereo/arm cameras are awful
quality compared to current camera sensors so you might as well
consider disabling them altogether to save the bandwidth.
I believe the pre-indigo PR2 had load balancing setup somehow between
lan0, lan2 and lan3, but it might be one of these things that should
have worked but actually didn't aside from some test systems.
As I wrote before, we don't use lan2 and lan3 at all and do not see
noticeable issues.
> If we do upgrade the OS on both machines, wouldn't it be best to have
> each one be an independent computer on the same network? Where one
> will be the ROS Master and the other simply runs nodes to interface
> which whatever is allocated to it (usb and ethernet devices). Maybe
> even upgrade the computers and run everything on one machine?
That's what I proposed in my last mail. The main advantages of the
unionfs/netboot setup are that you don't have to think of c2 as a
separate computer. It's just an extension of c1. If you modify /usr or
/etc on c1 (e.g., by installing packages) the changes will be there on
c2 - unless you hit one of the files from the unionfs overlay which are
located somewhere in c1:/unionfs/overlay (off the top of my head).
The only thing you need to do is offload computation from c1 is to add
a `machine="c2"` attribute to your launch file. These are huge
advantages. It's just that overlayfs + netboot also make administration
much harder and the whole thing needs a rather fragile configuration.
Also with the additional pr2-head NUC necessary for a kinect sensor you
have another independent system already and need to keep that in mind
during updates...
Best,
Michael
On Thu, 1 Sep 2022 18:08:32 -0700 (PDT)
> >> Phone: +49 40 42883-2432 <+49%2040%20428832432>
One thing ahead: The PR2 ecosystem has a lot of dead/rarely used code.
Some packages that can help with robot maintenance are still not ported
to catkin (and anyone who would use them would want to do that), not to
mention python3. Also, even if it looks like a feature might have
worked in ros fuerte it might have been broken even back then with
nobody noticing. It's just a lot of features that come together here.
So of course it's a good idea to check all the documentation you can
get your hands on (and documentation videos) but don't rely on anything
unless you explicitly tested it in full operation. Sorry for the
pessimistic view, I just find it an healthy attitude with ROS
altogether. :-)
> One more thing. Assuming I decide to start from fresh and reinstall
> the OS. Will following the ROS wiki instructions here
> robot? Of course, I'd need to sort out the network problems first.
No. I would definitely not rely on it.
The "indigo upgrade" by clearpath we received already had various
system issues that took time to fix.
Also Ubuntu 14.04 is out of LTS by now (after all it's been 8 years),
so you can't easily pull from the ubuntu deb repositories anymore.
> I love the idea of using docker, It's an elegant, modern solution!
quite some overhead and comes with disadvantages. No systemd, openvpn
in the Ubuntu 14.04 version (which is a version that does not include
WG's modifications yet to allow for floating connections), ...
If we would do it again, we definitely would setup c1 and c2 separately
at least with Ubuntu 20.04 first, but for now what we have works
sufficiently well.
> In your opinion, is it better to use a later OS (22.04), run docker
> and use Ubuntu 14.04?
so you want to use current versions and fix issues there. I don't see
any upside to try to run anything older than ROS noetic by now unless
you have it and don't want to change. And yes, you might also find
python3 migration issues there that have not been pushed upstream by
anyone. Please contribute to the repositories at https://github.com/PR2
if you fix something.
Considering whether to setup Ubuntu 20.04 or 22.04 as base system I'm
undecided. Open Robotics's ROS(1) distributions do not support Ubuntu
22.04 anymore, so you would probably want to run your ROS noetic on
20.04 anyway. If you directly setup a 20.04 you will not need any
docker at all for the near future (which I consider a good thing), but
of course a 22.04 base system might give you some benefits simply
because it's newer. (I know at least with 14.04 base images we have
problems with PS4 controller connections, there is always something
that works better with newer versions and always a lot that breaks).
Of course you can also compile your ROS from scratch on 22.04
(the keyword would be ROS-O) or use RoboStack's noetic stack, but you
might not want to start out with it if you don't have a working PR2
right now.
> On that note, it would be very helpful if you can provide a docker
> image of what you have running currently. I'll also do my own
> research again with docker in mind.
with whatever we need. I don't think there is much sense in sharing
this with you.
> One thing I realized while working on the robot is that the manual
> says the c2 netboots off of c1, and c2 does not have its own OS. But
> with the current setup, I see that there is an OS on both machines
> separately. I don't think any netbooting is happening, they also
> don't share a filesystem and c2 doesn't even have ROS installed.
> Maybe I'm confused as to what netbooting actually means or maybe the
> Indigo upgrade did away with netbooting altogether?
any modifications. For the canonical PR2 setup this system should never
boot and they might as well have swiped the disk.
However, if C1 does not start its dnsmasq to offer the pxe boot on
the correct interface, or if the bios on c2 is not configured properly
for netboot only, C2 might eventually boot into that system.
> I have gotten access to c1 by removing the lan card and placing a
> small graphics card in its place, it's only temporary though. I don't
> understand why c1 needs 7 lan ports. Is it a bandwidth limitation?
lan with 10.69.0.0/24 interfaces. you could even map both 10.68 and
10.69 to the same lan interface but you would have to monitor (iftop et
al) the interface to see whether it overloads. I believe the cameras
were split off because of load issues. On the other hand the original
hardware design is over 12 years old now and the new network interfaces
might be able to handle more. Also the stereo/arm cameras are awful
quality compared to current camera sensors so you might as well
consider disabling them altogether to save the bandwidth.
I believe the pre-indigo PR2 had load balancing setup somehow between
lan0, lan2 and lan3, but it might be one of these things that should
have worked but actually didn't aside from some test systems.
As I wrote before, we don't use lan2 and lan3 at all and do not see
noticeable issues.
> If we do upgrade the OS on both machines, wouldn't it be best to have
> each one be an independent computer on the same network? Where one
> will be the ROS Master and the other simply runs nodes to interface
> which whatever is allocated to it (usb and ethernet devices). Maybe
> even upgrade the computers and run everything on one machine?
unionfs/netboot setup are that you don't have to think of c2 as a
separate computer. It's just an extension of c1. If you modify /usr or
/etc on c1 (e.g., by installing packages) the changes will be there on
c2 - unless you hit one of the files from the unionfs overlay which are
located somewhere in c1:/unionfs/overlay (off the top of my head).
The only thing you need to do is offload computation from c1 is to add
a `machine="c2"` attribute to your launch file. These are huge
advantages. It's just that overlayfs + netboot also make administration
much harder and the whole thing needs a rather fragile configuration.
Also with the additional pr2-head NUC necessary for a kinect sensor you
have another independent system already and need to keep that in mind
during updates...
Best,
Michael
On Thu, 1 Sep 2022 18:08:32 -0700 (PDT)
Reply all
Reply to author
Forward
0 new messages