NTP Global alteration.
Drew White
I want to set the NTP protocol to target the parent VM and on the NetVM or Sys-Firewall have that as the NTP server that feeds everything under it.
Thus only one VM calls the external source at a lesser interval to do the requests.
How, in this system, do I perform this to get that to work please?
The "ClockVM" does not seem to be operating the way I would have thought a "ClockVM" would.
Is there a bug in it?
Sincerely,
Drew.
Jean-Philippe Ouellet
> Hi folks,
Hi,
> I want to set the NTP protocol to target the parent VM and on the NetVM or Sys-Firewall have that as the NTP server that feeds everything under it.
> Thus only one VM calls the external source at a lesser interval to do the requests.
> How, in this system, do I perform this to get that to work please?
https://github.com/QubesOS/qubes-core-agent-linux/blob/master/qubes-rpc/qubes.SetDateTime
https://github.com/QubesOS/qubes-core-agent-linux/blob/master/qubes-rpc/qubes.SyncNtpClock
https://github.com/QubesOS/qubes-core-agent-linux/blob/master/qubes-rpc/sync-ntp-clock
> The "ClockVM" does not seem to be operating the way I would have thought a "ClockVM" would.
dom0. The rest of the VMs get their time set by dom0 via
qubes.SetDateTime service.
There are many reasons for this, including eliminating redundant
network traffic, and the fact that it is desirable for time to be
correct in all VMs (including those intentionally without any network
access).
> Is there a bug in it?
https://github.com/QubesOS/qubes-issues/issues?q=is%3Aissue%20is%3Aopen%20ntp
https://github.com/QubesOS/qubes-issues/issues?q=is%3Aissue%20is%3Aopen%20clockvm
doesn't look like it!
> Sincerely,
> Drew.
Sincerely,
Jean-Philippe.
Drew White
> On Sun, Mar 12, 2017 at 9:19 PM, Drew White <drew....@gmail.com> wrote:
> > Hi folks,
>
> Hi,
>
> > I want to set the NTP protocol to target the parent VM and on the NetVM or Sys-Firewall have that as the NTP server that feeds everything under it.
>
> No, you don't want that.
Why don't I want what I want?
>
> > Thus only one VM calls the external source at a lesser interval to do the requests.
>
> That is already how it works.
Then why does EVERY GUEST call pool.ntp.org? (unless I change it in the template for every VM)
>
> > How, in this system, do I perform this to get that to work please?
>
> Well, one would start by reading and understanding the relevant source:
>
> https://github.com/QubesOS/qubes-core-agent-linux/blob/master/qubes-rpc/qubes.SetDateTime
> https://github.com/QubesOS/qubes-core-agent-linux/blob/master/qubes-rpc/qubes.SyncNtpClock
> https://github.com/QubesOS/qubes-core-agent-linux/blob/master/qubes-rpc/sync-ntp-clock
I read all that, that's why I found out how to change it in the first place, but every time I do something like add a NewGuest and install, with it's defaults to pool.ntp.org, it goes off and gets the NTP from an outside source. (not very secure), so I have to keep changing it to be the local server. I want to capture it all so only the NetVM performs that action.
>
> > The "ClockVM" does not seem to be operating the way I would have thought a "ClockVM" would.
>
> Only the ClockVM to uses NTP at all, and it sends the time back to
> dom0. The rest of the VMs get their time set by dom0 via
> qubes.SetDateTime service.
So the ClockVM ONLY interacts with Dom0. Fair enough. Then it would be a good addition to allow it to update each Guest.
> There are many reasons for this, including eliminating redundant
> network traffic, and the fact that it is desirable for time to be
> correct in all VMs (including those intentionally without any network
> access).
redundant network traffic... so every 10 minute PER GUEST, it contacts pool.ntp.org and gets the time. That isn't redundant network traffic.
> > Is there a bug in it?
>
> Lets see...
>
> https://github.com/QubesOS/qubes-issues/issues?q=is%3Aissue%20is%3Aopen%20ntp
> https://github.com/QubesOS/qubes-issues/issues?q=is%3Aissue%20is%3Aopen%20clockvm
>
> doesn't look like it!
Well, none that have been reported by anyone other than myself when asking questions in the first place about it. But none opened a bug about it because it's "not a bug" even though it is, (in my personal opinion) a very big bug to have EVERY GUEST contact pool.ntp.org every 10 minutes. wether it's a guest that's behind a proxy, or the proxy itself, or the net vm.
This is a security concern, and a big one at that.
for all unix types, the clock VM should contact the NTP server once every 6 hours (or on boot and then every 6 hours), and every guest should be updated by that guest for time, unless set to otherwise update from elsewhere.
I have my own NTP server, and yet I install things, and I just want to capture all NTP from everything behind the NetVM and make it all get the NTP from the NetVM. Unless it's requesting to the designated Network NTP server.
> > Sincerely,
> > Drew.
>
> Sincerely,
> Jean-Philippe.
Jean-Philippe Ouellet
> On Monday, 13 March 2017 12:36:55 UTC+11, Jean-Philippe Ouellet wrote:
>> On Sun, Mar 12, 2017 at 9:19 PM, Drew White <drew....@gmail.com> wrote:
>> > I want to set the NTP protocol to target the parent VM and on the NetVM or Sys-Firewall have that as the NTP server that feeds everything under it.
>>
>> No, you don't want that.
>
> Why don't I want what I want?
understand. Only the ClockVM is intended to generate any NTP traffic
which leaves your machine.
The rest of the VMs are synchronized not via NTP, but via a qrexec
service. This works even when the VMs are not networked, whereas NTP
to a proxy NTP server in sys-net (or somewhere) would not.
>> > Thus only one VM calls the external source at a lesser interval to do the requests.
>>
>> That is already how it works.
>
> Then why does EVERY GUEST call pool.ntp.org? (unless I change it in the template for every VM)
output from:
[user@sys-firewall ~]$ sudo tcpdump -ni eth0 'udp port ntp'
Have you changed every guest on your system to do that or something?
>> > How, in this system, do I perform this to get that to work please?
>>
>> Well, one would start by reading and understanding the relevant source:
>>
>> https://github.com/QubesOS/qubes-core-agent-linux/blob/master/qubes-rpc/qubes.SetDateTime
>> https://github.com/QubesOS/qubes-core-agent-linux/blob/master/qubes-rpc/qubes.SyncNtpClock
>> https://github.com/QubesOS/qubes-core-agent-linux/blob/master/qubes-rpc/sync-ntp-clock
>
> I read all that, that's why I found out how to change it in the first place, but every time I do something like add a NewGuest and install, with it's defaults to pool.ntp.org, it goes off and gets the NTP from an outside source. (not very secure), so I have to keep changing it to be the local server. I want to capture it all so only the NetVM performs that action.
services which are not running?
>> > The "ClockVM" does not seem to be operating the way I would have thought a "ClockVM" would.
>>
>> Only the ClockVM to uses NTP at all, and it sends the time back to
>> dom0. The rest of the VMs get their time set by dom0 via
>> qubes.SetDateTime service.
>
> So the ClockVM ONLY interacts with Dom0. Fair enough. Then it would be a good addition to allow it to update each Guest.
this periodically. This is better than what I assume you suggest
(ClockVM directly invoking qubes.setDateTime in each guest) because
the service invocations are implicitly rate-limited and contents
filtered by dom0. It is also not desired for the ClockVM VM to even
know which other VMs exist, let alone know which ones are running and
need their clock set.
>> There are many reasons for this, including eliminating redundant
>> network traffic, and the fact that it is desirable for time to be
>> correct in all VMs (including those intentionally without any network
>> access).
>
> redundant network traffic... so every 10 minute PER GUEST, it contacts pool.ntp.org and gets the time. That isn't redundant network traffic.
>> > Is there a bug in it?
>>
>> Lets see...
>>
>> https://github.com/QubesOS/qubes-issues/issues?q=is%3Aissue%20is%3Aopen%20ntp
>> https://github.com/QubesOS/qubes-issues/issues?q=is%3Aissue%20is%3Aopen%20clockvm
>>
>> doesn't look like it!
>
> Well, none that have been reported by anyone other than myself when asking questions in the first place about it. But none opened a bug about it because it's "not a bug" even though it is, (in my personal opinion) a very big bug to have EVERY GUEST contact pool.ntp.org every 10 minutes. wether it's a guest that's behind a proxy, or the proxy itself, or the net vm.
> This is a security concern, and a big one at that.
> for all unix types, the clock VM should contact the NTP server once every 6 hours (or on boot and then every 6 hours), and every guest should be updated by that guest for time, unless set to otherwise update from elsewhere.
pool recommendations [2] suggest this.
[1]: https://tools.ietf.org/html/rfc1305
[2]: http://www.pool.ntp.org/tos.html
> I have my own NTP server, and yet I install things, and I just want to capture all NTP from everything behind the NetVM and make it all get the NTP from the NetVM. Unless it's requesting to the designated Network NTP server.
and enabled an ntp client in my default template"? That might explain
some of your confusion.
Drew White
> On Sun, Mar 12, 2017 at 10:24 PM, Drew White <drew....@gmail.com> wrote:
> > On Monday, 13 March 2017 12:36:55 UTC+11, Jean-Philippe Ouellet wrote:
> >> On Sun, Mar 12, 2017 at 9:19 PM, Drew White <drew....@gmail.com> wrote:
> >> > I want to set the NTP protocol to target the parent VM and on the NetVM or Sys-Firewall have that as the NTP server that feeds everything under it.
> >>
> >> No, you don't want that.
> >
> > Why don't I want what I want?
>
> For the reasons I already stated, and that you appear to already
> understand. Only the ClockVM is intended to generate any NTP traffic
> which leaves your machine.
>
> The rest of the VMs are synchronized not via NTP, but via a qrexec
> service. This works even when the VMs are not networked, whereas NTP
> to a proxy NTP server in sys-net (or somewhere) would not.
> >> > Thus only one VM calls the external source at a lesser interval to do the requests.
> >>
> >> That is already how it works.
> >
> > Then why does EVERY GUEST call pool.ntp.org? (unless I change it in the template for every VM)
>
> That is not the behavior I observe on my system, confirmed by lack of
> output from:
>
> [user@sys-firewall ~]$ sudo tcpdump -ni eth0 'udp port ntp'
>
> Have you changed every guest on your system to do that or something?
Nope, I altered the sync-ntp-clock file. I changed it from pool.ntp.org to the local server in each guest.
Then every guest I changed stopped trying to get the time via ntp from pool.ntp.org.
Until I changed that in each guest, it kept doing it EVERY 10 MINUTES from EVERY Guest that was running.
So that was about 15 requests every 10 minutes. Sometimes more.
One request for every guest.
Thus, it kept running /usr/sbin/ntpdate pool.ntp.org
Until I changed that, it was futile.
> >> > How, in this system, do I perform this to get that to work please?
> >>
> >> Well, one would start by reading and understanding the relevant source:
> >>
> >> https://github.com/QubesOS/qubes-core-agent-linux/blob/master/qubes-rpc/qubes.SetDateTime
> >> https://github.com/QubesOS/qubes-core-agent-linux/blob/master/qubes-rpc/qubes.SyncNtpClock
> >> https://github.com/QubesOS/qubes-core-agent-linux/blob/master/qubes-rpc/sync-ntp-clock
> >
> > I read all that, that's why I found out how to change it in the first place, but every time I do something like add a NewGuest and install, with it's defaults to pool.ntp.org, it goes off and gets the NTP from an outside source. (not very secure), so I have to keep changing it to be the local server. I want to capture it all so only the NetVM performs that action.
>
> I get the impression that maybe you are just changing config files of
> services which are not running?
Actually, it all came back to sync-ntp-clock file, as I said in previous.
I looked for other config files first, and nothing changed no matter what I changed. the system wasn't getting the NTP from the server or the router, or discovering the NTP server on it's own, I found it hard-coded there.
> >> > The "ClockVM" does not seem to be operating the way I would have thought a "ClockVM" would.
> >>
> >> Only the ClockVM to uses NTP at all, and it sends the time back to
> >> dom0. The rest of the VMs get their time set by dom0 via
> >> qubes.SetDateTime service.
> >
> > So the ClockVM ONLY interacts with Dom0. Fair enough. Then it would be a good addition to allow it to update each Guest.
>
> No. That would be a bad design for several reasons. Dom0 already does
> this periodically. This is better than what I assume you suggest
> (ClockVM directly invoking qubes.setDateTime in each guest) because
> the service invocations are implicitly rate-limited and contents
> filtered by dom0. It is also not desired for the ClockVM VM to even
> know which other VMs exist, let alone know which ones are running and
> need their clock set.
I was more thinking the ClockVM (CVM) gets the time, then Dom0 gets the time, then Dom0 updates everything, it would all be via Dom0, but the CVM gets the time initially, and if it has a difference in the NTP compared to the time set in the CVM it then proceeds to update each guests time without calling an external NTP server, and keeps it all inside the Guest regime.
> >> There are many reasons for this, including eliminating redundant
> >> network traffic, and the fact that it is desirable for time to be
> >> correct in all VMs (including those intentionally without any network
> >> access).
> >
> > redundant network traffic... so every 10 minute PER GUEST, it contacts pool.ntp.org and gets the time. That isn't redundant network traffic.
>
> Again. I do not observe this. Have you verified with an unmodified template?
Yes, brand new installation.
> >> > Is there a bug in it?
> >>
> >> Lets see...
> >>
> >> https://github.com/QubesOS/qubes-issues/issues?q=is%3Aissue%20is%3Aopen%20ntp
> >> https://github.com/QubesOS/qubes-issues/issues?q=is%3Aissue%20is%3Aopen%20clockvm
> >>
> >> doesn't look like it!
> >
> > Well, none that have been reported by anyone other than myself when asking questions in the first place about it. But none opened a bug about it because it's "not a bug" even though it is, (in my personal opinion) a very big bug to have EVERY GUEST contact pool.ntp.org every 10 minutes. wether it's a guest that's behind a proxy, or the proxy itself, or the net vm.
>
> Things do not work as you claim they do.
Well, we need to modify it so it does then?
> > This is a security concern, and a big one at that.
>
> Nope.
>
> > for all unix types, the clock VM should contact the NTP server once every 6 hours (or on boot and then every 6 hours), and every guest should be updated by that guest for time, unless set to otherwise update from elsewhere.
>
> Where do you get this 6 hours figure from? Neither the RFC [1] or the
> pool recommendations [2] suggest this.
>
> [1]: https://tools.ietf.org/html/rfc1305
> [2]: http://www.pool.ntp.org/tos.html
>
> > I have my own NTP server, and yet I install things, and I just want to capture all NTP from everything behind the NetVM and make it all get the NTP from the NetVM. Unless it's requesting to the designated Network NTP server.
>
> So... perhaps by "I have my own NTP server" do you mean "I installed
> and enabled an ntp client in my default template"? That might explain
> some of your confusion.
No, NTP server, not on this PC, on the actual physical network.
Jean-Philippe Ouellet
> Question: Why does it not work properly then?
It *does* appear to work properly by default, confirmed by not seeing
the NTP traffic you describe on two Qubes machines, one of which is
literally a perfectly unmodified default install of 3.2 I have just
for testing things.
Feel free to send patches to allow users to easily specify an ntp server(s).
>> >> > The "ClockVM" does not seem to be operating the way I would have thought a "ClockVM" would.
>> >>
>> >> Only the ClockVM to uses NTP at all, and it sends the time back to
>> >> dom0. The rest of the VMs get their time set by dom0 via
>> >> qubes.SetDateTime service.
>> >
>> > So the ClockVM ONLY interacts with Dom0. Fair enough. Then it would be a good addition to allow it to update each Guest.
>>
>> No. That would be a bad design for several reasons. Dom0 already does
>> this periodically. This is better than what I assume you suggest
>> (ClockVM directly invoking qubes.setDateTime in each guest) because
>> the service invocations are implicitly rate-limited and contents
>> filtered by dom0. It is also not desired for the ClockVM VM to even
>> know which other VMs exist, let alone know which ones are running and
>> need their clock set.
>
> I was more thinking the ClockVM (CVM) gets the time, then Dom0 gets the time, then Dom0 updates everything, it would all be via Dom0, but the CVM gets the time initially, and if it has a difference in the NTP compared to the time set in the CVM it then proceeds to update each guests time without calling an external NTP server, and keeps it all inside the Guest regime.
From my quick reading of the source and observations of my systems,
that appears to be exactly how it is implemented right now.
Note also that this is not what you initially described in your first email.
Drew White
> On Sun, Mar 12, 2017 at 10:59 PM, Drew White <drew....@gmail.com> wrote:
> > Question: Why does it not work properly then?
>
> Answer: Must be because of something you changed on your system.
>
> It *does* appear to work properly by default, confirmed by not seeing
> the NTP traffic you describe on two Qubes machines, one of which is
> literally a perfectly unmodified default install of 3.2 I have just
> for testing things.
>
> > Thus, it kept running /usr/sbin/ntpdate pool.ntp.org
> >
> > Until I changed that, it was futile.
>
> Feel free to send patches to allow users to easily specify an ntp server(s).
If I myself provided applications/services to enable this, then that would take me a bit of time to achieve properly due to the fact that it would have to work in version 4 as well.
> >> >> > The "ClockVM" does not seem to be operating the way I would have thought a "ClockVM" would.
> >> >>
> >> >> Only the ClockVM to uses NTP at all, and it sends the time back to
> >> >> dom0. The rest of the VMs get their time set by dom0 via
> >> >> qubes.SetDateTime service.
> >> >
> >> > So the ClockVM ONLY interacts with Dom0. Fair enough. Then it would be a good addition to allow it to update each Guest.
> >>
> >> No. That would be a bad design for several reasons. Dom0 already does
> >> this periodically. This is better than what I assume you suggest
> >> (ClockVM directly invoking qubes.setDateTime in each guest) because
> >> the service invocations are implicitly rate-limited and contents
> >> filtered by dom0. It is also not desired for the ClockVM VM to even
> >> know which other VMs exist, let alone know which ones are running and
> >> need their clock set.
> >
> > I was more thinking the ClockVM (CVM) gets the time, then Dom0 gets the time, then Dom0 updates everything, it would all be via Dom0, but the CVM gets the time initially, and if it has a difference in the NTP compared to the time set in the CVM it then proceeds to update each guests time without calling an external NTP server, and keeps it all inside the Guest regime.
>
> Exactly.
>
> From my quick reading of the source and observations of my systems,
> that appears to be exactly how it is implemented right now.
In other words, the way it is right now is in a form in which it is yet to be complete because they have not instantiated the way that it's meant to work?
> Note also that this is not what you initially described in your first email.
It is not what I initially described because I was making an enquiry not providing details on what I had to do to change it per guest and not on a global level in such a way to reduce the impact on the system and the NTP server and DNS server.
I posted on the forums, not sent an email, but I understand that you may think so because it's a forum that has a mailing list on it available.
Jean-Philippe Ouellet
> On Monday, 13 March 2017 15:02:42 UTC+11, Jean-Philippe Ouellet wrote:
>> From my quick reading of the source and observations of my systems,
>> that appears to be exactly how it is implemented right now.
>
> In other words, the way it is right now is in a form in which it is yet to be complete because they have not instantiated the way that it's meant to work?
your system to how it was before) because a default system already
behaves in the correct manner.
>> Note also that this is not what you initially described in your first email.
>
> It is not what I initially described because I was making an enquiry not providing details on what I had to do to change it per guest and not on a global level in such a way to reduce the impact on the system and the NTP server and DNS server.
>
> I posted on the forums, not sent an email, but I understand that you may think so because it's a forum that has a mailing list on it available.
Drew White
> On Mon, Mar 13, 2017 at 12:54 AM, Drew White <drew....@gmail.com> wrote:
> > On Monday, 13 March 2017 15:02:42 UTC+11, Jean-Philippe Ouellet wrote:
> >> From my quick reading of the source and observations of my systems,
> >> that appears to be exactly how it is implemented right now.
> >
> > In other words, the way it is right now is in a form in which it is yet to be complete because they have not instantiated the way that it's meant to work?
>
> No, in other words nothing needs to be done (except perhaps reverting
> your system to how it was before) because a default system already
> behaves in the correct manner.
Unfortunately, what you are informing me is wrong.
> >> Note also that this is not what you initially described in your first email.
> >
> > It is not what I initially described because I was making an enquiry not providing details on what I had to do to change it per guest and not on a global level in such a way to reduce the impact on the system and the NTP server and DNS server.
> >
> > I posted on the forums, not sent an email, but I understand that you may think so because it's a forum that has a mailing list on it available.
>
> /me stops feeding the troll now
No troll here, sorry.
Drew White
> On Monday, 13 March 2017 19:48:19 UTC+11, Jean-Philippe Ouellet wrote:
> > On Mon, Mar 13, 2017 at 12:54 AM, Drew White <drew....@gmail.com> wrote:
> > > On Monday, 13 March 2017 15:02:42 UTC+11, Jean-Philippe Ouellet wrote:
> > >> From my quick reading of the source and observations of my systems,
> > >> that appears to be exactly how it is implemented right now.
> > >
> > > In other words, the way it is right now is in a form in which it is yet to be complete because they have not instantiated the way that it's meant to work?
> >
> > No, in other words nothing needs to be done (except perhaps reverting
> > your system to how it was before) because a default system already
> > behaves in the correct manner.
>
> So having it as standard works the way I said it does, and you are telling me it doesn't when it does?
>
> Unfortunately, what you are informing me is wrong.
>
As I stated earlier, the ntp is always retrieved from pool.ntp.org, the site that is DEFINED in the file. IF the file did NOT have that set, then it would look on the network to get the ntp settings that are defined by the network.
Thus, you saying that it should do that, when the pool.ntp.org domain is defined as the target, conveys that you are in fact incorrect.
Jean-Philippe Ouellet
other domains via qrexec services, so your claim about NTP traffic
coming from multiple VMs on a default system is false.
As for changing the NTP server used, feel free to submit patches.
Drew White
Then can you tell me why changing it in all VMS that aren't the clock VM allows the requests to decrease by 1 for every one that I changed?
Can you tell me why changing the template altered all Guests and also altered the clockVM it stopped looking at pool.ntp.org and looked where I told it to?
IF I commented out the manual setting to request from pool.ntp.org in a specific guest, that guest didn't get the time updated in that guest.
How do you explain all that?
Sorry if anything I'm saying is coming across in a seemingly wrong way, it is not meant as such, I'm just trying to understand why it's not working the way that you said it's supposed to work when standard works the way that you say it doesn't.
Drew White
ClockVM (ClockVM) [Not necessarily sys-net]
Okay, so you are telling me that the data coming from each guest connecting through the monitor, connecting to pool.ntp.org to get the time is not the way it works even though that is the way it worked since it was initially installed?
I monitored, I checked, I verified, I checked the DNS server... I checked it all BEFORE I posted here.
Drew White
If I was to submit patches, they wouldn't be in Python Code. So I can't submit any changes because my applications aren't written in Python.
cooloutac
the amount of times qubes contacts the dns servers is same on baremetal systems. I always eyeball etherape from sys-net.
Drew White
Actually, it's a lot more often.
I run many things like that, but I just mainly watch the DNS server.. I got too many requests for the machines to be even doing it once every 10 mins. how to fix?