VoIP always has high latencies. If any part of the network was
overloaded, these could become much worse. The receiving phone will
have been using a jitter buffer to account for variations in these, but
it may well have only increased the jitter buffer length, so the delay
would reflect the maximum delay so far.
You'd need a non-internet derived, audio, time signal, to work out which
direction(s) was/were accumulating the delays.
(The first reasons for concerns with Digital Voice and social alarms
were not the resilience ones that come up here about every three weeks,
but that the round trip delays would be too long for the alarm modems
and alarm signalling protocols.)