Working with dates and time?

[Björn Buckwalter]

Hi all,

I'm wondering how you all manage and calculate with time when you need better precision than 1 second?

More or less what I am looking for is equivalents to os.date, os.time, and os.difftime with support for fraction of seconds. Effectively the same API would be OK with seconds represented as float (probably accurate enough for my needs) instead of integer.

Let me know what libraries you use and their pros/cons!

Many thanks,

Bjorn

[Kartik Agaram]

luasocket has socket.gettime(), though I've always wondered why that's in a socket library..

Also apparently doesn't get milliseconds on Windows: https://stackoverflow.com/questions/463101/lua-current-time-in-milliseconds

[Sainan]

Something like this?

function os.millis()
return os.time() * 1000
end

> --
> You received this message because you are subscribed to the Google
> Groups "lua-l" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to lua-l+un...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/lua-l/6ebe99e4-b6f9-4643-855b-8135b38091c4n%40googlegroups.com
> [1].
>
>
> Links:
> ------
> [1]
> https://groups.google.com/d/msgid/lua-l/6ebe99e4-b6f9-4643-855b-8135b38091c4n%40googlegroups.com?utm_medium=email&utm_source=footer

[Sainan]

Sorry, typo here, I meant `os.clock` instead of `os.time`.

[Frank Kastenholz]

In past projects we had project-specific modules that could access finer grained clocks. Millisecond is usually adequate in Lua code - microsecond is more than adequate. 

 Events that occur and are fist handled in C could get timestampled down in C, or even by hardware, and that could be NS or less - so the Lua. COde had to handle very large integefs ….

Frank Kastenholz

On Mar 16, 2024, at 12:39 PM, Björn Buckwalter <bjorn.bu...@gmail.com> wrote:

Hi all,

--

You received this message because you are subscribed to the Google Groups "lua-l" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lua-l+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lua-l/6ebe99e4-b6f9-4643-855b-8135b38091c4n%40googlegroups.com.

[Francisco Olarte]

On Sat, 16 Mar 2024 at 18:29, Sainan <sai...@calamity.gg> wrote:
> Sorry, typo here, I meant `os.clock` instead of `os.time`.

Be aware that clock() uses ISO C clock(3), as documented, which has a
lot of caveats to be used as a clock.

Main one is it measures used cpu time, not wall time:

> print(os.time(), os.clock()); io.read();print(os.time(), os.clock());
1710612282 0.019548

1710612295 0.019588

I delayed a bit pressing enter. time() shows io.read() took about 13
seconds, but used only about 40 CPU microseconds.

It can also happen the other way round. I've got a lot of programs
which stream process huge files. I store them compressed as, among
other things, I have found it is faster to read less and decompress,
process, recompress and write less, specially with current multicore
machines where with a little threading and buffering extra CPU is
basically free. They routinely show >200% CPU usages ( max will be
300%, as I use one CPU to read & decompress, one to process, one to
recompress result ).

Francisco Olarte.

[Denis Dos Santos Silva]

Lua dont provide a high resolution get time function

checkout:

a) luasocket package have a function to get time with high resolution precision*

socket.gettime()

https://lunarmodules.github.io/luasocket/reference.html

b) luaposix - https://github.com/luaposix/luaposix

c) luatz - https://github.com/daurnimator/luatz/blob/master/luatz/gettime.lua

* https://man7.org/linux/man-pages/man2/gettimeofday.2.html or related OS

[Francisco Olarte]

On Sat, 16 Mar 2024 at 17:55, Kartik Agaram <a...@akkartik.com> wrote:
> luasocket has socket.gettime(), though I've always wondered why that's in a socket library..

Lots of sockets libraries use select() ( which uses timeval, with
microseconds, for timeouts ) and similar stuff, and provide time
sources which similar precision if the host language does not have one
built in, just in case. In socket algorithms ( protocols, and similar
stuff ) milliseconds timeouts are common, and you normally need a same
precission clock to suplement and make extra calculations.

Francisco Olarte.

[Sainan]

Fair. I'm mainly a Windows user, and here CLOCKS_PER_SECOND is 1000, and
it maps perfectly onto milliseconds:

1710613111 0.0

1710613114 3.262

I suppose a standard library time millis function would be nice either
way.

[Bjorn Buckwalter]

Thanks all for the suggestions! I should clarify, though: I am working with files of time series data (satellite ephemeris) and as such am more interested in manipulating (parsing, arithmetic, formatting) date and time data rather than accurately determining the current time or measuring elapsed time. I could probably wrap something around the os module functions adding a nanosecond field, but thought something might be out there already (Denis pointed to luatz, but from the docs alone it isn't obvious to me if fractional seconds are supported).

--
You received this message because you are subscribed to a topic in the Google Groups "lua-l" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lua-l/R3qYEXkH22U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lua-l+un...@googlegroups.com.

[Francisco Olarte]

Hi Sainan:

On Sat, 16 Mar 2024 at 19:22, Sainan <sai...@calamity.gg> wrote:
> Fair. I'm mainly a Windows user, and here CLOCKS_PER_SECOND is 1000, and
> it maps perfectly onto milliseconds:
> 1710613111 0.0
> 1710613114 3.262

Or your lua exe may be busywaiting on io.read.

> I suppose a standard library time millis function would be nice either way.

I mainly use lua embeded, one of the first thing I push in my states
is gettimeofday ( which can be wrapped in lua to return either a
double fractional os.time compatible seconds, a 64 bit microseconds or
normally both ) and usleep().

Once you have that wrapping os.time(t) to return microseconds and
similar stuff is a piece of cake.

For other things I just go to 64 bit nanoseconds, precise enough and
with enough range to last till I retire.

Francisco Olarte.

[Francisco Olarte]

On Sat, 16 Mar 2024 at 19:25, Bjorn Buckwalter
<bjorn.bu...@gmail.com> wrote:
> Thanks all for the suggestions! I should clarify, though: I am working with files of time series data (satellite ephemeris) and as such am more interested in manipulating (parsing, arithmetic, formatting) date and time data rather than accurately determining the current time or measuring elapsed time. I could probably wrap something around the os module functions adding a nanosecond field, but thought something might be out there already (Denis pointed to luatz, but from the docs alone it isn't obvious to me if fractional seconds are supported).

I do similar stuff with CDRs, but my external representation are
normally standarized to ISO 8601 ( with explicit +00 timezone ) so
parsing is easy.

Leaving aside parsing from text, you can use fractional seconds if 53
bits is enough for you, this should go to about microseconds, in this
times, wrap os.date() to handle that:

> function tdump(t) for k,v in pairs(t) do print(k,v) end endAnd you can wrap it into a function adding the fractional part into it:
> function explode(t) s,f=math.modf(t) res=os.date('*t',s); res.usec=math.floor(f*1000000); return res; end
> print(tdump(explode(os.time()+0.1234567)))
month 3
min 57
day 16
wday 7
year 2024
yday 76
usec 123456
isdst false
hour 19
sec 12
Backwards conversion is easy, os.time() does not care about usec
field, just wrap and add.

I, personally, would go for a 64 bit nanoseconds count ( which gives
you...2^63/365.2425/86400/1e9= 292.27702462692771494535 years range )
and just do the div/mod mul/add adjustements in a wrapper. Arithmetic
is then easy.

If you want to roll your own conversions from/to parts/strings, for
complete control, I did a java class for that clocking at 583 lines /
189 semicolons, with a lot of unneeded functionality ( it is a library
), I have some in perl in less than 30 lines for special purposes, lua
should have a similar count. I have found UTC date/time conversions to
numbers just have two tricks, use two arrays for acumulated day count
before start of month, one for leap, or for other years, and
precalculate a numeric value for days since 01-01-year 0 ( which does
not exist but is useful ) to the epoch.

Francisco Olarte.

[George Georgalis]

On Sat, Mar 16, 2024 at 11:25 AM Bjorn Buckwalter <bjorn.bu...@gmail.com> wrote:

Thanks all for the suggestions! I should clarify, though: I am working with files of time series data (satellite ephemeris) and as such am more interested in manipulating (parsing, arithmetic, formatting) date and time data rather than accurately determining the current time or measuring elapsed time. I could probably wrap something around the os module functions adding a nanosecond field, but thought something might be out there already (Denis pointed to luatz, but from the docs alone it isn't obvious to me if fractional seconds are supported).

I generally build precise time logging around DJB's public domain implementation of tai64n.

C libraries are available as well as a standalone binary, I've modified the daemontools package

to only build the tai64n part (because I use runit/socklog a daemontools replacement).

Wikipedia and Dan's page explain the standard; pico second resolution with astronomical range.

A 24 bit representation in hex is 40000000, followed by unix seconds, followed by 60ths (up to

not including 0x3c), which are further subdivided by 6 bits (000000..ffffff).

In practice the cli is really easy to work with.

echo | tai64n

@4000000065f5e9be25c6288c

Typically I lop off the @40000000 part, but sometimes I use it to represent mock data,

eg @4000c000, @4000c001 and @4000c002 might represent prefixes of fictitious data.

In db transactions I often use tai64n as a uid. Be careful, modern high performance CPUs have

the potential to read their onboard clock faster than it increments, eg two timestamps, same time.

(generally not a problem though)

tai64nlocal converts to local time

echo @4000000065f5e9be25c6288c | tai64nlocal
2024-03-16 11:49:24.633743500

Once you are familiar with the character alignment, processing is easy.

Obviously if you have unix seconds, just convert to hex for a tai64n approximation.

@40000000 65f5e9be 25c6288c

unix seconds in the middle (sans leap seconds) followed by 0x25 60ths of a second...

The center 8 bits are not exactly unix seconds because leap seconds are handled differently.

For me, that was the start of managing time inaccuracies gracefully. Basically timestamps

are 24 bits beginning with an @, or they are something less accurate ;-)

I am in the habit of just using some significant digits to represent approximate time

eg there is about an 8 hour window using 65f5, and using 65f5e gives about :30 minute accuracy.

These journal entries are easy to sort and their relative spans are easy to "see", once familiar.

Functions to convert limited resolution to local time are pretty easy.

In practice you can use tai64nlocal, but the utility is simplicity, not conversion to human format.

I have local functions that generate this format 65f5e9a3-20240316_114857 for interchange.

It keeps me oriented to hex time and has enough human resolution...

http://cr.yp.to/libtai.html

http://cr.yp.to/libtai/tai64.html

http://cr.yp.to/daemontools/tai64n.html

http://cr.yp.to/daemontools/tai64nlocal.html

https://en.wikipedia.org/wiki/International_Atomic_Time

I'm not crazy enough to get into debates about time tracking, but this system

works well for me and I'm not expecting it to change (across platforms or otherwise). :-)

-George

--

George Georgalis, (415) 894-2710, http://www.galis.org/

[George Georgalis]

oops, there are a few more than 24 bits there... but the resolution should be clear ;-)

[Paul Merrell]

You might take a look at lua-time, <https://github.com/leite/lua-time>. I haven't used it. The page says it's for Lua 5.1 or LuaJit. 

Best regards,

Paul

 

--

[Notice not included in the above original message:  The U.S. National Security Agency neither confirms nor denies that it intercepted this message.]
                                                ¯\_(ツ)_/¯

[bil til]

Am Sa., 16. März 2024 um 19:25 Uhr schrieb Bjorn Buckwalter
<bjorn.bu...@gmail.com>:

>
> Thanks all for the suggestions! I should clarify, though: I am working with files of time series data (satellite ephemeris) and as such am more interested in manipulating (parsing, arithmetic, formatting) date and time data rather than

In 32bit systems, in contrast to 64bit unix time (Seconds since 1070),
I perfer to use two float variables "number of days" (since 1970
similar to Unix time, or since 2020 or 2000, as you prefer), and the
seconds per day.

A day has 3600x24 seconds, so ca. 80000-100 000 sec. Float 32bit has
bit resolution of 22 bits, so 4*2E10*2'E10 or 4 Mio. . So you get
1/40sec with exact bit resolution by such float representation.

200 years are ca. 100000 days, so your day span then is 200x40= 8000
years by this calculation.

If you want a factor 20 better sec resolution, and 400 (=8000/20)
years are long enough for your timers, you could also use 2 float
variables for "number of hours" and "number of seconds by hour", then
the seonds variable has exact bit resolution of 1/800, or ca. 1msec.

... using such a "segmentation", time and date differences are failry
easy to calculate in 32 bit systems with 32bit floating point support
(e. g. ARM Cortex M3 upwards microcontrollers).

[Denis Dos Santos Silva]

LuaDate is a Lua module for date and time calculation and retrieval using the Gregorian Date system.

https://tieske.github.io/date/

[Frank Kastenholz]

> On Mar 16, 2024, at 2:25 PM, Bjorn Buckwalter <bjorn.bu...@gmail.com> wrote:
>
> Thanks all for the suggestions! I should clarify, though: I am working with files of time series data (satellite ephemeris) and as such am more interested in manipulating (parsing, arithmetic, formatting) date and time data rather than accurately determining the current time or measuring elapsed time. I could probably wrap something around the os module functions adding a nanosecond field, but thought something might be out there already (Denis pointed to luatz, but from the docs alone it isn't obvious to me if fractional seconds are supported).

I had to do something like this in a signal processing application I worked on a long time ago. It received samples which included a very precise timestamp (generated internally by the hardware). We then had to do basic arithmetic, etc, with the timestamps. We wrote a C/Lua module that, among other things, provided the timestamp as a Lua userdata, with the necessary __add/etc metafunctions. The one problem we ran into was that Lua had a couple of places where it does not allow mixed operations, such as <userdata> <operation> <number>, or the like - it didn’t do what we wanted with regard to calling the metafunctions of the userdata, which could look at the other operand and Do The Right Thing based on the operand’s type. This was desired to make the Lua code easier to write and clearer. For example, we wanted to do things like
If <timestamp_userdata> == 12345
I’m afraid I don’t recall the places this happened - but it was fairly easy to find and fix them when we ran into it.

Anyway, I think that you’d be best off doing the same thing since you then can get _exactly_ the functionality, etc, that you want rather than having to coerce your data into existing libraries, etc.

Frank Kastenholz

[Luiz Henrique de Figueiredo]

> The one problem we ran into was that Lua had a couple of places where it does not allow mixed operations, such as <userdata> <operation> <number>, or the like - it didn’t do what we wanted with regard to calling the metafunctions of the userdata, which could look at the other operand and Do The Right Thing based on the operand’s type.

That depends on how you code the metafunctions. Lua does call a
metamathod regardless of the order. Try this code:
a=setmetatable({},{__add=print})
x=a+4
x=4+a
It should output something like
table: 0x7fe9dfc09480 4
4 table: 0x7fe9dfc09480

[Frank Kastenholz]

Thanks for the response

After a bit of searching my memory, the issue was with comparing for equality. We had a case where we had a C structure similar to a Unix timeval. This structure was managed by a userdata and the userdata had a metastable associated with it.

The application programmers wanted to be able to write something like:
If (timeval==5) then … else … end
And have that be shorthand for
If (timeval.tv_sec == 5 and timeval.tv_usec == 0) …

It didn’t work because we were comparing a userdata and a number — and Lua says that if the types are different, they are not equal (we assumed that the __eq metamethod would override this). We modified luaV_equalobj to allow this.

We never proposed changing Lua to support this since A) we felt that it was of limited use (only a few places would benefit), B) it was a shortcut for a fairly simple alternative, and C) we could imagine that it could break other Lua code that might (in some weird way) depend in the existing definition of Lua’s == behavior.

Thanks
Frank Kastenholz

[Luiz Henrique de Figueiredo]

> the issue was with comparing for equality.

Indeed, that is a limitation: equality first tests types and so never
reaches metamethods.

[Thijs Schreijer]

LuaDate [1] is what I typically use to do date/time calculations. This is pure Lua so is limited in handling TZ conversions. LuaTZ would be better if you require TZ conversions.

luasystem [2] : a cross platform library (mac/unix/windows) that has functions that Lua lacks due to its ansi-C nature. It includes getting system time (without pulling in a socket library) and also features monotonic time.

<shameless plug> besides that we recently added environment variable setting/listing, solid random numbers, and 'isatty' all cross-platform</shameless plug>

regards

Thijs

[1]: https://github.com/tieske/date

[2]: https://github.com/lunarmodules/luasystem