Rounding error in dates?

[Grunthos]

I have a local file with a date as reported by ```ls -l --time-style=full-iso``` of "2020-05-09 00:36:07.999999900 +0000".

When I copy it locally using ```cp -a``` on a local btrfs file system with Ubuntu 20.4.1, the date is preserved.

When I copy it to a local S3QL file system, one second is added to the date. and it is reported as "2020-05-09 00:36:08.999999900".

When I copy it back, the date retains the '08' seconds. If I copy it back again to S3QL, it advances one more second.

It looks like a rounding error, since dates ending in " 999999000" work as expected.

Console output as follows:

root@xxx:/home/usr# touch -d "2013-01-13 13:13:13.9999" q.q
root@xxx:/home/usr# cp -a q.q /mnt
root@xxx:/home/usr# ls -l --time-style=full-iso /mnt/q.q
-rw-r--r-- 1 root usr 96 2013-01-13 13:13:13.999900000 +0000 /mnt/q.q
root@xxx:/home/usr# touch -d "2013-01-13 13:13:13.99999" q.q
root@xxx:/home/usr# cp -a q.q /mnt
root@xxx:/home/usr# ls -l --time-style=full-iso /mnt/q.q
-rw-r--r-- 1 root usr 96 2013-01-13 13:13:13.999990000 +0000 /mnt/q.q
root@xxx:/home/usr# touch -d "2013-01-13 13:13:13.999999" q.q
root@xxx:/home/usr# cp -a q.q /mnt
root@xxx:/home/usr# ls -l --time-style=full-iso /mnt/q.q
-rw-r--r-- 1 root usr 96 2013-01-13 13:13:13.999999000 +0000 /mnt/q.q
root@xxx:/home/usr# touch -d "2013-01-13 13:13:13.9999999" q.q
root@xxx:/home/usr# cp -a q.q /mnt
root@xxx:/home/usr# ls -l --time-style=full-iso /mnt/q.q
-rw-r--r-- 1 root usr 96 2013-01-13 13:13:14.999999900 +0000 /mnt/q.q

[Grunthos]

This is with S3Ql 3.3.2 (Ubuntu default)

[Nikolaus Rath]

On Nov 22 2020, Grunthos <philip.jo...@gmail.com> wrote:
> I have a local file with a date as reported by ```ls -l
> --time-style=full-iso``` of "2020-05-09 00:36:07.999999900 +0000".
>
> When I copy it locally using ```cp -a``` on a local btrfs file system with
> Ubuntu 20.4.1, the date is preserved.
>
> When I copy it to a local S3QL file system, one second is added to the

> date. and it is reported as "2020-05-09 00:36:*08*.999999900".

>
> When I copy it back, the date retains the '08' seconds. If I copy it back
> again to S3QL, it advances one more second.
>
> It looks like a rounding error, since dates ending in " 999999000" work as
> expected.

Thanks for the report! This is probably a bug in how pyfuse3 converts
timestamps from (or to) nanoseconds - see
https://github.com/libfuse/pyfuse3/blob/master/src/pyfuse3.pyx#L298 and https://github.com/libfuse/pyfuse3/blob/master/src/macros.c#L17.

Patches and unit tests welcome! :-)

Best,
-Nikolaus

--
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

»Time flies like an arrow, fruit flies like a Banana.«

[Grunthos]

Hmm...if I can get my head around exactly how the st_mtime structure/object is formed. But, in the mean time, if there is a tight linkage with the C struct, then one wonders why it uses "int(self.attr.st_mtime )" instead of "(stbuf)->st_mtim.tv_sec"  (to mirror the behaviour of the macro). Further, if the value of "self.attr.st_mtime" is a float(??), why not use floor?

[Grunthos]

*definitely* looks like a rounding problem with using a double-precision value to represent seconds + ns since 1970. For a standard double, only 6 decimal digits work...for ns, you need 9 places.

Try this in python:

>>> 1577880000.9999999

<- this is the source of the problem.

[Grunthos]

Further, if my analysis is correct, then it suggests anything that uses the pyfuse composite (double/float) field may have breakage.

[Grunthos]

I've added a bug to pyfuse3; the ability for me to consider a patch depends in very large part on what level of API compatibility you want to break!

Since 64 bits is insufficient to represent a nano-second timestamp, something has to become incompatible. I think the choice is down to:

• add numpy (or similar) to the pyfuse dependencies
• remove the *_ns properties (or at least make them return a struct/object instead)
  

[Grunthos]

This of course potentially impacts S3QL **if** it stores FS dates as numeric values...since SQLite suffers the same limitation.

[Grunthos]

OK...I know I'm talking to myself here, but it's worth documenting the thoughts in case they are useful...

If pyfuse returned a valid representation of nano-seconds since 1970 (say, for example, a python3 INT value -- which is unbounded), then S3QL would need to store them differently, since SQLite will do nothing more than 64 bit numeric values. There are three broad possibilities, I think:

• store as strings. This would be a resounding 'no' since comparisons would not work as expected, and it would be deeply inefficient.
  
• store as blobs: maybe. Research required to (a) ensure sqlite compares blobs with correct endianness and (b) has committed to continue to do so. Both seem likely, but conversion to/from might be painful.
  
• store as two values (eg. seconds and nanoseconds), retrieve and reassemble as a pythin INT.
  

[Grunthos]

If I were to provide a patch, it would have to be to both pyfuse3 and s3ql. The optimal solution based on analysis so far seems to be:

• pyfuse returns only integer values for dates, the values are nano-seconds since epoch.
• s3ql stores these values in the database as two integer values (_sec and _ns), derived from the pyfuse3 value using integer division and modulus.
• s3ql retrieves these values and reconstructs them.
• In-database date comparisons need to be modified to compare the two values appropriately.

This would fix the problem I think, but produces potential issues for anything that reads the s3ql database directly, and anything that relies on pyfuse _ns methods returning dates as a real-value number of seconds.

Over to you...I'll keep quiet for a while...

[Grunthos]

The patch I have submitted to pyfuse3 seems to fix the problem on my test machine...

[Nikolaus Rath]

Hi,

Thanks for looking into this! Let me try to help by clarifying some of
the questions that you've raised:

- I think it's fine to break pyfuse3 backwards compatibility, as long
as we are explicit about it and increment the major version
number. That is what it's there for.

- I would rather not break backwards compatibility in S3QL, since this
means that we have to write upgrade code and everyone has to upgrade
their filesystem.

- I do not think S3QL needs to support nanosecond resolution (the
VFS does not expect this from filesystems in general
either). However, we should do better than seconds.

- Currently, S3QL stores data as an int64 of nanoseconds (in both
Python and SQLite). I think this should be fine - in 64 bits we can
store 9223372036854775808 ns which is more than 292 years (starting
at 1970).

- pyfuse3 is not intended to use floating point values at all (see
e.g. http://www.rath.org/pyfuse3-docs/data.html#pyfuse3.EntryAttributes). My
best guess is that code that uses "1e9" should read "1 000 000 000"
and that I (or whoever wrote the code) forgot that 1e9 gives a float
rather than an integer.


All the best,
-Nikolaus

On Nov 23 2020, Grunthos <philip.jo...@gmail.com> wrote:
> I've added a bug to pyfuse3; the ability for me to consider a patch depends
> in very large part on what level of API compatibility you want to break!
>
> Since 64 bits is insufficient to represent a nano-second timestamp,
> something has to become incompatible. I think the choice is down to:
>

> - add numpy (or similar) to the pyfuse dependencies
> - remove the *_ns properties (or at least make them return a

> struct/object instead)
>
> On Monday, November 23, 2020 at 10:42:47 PM UTC+11 Grunthos wrote:
>
>> Further, if my analysis is correct, then it suggests anything that uses
>> the pyfuse composite (double/float) field may have breakage.
>>
>> On Monday, November 23, 2020 at 10:30:52 PM UTC+11 Grunthos wrote:
>>
>>> *definitely* looks like a rounding problem with using a double-precision
>>> value to represent seconds + ns since 1970. For a standard double, only 6
>>> decimal digits work...for ns, you need 9 places.
>>>
>>> Try this in python:
>>>
>>> >>> 1577880000.9999999
>>>
>>> <- this is the source of the problem.
>>>
>>
>

> --
> You received this message because you are subscribed to the Google Groups "s3ql" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to s3ql+uns...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/s3ql/0a4a5cb4-bf43-4f72-bacd-69636d26d37en%40googlegroups.com.

[Grunthos]

Thanks for the info! Not sure how I managed to miscalculate the integer representation, but that's great news...it means all that needs fixing is pyfuse3, which is pretty easy. Ditto llfuse (the fix I sent for pyfuse3 also works on llfuse).