file-position in Win32 not working beyond 1 GB

[Dmitry Pavlov]

Hello,

On a fresh 32-bit Racket 7.5 install on 32-bit Windows 7,
(file-position port number) does not work when number
is more that 1 GB.

I can not now say exactly what the error message was,
because I am away from that system, but IIUC it
was something about the position being "too large".

The size of the file, though, is definitely large enough
to have that position, and (file-position port number)
on it worked fine until the number grew beyond
the said limit.

On 64-bit Linux, everything is fine.

Can I work this problem around somehow?

Best regards

Dmitry

[George Neuner]

On 12/20/2019 9:39 AM, Dmitry Pavlov wrote:

On a fresh 32-bit Racket 7.5 install on 32-bit Windows 7,
(file-position port number) does not work when number
is more that 1 GB.

On 32-bit machines, the largest fixnum is 2^30  which is 1GB.  I suspect that  file-position  really wants a fixnum rather than a non-negative-integer as is documented.

Can I work this problem around somehow?

I don't believe there's a way within Racket.  Potentially you could use FFI and drop into Windows own file functions - the 32-bit API contains an extended seek function which works with 64 bit offsets.


George

[Dmitry Pavlov]

On a fresh 32-bit Racket 7.5 install on 32-bit Windows 7,
(file-position port number) does not work when number
is more that 1 GB.

On 32-bit machines, the largest fixnum is 2^30  which is 1GB.  I suspect that  file-position  really wants a fixnum rather than a non-negative-integer as is documented.

That would be sad because it essentially renders my program non-runnable on 32-bit Windows.

To those who know about how (file-position) works: has it always been that way?
I know that 32-bit systems are not cool nowadays, but they are still in use on some of my colleagues' machines.

Also, I do not quite buy the 1 GB limit even on a 32-bit system.
I would have understood maximum the limit of maximum signed integer 2^31 = 2 GB (and my program would be fine with 2 GB limit).

Best regards,

Dmitry

[Dmitry Pavlov]

> I would have understood maximum the limit of maximum signed integer
> 2^31 = 2 GB (and my program would be fine with 2 GB limit).

meant to be "I would have understood the limit of maximum signed integer
..."

[Matthew Flatt]

The Racket-imposed limit should be 64 bits (more than enough) on all
platforms. I can try to replicate the problem later today, but more
information on the error message would be helpful.

> --
> You received this message because you are subscribed to the Google Groups
> "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to racket-users...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/racket-users/22f36172-daf8-7e6f-5f55-9e1c562b
> 8b61%40iaaras.ru.

[Shu-Hung You]

Could it be that the Racket layer importing rktio C code directly uses
get-ffi-obj and does not convert scheme_bignum_type to _int64?

@Dmitry: I guess Racket uses 1 bit for type tagging, leaving only
30+sign (or 62+sign) bits for the actual value.

> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/5dfd2879.1c69fb81.2bbe.ad80SMTPIN_ADDED_MISSING%40gmr-mx.google.com.

[Shu-Hung You]

On Fri, Dec 20, 2019 at 2:05 PM Shu-Hung You
<shu-hu...@eecs.northwestern.edu> wrote:
>
> Could it be that the Racket layer importing rktio C code directly uses
> get-ffi-obj and does not convert scheme_bignum_type to _int64?
>

Never mind this -- I got the FFI part wrong.

[Dmitry Pavlov]

> The Racket-imposed limit should be 64 bits (more than enough) on all
> platforms. I can try to replicate the problem later today, but more
> information on the error message would be helpful.

I do not have access to that Windows 7 machine until Monday.
I managed to reproduce the problem, though, on a 32-bit Windows XP with
Racket 7.1.

file-position: new position is too large
  port: #<input-port:E:\path\to\my\file>
  position: 1122398240

Best regards,

Dmitry

[George Neuner]

The fixnum tag is just 1 bit, but you are forgetting about the sign.  32
- 1 (tag) - 1 (sign) = 30.

George

[Matthew Flatt]

I'm not able to replicate the error on a fresh(!) 32-bit Windows 7
installation using v7.1 or v7.5. I also don't think it should happen
based on the implementation. So, I'm missing something.

That error string only appears here:

https://github.com/racket/racket/blob/master/racket/src/racket/src/port.c#L4093

There could be something wrong with `scheme_get_long_long_val`, but
it's surprising that it could go wrong in a machine-specific way.

Does the error happen for you even in a very short program that tries
to set the file position to 1122398240, or does it only happen in your
full program?

Matthew

[George Neuner]

On 12/20/2019 6:59 PM, Matthew Flatt wrote:
> At Fri, 20 Dec 2019 23:39:30 +0300, Dmitry Pavlov wrote:
> > > The Racket-imposed limit should be 64 bits (more than enough) on all
> > > platforms. I can try to replicate the problem later today, but more
> > > information on the error message would be helpful.
> >
> > I do not have access to that Windows 7 machine until Monday.
> > I managed to reproduce the problem, though, on a 32-bit Windows XP with
> > Racket 7.1.
> >
> > file-position: new position is too large
> >   port: #<input-port:E:\path\to\my\file>
> >   position: 1122398240
>
> I'm not able to replicate the error on a fresh(!) 32-bit Windows 7
> installation using v7.1 or v7.5. I also don't think it should happen
> based on the implementation. So, I'm missing something.

I tried with 32-bit  7.3, but I also could not create a ~1GB file
position size error.


One strange thing I did notice though:
My 32-bit Racket happens to be on a 64-bit Win10 host.  I dashed off the
little test program below, then compiled it to a (13MB) distribution
executable for the 32-bit Win7 machine (which does not have Racket
available).  When I tried it, I was quite surprised that it took over a
minute to run.  I deleted the data file and ran it again, this time
monitoring the file size in Explorer.  The file began small and steadily
increased in size until the program ended. Being curious, I then tried
the executable on the 64-bit Win10 machine and got the similar results -
it ran faster, but still took much longer than I expected.

Is Racket really writing billions of zeroes rather than creating a
sparse file?  It seems to me that this file should only create 2 actual
data blocks, and (modulo JIT) the program should finish almost
instantly.  What am I missing?

George

  -----------------------------------------------------
#lang racket

(define (fout) (current-output-port))

(with-output-to-file "test.dat"
  (lambda ()
    (write "hello")
    (file-position (fout) (* 3 1024 1024 1024))
    (write "there")
    )
  #:mode   'binary
  #:exists 'can-update
  )

[Dmitry Pavlov]

> There could be something wrong with `scheme_get_long_long_val`, but
> it's surprising that it could go wrong in a machine-specific way.
>
> Does the error happen for you even in a very short program that tries
> to set the file position to 1122398240, or does it only happen in your
> full program?

I just did some debug printing.

The problem is more complicated than I thought.
I was misled by the "too large" diagnostics message and thought that the problem is just the number, but it is not, or at least not only the number.

The error pops out not on the first or second positioning exceeding 1 GB. In fact, it happens on different moments in different runs.

This time, for instance, it happened on position 1120708256.
Interesntigly, moments ago the positioning on the same number, and also with different numbers, on the same file, several times, did not raise an error. Here is the tail of the sequence of positioning numbers on the same file that ended with the error:

1120708256
46696208
46696208
46696208
1120708256
46696208
46696208
46696208
1120708256
46696208
46696208
46696208
1120708256 ---> error

(other files were also positioned alternately with this one; not sure if it is important).

Despite the presence of numbers < 1 GB in the sequence, the error always happens on numbers greater than 1 GB

I never had a problem with the same program on 64-bit Linux.

I can do some more tests if you need them. I could use a hint about what kind of test would be useful to track down the problem.

Best regards,

Dmitry

[Matthew Flatt]

At Sat, 21 Dec 2019 00:22:19 -0500, George Neuner wrote:
> Is Racket really writing billions of zeroes rather than creating a
> sparse file?  It seems to me that this file should only create 2 actual
> data blocks, and (modulo JIT) the program should finish almost
> instantly.  What am I missing?

Although NTFS supports sparse files, it looks like the OS creates them
only when an application specifically requests sparse mode. Racket does
not currently request sparse mode for any files that it creates.

[Matthew Flatt]

At Sat, 21 Dec 2019 11:06:33 +0300, Dmitry Pavlov wrote:
> The error pops out not on the first or second positioning exceeding 1 GB. In
> fact, it happens on different moments in different runs.

Thanks! It really is a bug in `scheme_get_long_long_val`, where the
extraction can read past the end of a bignum on a 32-bit platform.

Repair pushed.

[Dmitry Pavlov]

> Thanks! It really is a bug in `scheme_get_long_long_val`, where the
> extraction can read past the end of a bignum on a 32-bit platform.
>
> Repair pushed.

Great, thank you!
Given that I do not normally build Racket, should I wait for the next snapshot from UoU or Northwestern to check out the repaired version?

Or any of the two?

Best regards,

Dmitry

[Matthew Flatt]

Yes --- but the Northwestern snapshot site is having trouble, so it
will be at the Utah site in about 24 hours.

In case it helps to run earlier, here's a one-off snapshot:

http://www.cs.utah.edu/~mflatt/tmp/racket-7.5.0.14-i386-win32.exe

[Dmitry Pavlov]

Matthew,

Only today I managed to get back to that problem.
But your effort of making a special executable for me still turned out to be relevant.
Because on my good ol' Windows XP machine, the Utah i386 nigtly build, while installing successfully, resulted in a Racket executable that the system refused to run, saying that it is not a correct executable. (The message was localized in Russian so I am not sure what is the exact English original message). Same situation with raco.exe so it must be something systematic.

And your special build works fine. And older stock Racket 7.1 worked fine too.

I can totally live without WinXP, just curious whether it is a planned behaviour.

Best regards,

Dmitry

[Matthew Flatt]

At Fri, 27 Dec 2019 14:01:33 +0300, Dmitry Pavlov wrote:
> Because on my good ol' Windows XP machine, the Utah i386 nigtly build, while
> installing successfully, resulted in a Racket executable that the system
> refused to run, saying that it is not a correct executable. (The message was
> localized in Russian so I am not sure what is the exact English original
> message). Same situation with raco.exe so it must be something systematic.
>
> And your special build works fine. And older stock Racket 7.1 worked fine too.
>
> I can totally live without WinXP, just curious whether it is a planned
> behaviour.

The release builds and that special build are cross-compiled using
MinGW. The one that doesn't run from the Utah snapshot is built with
MSVC 2017. I think that must be the difference, with modenr MSVC doing
something to the executable that XP doesn't recognize.

The release builds will continue to use MinGW, so they'll continue to
work on XP.

... at least for "Racket.exe" and "Raco.exe". I see that "DrRacket.exe"
doesn't work on XP, anymore, because glib and related GUI dependencies
use system functions that became available only in Vista. (If someone
really needed to make that work, it's possible that substituting
libraries from an old Racket release work work, but I'm not sure.)

[George Neuner]

On Wed, 1 Jan 2020 15:50:10 -0700, Matthew Flatt
<mfl...@cs.utah.edu> wrote:

>The release builds and that special build are cross-compiled using
>MinGW. The one that doesn't run from the Utah snapshot is built with
>MSVC 2017. I think that must be the difference, with modenr MSVC doing
>something to the executable that XP doesn't recognize.

IIRC VS2015 could build for XP if some extension was installed: it
produced an execuable that could run with the VS2013 library.

I'm pretty sure VS2017 and later don't support XP at all.

VS2013 will work for XP assuming that the code doesn't require any
functions that are only in later systems. Maybe someone has a copy
that they can donate?

George