file-position does not extend file as expected

[David Storrs]

Consider the following test script:

#lang racket

(define (say . args) (displayln (apply ~a args)))
(define the-path "./test-file")
(when (file-exists? the-path)  (delete-file the-path))
(say "before open, file exists?: " (file-exists? the-path))

(let ([the-port (open-output-file the-path #:mode 'binary #:exists 'append)])
  (say "after open, file exists?: " (file-exists? the-path))
  (say "file size, position: " (file-size the-path) ", " (file-position the-port))
  (file-position the-port 1000); should extend the file and fill the intervening space with 0  
  (say "file size, position: " (file-size the-path) ", " (file-position the-port))
  (flush-output the-port) ; ensure that the filling 0s have been written, just to be sure      
  (say "file size, position: " (file-size the-path) ", " (file-position the-port)))

The output is:

before open, file exists?: #f
after open, file exists?: #t
file size, position: 0, 0
file size, position: 0, 1000

file size, position: 0, 1000

The file on disk is in fact 0B long.  The docs for file-position (https://docs.racket-lang.org/reference/port-buffers.html#%28def._%28%28quote._~23~25kernel%29._file-position%29%29) include the following:

"When file-position sets the position pos beyond the current size of an output file or (byte) string, the file/string is enlarged to size pos and the new region is filled with 0 bytes. If pos is beyond the end of an input file or (byte) string, then reading thereafter returns eof without changing the port’s position."

Given the above, I was expecting the file to end up being 1000 bytes.  I'm using Racket 7.1 on OSX 10.11; is this an OS issue or am I misunderstanding something about Racket?

[Jon Zeppieri]

I *think* that the docs are wrong here. `file-position` maps onto
`lseek` on posix systems (pretty sure about this after a quick perusal
of the code) and `lseek` docs say:

> The lseek() function shall allow the file offset to be set beyond the end of the existing data in the file. If data is later written at this point, subsequent reads of data in the gap shall return bytes with the value 0 until data is actually written into the gap.

So the file isn't really extended until you write something at that
position. But I'm sure Matthew can dive a definitive answer here.

- Jon

> --
> You received this message because you are subscribed to the Google Groups "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/CAE8gKofKWtAKU4ZQk6ai-ydZx7WmY8%3D1GwDUFbW15BpxUSQ1qw%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

[Jon Zeppieri]

On Mon, Jun 24, 2019 at 4:51 PM Jon Zeppieri <zepp...@gmail.com> wrote:
>
> `lseek` docs say:
>
> > The lseek() function shall allow the file offset to be set beyond the end of the existing data in the file. If data is later written at this point, subsequent reads of data in the gap shall return bytes with the value 0 until data is actually written into the gap.
>

And the Windows call `SetFilePosition` is similar:

> It is not an error to set a file pointer to a position beyond the end of the file. The size of the file does not increase until you call the SetEndOfFile, WriteFile, or WriteFileEx function. A write operation increases the size of the file to the file pointer position plus the size of the buffer written, which results in the intervening bytes uninitialized.

[David Storrs]

I actually did try writing something at the extended position, although I removed it before posting the above.  Even with a write, it still does not do as expected.  In fact, the results are even stranger:

#lang racket

(define (say . args) (displayln (apply ~a args)))
(define the-path "./test-file")
(when (file-exists? the-path)  (delete-file the-path))
(say "before open, file exists?: " (file-exists? the-path))

(let ([the-port (open-output-file the-path #:mode 'binary #:exists 'append)])
  (say "after open, file exists?: " (file-exists? the-path))

  (say "before repos, file size, position: " (file-size the-path) ", "

       (file-position the-port))
  (file-position the-port 1000); should extend the file and fill the intervening space with 0  

  (say "before flush, file size, position: " (file-size the-path) ", "

       (file-position the-port))
  (flush-output the-port) ; ensure that the filling 0s have been written, just to be sure      

  (say "before write, file size, position: " (file-size the-path) ", "
       (file-position the-port))

  (write "foo" the-port)
  (say "before 2nd flush, file size, position: "

       (file-size the-path) ", " (file-position the-port))
  (flush-output the-port)  

  (say "finally, file size, position: " (file-size the-path) ", " (file-position the-port))
  )

Ouptut:

before open, file exists?: #f
after open, file exists?: #t

before repos, file size, position: 0, 0
before flush, file size, position: 0, 1000
before write, file size, position: 0, 1000
before 2nd flush, file size, position: 0, 1005

finally, file size, position: 5, 5

Personally, i find this even weirder.  I wrote 3 characters, so why is the file size 5?  (Which it is, on disk.)

[Matthew Flatt]

I think that part is because you've opened the file in 'append mode.

At the OS level, 'append means "move to the end of the file before each
write". So, 'append and `file-position` to extend a file size just
don't work helpfully together.

I'll improve the docs.

> --
> You received this message because you are subscribed to the Google Groups
> "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to racket-users...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/racket-users/CAE8gKodw3c2wqUBdAPKRkX3hqin4fhZ
> 8kjeFKh3LouFhY4-8XQ%40mail.gmail.com.

[Shu-Hung You]

FWIW, (write "foo") writes "\"foo\"" to the file. Maybe you want
write-string or write-bytes?

> On Mon, Jun 24, 2019 at 4:57 PM Jon Zeppieri <zepp...@gmail.com> wrote:
>>
>> On Mon, Jun 24, 2019 at 4:51 PM Jon Zeppieri <zepp...@gmail.com> wrote:
>> >
>> > `lseek` docs say:
>> >
>> > > The lseek() function shall allow the file offset to be set beyond the end of the existing data in the file. If data is later written at this point, subsequent reads of data in the gap shall return bytes with the value 0 until data is actually written into the gap.
>> >
>>
>> And the Windows call `SetFilePosition` is similar:
>>
>> > It is not an error to set a file pointer to a position beyond the end of the file. The size of the file does not increase until you call the SetEndOfFile, WriteFile, or WriteFileEx function. A write operation increases the size of the file to the file pointer position plus the size of the buffer written, which results in the intervening bytes uninitialized.
>

> --
> You received this message because you are subscribed to the Google Groups "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/CAE8gKodw3c2wqUBdAPKRkX3hqin4fhZ8kjeFKh3LouFhY4-8XQ%40mail.gmail.com.

[David Storrs]

On Mon, Jun 24, 2019 at 5:19 PM Shu-Hung You <shu-hu...@eecs.northwestern.edu> wrote:

FWIW, (write "foo") writes "\"foo\"" to the file. Maybe you want
write-string or write-bytes?

*facepalm*

I think I want a better brain.  Thank you.

Matthew, thanks for the clarification about append mode.  Sure enough, as soon as I take it out of 'append everything is fine.

Dave