Review fseek on MinGW / MinGW-w64 on Windows prior to Lua 5.5.0 official release

[Luau Project]

In a temporary repository ( https://github.com/luau-project/lua-fseek-analysis ), through GitHub Actions, I performed an analysis on the state of fseek support on Lua 5.5.0 (beta) announced at https://groups.google.com/g/lua-l/c/N1MMWqG4Ad0/m/2CIoXFJyAwAJ , in the hope that the proposed changes get incorporated on Lua 5.5.0 official release.

Almost one decade ago, the fseek problem on Windows was reported at http://lua-users.org/lists/lua-l/2015-05/msg00315.html . Later, Roberto provided an answer to the problem at http://lua-users.org/lists/lua-l/2015-05/msg00370.html , often used by the community as a work around with MinGW / MinGW-w64.

For the analysis, the repository downloads a huge file ( 3.7 GB - Debian 12.11 iso from https://cdimage.debian.org/debian-cd/current/amd64/iso-dvd/debian-12.11.0-amd64-DVD-1.iso ) and performs simple fseek tests on different C toolchains on Windows:

• MinGW-w64 provided by the MSYS2 project using GCC 15.1.0. Also includes environments using clang 20.1.7 from llvm-mingw ( https://github.com/mstorsjo/llvm-mingw );
• MinGW-w64 already installed on GitHub machines using GCC 12.2.0;
• MinGW (32-bit) using GCC 4.8.1 and GCC 6.3.0, considered a defunct project, which is only available at https://sourceforge.net/projects/mingw/ . However, it is still relevant due a high number of weekly downloads;
• MSVC (19.44.35209) from Visual Studio 2022;
• clang-cl (19.1.5), which is clang MSVC-compatible.

At the moment, three kind of tests are run:

• tests employing a patch provided by Roberto;
• tests employing a custom patch provided by myself;
• tests without fseek patches at all.

Conclusion:

Tip: Anyone logged in to GitHub can inspect the results of the tests: https://github.com/luau-project/lua-fseek-analysis/actions/runs/16128344144

* In the GitHub repository https://github.com/luau-project/lua-fseek-analysis , I proposed a solution that works on all MinGW / MinGW-w64 toolchains under tests.

PS.: Since MinGW / MinGW-w64 has become the standard choice for developers planning to use GCC on Windows, I think it an addition worth to incorporate on Lua codebase.

In my opinition, this is a suitable opportunity to review fseek for relevant MinGW / MinGW-w64 providers on Windows before the final Lua 5.5.0 release.

[Sainan]

Using Windows' 64-bit versions of fseek seems reasonable for files >4GB.

However, it seems your issue is with a file in the 2GB-4GB range meaning the high bit of a 32-bit integer being set, so I would suspect some signed/unsigned problem in MinGW. As you say yourself, you're using a defunct version of it, so any such issues are totally expectable and your own fault for not updating your MinGW.

-- Sainan

[Luau Project]

Unfortunately, due to limited text formatting capabilities here, I decided to keep the initial post small.
Most of the details are in the GitHub repository ( https://github.com/luau-project/lua-fseek-analysis ). You have to visit it to understand the problem.

In case you are not willing to check the details, then I'll put it here in short terms: the current unpatched Lua 5.5.0 fseek is not working correctly on MinGW / MinGW-w64. Moreover, it is not working correctly since 5.3 on MinGW / MinGW-w64, regardless of a recent / ancient MinGW or MinGW-w64 version.

One may argue that it is a MinGW / MinGW-w64 fault. Yes, I agree on that. On the other hand, Lua has MSVC-specific code too. Grounded on the fact that MinGW is a target in the Makefile, I can suppose it should receive some attention.

Anyone willing to better understand the problem shall:

1. Have a read in the project page on GitHub;
2. Login to GitHub and inspect the runs of the actions.

[sur-behoffski]

G'day,

I'm still wrestling with VM issues, but hope to get to my main target,
testing compatibility of Lua-5.5.0-beta with LuaRocks 3.12.2, later
today.

In the meantime, I can report that David Scilia's "continue" keyword
patch from 2023, works with Lua 5.4.2-5.4.8. (Lua-5.4.5 was withdrawn
due to a separate environment issue, and is not supported by lglicua.)

Cheers, b

[Denis Dos Santos Silva]

if you are using gcc, checkout

gcc/clib dont `deal` automatically based *on bits*, but, standart.

_FILE_OFFSET_BITS

 _LARGEFILE64_SOURCE

$getconf LFS_CFLAGS

wrapped 64-bit
off_t -> off64_t
fseeko -> fseeko64

.... 
open -> open64

#define _FILE_OFFSET_BITS 64 /// enable 64-bit ***wrapped*** 64-bit functions //
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>

int main() {
    int fd = open("arquivo_grande.bin", O_RDONLY);
    if (fd == -1) {
        perror("open");
        return 1;
    }

    off_t offset = lseek(fd, 0, SEEK_END);
    printf(" %lld bytes\n", (long long)offset);
    close(fd);
    return 0;
}

[Sainan]

I think you replied to the wrong thread.

-- Sainan

[Sainan]

Oh wait, sur-behoffski was replying to the mingw fseek thread but changed the subject so it shows up in different threads in my email client, but it's the same thread on Google groups. So strange.

-- Sainan

[Roberto Ierusalimschy]

(A somewhat late reply...)

The "patch provided by Roberto" is now mostly incorporated in the
code base, except that it does not check for MinGW:

------------------------------------------
$ tail -n -727 liolib.c | head
#if !defined(l_fseek) /* { */

#if defined(LUA_USE_POSIX) /* { */

#include <sys/types.h>

#define l_fseek(f,o,w) fseeko(f,o,w)
#define l_ftell(f) ftello(f)
#define l_seeknum off_t

------------------------------------------

Maybe we could only change the test 'defined(LUA_USE_POSIX)',
for instance adding another flag (e.g. LUA_USE_LARGESEEK)?

-- Roberto

[Luau Project]

> Maybe we could only change the test 'defined(LUA_USE_POSIX)',
> for instance adding another flag (e.g. LUA_USE_LARGESEEK)?

Based on the fact that 64-bit operating systems (and their large files) are widespread nowadays, the addition of a `LUA_USE_LARGESEEK` flag sounds interesting. Going that route, I think it would be possible to fix MinGW / MinGW-64 accounting for that flag in the src/Makefile. 

[Luau Project]

Roberto,

I noticed the commit ( 9a3940380a2a1540dc500593a6de0c1c5e6feb69 ) in the GitHub mirror adding the `LUA_USE_OFF_T` compilation flag allowing MinGW / MinGW-w64 to use POSIX-like API for file offsets. Thank you for the addition.

If my guess is correct, in the upcoming version of Lua 5.5.0, I think that you are going to change `src/Makefile` at line 132 to use
    "SYSCFLAGS=-DLUA_BUILD_AS_DLL -DLUA_USE_OFF_T" "SYSLIBS=" "SYSLDFLAGS=-s" lua.exe

That way, the fseek problem would be solved for MinGW-w64 ( https://sourceforge.net/projects/mingw-w64/ ), but not for the legacy 32-bit MinGW ( https://sourceforge.net/projects/mingw/ ).

Not sure if you are interested, but in case a fix for MinGW is also desired, I have a suggestion.

DISCLAIMER: The proposed changes in the file `src/Makefile` below, I am placing under the same license found at https://www.lua.org/license.html

Even though I have read `liolib.c` at least a dozen times, I just noticed the line `#if !defined(l_fseek)` today. Then, I realized that it allows the `fseek` problem to be solved just by changing `src/Makefile` from lua-5.5.0-beta tarball at line 132:

where it is:
    "SYSCFLAGS=-DLUA_BUILD_AS_DLL" "SYSLIBS=" "SYSLDFLAGS=-s" lua.exe

can be replaced by:
    "SYSCFLAGS=-DLUA_BUILD_AS_DLL -Dl_fseek=fseeko64 -Dl_ftell=ftello64 -Dl_seeknum=off64_t" "SYSLIBS=" "SYSLDFLAGS=-s" lua.exe

Applying such replacement above, it fixes the fseek problem for both MinGW / MinGW-w64 (I tested the changes locally).

Note 1: If you want, I can rerun the GitHub Action tests applying such replacement in the Makefile.

Note 2: I also noticed that my `lua-fseek-analysis` repository was MIT licensed. In the next commit to test the upcoming changes on Lua 5.5.0 (beta-2 or RC or whatever), I'll relicense it to use https://www.lua.org/license.html . This move has the intent to not cause any worries in the use of the proposed changes by the Lua team, which may arise due the copyright notice required by the MIT license.

[Roberto Ierusalimschy]

> Even though I have read `liolib.c` at least a dozen times, I just noticed
> the line `#if !defined(l_fseek)` today. Then, I realized that it allows the
> `fseek` problem to be solved just by changing `src/Makefile` from
> lua-5.5.0-beta tarball at line 132:
>
> where it is:
> "SYSCFLAGS=-DLUA_BUILD_AS_DLL" "SYSLIBS=" "SYSLDFLAGS=-s" lua.exe
>
> can be replaced by:
> "SYSCFLAGS=-DLUA_BUILD_AS_DLL -Dl_fseek=fseeko64 -Dl_ftell=ftello64
> -Dl_seeknum=off64_t" "SYSLIBS=" "SYSLDFLAGS=-s" lua.exe

Or else you add these defines in luaconf.h.

-- Roberto

[Edoardo Lolletti]

Regarding this issue, wouldn't be easier to use the windows api for
these cases? In this scenario, only fseek and ftell would have to be
changed to use SetFilePointer.
In this case, to get the windows HANDLE from the FILE* the chain of calls
HANDLE handle = (HANDLE)_get_osfhandle(fileno(f));
would have to be done.
Something akin to

long long l_fseek(FILE* f, lua_Integer offset, int origin) {
HANDLE handle = (HANDLE)_get_osfhandle(fileno(f));
LARGE_INTEGER i;
i.QuadPart = offset;
i.LowPart = SetFilePointer(handle, i.LowPart, &i.HighPart, origin);
return (i.LowPart == INVALID_SET_FILE_POINTER) ? 0 : -1;
}
long long l_ftell(FILE* f) {
HANDLE handle = (HANDLE)_get_osfhandle(fileno(f));
LARGE_INTEGER i;
i.QuadPart = 0;
i.LowPart = SetFilePointer(handle, i.LowPart, &i.HighPart, FILE_CURRENT);
return i.QuadPart;
}

(with the mapping from windows errors to errno values

[Luau Project]

> Regarding this issue, wouldn't be easier to use the windows api for
> these cases?

Using specific Windows API is also a possibility, but speaking for myself, I am fine with the proposed changes, because it allows most Windows-specific use cases ( (1) MSVC / clang-cl; and (2) recent MinGW-w64 for GCC and clang as well ).

As far as I am aware, Lua does a great job to keep most of its functionality closely related to plain C89, deviating from that only in some tiny areas (e.g.: loading shared libraries for Lua modules) for some well known groups of operating systems.

From my own experience, with a C89 compliant compiler, I know it is possible to compile, build and run most Lua versions almost unmodified on very old 16-bit operating systems. I did test Lua 5.1 on DOSBox ( check it here https://github.com/luau-project/lua-pcg/blob/main/docs/DOSBox.md if you have interest ), an open source MSDOS, and I know that other Lua versions (5.4) also work there with minor changes to the source code.