init_by_lua and io.open

[Victor]

In an "init_by_lua" block I'm getting a pointer to a file content:

local fd = assert ( io.open ("/var/local/test,dat" , "rb" ) )
records = assert ( fd:read ( "*a" ) )

and can use the "records" global in different locations and "content_by_lua":

data = records:byte(offset+1)
--ffi casts

Works great.

But when and how should I release the "fd" file handle ?

Is there a known trick maybe ? :-)

Any help would be welcomed!

Victor 

[Robert Paprocki]

Hi,

io.read returns the contents of the file as a string, not a file pointer. so here your variable `records` contains the whole contents of the file as a string, not a pointer. And once this data is read, you don't need to keep the file handle open, so you can close it immediately following your call to fd:read() (though i believe the file is closed once the variable associated with it falls out of scope).

As for how: fd:close(). Easy :)

[Wiktor Sadowski]

Thanks a lot!
I knew about "fd:close()". But I'm not that happy that io.read returns string and a pointer :-)

For a file over 200MB a pointer would be more sensible in my opinion.

So maybe a memory map file ? Would it make sense ? And how to make it released on nginx exit ?

--
You received this message because you are subscribed to a topic in the Google Groups "openresty-en" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/openresty-en/UtCgggxmoLY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to openresty-en+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Robert Paprocki]

Hi,

On Mon, Aug 22, 2016 at 6:50 PM, Wiktor Sadowski <dsgnv...@gmail.com> wrote:

Thanks a lot!
I knew about "fd:close()". But I'm not that happy that io.read returns string and a pointer :-)

For a file over 200MB a pointer would be more sensible in my opinion.

How exactly would this work? Would you be reading from various portions of the file from disk for every access? That would absolutely destroy performance.

 

So maybe a memory map file ? Would it make sense ? And how to make it released on nginx exit ?

I don't think Lua provides an interface to mmap a file. You will probably need to write a native C module for something like that.

You also may want to examine your approach here. Something like redis might make a lot more sense than seeking all over a bare file, even if loaded entirely into memory. Maybe describe a bit more of what you're trying to accomplish so we can better understand the problem? In general, native Lua file IO is not performant resty ecosystem, and should be avoided.

[Victor]

Thank you very much again.
I have quite big data file with structure inspired by MaxMind DB - https://maxmind.github.io/MaxMind-DB/. Access to records is very fast with pointer math and there is no memory overhead - I already tried in C.
You might be right suggesting redis but allocating enough memory with ffi and moving there the content of the file on "init_by_lua" would work better in such cases. Faster and less resources eaten probably. 
But will try both solutions :-)

v