Accessing a global table across threads

[John Logsdon]

Folks

I am using torch mainly (at the moment) to access the threads module and
this works very well on my AMD Threadripper 16 core box (Ubuntu 16.04).

However I have to read the same data file every time a particular function
(called Pass) is called which, since the file is of the order 1MB and
called ~ 10^6 times, makes it rather inefficient, despite luajit's
legendary speed.

The n*p data of numbers was stored as a series of loadstrings (yuk but it
worked) but now I store it as a table of 8 byte reals using ffi, which
approximately halves the data load.

I have tried using mmap but the file handle and data are then cdata which
cannot be serialised as upvalues by threads and also use an address >4G
which would not be within the child thread luajit's space.

I use /tmpfs to store the data file in memory so it would probably not
help that much.

However, as the data are not modified at all during the process, this is
all very wasteful so I was playing with modules to see if I can
effectively pre-load the data into the global space in the instantiated
pool just once (well 32 times but not >10^5 times!)

Here is the conundrum:

If I require a module like torch, I only have to add one require in the
parent thread for torch to be visible in the child threads. If I don't
require torch anywhere, then the programs barfs.

But if I generate a global table as a module and write it out as eg
mytable.lua then require "mytable", I need to require it also in the Pass
function so it will I think still load the data each time, back to the
same problem. :(

So how can I load the module/table once in the parent thread so that when
the child threads are created, it includes the mytable in the same way
that (eg) torch is included?

It must be something to do with the module creation - some flag that tells
luajit which modules to include. If it needs a bit of C then so be it!!
I have to learn C one day!

Do I need to #include a dummy table in the source and recompile?

Any ideas?

John

John Logsdon
Quantex Research Ltd
t+44161454951/m+447717758675