file io within NGINX and OpenResty - blocking ?

[Stefan Parvu]

Hi,

If I understood right the way nginx and openresty handles access to disk and files is by blocking at the worker level. Correct ?

Suppose I have a simple LUA function which opens a JSON file, writes some info to the file, closes it, for example when
a new user has been registered to our system. Now if I have an OpenResty installation where NGINX has been configured
to run on 4 or 6 workers, how can I ensure consistency to the file ?

Questions:

1. How one would ensure data consistency when having NGINX running on 2,4 workers ?

2. Would be easier to handle the access to the file in-memory ? Like for example memcached or redis ?
Anyway at some point data should arrive to the flile itself ? For example in a case of a system crash or
whatever else ...

3. What other suggestions ?

Many thanks,
Stefan

[Yichun Zhang (agentzh)]

Hello!

On Wed, Aug 20, 2014 at 9:48 AM, Stefan Parvu wrote:
> If I understood right the way nginx and openresty handles access to disk and
> files is by blocking at the worker level. Correct ?
>

Yes, unless AIO is used (but some file related syscalls like stat can
still block) because there is no such thing as "nonblocking file IO".

> Suppose I have a simple LUA function which opens a JSON file, writes some
> info to the file, closes it, for example when
> a new user has been registered to our system.

Maybe you should really use an external database or a simpler
key-value store exposing a socket interface for it?

> Now if I have an OpenResty
> installation where NGINX has been configured
> to run on 4 or 6 workers, how can I ensure consistency to the file ?
>

If you really want to write to files in OpenResty, you need to use
lua-resty-lock to protect it:

https://github.com/openresty/lua-resty-lock

Again, this can be way less efficient than using a socket-based
external data service as suggested above.

> 1. How one would ensure data consistency when having NGINX running on 2,4
> workers ?
>

See above.

> 2. Would be easier to handle the access to the file in-memory ? Like for
> example memcached or redis ?

Yes, sure. Also, redis has persistence support itself:
http://redis.io/topics/persistence

> Anyway at some point data should arrive to the flile itself ? For example in
> a case of a system crash or
> whatever else ...
>

See above.

> 3. What other suggestions ?
>

Avoid file IO wherever possible in the context of nginx.

Regards,
-agentzh

[Stefan Parvu]

Yes, unless AIO is used (but some file related syscalls like stat can
still block) because there is no such thing as "nonblocking file IO".

correct. we dont use AIO.

If you really want to write to files in OpenResty, you need to use
lua-resty-lock to protect it:

    https://github.com/openresty/lua-resty-lock

Right. Thanks. I will read about it. The more I think it looks clearly we need probable to use Redis and
change the way we handle IO for this part. We need to move away from accessing the files within
OpenResty.

Thanks a lot for comments and help,
Stefan

[Aapo Talvensaari]

On Wednesday, August 20, 2014 10:52:04 PM UTC+3, Stefan Parvu wrote:

If you really want to write to files in OpenResty, you need to use 

lua-resty-lock to protect it:

    https://github.com/openresty/lua-resty-lock

Right. Thanks. I will read about it.

There are some threads in this forum where people ask about using SQLite. SQLite handles the locking and has busy timeout in it's API:

http://www.sqlite.org/c3ref/busy_timeout.html

But you have to use the same connection for busy_timeout to work (e.g. multiple writers wait). You should also check out WAL mode:

http://www.sqlite.org/wal.html (e.g. you may use multiple threads, but use jus one connection).

There are also other tunings:

http://www.sqlite.org/pragma.html

If you need more scalability, you should set up a server, but SQLite is probably the next best bet if you want to have it serverless.

[Aapo Talvensaari]

On Thursday, August 21, 2014 2:15:37 AM UTC+3, Aapo Talvensaari wrote:

On Wednesday, August 20, 2014 10:52:04 PM UTC+3, Stefan Parvu wrote:

If you really want to write to files in OpenResty, you need to use 

lua-resty-lock to protect it:

    https://github.com/openresty/lua-resty-lock

Right. Thanks. I will read about it.

There are some threads in this forum where people ask about using SQLite. SQLite handles the locking and has busy timeout in it's API:

http://www.sqlite.org/c3ref/busy_timeout.html

I was also thinking that you could maybe ngx.timer.at your SQLite access. This way it doesn't block the Nginx worker.

[Yichun Zhang (agentzh)]

Hello!

On Wed, Aug 20, 2014 at 4:43 PM, Aapo Talvensaari wrote:
>
> I was also thinking that you could maybe ngx.timer.at your SQLite access.
> This way it doesn't block the Nginx worker.
>

No way. ngx.timer.at() creates a collaborative "light thread" (after
the specified delay), not an OS thread. To prevent SQLite from
blocking the nginx event loop, you'll have to use a separate OS thread
(caution, OS threads have their own complexity and overhead).

Regards,
-agentzh

[Stefan Parvu]

> No way. ngx.timer.at() creates a collaborative "light thread" (after
> the specified delay), not an OS thread. To prevent SQLite from
> blocking the nginx event loop, you'll have to use a separate OS thread
> (caution, OS threads have their own complexity and overhead).

After more reading we will go for Redis for keeping all user information
mapped in memory and have Redis to keep the files hot using AOF 1sec.

All further requests will basically mean socket operations to NGINX / OpenResty
and that should solve our concurrency issues. No threads or complex processing
which create an overhead on the system overall.

Thanks all for comments.

--
Stefan Parvu <spa...@systemdatarecorder.org>

[Alexis Okuwa]

I know this question is old but if I want to use file Io. Can I use aio to get it "less blocking" and does that aio work from within lua code?

[Yichun Zhang (agentzh)]

Hello!

On Tue, Jul 12, 2016 at 2:21 AM, Alexis Okuwa wrote:
> I know this question is old but if I want to use file Io. Can I use aio to get it "less blocking" and does that aio work from within lua code?
>

Well, there is plan to expose nginx's AIO API to the Lua land in
ngx_lua. But it would require Direct IO on Linux due to Linux's
limitation. It would be much more useful on other operating systems
like FreeBSD I think.

Patches welcome! Volunteers welcome!

Best regards,
-agentzh