Appending data to a log file

[Marco Palladino]

Hi,

I am building a custom logger that appends data to a specific file on disk, and I was wondering if the following approach would be recommended:

log_by_lua '

  local function log(premature, message)

    local f = io.open("/some/path.log", "a+")

    f:write(message.."\n")

    f:close()

  end

  local ok, err = ngx.timer.at(0, log, "Test Message")

  if not ok then

    ngx.log(ngx.ERR, "Failed to create timer: ", err)

  end

';

Specifically I am wondering if under heavy load opening and closing the file every time would be recommended, or a different approach should be taken because it would be too expensive.

Thanks

[Vladislav Manchev]

I guess you're better off with lua-resty-logger-socket [0] for non-blocking logging.

Opening and closing the file would involve syscalls that might be the most expensive part of your code above, so maybe you should reconsider your approach.

[0]: https://github.com/cloudflare/lua-resty-logger-socket

Best,

Vladislav

--
You received this message because you are subscribed to the Google Groups "openresty-en" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openresty-en...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Marco Palladino]

Unfortunately that module doesn't support logging to a third party file on disk. I was wondering if I opening the file once, and storing the handler in a global variable to be re-used every time, would work?

[Yichun Zhang (agentzh)]

Hello!

On Fri, May 29, 2015 at 3:32 PM, Marco Palladino wrote:
> Unfortunately that module doesn't support logging to a third party file on
> disk. I was wondering if I opening the file once, and storing the handler in
> a global variable to be re-used every time, would work?
>

Yes, you could open the file in append mode and use LuaJITFFI to call
the write() or writev() syscalls directly without going through libc's
buffered I/O layer. You could cache and share the file descriptor on
the nginx worker level via the following technique:

https://github.com/openresty/lua-nginx-module#data-sharing-within-an-nginx-worker

Just my 2 cents :)

Best regards,
-agentzh

[Vladislav Manchev]

It won't be possible to share a file descriptor between workers, so it's a dead end I think.

Also, not sure what's stopping you from logging to a third party file on disk with lua-resty-logger-socket. You can just log to a localhost syslog-ng instance (although I never tried that).

Best,

Vladislav

[Yichun Zhang (agentzh)]

Hello!

On Fri, May 29, 2015 at 3:55 PM, Vladislav Manchev wrote:
> It won't be possible to share a file descriptor between workers, so it's a
> dead end I think.
>

We don't need to share the log file descriptor across workers in the
first place. Even nginx's builtin logger does not do this. Thanks to
modern operating systems' atomic file appending feature ;)

> Also, not sure what's stopping you from logging to a third party file on
> disk with lua-resty-logger-socket. You can just log to a localhost syslog-ng
> instance (although I never tried that).
>

Sure, AFAIK, syslog-ng server can further log into the file system on
our behalf.

Regards,
-agentzh

[Marco Palladino]

I decided to use "fopen", "fprintf" and "fclose":

log_by_lua '

  local ffi = require "ffi"

  ffi.cdef[[

  typedef struct {

    char *fpos;

    void *base;

    unsigned short handle;

    short flags;

    short unget;

    unsigned long alloc;

    unsigned short buffincrement;

  } FILE;

  FILE *fopen(const char *filename, const char *mode);

  int fprintf(FILE *stream, const char *format, ...);

  int fclose(FILE *stream);

  ]]

  local function log(premature, message)

    local f = ffi.C.fopen("/some/path.log", "a+")

    ffi.C.fprintf(f, message)

    ffi.C.fclose(f)

  end

  local ok, err = ngx.timer.at(0, log, "Test Message")

  if not ok then

    ngx.log(ngx.ERR, "Failed to create timer: ", err)

  end

';

The example above still opens and closes the file on every request, I will go ahead and implement the @agentzh suggestion for sharing the file handler.

[Yichun Zhang (agentzh)]

Hello!

On Tue, Jun 2, 2015 at 9:15 AM, Marco Palladino wrote:
> I decided to use "fopen", "fprintf" and "fclose":
>

As mentioned previously, these libc functions are not safe at all in
this context because of its own userland buffering. Use the thin libc
wrappers for the syscalls like write() and writev() directly.

Regards,
-agentzh