[erlang-questions] run_erl and fsync

[Damienuk Davis]

Hi all,

In one of my projects I used run_erl to launch Erlang VM in daemon mode, rotate logs, etc.

But I discovered unexpected performance problem running production application with run_erl. Application caused high iowait.

After some investigation I found that run_erl does fsync on every log entry, causing disk IO load. Do we need fsync on every log entry.

Appreciate your valuable comments.

Thanks

Damien

[dmitry kolesnikov]

Does fsync block any VM scheduler? Or is it executed within async thread?

Or none of them :-)

I do not think that flush log is required for majority of Web systems. I believe this is a TeleCo legacy... Should it be configurable at least build time?

Best Regards,

Dmitry >-|-|-*>

_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

[Damienuk Davis]

In my project there are fsms that communicates with c via erl driver (port). There are io:fwrite's in the erlang side. This program along works fine. But when you run some other process with high disk access, it causes erlang program to halt even if there is enough memory and processing. If I comment the fsync is run_erl.c file then program works fine even with higher load on hard disk. 

static void write_to_log(int* lfd, int* log_num, char* buf, int len)

{

  int size;

  /* Decide if new logfile needed, and open if so */

  size = lseek(*lfd,0,SEEK_END);

  if(size+len > log_maxsize) {

    close(*lfd);

    *log_num = next_log(*log_num);

    *lfd = open_log(*log_num, O_RDWR|O_CREAT|O_TRUNC|O_SYNC);

  }

  /* Write to log file */

  if (write_all(*lfd, buf, len) < 0) {

    status("Error in writing to log.\n");

  }

#if USE_FSYNC

  fsync(*lfd); //after commenting this line it works fine 

#endif

}

Thanks

Damien

---

From: dmitry kolesnikov <dmkole...@gmail.com>
To: Damienuk Davis <dami...@ymail.com>
Cc: "erlang-q...@erlang.org" <erlang-q...@erlang.org>
Sent: Thursday, July 26, 2012 5:44 PM
Subject: Re: [erlang-questions] run_erl and fsync

[Dmitry Kolesnikov]

I am curious what platform you are build the staff and what is the config?

fsync is behind USE_FSYNC but 

#ifndef O_SYNC

#define O_SYNC 0

#define USE_FSYNC 1

#endif

- Dmitry

[Damienuk Davis]

It's linux (Redhat ES 5.2) platform with following server configuration

DL370 with 2*6core and 12GB RAM 

Is it safe to define O_SYNC (this will cause to bybass fsync call)? What can be the impact to the system?

---

From: Dmitry Kolesnikov <dmkole...@gmail.com>

To: Damienuk Davis <dami...@ymail.com>
Cc: "erlang-q...@erlang.org" <erlang-q...@erlang.org>

Sent: Thursday, July 26, 2012 8:48 PM

[Michael Santos]

On Thu, Jul 26, 2012 at 09:27:56AM -0700, Damienuk Davis wrote:
> It's linux (Redhat ES 5.2) platform with following server configuration
> DL370 with 2*6core and 12GB RAM 
> Is it safe to define O_SYNC (this will cause to bybass fsync call)?

According to the man page, fsync() will result in 2 writes (data +
mtime). So, O_SYNC should be faster.

> What can be the impact to the system?

Some of the log might be lost in a crash if sync writes are
disabled.

Here's an untested patch to run_erl that gives the option of disabling
sync writes. It'd be nice to add an option to use syslog too.

diff --git a/erts/etc/unix/run_erl.c b/erts/etc/unix/run_erl.c
index 6b350e8..e2de798 100644
--- a/erts/etc/unix/run_erl.c
+++ b/erts/etc/unix/run_erl.c
@@ -169,6 +169,7 @@ static int log_generations = DEFAULT_LOG_GENERATIONS;
static int log_maxsize = DEFAULT_LOG_MAXSIZE;
static int log_alive_minutes = DEFAULT_LOG_ALIVE_MINUTES;
static int log_activity_minutes = DEFAULT_LOG_ACTIVITY_MINUTES;
+static int log_sync_on_write = 1;
static int log_alive_in_gmt = 0;
static char log_alive_format[ALIVE_BUFFSIZ+1];
static int run_daemon = 0;
@@ -296,6 +297,9 @@ int main(int argc, char **argv)
if (log_maxsize < LOG_MIN_MAXSIZE)
ERROR1(LOG_ERR,"Minimum RUN_ERL_LOG_MAXSIZE is %d", LOG_MIN_MAXSIZE);
}
+ if ((p = getenv("RUN_ERL_DISABLE_SYNC_ON_WRITE"))) {
+ log_sync_on_write = 0;
+ }

/*
* Create FIFOs and open them
@@ -480,6 +484,7 @@ static void pass_on(pid_t childpid)
int maxfd;
int ready;
int got_some = 0; /* from to_erl */
+ int flags = O_RDWR|O_APPEND|O_CREAT;

/* Open the to_erl pipe for reading.
* We can't open the writing side because nobody is reading and
@@ -495,9 +500,11 @@ static void pass_on(pid_t childpid)
#endif

/* Open the log file */
+ if (log_sync_on_write)
+ flags |= O_SYNC;

lognum = find_next_log_num();
- lfd = open_log(lognum, O_RDWR|O_APPEND|O_CREAT|O_SYNC);
+ lfd = open_log(lognum, flags);

/* Enter the work loop */

@@ -842,7 +849,8 @@ static int open_log(int log_num, int flags)

status("Error in writing to log.\n");

#if USE_FSYNC

- fsync(lfd);
+ if (log_sync_on_write)
+ fsync(lfd);
#endif

return lfd;
@@ -855,14 +863,19 @@ static int open_log(int log_num, int flags)

static void write_to_log(int* lfd, int* log_num, char* buf, int len)
{
int size;

+ int flags = O_RDWR|O_CREAT|O_TRUNC|O_SYNC;

/* Decide if new logfile needed, and open if so */

size = lseek(*lfd,0,SEEK_END);
if(size+len > log_maxsize) {

+ if (!log_sync_on_write) {
+ fsync(*lfd);
+ flags &= ~O_SYNC;
+ }
sf_close(*lfd);
*log_num = next_log(*log_num);
- *lfd = open_log(*log_num, O_RDWR|O_CREAT|O_TRUNC|O_SYNC);
+ *lfd = open_log(*log_num, flags);

}

/* Write to log file */

@@ -872,7 +885,8 @@ static void write_to_log(int* lfd, int* log_num, char* buf, int len)
}

#if USE_FSYNC
- fsync(*lfd);
+ if (log_sync_on_write)
+ fsync(*lfd);
#endif

[Damienuk Davis]

thanks for the information. So disable fsync may not cause huge issue in logs, but will reduce the io wait.

Thanks

Damien

---

From: Michael Santos <michael...@gmail.com>
To: Damienuk Davis <dami...@ymail.com>
Cc: Dmitry Kolesnikov <dmkole...@gmail.com>; "erlang-q...@erlang.org" <erlang-q...@erlang.org>
Sent: Friday, July 27, 2012 6:04 PM

Subject: Re: [erlang-questions] run_erl and fsync