os.WriteFile not writing files in long running process

[Paweł Szczur]

Hi,

I have a long running hobby program with a code: 

var (
   url      = "https://example.com"
   lastBody []byte
)

func get(client *http.Client, dir) (changed bool, data []byte, err error) {

   resp, err := client.Get(url)
   if err != nil {
      return false, nil, err
   }
   if resp.StatusCode != http.StatusOK {
      log.Printf("status code: %d", resp.StatusCode)
      return false, nil,  nil
   }
   body, err := ioutil.ReadAll(resp.Body)
   if err != nil {
      return false, nil, err
   }
   if bytes.Compare(body, lastBody) == 0 {
      logrus.Info("data is equal")
      return false, data, nil
   }

   lastBody = body
   log.Printf("got %dB data", len(body))

   dir = path.Join(dir, t.Format("2006/01/02"))
   if err = os.MkdirAll(dir, defaultDirPerm); err != nil {
      log.Print("failed to create a dir")
   }
   p := path.Join(dir, fmt.Sprintf("%d.json", t.Unix()))

   return true, body, ioutil.WriteFile(p, body, 0640)
}

After around 14 days the process stops writing files. However, it executes os.WriteFile and does not return any error.

The function is called repeatedly every N second (~40-60sec.). It produces around 1700 files a day.

Any ideas?

I've found https://github.com/golang/go/issues/14491 and some other threads but not a conclusive result.

Cheers, Paweł Szczur

[Tamás Gulácsi]

So, the directories are created?
Anyway, a resp.Body.Close() is missing.

[Manlio Perillo]

Here, if os.MkdirAll fails, the function should return with an error.

   p := path.Join(dir, fmt.Sprintf("%d.json", t.Unix()))

   return true, body, ioutil.WriteFile(p, body, 0640)
}

Also, you should probably pass lastBody as a function argument, and not as a global variable.

The variable t is also not defined in the code you posted.

 

 

After around 14 days the process stops writing files. However, it executes os.WriteFile and does not return any error.

The function is called repeatedly every N second (~40-60sec.). It produces around 1700 files a day.

The problem may be triggered when the code fails to create a new directory; however WriteFile should return an error in this case.

> [...]

Manlio 

[Paweł Szczur]

Thanks. Both things you mentioned are already fixed. The MakeDir may of course fail and now I handle it, but in described situation it was not an issue.
The files were written for most of the day successfully and suddenly they stopped to appear.
The disk was and is not full. There was no error returned by any function (I've examined logs).

[Manlio Perillo]

Did you checked the system/kernel logs?

If you still are unable to find the cause of the problem, and assuming you are on an UNIX system, you can try to change the code to make sure that both the new directory entry, and the new file data is flushed to disk.

For the directory entry you have to open the directory with os.OpenFile, passing the os.O_RDONLY|unix.O_DIRECTORY flags, and call Sync after the new file is created.

See http://danluu.com/file-consistency/

Manlio

[dja...@gmail.com]

Hi,

did you write files in /tmp on linux ?

(and there is daemon that clean old files in /tmp ?)

Regards,

Djadala

[Paweł Szczur]

No.

[Paweł Szczur]

If you're asking about dmesg, I'm not seeing anything related.

Yes, I'm on linux. The dir was created successfully, I guess it was not a problem. Also, I believe the file is flushed on Close.

I will look into the link.

Paweł

[Manlio Perillo]

No, the file is not flushed on close.

https://linux.die.net/man/2/close

"A successful close does not guarantee that the data has been successfully saved to disk, as the kernel defers writes. It is not common for a file system to flush the buffers when the stream is closed. If you need to be sure that the data is physically stored use fsync(2). (It will depend on the disk hardware at this point.)"

You can just Sync the file, to make sure this not the problem.

Manlio

[Matt Harden]

From the program's perspective the file is indeed flushed on close. The kernel deferring writes to disk can only have any effect if the system crashes, and the files will be flushed to disk within a short time anyway. You absolutely do NOT need to call fsync every file on close unless you require the data to be written to disk synchronously. In the vast majority of cases you don't need that, and calling fsync takes a long time and makes the system less efficient.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Manlio Perillo]

If files are created but are empty, there is a very simple explanation: the body of the GET response is empty.

Check if resp.ContentLength is equal to 0.

Also, if resp.StatusCode is not StatusOK, I suggest to return a not nil error.

Manlio

[Reinhard Luediger]

Maybe you are running out of sockets due to the missing close on the response body after read