RStan error message after concluding sampling

[H. X.]

Hi All,
After finishing sampling, I got the error message:

 Elapsed Time: 757694 seconds (Warm-up)
               930398 seconds (Sampling)
               1.68809e+06 seconds (Total)

Error in file(file, "r") : cannot open the connection
Calls: stan -> sampling -> sampling -> .local -> scan -> file
In addition: Warning message:
In file(file, "r") :
  cannot open file '/var/folders/15/2v3s52zj6px1zmw2y17vvtpr0000gp/T//Rtmp1QXevN/file8506f3e5a4b': No such file or directory
Execution halted

I tested my code in smaller iteration runs, and it did not return error messages then, so I am wondering what the source of this error message could be. My code is attached.


Thanks

[Ben Goodrich]

On Wednesday, September 28, 2016 at 10:38:43 AM UTC-4, H. X. wrote:

Hi All,
After finishing sampling, I got the error message:

 Elapsed Time: 757694 seconds (Warm-up)
               930398 seconds (Sampling)
               1.68809e+06 seconds (Total)

Error in file(file, "r") : cannot open the connection
Calls: stan -> sampling -> sampling -> .local -> scan -> file
In addition: Warning message:
In file(file, "r") :
  cannot open file '/var/folders/15/2v3s52zj6px1zmw2y17vvtpr0000gp/T//Rtmp1QXevN/file8506f3e5a4b': No such file or directory
Execution halted

It looks as if that file disappeared before R tried to read the messages in it. Maybe files in that directory are auto-cleaned after a certain amount of time? You might be able to get it work by specifying open_progress = FALSE in the call to stan() or sampling().

Ben

 

[Emmanuel Charpentier]

Hmmm... the reported times are suspicious. Did your code *really* ran for three bloody weeks ?

--
Emmanuel Charpentier

[Wally Xie]

Embarrassingly, yes.
I have no idea why it ran so long. Using an i7 2014 Macbook. On subsequent reruns of the code, it has been much faster.

---

From: Emmanuel Charpentier
Sent: ‎9/‎29/‎2016 2:15
To: Stan users mailing list
Subject: [stan-users] Re: RStan error message after concluding sampling

--
You received this message because you are subscribed to a topic in the Google Groups "Stan users mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/stan-users/q4km8azfxok/unsubscribe.
To unsubscribe from this group and all its topics, send an email to stan-users+...@googlegroups.com.
To post to this group, send email to stan-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Bob Carpenter]

You may have been running for so many iterations that
it wouldn't fit into memory when you were done.

Always start small and build out until you get R-hat near
1 and n_eff > 100 for all parameters (and lp__). The 2000
iteration default is probably too high.

- Bob

> On Sep 29, 2016, at 10:55 AM, Wally Xie <yaois...@gmail.com> wrote:
>
> Embarrassingly, yes.
> I have no idea why it ran so long. Using an i7 2014 Macbook. On subsequent reruns of the code, it has been much faster.

> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.

[Gary Venter]

I have been getting the same error on long runs on the same computer. The last time I was running 3 chains and I only got it on one node. Two of the three finished after 250,000 seconds but the third took 868,500 seconds. Before I have gotten it on more than one node. It looks like it indeed has something to do with files disappearing after a long run time. If open_progress = FALSE will solve this I will try it. But I don't know what that does.

On Wednesday, September 28, 2016 at 10:38:43 AM UTC-4, H. X. wrote:

[Gary Venter]

Actually I got the error after sampling but at a slightly different call:

Error in checkForRemoteErrors(val) : 

  one node produced an error: cannot open the connection

Warning message:

In file(file, "r") :

  cannot open file '/var/folders/st/6g35z0m12j18szfc1_630p340000gn/T//RtmpHyATLP/file4c2a3c1adf35': No such file or directory

On Wednesday, September 28, 2016 at 10:38:43 AM UTC-4, H. X. wrote:

[Gary Venter]

Do you use CleanMyMac? I'm suspecting that to be deleting these temp files after a week or so, which would kill these long runs once they finish sampling.

On Wednesday, September 28, 2016 at 10:38:43 AM UTC-4, H. X. wrote:

[Wally Xie]

I do not use CleanMyMac or any cleaning software. Shorter runs still work fine. Does OSX itself clean temp files on a regular basis?

---

From: Gary Venter
Sent: ‎10/‎10/‎2016 9:31

To: Stan users mailing list
Subject: [stan-users] Re: RStan error message after concluding sampling

[Gary Venter]

It seems likely. I was hoping it was CleanMyMac then I would have some control. But I've gotten this "no file" error about 4 times now, and every time the number of missing files was the number of chains that took more than a week to run. Then I lose the fast chains too. It would be helpful if Stan wrote out each chain as it finished so I would at least get something.

[Daniel Lee]

Hi Wally and Gary,

Perhaps the most prudent thing to do is to run through CmdStan if there are things happening out of your control. It streams output to a file and there's nothing else written out to any temp space. You should get the same results with the same seeds on the same architecture, compiler, etc. (and you probably won't get the same results if you use different architectures or compiler settings).

Daniel

--
You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+unsubscribe@googlegroups.com.

[Ben Goodrich]

On Monday, October 10, 2016 at 5:28:36 PM UTC-4, Gary Venter wrote:

It seems likely. I was hoping it was CleanMyMac then I would have some control. But I've gotten this "no file" error about 4 times now, and every time the number of missing files was the number of chains that took more than a week to run. Then I lose the fast chains too. It would be helpful if Stan wrote out each chain as it finished so I would at least get something.

For the next version of RStan, we'll check for the (continued) existence of the file before trying to read it. Although if it disappears during a weak-long run, you won't be able to see run-time problems with that chain. For now, I guess you have to specify the sample_file argument to write each iteration to the disk.

Ben

[Gary Venter]

Thanks Daniel and Ben. Very useful suggestions.

[Bob Carpenter]

Is there a way to stream the draws to disk in RStan
without writing the draws to memory?

If we can free up some cycles for new features, we might
be able to add something that'll checkpoint and allow restarts
even in the face of a crash.

- Bob

> --
> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.

[Ben Goodrich]

On Monday, October 10, 2016 at 11:23:22 PM UTC-4, Bob Carpenter wrote:

Is there a way to stream the draws to disk in RStan

 

There has been since before we moved to GitHub: Specify the sample_file argument to sampling() or stan()

An optional character string providing the name of a file. If specified the draws for all parameters and other saved quantities will be written to the file. If not provided, files are not created. When the folder specified is not writable, tempdir() is used. When there are multiple chains, an underscore and chain number are appended to the file name.

Ben

[Ben Goodrich]

On Monday, October 10, 2016 at 6:05:13 PM UTC-4, Ben Goodrich wrote:

For the next version of RStan, we'll check for the (continued) existence of the file before trying to read it. Although if it disappears during a weak-long run, you won't be able to see run-time problems with that chain. For now, I guess you have to specify the sample_file argument to write each iteration to the disk.

You can now install rstan from GitHub with this fix.

Ben

[Wally Xie]

Thank you!

---

From: Ben Goodrich
Sent: ‎10/‎10/‎2016 20:59

To: Stan users mailing list

Subject: Re: [stan-users] Re: RStan error message after concluding sampling

[Bob Carpenter]

I know it's possible to stream the draws, but what I was asking
was whether it's possible to turn off saving them in memory.

- Bob

P.S. My own preference would be to fail if the user supplies a
file path that can't be opened for writing, but maybe that's not
how R normally works.

[Jiqiang Guo]

Sort of. You can specify pars=character(0) when calling Stan.

Jiqiang 

[Bob Carpenter]

Neat workaround---thanks.

- Bob