implausibly old time stamp 1969-12-31 18:00:00

[Igor Yakushin]

Hello All,

I am trying to run on our GPU cluster using Singularity 2.4:

singularity exec --nv docker://tensorflow/tensorflow:latest-gpu python ./models/tutorials/image/mnist/convolutional.py

It runs fine on my laptop, but not on the cluster:

===========

Docker image path: index.docker.io/tensorflow/tensorflow:latest-gpu
Cache folder set to /home/ivy2/.singularity/docker
[19/19] |===================================| 100.0%  
Creating container runtime...
tar: usr/local/cuda-8.0/.wh..wh..opq: implausibly old time stamp 1969-12-31 18:00:00
tar: usr/share/doc/cuda-cublas-8-0/.wh..wh..opq: implausibly old time stamp 1969-12-31 18:00:00
tar: usr/share/doc/cuda-cudart-8-0/.wh..wh..opq: implausibly old time stamp 1969-12-31 18:00:00
...

pciBusID 0000:09:00.0
Total memory: 11.17GiB
Free memory: 11.10GiB
2017-10-24 19:50:42.953324: I tensorflow/core/common_runtime/gpu/gpu_device.cc:976] DMA: 0  
2017-10-24 19:50:42.953396: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 0:   Y  
2017-10-24 19:50:42.953466: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla K80, pci bus id: 0000:09:00.0)
Killed

===========

My laptop does not complain about "old time stamp".

Any ideas?

Thank you,

Igor

[Igor Yakushin]

If I first build the image from docker (it still complains about the dates) and then use the local image, it runs.

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--

Igor Yakushin, Ph.D.

Computational Scientist

Kavli Institute for Cosmological Physics, ERC #413

Research Computing Center, room #2

The University of Chicago

773-834-5313

https://rcc.uchicago.edu

[vanessa s]

We might need to add a flag to ignore the old timestamp warning (--warning=no-timestamp and see here). I'm assuming from the "killed" that the container doesn't work?  It might also be an issue of a bad download - we should maybe think about checking the hash of the files.

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

[John Hearns]

Six hours before the Epoch.... when the Universe was waiting for that vacuum fluctuation to happen....

I guess you are in a timezone six hours behing GMT?

I guess also we are all aware what happens when that integer overflows...

https://en.wikipedia.org/wiki/Year_2038_problem

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

[Igor Yakushin]

On Tue, Oct 24, 2017 at 4:48 PM, 'John Hearns' via singularity <singu...@lbl.gov> wrote:

Six hours before the Epoch.... when the Universe was waiting for that vacuum fluctuation to happen....

I guess you are in a timezone six hours behing GMT?

Chicago, Central Time.

[Igor Yakushin]

Hi Vanessa,

We might need to add a flag to ignore the old timestamp warning (--warning=no-timestamp and see here). I'm assuming from the "killed" that the container doesn't work?  

Yes, it does not work when I try to run it on the cluster directly from docker hub.

It usually works on my laptop, although not always.

It works on the cluster if one first builds the image from docker and then use the image.

So, I guess, corruption during download might be a reasonable hypothesis.

Thank you,

Igor

[Gregory M. Kurtzer]

It should just reset the timestamp on those files to something feasible or the current time of unpacking. I would not think that this would break any functionality, can you confirm?

Thank you!

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--

Gregory M. Kurtzer

CEO, SingularityWare, LLC.

Senior Architect, RStor

[Igor Yakushin]

On Tue, Oct 24, 2017 at 5:04 PM, Gregory M. Kurtzer <gmku...@gmail.com> wrote:

It should just reset the timestamp on those files to something feasible or the current time of unpacking. I would not think that this would break any functionality, can you confirm?

Thank you!

On Tue, Oct 24, 2017 at 3:00 PM, Igor Yakushin <iv...@uchicago.edu> wrote:

Hi Vanessa,

We might need to add a flag to ignore the old timestamp warning (--warning=no-timestamp and see here). I'm assuming from the "killed" that the container doesn't work?  

Yes, it does not work when I try to run it on the cluster directly from docker hub.

It usually works on my laptop, although not always.

Forgot to mention: on my laptop it never complains about timestamp. On my laptop I am using Ubuntu 16.04 and on the cluster Scientific Linux 7.2.

 

It works on the cluster if one first builds the image from docker and then use the image.

So, I guess, corruption during download might be a reasonable hypothesis.