How much temporary disk usage is allowed on an m1 node?

[Geoffrey Irving]

Hello,

I am preparing for a fairly large out-of-core computation, the most
expensive tasks of which will consume 4 ~100 GB files and produce
another ~100 GB file. It's possible to structure the computation so
that the inputs and output are streamed to/from S3 in pieces, but the
task also needs about ~100 GB of temporary space which I'd prefer not
to have to send to S3 only to quickly read it back onto the same node.
Thus, my question is:

How much temporary disk usage is allowed on an m1 node? Is it safe to
generate one (or more?) 100 GB files while the computation runs, or
are the disks potentially "tiny"?

Thanks,
Geoffrey

[Ken Elkabany]

Hi Geoffrey,

An m1 has a soft limit of 140 GB of temporary space, so you'll have to stream your inputs and outputs. Just wondering, what's your planned application?

Best,

Ken

[Geoffrey Irving]

Thanks: a soft limit of 140 GB should be workable.

I'm working on solving the board game pentago, which requires
computing win/loss/tie values for 6,359,326,373,893,888 (about
6.36e15) positions.

https://github.com/girving/pentago

It's not yet clear whether I can compress and optimize it down to the
point where I feel like paying for it out of pocket, though (with my
existing code it would probably take several thousand dollars). zlib
and I/O are slow. :)

Geoffrey