Performance question for writes on SSD/external hard drive

[Bradley Neuberg]

I'm writing a fairly large dataset into a LevelDB database (a bit larger than 1 TB) that I'm then feeding into a library named Caffe to do deep learning.

I have an SSD drive that is fast but has only about 200 GBs, and an external large drive that is 3 TBs over a USB 3 connection but slower. I am currently completely writing out the database to the large external drive currently; it takes about 5 hours, both due to write time and protobuf serialization. I'd like to speed up the write portion (I'm investigating other things for the protobuf serialization).

While writing the LevelDB files I was wondering if its possible to write out segments of the database to the SSD for much faster write performance and then move the pieces over to the larger external hard drive during the whole process? Any advice on doing this? I'm not sure how LevelDB will respond if I move a file out from under it after its finished writing.

I am currently using the Python LevelDB libraries.

Here's a gist with the method doing the writes for reference:

https://gist.github.com/BradNeuberg/c47a07cce0230cfc2f5d

Best,

  Brad Neuberg