memory mapping and large float32 arrays

[John-Philip Butler]

I am currently working with collections of gigabyte-sized multi-dimensional float32 arrays generated by another program stored as raw data files (with the correct endianness), located on a samba-mounted file server for space reasons. Subsequent processing can then mostly be done by using subarrays.

My current workflow boils down to:

JCHAR map_jmf_ 'MyCube' ; DataFileF32

Cube=: Shape $ _1 (3!:5) MyCube

I was wondering:

* is my workflow is optimal? e.g. would it be worthwhile trying to save the files as float64's to enable memory mapping to work directly? Actually I am about to try changing the units and using integers instead, which is reasonable for my current application.

* are there plans to support direct use of float32's in the future?

[Henry Rich]

1. It might well be worthwhile to memory-map the float8 array.  That would shift the expense of conversion from your program to the creator of the array.

2. Taking a subarray creates a copy of the data unless the data is contiguous; then it gives a virtual array.  The virtual is realized only when boxed or assigned.

3. u;._3 on the top 2 dimensions trips lightly through the array with minimal copying.

4. float4 support is planned but not scheduled.  Don't expect it in J9.7.

Henry Rich

To unsubscribe from this group and stop receiving emails from it, send an email to forum+un...@jsoftware.com.

[John-Philip Butler]

Thanks Henry

I think u ;._3 will be perfect!

[Henry Rich]

That would be great.  I've long hoped someone would need it in a big application.  It makes one copy of the array, in small pieces, and shifts the data origin as it goes along to pan through the array vertically and horizontally.

Henry Rich

[John-Philip Butler]

This time round I could just divide and conquer, but I soon have a project comming up that will should give me a chance to stress test this!