Memory limit and SparseFieldIO

[jam...@moving-picture.com]

I'm encountering a crash during rendering a large number of large volumes, with call stack:

Field3D::v1_4::Sparse::SparseBlock<Imath::Vec3<float> >::resize(int), field3d/SparseField.h:249
bool Field3D::v1_4::SparseFieldIO::readData<Imath::Vec3<float> >(int, int, std::string const&, std::string const&, Field3D::v1_4::SparseField<Imath::Vec3<float> >::Ptr), field3d/SparseFieldIO.h:410
Field3D::v1_4::SparseFieldIO::read(int, std::string const&, std::string const&, Field3D::v1_4::DataTypeEnum), src/SparseFieldIO.cpp:206
Field3D::v1_4::Field<Imath::Vec3<float> >::Ptr Field3D::v1_4::readField<Imath::Vec3<float> >(std::string const&, int, std::string const&, std::string const&), field3d/Field3DFile.h:1572
Field3D::v1_4::Field<Imath::Vec3<float> >::Ptr Field3D::v1_4::Field3DInputFile::readLayer<Imath::Vec3<float> >(std::string const&, std::string const&, bool) const, field3d/Field3DFile.h:1076
Field3D::v1_4::Field<Imath::Vec3<float> >::Ptr Field3D::v1_4::Field3DInputFile::readVectorLayer<float>(std::string const&, std::string const&) const, field3d/Field3DFile.h:1285
Field3D::v1_4::Field<Imath::Vec3<float> >::Vec Field3D::v1_4::Field3DInputFile::readVectorLayers<float>(std::string const&, std::string const&) const, field3d/Field3DFile.h:1012

This is with the new 'dev-1.4.2' branch, and a memory limit of 1.1Gb. 
Maybe 
The crash appears to happen in this section of code:

template <class Data_T>
bool SparseFieldIO::readData(hid_t location, 
                             int numBlocks, 
                             const std::string &filename, 
                             const std::string &layerPath, 
                             typename SparseField<Data_T>::Ptr result)
{
    // ..snip

 SparseBlock<Data_T> *blocks = result->m_blocks;

  // ... Read the isAllocated array
  {
    vector<char> isAllocated(numBlocks);
    readSimpleData<char>(location, "block_is_allocated_data", isAllocated);
    for (int i = 0; i < numBlocks; ++i) {
      blocks[i].isAllocated = isAllocated[i];
      if (isAllocated[i]) {
        blocks[i].resize(valuesPerBlock);
      }
    }
  }

   //  ..snip
}


I may be misinterpreting the code, but here is seems that the memory for all non-empty blocks is allocated in this loop, regardless of memory limit.
And this happens on loading every volume, which renders the memory limit useless. Or am I missing something?

Jamie

[Magnus Wrenninge]

You're absolutely right, this is a regression that was introduced with the 1.4 version. Could you try making this adjustment?

// Only allocate if not using dynamic loading
if (!dynamicLoading && isAllocated[i]) {
  blocks[i].resize(valuesPerBlock);
}

Magnus

--
You received this message because you are subscribed to the Google Groups "Field3D dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to field3d-dev...@googlegroups.com.
To post to this group, send email to field...@googlegroups.com.
Visit this group at http://groups.google.com/group/field3d-dev.
For more options, visit https://groups.google.com/groups/opt_out.

[jam...@moving-picture.com]

Hi Magnus,

That adjustment seems to fix the problem (at least based on tests so far). Thanks for the quick fix!

Jamie

[Magnus Wrenninge]

I just committed v1.4.2 and v1.4.3 which address the memory limit and the regression bug, respectively. The master branch is at 1.4.3 so you should be able to go back to that.

Magnus

[jam...@moving-picture.com]

Great, thanks for the quick turnaround  :)

Jamie