Multiple image sizes as input for memory data layer

[venk hioe]

In https://groups.google.com/forum/#!topic/caffe-users/g1L9VhGoOFM I already asked how to handle simultaneous input of multiple source images. Thanks to @Jan C Peters at that point! However I now have a further question concerning the data input using memory data:

Suppose my net produces an output of one image patch. As input it takes multiple sets of image patches (let's say M sets), each set containing D patches of same size. The amount D of patches per set stays constant among the sets. However the patch sizes (N_i x N_i with i = [1:M] ) differ among the sets.
Could anyone give me an advice how to handle such an input by the memory data layer? I had already a look at the memory_data_layer.cpp and I am not sure if it can handle an input of differently sized images per batch. So I am now considering to adapt the memory for my specific purpose. Is this a good idea or can the memory data layer handle "intricately-shaped" input?

Sorry for this question (at the risk to come out as a caffe-noob that I am), but  it possible to pass memory data to multiple data layers at all? Is there any sample (best in C++)?

Thanks