Normally Caffe layers contain parameter blobs and output data blobs - never otherwise. But I
think this might be possible using the
Parameter layer. For each of these InnerProduct layers define a corresponding Parameter layer that would have an internal blob of the exact same dimension as the InnerProduct. Then make these layers share parameters with each other (that is, InnerProduct shares with its Parameter). In theory, this would let you get access to the InnerProduct's parameter blob as the output of the corresponding Parameter layer. Then you could Eltwise them or do whatever you want to compute some loss. I haven't tested it though, it just came to my mind. If you decide to try it out, be sure to let me know if it worked ;)