Hi,
> I can figure out a number of ways to copy to *Dense and than do the
operations but was wondering if you have any pointers to do this the
most efficient manner.
Using mat.Dense will work but won't be efficient because you'll doing operations on zeroes from the unused triangle and from elements outside of the band.
> first I create C as a Dense, and use a.T() and b.T(), the Dense gets the result I expect.
What you're doing in the test code is not correct and works only by chance thanks to a TriBandDense with n=5,k=4 having the same memory layout as a Dense with m=n=5.
> result := mat.NewTriBandDense(n, k, mat.Upper, /*NEED something else*/ c.RawMatrix().Data)
As you've also apparently realized, you cannot do that, the storage formats are not compatible.
> Any advice would be greatly appreciated.
Since we're currently not providing a MulElem function that takes TriBandDense and returns TriBandDense (it's a rather unusual operation to do with triangular band matrices), you can implement it yourself using At and SetTriBand. You must take care to only set elements within the triangle and band, otherwise you'll get panics. If you wait for the next release, you could simplify that by using the DoNonZero method
https://github.com/gonum/gonum/blob/a4dda6a99c494413c752a2fd35338c59448f2207/mat/triband.go#L366
Alternatively, you could implement the operations by working directly with the raw matrix data obtained with RawTriBand. It may be more efficient because you avoid our internal bound checks but also more dangerous because you work with raw data and you avoid our internal bound checks. And you must understand the underlying matrix layout.
Vladimir