You're trying to set the view matrix on the camera which may get overwritten by the camera manipulator at the beginning of each frame. Instead, check your camera
manipulator (e.g. TrackballManipulator) if it has a member function that
accepts a view matrix.
Some camera manipulators (like the Terrain manipilator) keep a center position, a distance and an Euler angle with respect to the center. Setting these values might work, while setting the view matrix directly may not work.
Also in the beginning the camera manipulators compute a default home position and use it. Make sure you apply your own view matrix at a later point in time.
Christian