Thinking about this again, having TryCancel() on the call context is truly surprising to me. It's spooky action at a distance. The call context is used once when the stream is established, and then typically forgotten, and the data is received by calling a method on the stream. So, why isn't TryCancel() on the stream?
What I find particularly jarring is that WritesDone() is on the stream, but TryCancel() is not. It seems that both methods should be in the same place (preferably the stream, rather than the context).