First up, I'd advise against mocking out a Dictionary. I've found it much easier to use a real one and check the behaviour of the class under test instead (say, by asserting on the real dictionary's content). I've found that avoiding mocking types I don't own has really help improved my tests (YMMV :)).
To answer your question, the Arg.Any/Is methods only work within the context of specifying a call (e.g. mockCol.Received().TryGetValue(Arg.Any...)). This makes things a bit clumsy when trying to use out params, but the following should work ok (depending on what returnObject you're really passing in):
mockCol.TryGetValue(expectObject.GetType(), out returnObject);
mockCol.Received().TryGetValue(expectObject.GetType(), out returnObject);
}
Another way to simulate this is to set the output using When..Called, which implicitly tests the call was received:
[Test]
public void IEnumerableGetEnumeratorCallsValueCollectionGetEnumerator2()
{