Why is IMongoQueryProvider internal?

[Potiguar Faga]

Is there any special reason why IMongoQueryProvider has been marked internal?

We're building a framework that has linq async support built-in and different data adapters. For our mongo data adapter it would be extremely helpful if we could have access to this interface, casting the provider of a Linq query to it would give us access to the ExecuteAsync<T> method simplifying the code quite a lot.

Any chance that it could be marked public? 

[Mai Funaccount]

I have to agree. Here I am with the latest 2.2.2 drivers, and I'm in mocking hell trying to figure this out under test.

System.ArgumentNullException: Value cannot be null.

Parameter name: source

   at MongoDB.Driver.Core.Misc.Ensure.IsNotNull[T](T value, String paramName)

   at MongoDB.Driver.IAsyncCursorExtensions.ToList[TDocument](IAsyncCursor`1 source, CancellationToken cancellationToken)

   at MongoDB.Driver.IAsyncCursorSourceExtensions.ToList[TDocument](IAsyncCursorSource`1 source, CancellationToken cancellationToken)

   at Revionics.Markdown.MongoDb.MongoDataAccess.Where[T](IMongoDatabase mongoDatabase, Expression`1 predicate) in C:\dev\tfs\Main\Portal\Markdown\Revionics.Markdown.MongoDb\MongoDataAccess.cs:line 79

[Craig Wilson]

We don't expose things that don't seem useful to a client so that we can change them in the future if necessary. It's impossible to predict how users will use a library and the larger the API surface, the less flexibility we have for adjusting to valuable scenarios users need. I personally have encountered something in either the framework or other libraries that I also wish was public. Feel free to file a Jira ticket at jira.mongodb.org under the CSHARP project and make your case.  

Potiguar's scenario makes sense to me. However, Mai, I'm not sure I'm in any sort of agreement about mocking the driver itself, particularly related to linq. By mocking the linq provider, you've effectively isolated yourself from an already tenuous query abstraction. Unless you actually run using the driver, you'll never know until production that a certain expression isn't supported by the driver.  I'd highly suggest not ever mocking a linq provider. In fact, you really shouldn't mock things you don't own, but that is ivory tower-ish.

Craig

[Robert Stam]

What does the line of code in MongoDatabaseAccess.Where look like? It seems like you might be calling

    xyz.ToList()

where xyz is some variable/value of type IAsyncCursorSource but whose value is null (hence the ArgumentNullException).

--
You received this message because you are subscribed to the Google Groups "mongodb-csharp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-cshar...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Robert Stam]

Sorry, looking more closely it's not the cursorSource that is null, but rather that the cursorSource's ToCursor method returned null.

So you have some line that looks like:

    cursorSource.ToList()

but then in the implementation of ToList (inside the driver):

    cursorSource.ToCursor() => returns null

Which ultimately results in the ArgumentNullException.

So it would still be helpful to see what your line of code looks like to figure out why ToCursor returned null.

I'm assuming your cursorSource is probably a LINQ query, so we might need to reproduce this. Could be a bug in the LINQ implementation.

[Mai Funaccount]

Hi Guys,

Thanks for looking. I am calling the ToList extension method... at least that's what code shows. I've also a mock of ToCursor result, but it comes back null. I'm sure it's partly me not know what this instance should look like in the system.

As another note, I've pulled the source code and run the tests, but many of them fail; making it hard to compare.

Fincally, I tried using MongoCollection's Find and Aggregate methods to get to calling ToList().

[TestClass]

    public class MongoDataAccessTests

    {

        private MongoCollectionSettings collectionSettings;

        private MongoClientSettings clientSettings;

        private MongoDatabaseSettings databaseSettings;

        private string connection;

        private Mock<IAsyncCursor<PlanDay>> asyncCursor;

        private Mock<IMongoQueryable<PlanDay>> mongoQueryable;

        [TestInitialize]

        public void Setup()

        {

            connection = Guid.NewGuid().ToString();

            collectionSettings = new MongoCollectionSettings

            {

                GuidRepresentation = GuidRepresentation.Standard,

                ReadEncoding = new UTF8Encoding(),

                ReadConcern = ReadConcern.Default,

                ReadPreference = new ReadPreference(ReadPreferenceMode.Nearest, new List<TagSet>()),

                WriteConcern = new WriteConcern(),

                WriteEncoding = new UTF8Encoding()

            };

            clientSettings = new MongoClientSettings

            {

                ReadPreference = collectionSettings.ReadPreference,

                GuidRepresentation = collectionSettings.GuidRepresentation,

                ReadConcern = collectionSettings.ReadConcern,

                ReadEncoding = collectionSettings.ReadEncoding,

                WriteConcern = collectionSettings.WriteConcern,

                WriteEncoding = collectionSettings.WriteEncoding

            };

            databaseSettings = new MongoDatabaseSettings

            {

                ReadPreference = collectionSettings.ReadPreference,

                GuidRepresentation = collectionSettings.GuidRepresentation,

                ReadConcern = collectionSettings.ReadConcern,

                ReadEncoding = collectionSettings.ReadEncoding,

                WriteConcern = collectionSettings.WriteConcern,

                WriteEncoding = collectionSettings.WriteEncoding

            };

            // I wish there was a better understanding of this area in code.

            asyncCursor = new Mock<IAsyncCursor<PlanDay>>();

            asyncCursor.Setup(x => x.MoveNext(It.IsAny<CancellationToken>())).Returns(false); // i've tried true as well

            asyncCursor.Setup(x => x.MoveNextAsync(It.IsAny<CancellationToken>())).Returns(Task.FromResult(false)); // i've tried true as well

            asyncCursor.SetupGet(x => x.Current).Returns(new List<PlanDay>());

            asyncCursor.SetupAllProperties();

            mongoQueryable = new Mock<IMongoQueryable<PlanDay>>();

            mongoQueryable.Setup(x => x.ToCursor(It.IsAny<CancellationToken>())).Returns(asyncCursor.Object);

            mongoQueryable.Setup(x => x.ToCursorAsync(It.IsAny<CancellationToken>())).Returns(Task.FromResult(asyncCursor.Object));

            mongoQueryable.Setup(x => x.GetExecutionModel()).Returns(It.IsAny<QueryableExecutionModel>());

            var queryProvider = new Mock<IQueryProvider>(); 

            queryProvider.Setup(x => x.Execute(It.IsAny<Expression>()));

            mongoQueryable.SetupGet(x => x.Provider).Returns(queryProvider.Object);

        }

        

        [TestMethod]

        public void WhereShouldReturnCollectionWhenDatabaseSupplied()

        {

            var access = new MongoDataAccess(

                connection,

                Guid.NewGuid().ToString(),

                new Mock<IBsonSerializer>().Object);

            

            var mongoDatabase = CreateMockDatabase(clientSettings, databaseSettings, connection);

            var mongoCollection = CreateMockCollection<PlanDay>(mongoDatabase.Object, collectionSettings).Object;

            

            mongoDatabase.Setup(x => x.GetCollection<PlanDay>(It.IsAny<string>(), It.IsAny<MongoCollectionSettings>()))

                         .Returns(mongoCollection);

            var result = access.Where<PlanDay>(mongoDatabase.Object, x => x.ForecastDate.Year >= 2014);

            Assert.AreEqual(result.Count, 1);

        }

        private static Mock<IMongoDatabase> CreateMockDatabase(MongoClientSettings clientSettings, MongoDatabaseSettings databaseSettings, string connection)

        {

            var mongoDatabase = new Mock<IMongoDatabase>();

            var client = new Mock<IMongoClient>();

            client.SetupGet(x => x.Settings).Returns(clientSettings);

            client.SetupAllProperties();

            mongoDatabase.SetupGet(x => x.Client).Returns(client.Object);

            mongoDatabase.Setup(x => x.Settings).Returns(databaseSettings);

            mongoDatabase.Setup(x => x.DatabaseNamespace).Returns(new DatabaseNamespace(connection));

            return mongoDatabase;

        }

        private static Mock<IMongoCollection<T>> CreateMockCollection<T>(IMongoDatabase database, MongoCollectionSettings mongoCollectionSettings) where T : new()

        {

            var serializer = new Mock<IBsonSerializer<T>>();

            serializer.Setup(x => x.Deserialize(It.IsAny<BsonDeserializationContext>(), It.IsAny<BsonDeserializationArgs>())).Returns(new T());

            serializer.Setup(x => x.Serialize(It.IsAny<BsonSerializationContext>(), It.IsAny<BsonSerializationArgs>(), It.IsAny<object>()));

            serializer.Setup(x => x.ValueType).Returns(typeof(T));

            

            var collectionMock = new Mock<IMongoCollection<T>>();

            collectionMock.Setup(x => x.Database).Returns(database);

            collectionMock.Setup(x => x.DocumentSerializer).Returns(serializer.Object);

            collectionMock.Setup(x => x.CollectionNamespace).Returns(new CollectionNamespace(database.DatabaseNamespace, typeof(T).Name));

            collectionMock.Setup(x => x.Settings).Returns(mongoCollectionSettings);

            return collectionMock;

        }

    }

public class MongoDataAccess : IMongoDataAccess

    {

        private readonly string _connection;

        private readonly string _instanceName;

        private readonly IBsonSerializer _serializer;

        /// <exception cref="ArgumentException"><paramref name="connection"/> is <see cref="string.IsNullOrEmpty"/>.</exception>

        /// <exception cref="ArgumentException"><paramref name="instanceName"/> is <see cref="string.IsNullOrEmpty"/>.</exception>

        /// <exception cref="ArgumentNullException"><paramref name="serializer"/> is <see langword="null" />.</exception>

        public MongoDataAccess(string connection, string instanceName, IBsonSerializer serializer)

        {

            if (string.IsNullOrEmpty(connection))

            {

                throw new ArgumentException("connection is null or empty.", "connection");

            }

            if (string.IsNullOrEmpty(instanceName))

            {

                throw new ArgumentException("instanceName is null or empty.", "instanceName");

            }

            if (serializer == null)

            {

                throw new ArgumentNullException("serializer");

            }

            _connection = connection;

            _instanceName = instanceName;

            _serializer = serializer;

        }

        public IMongoDatabase GetMongoDatabase()

        {

            var mc = new MongoClient();

            return mc.GetDatabase(_connection);

        }

        public IBsonSerializer Serializer

        {

            get

            {

                return _serializer;

            }

        }

        public IList<T> Where<T>(Expression<Func<T, bool>> predicate = null)

        {

            return Where(GetMongoDatabase(), predicate);

        }

        public IList<T> Where<T>(IMongoDatabase mongoDatabase, Expression<Func<T, bool>> predicate = null)

        {

            var mongoCollection = mongoDatabase.GetCollection<T>(typeof(T).Name);

            if (mongoCollection == null)

            {

                throw new Exception("mongoCollection is null");

            }

            if (predicate == null)

            {

// throws error in test

                var aggregateFluent = mongoCollection.Aggregate();

                return aggregateFluent.ToList();

            }

            var mongoQueryable = mongoCollection.Find(predicate);

    // throws same error in another test

            return mongoQueryable.ToList();

        }

    }

Looking at mongodb codebase:

IMongoCollectionExtension.cs

public static IFindFluent<TDocument, TDocument> Find<TDocument>(this IMongoCollection<TDocument> collection, Expression<Func<TDocument, bool>> filter, FindOptions options = null)

    {

      Ensure.IsNotNull<IMongoCollection<TDocument>>(collection, "collection");

      Ensure.IsNotNull<Expression<Func<TDocument, bool>>>(filter, "filter");

      return IMongoCollectionExtensions.Find<TDocument>(collection, (FilterDefinition<TDocument>) new ExpressionFilterDefinition<TDocument>(filter), options);

    }

...

public static IFindFluent<TDocument, TDocument> Find<TDocument>(this IMongoCollection<TDocument> collection, FilterDefinition<TDocument> filter, FindOptions options = null)

    {

      FindOptions<TDocument, TDocument> options1;

      if (options == null)

      {

        options1 = (FindOptions<TDocument, TDocument>) new FindOptions<TDocument>();

      }

      else

      {

        FindOptions<TDocument> findOptions = new FindOptions<TDocument>();

        bool? allowPartialResults = options.AllowPartialResults;

        findOptions.AllowPartialResults = allowPartialResults;

        int? batchSize = options.BatchSize;

        findOptions.BatchSize = batchSize;

        string comment = options.Comment;

        findOptions.Comment = comment;

        int num = (int) options.CursorType;

        findOptions.CursorType = (CursorType) num;

        TimeSpan? maxAwaitTime = options.MaxAwaitTime;

        findOptions.MaxAwaitTime = maxAwaitTime;

        TimeSpan? maxTime = options.MaxTime;

        findOptions.MaxTime = maxTime;

        BsonDocument modifiers = options.Modifiers;

        findOptions.Modifiers = modifiers;

        bool? noCursorTimeout = options.NoCursorTimeout;

        findOptions.NoCursorTimeout = noCursorTimeout;

        bool? oplogReplay = options.OplogReplay;

        findOptions.OplogReplay = oplogReplay;

        options1 = (FindOptions<TDocument, TDocument>) findOptions;

      }

      return (IFindFluent<TDocument, TDocument>) new FindFluent<TDocument, TDocument>(collection, filter, options1);

    }

FindFluent`2.cs

public FindFluent(IMongoCollection<TDocument> collection, FilterDefinition<TDocument> filter, FindOptions<TDocument, TProjection> options)

    {

      this._collection = Ensure.IsNotNull<IMongoCollection<TDocument>>(collection, "collection");

      this._filter = Ensure.IsNotNull<FilterDefinition<TDocument>>(filter, "filter");

      this._options = Ensure.IsNotNull<FindOptions<TDocument, TProjection>>(options, "options");

    }

IAsyncCursorSourceExtensions.cs

public static List<TDocument> ToList<TDocument>(this IAsyncCursorSource<TDocument> source, CancellationToken cancellationToken = null)

    {

      using (IAsyncCursor<TDocument> source1 = source.ToCursor(cancellationToken))

        return IAsyncCursorExtensions.ToList<TDocument>(source1, cancellationToken);

[Craig Wilson]

Hi Mai,

The problems you are going through above are exactly why it is fundamentally a bad idea to mock things you don't own. You are getting a null exception because, as your own comment states, "[You] wish there was a better understanding of this area".  I'd highly suggest not mocking these interfaces and, instead, writing integration tests.  If you want to mock your own data access layer, then that's great. It's code you own in which you know how it works. In this case, however, you are leaking MongoDB abstractions through (IBsonSerializer, IMongoDatabase). I don't have a problem with this design as it lets you take advantage of low-level, MongoDB only things. However, once you do this, you can't really unit test it anymore. You need integration tests.

I'm not trying to be harsh, so forgive me if it comes across that way. Please feel free to follow-up with questions and, like I said above, feel free to file a feature request in Jira requesting access to certain interfaces/classes we've kept internal.

Craig

[Mai Funaccount]

Hi Craig,

I agree that I should, and I will do integration tests. I also feel this should be a unit tested too. Plus when pulled the codeI found many of these tests broken, which makes me want to verify the flow. I do this same stuff with SQL calls. It's the structure of code and internals within MongoDB that are making it difficult. Anyway this was all I intended:

1. I put in an expression (it doesn't matter at this point, I'm not validating the expression right now)
   
2. I get a mocked result. 
   

I can do this service / api end point tests and SQL tests with a similar strategy above. The MongoDB library has so much stuff under the hood going on, that I, and I'm sure others, struggle setting up something simple. With the MongoDB Drivers I found that it was always something new that came up which was a problem for it being null (not mocked out) and becoming 10 steps instead of 2 (at this point).

Take SQL for example. When I do a mock test with that, I just need to mock results with the SQLConnection, SQQLCommand, SQLReader, or SQLDataProvider (sorry, I'm forgetting the exact names right now) depending on the path to acquire data. As I started, I'm trying to do the two items above. 

With all of this said, are saying you don't know what to do? I love debate, but I also would love to keep this focused.

Thanks,
Mai

[Craig Wilson]

Hi Mai,

If you feel you must unit test, then that is your call. I can't stop you :). All of our tests pass when run in our CI server as well as on our local boxes. You need to ensure that enableTestCommands is turned on as some tests rely on some undocumented functions to ensure that the driver behaves correctly when encountering unexpected conditions. Perhaps you could provide the tests which are failing?

I'm sorry it's taking you 10 steps to mock things out. We can't change this to make it easier for users doing things they really shouldn't be doing.  Likely, you are encountering issues where your code is using an extension method and you can't mock the extension method, so you need to mock the stuff underneath. This design as made it easy for us to hook things together and unit test them internally, where we do use mocking. However, we already know how everything is supposed to fit together and we own the code. These are very large differences between what you are doing, where you don't own the code and don't really understand how everything hooks together. Hence, you end up with argument null exceptions and other oddities. When you encounter these issues, you're just going to have to work through them by looking at the source code and figuring it out. If you have a question about a particular design, feel free to ask.

I would lake to assuage your fears that our tests don't pass, so please follow-up with (a) how you are running the tests, (b) how you are starting the mongod server, and (c) which tests are failing. I'm not completely sure why these are relevant to you mocking our libraries, but happy to help you get those working.

Craig