AWS Lambda + MongoDB Atlas = very slow

[Benjamin Hüttinger]

Hi,

i just tried out AWS Lambda and MongoDB Atlast together, but so far i'm a bit disappointed by the bad performance.
Since i can't believe the results i'm getting are normal i would like to find out what i do wrong here so i can continue to use MongoDB.

- I have a MongoDB Atlas cluster (Free Tier) in the eu-west-1 region

- I also have a AWS Lambda Function in the same region

- Atlas is configured to allow access from everywhere at the moment, there is no VPC set up

- The code of the lambda is just simple nodeJS with query to an empty collection (see code below)

Execution Time for this lambda is typically 1000s+, while i had the same query with DynamoDB running at around or below 200ms.

I'm testing from the AWS Lambda web console, so no request overhead.

I suppose a lot of the execution time is for making a new connection to MongoDB with every execution.

Moving the connection in the initialization could improve this, to have a connection cached between requests.

But even then 1+ seconds execution time for a cold request is still pretty high compared to what DynamoDB got.

Is is the missing VPC? Is it slow throughput on the free Atlas tier? Or am i doing something wrong here?

Thanks a lot :)

'use strict';

const MongoClient = require('mongodb').MongoClient;

module.exports.handle = (event, context, callback) => {

  console.log('event : ', event);

  MongoClient.connect(process.env.MONGODB_URL, function(err, MongoDB) {

    console.log('Connected correctly to server');

    

    MongoDB.collection('items').find({}).toArray(function(error, items) {

      if (error) {

        console.error(error);

        

        callback(null, {

          statusCode: 500,

          body: 'error'

        });

      } else {

        console.log('data returned', items);

        

        callback(null, {

          statusCode: 200,

          body: 'success'

        });

        

            MongoDB.close();

      }

    });

  });

};

[Stephen Steneker]

On Sunday, 12 March 2017 19:15:05 UTC+11, Benjamin Hüttinger wrote:

i just tried out AWS Lambda and MongoDB Atlas together, but so far i'm a bit disappointed by the bad performance.

Hi Benjamin,

AWS Lambda has some "cold start" overhead when initializing a new container and establishing database connections; MongoDB Atlas requires authentication and SSL which adds to the initial connection time. MongoDB drivers typically amortise connection overhead using a persistent pool of active connections, but the stateless nature of AWS Lambda requires a different approach.

We are looking into how to improve MongoDB driver support for AWS Lambda, but in general Lambda is a trade-off between compute cost and persistence. For more predictable application latency you should use a standard persistent application environment (eg. EC2). Lambda is better suited for event-driven use cases where tasks are periodic or bursty, and not time-sensitive.

It would also be helpful to note which version of the MongoDB Node.js driver you are using as a reference point for others finding this discussion in future.

Regards,

Stennie

[matt.br...@10gen.com]

Hi Benjamin,

The reason this particular function is slow is because you are incurring a full connection sequence to atlas each time the handler is called (see Stephen's explanation for more here). Luckily, there is a way around this! AWS Lambda will actually "freeze" the environment for subsequent runs, meaning we can indeed reuse connections and dramatically improve performance - they simply need to be cached outside of the handler.

Here is an update of your provided example:

'use strict';

const MongoClient = require('mongodb').MongoClient;

const ATLAS_URI = <insert connection string here>;

let cachedDb = null;

function connectToDatabase(uri) {

console.log('=> connect to database');

if (cachedDb) {

console.log('=> using cached database instance');

return Promise.resolve(cachedDb);

}

return MongoClient.connect(uri)

.then(db => { cachedDb = db; return cachedDb; });

}

function queryDatabase(db) {

console.log('=> query database');

return db.collection('items').find({}).toArray()

.then(() => { return { statusCode: 200, body: 'success' }; })

.catch(err => { return { statusCode: 500, body: 'error' }; });

}

module.exports.handle = (event, context, callback) => {

console.log('event: ', event);

connectToDatabase(ATLAS_URI)

.then(db => queryDatabase(db))

.then(result => {

console.log('=> returning result: ', result);

context.succeed(result);

})

.catch(err => {

console.log('=> an error occurred: ', err);

context.failed(err);

});

};

In my local testing the initial run completed in ~2-5s, while subsequent runs ran in ~2-15ms (on a collection of 1M "random" documents).

HTH,

Matt

[Raphael Londner]

Hi Benjamin,

In the meantime, we have updated our guidance for AWS Lambda when used with Node.js and MongoDB Atlas. Feel free to find our performance optimization recommendations at https://www.mongodb.com/blog/post/optimizing-aws-lambda-performance-with-mongodb-atlas-and-nodejs

Best regards,

Raphael.