[MongoDB Charts] Stitch fails to start

Antoine Bellion

unread,

Jan 15, 2020, 1:52:57 PM1/15/20

to mongodb-user

I followed all the steps to install Charts on premise. Everything seems to be correct, the connection to the metadata database works, the secret containing the URI is fine, etc... However, the Stitch server fails to start and has the following logs :

root@charts:/mongodb-charts/logs# cat stitch-startup.log
{"level":"info","ts":1579086352.771636,"caller":"libmongo/libmongo.go:80","msg":"libmongo: using plugin"}
2020-01-15T11:05:52.818Z	INFO	main_server	server/main.go:367	Starting prometheus exporter on: localhost:8100
2020-01-15T11:05:52.834Z	INFO	main_server	server/main.go:84	Starting combined API & admin API server on :8080
2020-01-15T11:05:54.206Z	ERROR	main_server	server/main.go:88	error starting up servers: (BadValue) Must specify at least one cursor id in: { killCursors: "metadata", cursors: [], $clusterTime: { clusterTime: Timestamp(1579086354, 3), signature: { hash: BinData(0, 1CDE4BECEB0FDAFF7A15270CF139C90BFEFC6552), keyId: 6748863342644822017 } }, $db: "5e1eec00d5ec139bb76c4a85_app", $readPreference: { mode: "primaryPreferred" } }

It looks like Stitch tries to send a killCursors command to the server, which fails because no cursor is specified.

Thanks for your help.

Tom Hollander

unread,

Jan 17, 2020, 12:25:52 AM1/17/20

to mongodb-user

Hi Antoine -

Sorry to hear you're having problems. Could you help by letting me know your Charts version and MongoDB server version?

thanks

Tom

Antoine Bellion

unread,

Jan 17, 2020, 5:13:01 AM1/17/20

to mongodb-user

Hi Tom,

Thanks for getting back ! How could I have forgotten that !
I tried with Charts 19.06 and 19.12. The MongoDB server's version is 4.2.2 on Atlas.

Best,
Antoine

Tom Hollander

unread,

Jan 19, 2020, 5:45:44 PM1/19/20

to mongodb-user

Thanks Antoine. I just tested Charts 19.12.1 against MongoDB 4.2.2 on Atlas and it started fine. This is the first time I've heard about this problem so it may be something very specific about your setup. Have you tried starting more than once, and if so do you get the same error each time? Is there anything else potentially unusual about your setup?

Antoine Bellion

unread,

Jan 22, 2020, 10:24:19 AM1/22/20

to mongodb-user

Hi Tom,

I tried restarting the service many time but the logs are always the same. I searched for something unusual in my setup but couldn't find anything (it's a regular swarm cluster). Here are the complete logs :

"stitch-startup.log" :

2020-01-22T15:16:29.119Z	INFO	main_server	server/main.go:367	Starting prometheus exporter on: localhost:8100
2020-01-22T15:16:29.136Z	INFO	main_server	server/main.go:84	Starting combined API & admin API server on :8080
2020-01-22T15:16:29.352Z	ERROR	main_server	server/main.go:88	error starting up servers: (BadValue) Must specify at least one cursor id in: { killCursors: "metadata", cursors: [], $clusterTime: { clusterTime: Timestamp(1579706189, 1), signature: { hash: BinData(0, DF41526720201EC1A8B7C9504F4EEFEB311E9927), keyId: 6784299120644325379 } }, $db: "5e28602bff7a25f568e7bf66_app", $readPreference: { mode: "primaryPreferred" } }

"charts-cli.log" :

2020-01-22T15:16:26.057+00:00 INFO  called charts-cli startup with arguments {"_":["startup"],"debug":false,"help":false,"version":false,"with-test-facilities":false,"withTestFacilities":false,"d":"/mongodb-charts","directory":"/mongodb-charts","$0":"mongodb-charts/bin/charts-cli.js"}
2020-01-22T15:16:26.066+00:00 INFO  parsedArgs task success
2020-01-22T15:16:26.070+00:00 INFO  installDir task success ('/mongodb-charts')
2020-01-22T15:16:26.071+00:00 INFO  log task success
2020-01-22T15:16:26.071+00:00 INFO  salt task success
2020-01-22T15:16:26.073+00:00 INFO  productNameAndVersion task success ({ productName: 'MongoDB Charts Frontend', version: '1.9.1' })
2020-01-22T15:16:26.074+00:00 INFO  gitHash task success ('1a46f17f')
2020-01-22T15:16:26.074+00:00 INFO  supportWidgetAndMetrics task success ('off')
2020-01-22T15:16:26.074+00:00 INFO  tileServer task success (undefined)
2020-01-22T15:16:26.074+00:00 INFO  tileAttributionMessage task success (undefined)
2020-01-22T15:16:26.074+00:00 INFO  rawFeatureFlags task success (undefined)
2020-01-22T15:16:26.077+00:00 INFO  encryptionKeyPath task success
2020-01-22T15:16:26.078+00:00 INFO  featureFlags task success ({})
2020-01-22T15:16:26.098+00:00 INFO  lastAppJson task success ({})
2020-01-22T15:16:26.107+00:00 INFO  existingInstallation task success (false)
2020-01-22T15:16:26.108+00:00 INFO  tenantId task success ('295430a0-17c5-478a-9f57-e3e65289cc7d')
2020-01-22T15:16:26.111+00:00 INFO  tokens task success
2020-01-22T15:16:26.112+00:00 INFO  stitchMigrationsLog task success ({ completedStitchMigrations:   [ 'stitch-1332',     'stitch-1897',     'stitch-2041',     'migrateStitchProductFlag',     'stitch-2041-local',     'stitch-2046-local',     'stitch-2055',     'multiregion',     'dropStitchLogLogIndexStarted' ] })
2020-01-22T15:16:26.113+00:00 INFO  chartsMongoDBUri task success
2020-01-22T15:16:26.117+00:00 INFO  stitchConfigTemplate task success
2020-01-22T15:16:26.119+00:00 INFO  libMongoIsInPath task success (true)
2020-01-22T15:16:26.218+00:00 INFO  waiting for MongoDB, attempt #1 to connect to MongoDB at mongodb://admin:*******@cluster0-shard-00-00-ldklv.mongodb.net:27017,cluster0-shard-00-01-ldklv.mongodb.net:27017,cluster0-shard-00-02-ldklv.mongodb.net:27017/test?ssl=true&replicaSet=Cluster0-shard-0&authSource=admin&retryWrites=true&w=majority.
2020-01-22T15:16:26.413+00:00 INFO  waiting for MongoDB, successfully connected to MongoDB at mongodb://admin:*******@cluster0-shard-00-00-ldklv.mongodb.net:27017,cluster0-shard-00-01-ldklv.mongodb.net:27017,cluster0-shard-00-02-ldklv.mongodb.net:27017/test?ssl=true&replicaSet=Cluster0-shard-0&authSource=admin&retryWrites=true&w=majority after 1 attempts.
2020-01-22T15:16:26.419+00:00 INFO  mongoDBReachable task success (true)
2020-01-22T15:16:26.434+00:00 INFO  stitchMigrationsExecuted task success ([ 'stitch-1332',  'stitch-1897',  'stitch-2041',  'migrateStitchProductFlag',  'stitch-2041-local',  'stitch-2046-local',  'stitch-2055',  'multiregion',  'dropStitchLogLogIndexStarted' ])
2020-01-22T15:16:26.770+00:00 INFO  minimumVersionRequirement task success (true)
2020-01-22T15:16:26.773+00:00 INFO  stitchConfig task success
2020-01-22T15:16:26.782+00:00 INFO  stitchConfigWritten task success (true)
2020-01-22T15:16:26.792+00:00 INFO  stitchChildProcess task success
2020-01-22T15:16:26.894+00:00 INFO  waiting for Stitch to start, attempt #1 to connect to Stitch server at http://localhost:8080.
2020-01-22T15:16:26.912+00:00 WARN  waiting for Stitch to start, attempt #1 failed: connect ECONNREFUSED 127.0.0.1:8080
2020-01-22T15:16:27.012+00:00 INFO  waiting for Stitch to start, attempt #2 to connect to Stitch server at http://localhost:8080.
2020-01-22T15:16:27.015+00:00 WARN  waiting for Stitch to start, attempt #2 failed: connect ECONNREFUSED 127.0.0.1:8080
2020-01-22T15:16:27.036+00:00 INFO  indexesCreated task success (true)
2020-01-22T15:16:27.216+00:00 INFO  waiting for Stitch to start, attempt #3 to connect to Stitch server at http://localhost:8080.
2020-01-22T15:16:27.217+00:00 WARN  waiting for Stitch to start, attempt #3 failed: connect ECONNREFUSED 127.0.0.1:8080
2020-01-22T15:16:27.519+00:00 INFO  waiting for Stitch to start, attempt #4 to connect to Stitch server at http://localhost:8080.
2020-01-22T15:16:27.521+00:00 WARN  waiting for Stitch to start, attempt #4 failed: connect ECONNREFUSED 127.0.0.1:8080
2020-01-22T15:16:28.021+00:00 INFO  waiting for Stitch to start, attempt #5 to connect to Stitch server at http://localhost:8080.
2020-01-22T15:16:28.023+00:00 WARN  waiting for Stitch to start, attempt #5 failed: connect ECONNREFUSED 127.0.0.1:8080
2020-01-22T15:16:28.823+00:00 INFO  waiting for Stitch to start, attempt #6 to connect to Stitch server at http://localhost:8080.
2020-01-22T15:16:28.825+00:00 WARN  waiting for Stitch to start, attempt #6 failed: connect ECONNREFUSED 127.0.0.1:8080
2020-01-22T15:16:30.126+00:00 INFO  waiting for Stitch to start, attempt #7 to connect to Stitch server at http://localhost:8080.
2020-01-22T15:16:30.128+00:00 WARN  waiting for Stitch to start, attempt #7 failed: connect ECONNREFUSED 127.0.0.1:8080
2020-01-22T15:16:32.230+00:00 INFO  waiting for Stitch to start, attempt #8 to connect to Stitch server at http://localhost:8080.
2020-01-22T15:16:32.232+00:00 WARN  waiting for Stitch to start, attempt #8 failed: connect ECONNREFUSED 127.0.0.1:8080
2020-01-22T15:16:34.332+00:00 INFO  waiting for Stitch to start, attempt #9 to connect to Stitch server at http://localhost:8080.
2020-01-22T15:16:34.334+00:00 WARN  waiting for Stitch to start, attempt #9 failed: connect ECONNREFUSED 127.0.0.1:8080
2020-01-22T15:16:36.437+00:00 INFO  waiting for Stitch to start, attempt #10 to connect to Stitch server at http://localhost:8080.
2020-01-22T15:16:36.439+00:00 WARN  waiting for Stitch to start, attempt #10 failed: connect ECONNREFUSED 127.0.0.1:8080
2020-01-22T15:16:38.542+00:00 INFO  waiting for Stitch to start, attempt #11 to connect to Stitch server at http://localhost:8080.
2020-01-22T15:16:38.544+00:00 WARN  waiting for Stitch to start, attempt #11 failed: connect ECONNREFUSED 127.0.0.1:8080
2020-01-22T15:16:40.646+00:00 INFO  waiting for Stitch to start, attempt #12 to connect to Stitch server at http://localhost:8080.
2020-01-22T15:16:40.649+00:00 WARN  waiting for Stitch to start, attempt #12 failed: connect ECONNREFUSED 127.0.0.1:8080
2020-01-22T15:16:42.750+00:00 INFO  waiting for Stitch to start, attempt #13 to connect to Stitch server at http://localhost:8080.
2020-01-22T15:16:42.753+00:00 WARN  waiting for Stitch to start, attempt #13 failed: connect ECONNREFUSED 127.0.0.1:8080
2020-01-22T15:16:44.855+00:00 INFO  waiting for Stitch to start, attempt #14 to connect to Stitch server at http://localhost:8080.
2020-01-22T15:16:44.860+00:00 WARN  waiting for Stitch to start, attempt #14 failed: connect ECONNREFUSED 127.0.0.1:8080
2020-01-22T15:16:46.961+00:00 INFO  waiting for Stitch to start, attempt #15 to connect to Stitch server at http://localhost:8080.
2020-01-22T15:16:46.964+00:00 WARN  waiting for Stitch to start, attempt #15 failed: connect ECONNREFUSED 127.0.0.1:8080
2020-01-22T15:16:46.964+00:00 ERROR stitchServerRunning task failure: Can't connect to Stitch Server at http://localhost:8080. Too many failed attempts. Last error: connect ECONNREFUSED 127.0.0.1:8080
2020-01-22T15:16:46.966+00:00 ERROR startup failed

Best,
Antoine

Tom Hollander

unread,

Jan 22, 2020, 11:42:05 PM1/22/20

to mongod...@googlegroups.com

Hi Antoine -

Thanks for providing more information. I've done some more testing and it seems this problem occurs with a MongoDB 4.2 cluster smaller than M10 (including the M0 free tier). Charts starts fine when using an M10 cluster. I'm pretty sure this worked with 4.0 M0 clusters but these can no longer be deployed. This is a recent change so I'll need to look into the root cause.

Since you're an Atlas user, is it possible for you to use MongoDB Charts on Atlas and avoid the need to manage the Charts server yourself?

{ name     : "Tom Hollander",
  title    : "Principal Product Manager",
phone : "+61 407 224 034",
  location : "Sydney, Australia" }

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to a topic in the Google Groups "mongodb-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongodb-user/Tco0296vw8s/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mongodb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/e458962c-d246-4034-baf7-89bf5bf26952%40googlegroups.com.

Antoine Bellion

unread,

Jan 23, 2020, 7:33:18 AM1/23/20

to mongodb-user

Hi Tom,

I'm glad you have been able to reproduce the issue.
Do you have an idea how long it will take to resolve it ? So that I can inform my team who depends on it. If you need anything from me don't hesitate I would be happy to help.

Using charts on Atlas isn't an option for me because the database I read the data from isn't on Atlas.

Best,
Antoine

To unsubscribe from this group and all its topics, send an email to mongod...@googlegroups.com.

Tom Hollander

unread,

Jan 23, 2020, 9:39:46 PM1/23/20

to mongodb-user

Hi Antoine -

I've looked into this more, and it seems the problem has already been fixed in the latest unshipped builds, so the next update to Charts on-prem will support this scenario.