mongodb-connector syncing issue

[Maciej Bodek]

Hello,

I'm using the mongodb-connector from the MongoDB team to sync my local database with Algolia search engine.

https://github.com/10gen-labs/mongo-connector

Steps I do:

1. Fire up the mongod daemon with replica set (mongod --replSet "rs0")
2. Initiatate in mongo shell: mongo -> rs.initiate(); and check if working with rs.status();
3. Fire up the mongodb-connector: mongo-connector -m localhost:27017 -n myDb.myCollection -d ./doc_managers/algolia_doc_manager.py -t MyApplicationID:MyApiKey:MyIndex

However, it doesn't work as expected. Actions performed locally (insert, update) are not mirrored on a remote platform, as if the collection is empty to the remote service.

I'm hoping someone had similar issues with either Algolia or elasticsearch and found a workaround. Any help appreciated!

Best,

Maciek

[Luke Lovett]

Hi Maciek,

I have a few questions:
- What version of Mongo Connector are you using? Are you using Algolia's fork of Mongo Connector (https://github.com/algolia/mongo-connector)?
- What version of Algolia (server/client) are you using?
- Could you please provide the log output from Mongo Connector, either here or in a pastebin/gist?

Is there any way to configure how frequently new operations are committed in Algolia? This is a configurable parameter on similar projects like Elasticsearch and Solr, where recent operations aren't visible until they've been committed to the index. Often the default value for these is pretty high (i.e., several minutes). It might be that operations from MongoDB just haven't yet been committed.

I'm not very familiar with Algolia, but hopefully your answers will help me to dig in deeper to what's going on here.

Hope this helps,
Luke

[Maciej Bodek]

Hi Luke,

- I'm using the Algolia fork - that's correct.

- I'm using the latest cloned version of the client from the repository.

- Log output is as following - from the mongo connector:

https://gist.github.com/insmac/897e8942f88c1a3bb57f

Then I have some activity in mongod:

https://gist.github.com/insmac/4fe07ce3d443227a2eb4

At that point, I've tried to insert a new document to the collection locally, but still - no syncing.

Hope that helps.

Best,

Maciek

[Luke Lovett]

Hi Maciek,

I just set up Algolia for myself, and using the latest 'master' version from the Algolia fork, I was successfully able to replicate inserts, deletes, and updates from a MongoDB replica set. Here's what I did:

1. Start up a MongoDB replica set (I happened to have used port 50000 as a starting port).

2. Start Mongo Connector:

mongo-connector -m localhost:50000 \
                -t AlgoliaAppId:AlgoliaKey:IndexName \
                -d mongo_connector/doc_managers/algolia_doc_manager.py

I didn't provide any other configuration to Mongo Connector or the Algolia DocManager.

3. I added some documents to the "altest.c" collection. Documents had the following structure:

{"_id": <object id>, "s": <random 6-character string>, "i": <random integer value>}

I inserted 10 documents like the above. I updated 5 of them to increment "i" by 1000, then deleted the ones with "i" less than 5.

4. I got some log output from MC:

2014-10-17 20:34:40,871 - INFO - Beginning Mongo Connector
2014-10-17 20:34:40,896 - INFO - Algolia Connector: Start without filter.
2014-10-17 20:34:40,896 - INFO - Algolia Connector: Start without remapper.
2014-10-17 20:34:40,896 - INFO - Algolia Connector: Start without post processing.
2014-10-17 20:34:40,896 - INFO - MongoConnector: Can't find config.txt, attempting to create an empty progress log
2014-10-17 20:34:40,898 - INFO - MongoConnector: Empty oplog progress file.
2014-10-17 20:34:40,900 - INFO - OplogThread: Initializing oplog thread
2014-10-17 20:34:40,901 - INFO - MongoConnector: Starting connection thread MongoClient([u'llpro:50002', u'llpro:50001', u'llpro:50000'])
2014-10-17 20:35:10,898 - INFO - Starting new HTTPS connection (1): XXXXX-1.algolia.io
2014-10-17 20:44:22,265 - INFO - Resetting dropped connection: XXXXX-1.algolia.io
2014-10-17 20:45:13,284 - INFO - Resetting dropped connection: XXXXX-1.algolia.io

5. From Algolia logs:

 /1/indexes/TestIndex/settings
200 3 minutes ago POST
/1/indexes/TestIndex?query=&getRankingInfo=1&facets=*&attributesToRetrieve=*&maxValuesPerFacet=10&hitsPerPage=10 (6 hits)
200 3 minutes ago POST
/1/indexes/TestIndex?query=&hitsPerPage=100&attributesToRetrieve=* (6 hits)
200 3 minutes ago GET
/1/indexes/TestIndex/task/422668960/
200 3 minutes ago PUT
/1/indexes/TestIndex/settings
200 3 minutes ago POST
/1/indexes/TestIndex/batch
200 4 minutes ago GET
/1/indexes/TestIndex/settings
200 4 minutes ago POST
/1/indexes/TestIndex?query=&getRankingInfo=1&facets=*&attributesToRetrieve=*&maxValuesPerFacet=10&hitsPerPage=10 (10 hits)
200 4 minutes ago POST
/1/indexes/TestIndex?query=&hitsPerPage=100&attributesToRetrieve=* (10 hits)
200 4 minutes ago GET
/1/indexes/TestIndex/task/422650060/
...

I find it suspicious that no message was logged about establishing a new HTTPS connection to your Algolia instance in your logs. Are you sure that you have the app id and api key correct? Other than that, I can only suggest that you file an issue on Algolia's fork of Mongo Connector, since they probably have a lot more expertise troubleshooting the Algolia connector than I do.

https://github.com/algolia/mongo-connector/issues/new

[Maciej Bodek]

Hey Luke,

Thanks for your help! It works now :) The issue was a specific parameter I've included to point to other database other than test. It's syncing automatically as intended.

I'm really flattered you've done this just to help me out. Just another sign of a good choice - I've choosen Mongo for an enterprise-class website (a large catalog type) and it works like a charm.

Best,

Maciek