Trouble reindexing discussion forum posts

140 views
Skip to first unread message

ji...@opencraft.com

unread,
Jan 9, 2017, 10:35:21 PM1/9/17
to General Open edX discussion
Hello,

We're setting up a new edxapp instance for a client with existing courses and forum posts, and are having trouble getting ElasticSearch to return search results for old posts.  New posts/comments are indexed just fine though.

The new edxapp instance is running Eucalyptus.2, with data migrated from Dogwood.

Here's what I've tried:
  • Ran rake search:rebuild as recommended by cs_comment_service/wiki/Search-Indexes.
    • Had to apply this patch to get the task to run without error (will submit that upstream).
    • Output looked promising; there were a lot of CommentThread entries read from mongodb:
      forum@c:~/cs_comments_service$ rake search:rebuild
/edx/app/forum/cs_comments_service/lib/tasks/flags.rake:6: warning: already initialized constant ROOT
/edx/app/forum/cs_comments_service/lib/tasks/kpis.rake:7: warning: already initialized constant ROOT
W, [2017-01-09T04:21:07.666963 #24546]  WARN -- : Overwriting existing field _id in class User.
I, [2017-01-09T04:21:07.675829 #24546]  INFO -- : Creating new index: content_20170109042107...
I, [2017-01-09T04:21:07.787122 #24546]  INFO -- : Applying index mappings for CommentThread
I, [2017-01-09T04:21:07.833666 #24546]  INFO -- : Applying index mappings for Comment
I, [2017-01-09T04:21:07.848420 #24546]  INFO -- : ...done!
I, [2017-01-09T04:21:07.851193 #24546]  INFO -- : preparing to (re)build index
D, [2017-01-09T04:21:07.852430 #24546] DEBUG -- : MONGODB | Adding xx.xx.xx.xx:27017 to the cluster.
D, [2017-01-09T04:21:07.856389 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.getnonce | STARTED | {}
D, [2017-01-09T04:21:07.857681 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.getnonce | SUCCEEDED | 0.00074798s
D, [2017-01-09T04:21:07.858003 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.authenticate | STARTED | {}
D, [2017-01-09T04:21:07.859127 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.authenticate | SUCCEEDED | 0.000842973s
D, [2017-01-09T04:21:07.912745 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | STARTED | {"count"=>"contents", "query"=>{"_type"=>{"$in"=>["Comment", "CommentThread"]}, "updated_at"=>{"$lte"=>2017-01-09 04:21:07 UTC}}}
D, [2017-01-09T04:21:07.916484 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | SUCCEEDED | 0.003127465s
D, [2017-01-09T04:21:07.918459 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | STARTED | {"count"=>"contents", "query"=>{"_type"=>{"$in"=>["Comment", "CommentThread"]}, "updated_at"=>{"$lte"=>2017-01-09 04:21:07 UTC}}}
D, [2017-01-09T04:21:07.920659 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | SUCCEEDED | 0.0019071720000000002s
D, [2017-01-09T04:21:07.922103 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | {"find"=>"contents", "filter"=>{"_type"=>{"$in"=>["Comment", "CommentThread"]}, "updated_at"=>{"$lte"=>2017-01-09 04:21:07 UTC}}, "skip"=>0, "limit"=>500}
D, [2017-01-09T04:21:07.963584 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.041098094s
I, [2017-01-09T04:21:07.976211 #24546]  INFO -- : content_20170109042107: imported 0 of 280 (0.0% complete after 0.06 seconds)
D, [2017-01-09T04:21:07.978606 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | {"find"=>"contents", "filter"=>{"_id"=>BSON::ObjectId('5771941e6fa57b9dee000003'), "_type"=>{"$in"=>["CommentThread"]}}}
D, [2017-01-09T04:21:07.979844 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.0009901159999999999s
D, [2017-01-09T04:21:07.984179 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | {"find"=>"contents", "filter"=>{"_id"=>BSON::ObjectId('5771941e6fa57b9dee000003'), "_type"=>{"$in"=>["CommentThread"]}}}
D, [2017-01-09T04:21:07.985220 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.0007970890000000001s
D, [2017-01-09T04:21:07.986733 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | {"find"=>"contents", "filter"=>{"_id"=>BSON::ObjectId('5771941e6fa57b9dee000003'), "_type"=>{"$in"=>["CommentThread"]}}}
D, [2017-01-09T04:21:07.987790 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.000801455s
D, [2017-01-09T04:21:07.989637 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | {"find"=>"contents", "filter"=>{"_id"=>BSON::ObjectId('5771941e6fa57b9dee000003'), "_type"=>{"$in"=>["CommentThread"]}}}
D, [2017-01-09T04:21:07.990676 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.000781482s
D, [2017-01-09T04:21:07.994360 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | {"find"=>"contents", "filter"=>{"_id"=>BSON::ObjectId('5771941e6fa57b9dee000003'), "_type"=>{"$in"=>["CommentThread"]}}}
D, [2017-01-09T04:21:07.995228 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.000639859s
D, [2017-01-09T04:21:07.996512 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | {"find"=>"contents", "filter"=>{"_id"=>BSON::ObjectId('5771941e6fa57b9dee000003'), "_type"=>{"$in"=>["CommentThread"]}}}
D, [2017-01-09T04:21:07.997424 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.000684286s
D, [2017-01-09T04:21:07.999114 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | {"find"=>"contents", "filter"=>{"_id"=>BSON::ObjectId('5771941e6fa57b9dee000003'), "_type"=>{"$in"=>["CommentThread"]}}}
<snip 987 DEBUG lines similar to the last two>
D, [2017-01-09T04:21:10.053846 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | SUCCEEDED | 0.003505191s
D, [2017-01-09T04:21:10.055157 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | {"find"=>"contents", "filter"=>{"_type"=>{"$in"=>["Comment", "CommentThread"]}, "updated_at"=>{"$lte"=>2017-01-09 04:21:07 UTC}}, "skip"=>500, "limit"=>500}
D, [2017-01-09T04:21:10.057621 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.002178538s
I, [2017-01-09T04:21:10.057859 #24546]  INFO -- : content_20170109042107: finished importing 280 documents
I, [2017-01-09T04:21:10.063743 #24546]  INFO -- : alias already exists (will move): content_20170109042021
I, [2017-01-09T04:21:10.072781 #24546]  INFO -- : alias "content" now points to index content_20170109042107.
I, [2017-01-09T04:21:10.072937 #24546]  INFO -- : importing any documents that changed between 2017-01-09 04:21:07 UTC and now
D, [2017-01-09T04:21:10.074877 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | STARTED | {"count"=>"contents", "query"=>{"_type"=>{"$in"=>["Comment", "CommentThread"]}, "updated_at"=>{"$gte"=>2017-01-09 04:21:07 UTC}}}
D, [2017-01-09T04:21:10.077733 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | SUCCEEDED | 0.002418113s
D, [2017-01-09T04:21:10.079052 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | STARTED | {"count"=>"contents", "query"=>{"_type"=>{"$in"=>["Comment", "CommentThread"]}, "updated_at"=>{"$gte"=>2017-01-09 04:21:07 UTC}}}
D, [2017-01-09T04:21:10.081516 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | SUCCEEDED | 0.0021388329999999997s
D, [2017-01-09T04:21:10.082713 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | {"find"=>"contents", "filter"=>{"_type"=>{"$in"=>["Comment", "CommentThread"]}, "updated_at"=>{"$gte"=>2017-01-09 04:21:07 UTC}}, "skip"=>0, "limit"=>500}
D, [2017-01-09T04:21:10.085106 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.002042407s
I, [2017-01-09T04:21:10.085363 #24546]  INFO -- : content_20170109042107: finished importing 0 documents
  • And sure enough, the new content_20170109042107 index now has the main content alias. The other indexes must be my previous attempts (though I don't remember trying that many times):
    forum@c:~/cs_comments_service$ curl http://xx.xx.xx.xx:9200/_aliases
{
  "content_20170108125330":{"aliases":{}},
  "content_20170108125001":{"aliases":{}},
  "content_20170108125747":{"aliases":{}},
  "content_20170108125315":{"aliases":{}},
  "courseware_index":{"aliases":{}},
  "content_20170109041935":{"aliases":{}},
  "content_20170109042107":{"aliases":{}},
  "content_20170108125440":{"aliases":{}},
  "content_20170108125305":{"aliases":{}},
  "content_20170108125433":{"aliases":{}},
  "content_20170109045546":{"aliases":{"content":{}}},
  "content_20170109042021":{"aliases":{}},
  "content_20170108125518":{"aliases":{}}
}
  • But there's only two entries in in the content index, and they only appeared after I added a new comment to the post:
    forum@c:~/cs_comments_service$ curl http://xx.xx.xx.xx:9200/content/_search
{"took":2,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},
  "hits":{"total":2,"max_score":1.0,
    "hits":[
        {"_index":"content_20170109045546","_type":"comment","_id":"5873153298615406ec000000","_score":1.0, "_source" : {"body":"foxes","course_id":"course-v1:edX+DemoX+Demo_Course","created_at":"2017-01-09T04:44:34Z","updated_at":"2017-01-09T04:44:34Z","commentable_id":"cba3e4cd91d0466b9ac50926e495b76f","comment_thread_id":"587203839861540683000000","group_id":{},"context":"course","id":"5873153298615406ec000000","_type":"comment"}},
        {"_index":"content_20170109045546","_type":"comment_thread","_id":"587203839861540683000000","_score":1.0, "_source" : {"title":"Aeroplane videotape stomach","body":"Random words for the forum search test","course_id":"course-v1:edX+DemoX+Demo_Course","commentable_id":"cba3e4cd91d0466b9ac50926e495b76f","created_at":"2017-01-08T09:16:51Z","updated_at":"2017-01-09T04:44:34Z","context":"course","last_activity_at":"2017-01-09T04:44:34Z","id":"587203839861540683000000","group_id":{},"votes_point":0,"author_id":"2582","thread_id":"587203839861540683000000","_type":"comment_thread"}}
    ]
}}

Has anyone seen this before?  Any ideas?

--

Robert R

unread,
Jan 11, 2017, 10:00:01 AM1/11/17
to General Open edX discussion
Hi Jill.

I think reindex is currently broken.  We are working on a PR that switches away from Tire (used for ES), and I discovered that reindex happens to be broken yesterday as part of that work.  This PR is prep for an ES upgrade, but is compatible with the current version of ES.

Here is the PR, but it is not complete yet.  It is probably larger than what you need to fix your issue.  One problem that I know you will have is partially fixed in this commit: https://github.com/edx/cs_comments_service/pull/216/commits/2be90968e96af553fa9e15a6181a9d051d065a40.  The issue was that Rake did not have some fixes that were in app.rb.

I can discuss more if needed.  But for now, I'm going to concentrate on landing this work.

Robert

Robert R

unread,
Jan 11, 2017, 4:05:15 PM1/11/17
to General Open edX discussion
I added this (untested) PR with a bit more details:

Jill Vogel

unread,
Jan 11, 2017, 10:06:42 PM1/11/17
to edx-...@googlegroups.com
Hi Robert,

That PR did the trick for me!  Have left a comment there.

Thank you for the fix!

Cheers,
Jill

--

--
You received this message because you are subscribed to the Google Groups "General Open edX discussion" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/edx-code/1011dbab-cd51-4026-922e-0a6700943fea%40googlegroups.com.

Reply all
Reply to author
Forward
0 new messages