Search engine for course content

[Marceau C.]

Hi,

I am looking for a way to search inside the content of the course. And it would be even better if it could search inside the files linked inside the course (PDF, PPT, ...) like we can do on Sharepoint.

Is this possible ?

Thanks,
Marceau C.

[Don-edX Mitchell]

We've frequently discussed adding search but it's never made it high on the priority list. get_items has most of the machinery for doing search on xblock content (see the unit tests which pass regex's to it); however, we've not done anything for searching in assets. We are refactoring asset storage; so, it's probably not worthwhile to add anything right now, but I will pass this request onto the architect for that refactoring. HOWEVER, being open source, we'd strongly encourage you to volunteer to add search to the asset refactoring (existing proposal which may be inaccessible but will be published here soon)

[Yarko Tymciurak]

On Mon, Aug 11, 2014 at 8:14 AM, Don-edX Mitchell <dmit...@edx.org> wrote:

We've frequently discussed adding search but it's never made it high on the priority list. get_items has most of the machinery for doing search on xblock content (see the unit tests which pass regex's to it); however, we've not done anything for searching in assets. We are refactoring asset storage; so, it's probably not worthwhile to add anything right now, but I will pass this request onto the architect for that refactoring. HOWEVER, being open source, we'd strongly encourage you to volunteer to add search to the asset refactoring (existing proposal which may be inaccessible but will be published here soon)

... yes, I am finding that inaccessible...

[John Eskew]

All,

That page is now publicly accessible here:

https://openedx.atlassian.net/wiki/display/PLAT/GridFS+Replacement

 

[Nate Aune]

This appears to be available now with the latest Birch release as described in the official documentation:

http://edx.readthedocs.org/projects/open-edx-building-and-running-a-course/en/named-release-birch/developing_course/workflow.html#making-course-content-searchable

But when I create a new course in a Birch site, I don't see the "Search box in the upper-left corner of the Courseware tab"

Nor do I see the "Reindex Content in the upper-right corner of the Course Outline page"

Is there something that needs to be done on the server in order to enable the search capability within Birch, perhaps setup and configure an ElasticSearch instance?

thanks,

Nate

[Chris Dodge]

Hello Nate,

Yes, we've been putting together course search functionality over the last few months. I'm not sure exactly what is in Birch vs. what is currently on tip of master. I believe there is some configuration changes required (e.g. FEATURE flags). I'll point the right developers to this mail thread to fill in the answers to your questions.

Best wishes,

Chris

@edx.org

[Martyn James]

Hi Nate,

There are some FEATURE switches that you'll need to set to True in order to have search working.

For Studio, you'll want to enable the indexing using:

    ENABLE_COURSEWARE_INDEX

this exposes that button within the course structure page and allows elastic search to be populated

For LMS, you'll want to enable the search box using:

    ENABLE_COURSEWARE_SEARCH

this exposes the search box within the courseware tab within a course

Yes, you'll need to configure an elasticsearch instance to use, currently it is designed to use the same elasticsearch cluster as the forms service does today. If you are using devstack or fullstack then you already have this in place.

Marty

@edx.org

On Wednesday, April 1, 2015 at 10:02:43 PM UTC-4, Nate Aune wrote:

[Martyn James]

Apologies for confusing everybody. I am not certain that the search facility is included in the Birch release. This functionality is currently on the master tip, so expect it to appear in the Cypress release.

[Nate Aune]

Hi Martyn,

Thanks for your reply. My responses inline below...

 

There are some FEATURE switches that you'll need to set to True in order to have search working.

For Studio, you'll want to enable the indexing using:

    ENABLE_COURSEWARE_INDEX

this exposes that button within the course structure page and allows elastic search to be populated

So this is a button for the course author (or instructor?) to click that triggers the course content to be indexed and made searchable?

 

For LMS, you'll want to enable the search box using:

    ENABLE_COURSEWARE_SEARCH

this exposes the search box within the courseware tab within a course

What content in the course is searched? Just HTML components or also video transcripts?

 

Yes, you'll need to configure an elasticsearch instance to use, currently it is designed to use the same elasticsearch cluster as the forms service does today. If you are using devstack or fullstack then you already have this in place.

Is there any documentation for how to configure ElasticSearch to work with the courseware search, or will the one that comes with the forums Just Work? 

Does edx-search have any capability to search for courses among those in the course listing? Does it provide any faceted search capabilities like what you see on edx.org?

>Apologies for confusing everybody. I am not certain that the search facility is included in the Birch release. This functionality is currently on the master tip, so expect it to appear in the Cypress release.

Is there a particular branch or PR that I can cherrypick? Since all of my customers are running Aspen or Birch, I can't just pull from master.

thanks,

Nate

[Martyn James]

Hi Nate,

1. ENABLE_COURSEWARE_INDEX

Yes, this exposes a "Reindex" button on the courseware page in Studio which will index the course. The index will need to be populated for searches to find anything. You'll want to hit this button for each course that people might search, but after that the index will update as you change anything. In the latest master we now fully reindex courses in an asynchronous worker process triggered by the course_publish event; but I left the Reindex button in place in case one ever wants to trigger it manually again.

2. Currently, HTML components and Video transcripts are indexed, and thus are searchable. We are hoping to add indexing to other components as we go - this is done by implementing the `index_dictionary` method in the xblock.

3. We have made it so that it currently uses the same ElasticSearch cluster as forums currently uses.

4. Searching for courses in the course listing... coming soon, our group is working on it ;-)

5. Initial PR => https://github.com/edx/edx-platform/pull/6506; this was committed soon after Birch was released, so will probably merge well

A couple of later commits improve on things, but they depend upon other changes in the platform - e.g. course_publish event in place => asynchronous indexing as changes are made, and so these are likely to not merge in well - probably want to hold on until the next named release for those - and at the same time, I suspect that we'll have course discovery searches wrapped up too ;-)

I hope that this helps!

Martyn

[Nate Aune]

Hi Martyn,

We're responding to an RFI for a potential edX project, and it has the following requirements:

1. As a student, I can search for courses based on course title and course description
2. As a student, I can filter for courses based on subject, course type, and language
3. As a student, I can search for courses based on course content (course text and video transcripts), which will return a list of courses and the number of matches
4. As an enrolled student, I can directly navigate to the sections that contain the searched keyword(s)  

From this email thread, it appears that 3 is fulfilled by Cypress, but what about 1, 2 and 4? In your previous email you said re 1 "coming soon, our group is working on it".  Did this feature land in Cypress too?

thanks,

Nate

[Steve Magoun]

Hi Nate,

All 4 of these features are included in Cypress.

Steve

[Nate Aune]

That's great news Steve! Thanks :)

[Saikou]

Hello Guys,

I'm currently trying to enable EDX SEARCH. I set up the variables ENABLE_COURSE_SEARCH, ENABLE_DASHBOARD_SEARCH and ENABLE_COURSE_DISCOVERY. Unfortunately it's doesn't seem to work.

If I search among the course listing from LMS home, the button immediately redirect me to the "/courses" where all courses are listed.
If I search a world ou course from my dashboard, it immediately says that there is 'no result"

Could you please help me to figure out what 's wrong?

Thanks
Saikou

[Emrah Emirtekin]

Hello All

I made the adjustments for edx-search but it works only in the course. When i search the course name without logging result cannot be found. Should i need to make any extra adjustments.

/edx/app/edxapp/edx-platform/cms/envs/common.py

'ENABLE_COURSEWARE_INDEX': True,

'ENABLE_LIBRARY_INDEX': True,

# Default to no Search Engine

SEARCH_ENGINE = "search.elastic.ElasticSearchEngine"

ELASTIC_FIELD_MAPPINGS = {

    "start_date": {

        "type": "date"

    }

}

/edx/app/edxapp/edx-platform/lms/envs/common.py

# Courseware search feature

    'ENABLE_COURSEWARE_SEARCH': True,

 # Dashboard search feature

    'ENABLE_DASHBOARD_SEARCH': True,

 # Course discovery feature

    'ENABLE_COURSE_DISCOVERY': True,

# Use None for the default search engine

SEARCH_ENGINE = "search.elastic.ElasticSearchEngine"

# Use LMS specific search initializer

SEARCH_INITIALIZER = "lms.lib.courseware_search.lms_search_initializer.LmsSearchInitializer"

# Use the LMS specific result processor

SEARCH_RESULT_PROCESSOR = "lms.lib.courseware_search.lms_result_processor.LmsSearchResultProcessor"

# Use the LMS specific filter generator

SEARCH_FILTER_GENERATOR = "lms.lib.courseware_search.lms_filter_generator.LmsSearchFilterGenerator"

# Override to skip enrollment start date filtering in course search

SEARCH_SKIP_ENROLLMENT_START_DATE_FILTERING = False

/edx/bin/supervisorctl restart edxapp:

/edx/bin/supervisorctl restart edxapp_worker

8 Ağustos 2014 Cuma 16:58:53 UTC+3 tarihinde Marceau C. yazdı:

[apollinaire samiey]

i need help to enable edx-search on mitxvm (beta-release of open edx)

[Mahyar Damavand]

Try this documentation:
http://edx.readthedocs.io/projects/edx-installing-configuring-and-running/en/latest/configuration/edx_search.html

By default edx search will be installed automatically when you install the Open edX Platform. And you should just enable course indexing...

On Thu, Jun 23, 2016, 6:48 AM apollinaire samiey <apoos...@gmail.com> wrote:

i need help to enable edx-search on mitxvm (beta-release of open edx)

--
You received this message because you are subscribed to the Google Groups "General Open edX discussion" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/edx-code/3c224a7a-7ce4-4211-8373-04a7669b8fb8%40googlegroups.com.

[apollinaire samiey]

i already tried it using this documentation but in my cms common.py file i don't have the settings as "ENABLE_COURSEWARE_INDEX" or "ENABLE_LIBRARY_INDEX" and i don't have "SEARCH_ENGINE" too. The same with the lms common.py file, i don't have settings for edx-search configuration.


I'm using beta-release of open edx


this is the box vm i used to install my open edx mitxvm-edx-platform-02sep13a.box


and the link of the official page is http://people.csail.mit.edu/ichuang/edx/


--
You received this message because you are subscribed to a topic in the Google Groups "General Open edX discussion" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/edx-code/CABCzmJKLa6_KyWXBDA-ktoz25aXr2On%2BprwmSO-A4NA-%3DzSyJw%40mail.gmail.com.

[Peter Pinch]

Apollinaire, that VM is quite old -- it predates edX's first named release. 

The current search feature won't work with it. Have you considered upgrading?

Peter Pinch |  Associate Director of Engineering, Office of Digital Learning

Massachusetts Institute of Technology

One Main Street  |  Cambridge MA 02142

E pdp...@mit.edu | T 617-253-6256 | C 617-652-0183

To view this discussion on the web visit https://groups.google.com/d/msgid/edx-code/CAA6wd2%3D07VFS6ggnL_AwUpqJ9rCexZKgxOcr7esrD3tg8Q5_CQ%40mail.gmail.com.

[apollinaire samiey]

I didn't know that we could update this VM. How can i update it??

To view this discussion on the web visit https://groups.google.com/d/msgid/edx-code/CA9BBDF9-33DB-40AE-9A6C-1CA4EE3A87A0%40gmail.com.

[Peter Pinch]

Unfortunately, there's no documented process for upgrading it. 

If you don't need the student data to be migrated, I would recommend exporting the courses and importing them into a new (dogwood) instance of edX. The export OLX has been relatively stable and any compatibility issues can be addressed by editing the XML and JSON. 

- Peter

[apollinaire samiey]

ok thanks

--
You received this message because you are subscribed to a topic in the Google Groups "General Open edX discussion" group.

To view this discussion on the web visit https://groups.google.com/d/msgid/edx-code/24BC9BFC-653A-4D4A-9B18-0AEFB9B17D45%40mit.edu.

[apollinaire samiey]

Hello!!
finaly i installed the bitnami open edx on my linux host.
i used edx search configuration and created some courses but i can't foundany  one of them and the refine search is empty when i try to search a courses. Please help me with course_discovery configuration

[moh...@cloudypedia.com]

Did you manage to do it ?