Field-level search on manual import (new feature added)

[Serdar Tumgoren]

Hey everyone,

Wanted to send the heads up about a new feature that allows you to add field-level search using the manual_import (a.k.a. bulk import) management command.

As you all know, the standard method for adding field-level search to a dataset is to first upload it, and then re-index the data after specifying additional fields to index on. This works great for reasonably sized data sets. However, we ran into serious performance issues when re-indexing a data set that was about 18 million rows. I'm planning to send a message this week detailing those performance issues, since others may encounter them at some point and the core devs (or others) may have advice. Would be good to hash this out now that we've fixed the data set size bug, which makes it easier to ingest increasingly large files (this is what exposed the issue on my end).

Meantime, the code update has been folded in with this pull request. Once again, if you want/need the change right away, you can pull from the master branch on my repo.

Documentation for the new feature is here. Let me know if you have any questions. And please keep in mind that the pull request is outstanding and hasn't received any feedback yet, so this change may or may not ultimately be merged with PANDA core. You've been warned :)

Best,

Serdar