Help us test FoxTrot Search 7's performance increases

[CTM Development - FoxTrot Search]

Dear FoxTrot-Search customers,

Following many months of rewriting critical performance bottlenecks, we are pleased to release for testing to our group members FoxTrot version 7.0b2. 

If you want to help us out, kindly do the following before upgrading to 7.0b2:

We initially only ask that you rebuild  (not just update) one of your most commonly used index files with the existing version 6.6.3, and send us the indexer log file (which you can access from the Manage Indices window in Foxtrot Pro)

Ideally, this would be an index of a somewhat substantial size, one which would take 30 minutes or more to rebuild.

Please send the resulting logs NOT to this list, but instead to our support mail address by opening a Support case from the Help menu. 

In the subject, after the case number, type the word PERFORMANCE, and we will track this afterwards.

Once this is done, you may proceed to download the FoxTrot Search 7.0 beta 2 seeds, depending on your interest, either:

FoxTrot Professional Search: http://www.ctmdev.com/beta/foxtrot_professional.dmg

FoxTrot Search Server: http://www.ctmdev.com/beta/foxtrot_search_server.dmg

FoxTrot Personal Search: http://www.ctmdev.com/beta/foxtrot_personal.dmg

To contribute a report (strongly encouraged !), please use the Help: Report menu, or if you find the issue worthy of other’s attention, use the discussion group. The first report should be a version 7 rebuild of the same index as above.

THANK YOU !

The FoxTrot-search team

at CTM Development SA

PS: As a motivational hint, depending on configurations, we have seen indexing performance increases ranging between 180% and 700% in version 7 !

--- 

FoxTrot Pro 7.0b2 release notes

▾ Performance improvements

• Much improved indexing time on multicore machines, especially when the index is stored on an SSD. This is however highly dependent on the type and size of the indexed files, and on the hardware configuration.

• 7.0b2 makes a smarter use of available processor cores when extracting indexable text.

▾ New search features

• [Pro] Added an “All items of type” criterion to find all indexed files of a given file type (or even all indexed files, regardless of their file type), without searching for any particular text string. This can also be used in conjunction with the filters, to perform advanced filename, date, or regular expression searches.

• Added a “filename extension” categorizer that can be used in addition to “item kind”.

• Added a visible / invisible categorizer.

▾ Other searching or indexing improvements

• Added more options to decide whether to index or not files in packages, applications and frameworks.

▾ User interface improvements

• Dark mode support (macOS 10.14 Mojave and later).

• [Pro] The proximity slider now shows a more precise tooltip.

• [Pro] When multiple search criterion fields are visible in the search window, empty ones are now always ignored (you previously had to remove unused ones).

• [Server] Added contextual menu to reset login info (username and password) for a given server, in FoxTrot Admin.

• Added a “Search…” menu item, with command-L shortcut, to move the keyboard focus to the search field.

• The contextual menu “reveal in finder” now shows the complete path as a submenu.

▾ Miscellaneous improvements

• Use a modern API for language identification on macOS 10.13 and later.

• Highlighting found occurrences did not work for documents created with recent versions of Pages, Numbers and Keynote.

• Added a hidden preference to use the NaturalLanguage framework for language identification on macOS 10.14 and later (defaults write com.ctmdev.FoxTrot UseNaturalLanguageIdentification -bool YES). This is slower, but may be more accurate.

• [Pro / Server] More secure network encryption (TLS 1.3) for shared indices.

• Estimation of the remaining indexing time is now more reliable.

• Some files could previously be added to the blacklist, although they had made the extractor crash only once; some third-party extractors can crash randomly, even when processing valid files.

• Indexer log files now contain much less useless information.

▾ Bug fixes

• The indexer could become unresponsive, and could refuse to stop, when compacting a very large index stored on a hard drive.

▾ System requirements

• macOS 10.12 Sierra or later.

[Xiangping Liu]

Thanks for the invitation. But I have been busy with my work. I will spare some time late to help you. Your product is amazing. Hope you will do better.

Xiangping Liu

School of Geography

Beijing Normal University

No. 19 Xin Jie Kou Wai Street, Haidian District,

Beijing, 100875, P.R. China 

CTM Development - FoxTrot Search <ctm-...@ctmdev.com> 于2019年11月29日周五 上午4:58写道：

--

---
You received this message because you are subscribed to the Google Groups "foxtrot-search" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foxtrot-searc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/foxtrot-search/3FC8DCAD-2A02-4DD9-8645-DD09E756C455%40ctmdev.com.

[Ajk Sanders]

I'm very impressed by the speed of cataloguing with the new beta.

[Darren Ingram]

Yes, it made a difference on a very large index I have (from over a week to just under a day). If this is the "big bang" for v7 I'd love to see a nicer display of data for v8.

CTM should be very pleased with the hard work they have undertaken to make these changes.

Darren

On Wed, 11 Dec 2019 at 12:52, 'Ajk Sanders' via foxtrot-search <foxtrot...@googlegroups.com> wrote:

I'm very impressed by the speed of cataloguing with the new beta.

--

---
You received this message because you are subscribed to the Google Groups "foxtrot-search" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foxtrot-searc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/foxtrot-search/107cda55-d23d-4a26-a391-0348c3428142%40googlegroups.com.

[Foxtrot Engineering]

> Darren Ingram wrote:
>
> Yes, it made a difference on a very large index I have (from over a week to just under a day). If this is the "big bang" for v7 I'd love to see a nicer display of data for v8.

Please feel free to elaborate on this: what kind of “nicer display” would you love to see?

Thanks for your feedback

Jérôme - Foxtrot Engineering

[Foxtrot Engineering]

> Darren Ingram wrote:
>
> Yes, it made a difference on a very large index I have (from over a week to just under a day).

Did you manage to store this very large index on an SSD? Large index files (i.e. about 10 GB or more) are quite slow when stored on a hard drive; another option would be to create multiple indices instead of a large one, each being smaller than 10 GB, and build them one at a time.

Jérôme - Foxtrot Engineering

[Darren Ingram]

I tried once but "something" eat up the remaining free disk space. I suspected (without evidence) that FT used it up to update the indices (a sort of temporary file) and I have since not had time to try again and isolate it as a problem or not.

It is not possible to create smaller indices due to the underlying software used to create the source documents.

I will try and clear some space in my schedule to have another try.

Darren

--

---
You received this message because you are subscribed to the Google Groups "foxtrot-search" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foxtrot-searc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/foxtrot-search/3C952B68-81E4-4D6F-B338-B752B64151FA%40ctmdev.com.

[Ajk Sanders]

FWIW I have 12 folders on an external SSD catalogued. They total 1.9TB for 225,600 items.

The indexes are stored on the internal HD, which is a fusion drive (in the Library folder).

The indexes are 50GB in size, the biggest is 21GB and has indexed 111,500 items.

(I haven't yet tried rebuilding that one with the new beta).

Are you suggesting I would get better/faster results if all indexes were under 10GB?

[CTM Development - FoxTrot Search]

Hello,

The very first thing I would suggest is that you store your indices not on a fusion drive but on an SSD. That should make a difference.

And yes, remaining under 10GB per index, while not an exact value, will indeed produce better performance than 21GB.

Regards,

jean michel/ctm qa

On 12 Dec 2019, at 17:38, 'Ajk Sanders' via foxtrot-search <foxtrot...@googlegroups.com> wrote:



--

---
You received this message because you are subscribed to the Google Groups "foxtrot-search" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foxtrot-searc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/foxtrot-search/d478c3d4-e967-49f7-aff8-785d0f4f44e3%40googlegroups.com.

[Ajk Sanders]

Hi

I will keep the indices on the startup (Fusion) drive for now because I need to be able to search them when the SSD is not plugged in.

I have pared down the folders and now have more indices but smaller ones.

Thanks

On Thursday, 12 December 2019 17:28:27 UTC, foxtrot-search wrote:

Hello,

The very first thing I would suggest is that you store your indices not on a fusion drive but on an SSD. That should make a difference.

And yes, remaining under 10GB per index, while not an exact value, will indeed produce better performance than 21GB.

Regards,

jean michel/ctm qa

On 12 Dec 2019, at 17:38, 'Ajk Sanders' via foxtrot-search <foxtrot...@googlegroups.com> wrote:



FWIW I have 12 folders on an external SSD catalogued. They total 1.9TB for 225,600 items.

The indexes are stored on the internal HD, which is a fusion drive (in the Library folder).

The indexes are 50GB in size, the biggest is 21GB and has indexed 111,500 items.

(I haven't yet tried rebuilding that one with the new beta).

Are you suggesting I would get better/faster results if all indexes were under 10GB?

--

---
You received this message because you are subscribed to the Google Groups "foxtrot-search" group.

To unsubscribe from this group and stop receiving emails from it, send an email to foxtrot...@googlegroups.com.

[ctm-...@ctmdev.com]

Hello

If I may make a recommendation, it is that at the first time you build or rebuild indices you do so on the SSD. That will make the induction process much faster, then you can copy them over to your Fusion drive. Don’t forget to close the index before copying and re-open it from the new location after the copy is done.

regards

jean michel / ctm qa

On Dec 17, 2019, at 5:35 PM, 'Ajk Sanders' via foxtrot-search <foxtrot...@googlegroups.com> wrote:



To unsubscribe from this group and stop receiving emails from it, send an email to foxtrot-searc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/foxtrot-search/ab9c7b56-1860-452c-9405-d8b2068337ad%40googlegroups.com.