whoa! SFM builds on OS X without any fiddling!

[Tom Lee]

The superfastmatch bootstrap script works out of the box on OS X 10.6.8! Wow. What a pleasant surprise!

I haven't yet played around with the instance I've got running, but it's thrilling to see this work so easily with yesterday's commits.

(superfastmatch)toms-macbook:superfastmatch $ make check

g++ -I./src -I./tests -I/usr/local/include/ -Itests/utils/ -c -Wall -Wextra -funsigned-char -m64 -march=core2 -O3 -g -O0 tests/utils/gmock-gtest-all.cc -o tests/utils/gmock-gtest-all.o

g++ -I./src -I./tests -I/usr/local/include/ -Itests/utils/ -MM -Wall -Wextra -funsigned-char -m64 -march=core2 -O3 -g -O0 tests/utils/gmock-gtest-all.cc > tests/utils/gmock-gtest-all.d

ar rv gmock-gtest.a tests/utils/gmock-gtest-all.o

ar: creating archive gmock-gtest.a

a - tests/utils/gmock-gtest-all.o

g++ -I./src -I./tests -I/usr/local/include/ -Itests/utils/ -lpthread -Wall -Wextra -funsigned-char -m64 -march=core2 -O3 -g -O0 src/postline.cc tests/postline-unittest.cc gmock-gtest.a -o tests/postline-unittest 

g++ -I./src -I./tests -I/usr/local/include/ -Itests/utils/ -I./src -I./tests -I/usr/local/include/ -Itests/utils/ -lpthread -lkyotocabinet -lctemplate -Wall -Wextra -funsigned-char -m64 -march=core2 -O3 -g -O0 src/document.cc src/association.cc tests/document-unittest.cc gmock-gtest.a -o tests/document-unittest

src/association.cc:253: warning: unused parameter ‘dict’

g++ -I./src -I./tests -I/usr/local/include/ -Itests/utils/ -I./src -I./tests -I/usr/local/include/ -Itests/utils/ -lpthread -lkyotocabinet -lkyototycoon -lctemplate -Wall -Wextra -funsigned-char -m64 -march=core2 -O3 -g -O0 src/document.cc src/association.cc src/logger.cc tests/association-unittest.cc gmock-gtest.a -o tests/association-unittest

src/association.cc:253: warning: unused parameter ‘dict’

g++ -I./src -I./tests -I/usr/local/include/ -Itests/utils/ -I./src -I./tests -I/usr/local/include/ -Itests/utils/ -lpthread -lkyotocabinet -lkyototycoon -lctemplate -Wall -Wextra -funsigned-char -m64 -march=core2 -O3 -g -O0 src/document.cc src/posting.cc src/logger.cc src/association.cc src/postline.cc src/command.cc tests/posting-unittest.cc gmock-gtest.a -o tests/posting-unittest

src/association.cc:253: warning: unused parameter ‘dict’

g++ -I./src -I./tests -I/usr/local/include/ -Itests/utils/ -I./src -I./tests -I/usr/local/include/ -Itests/utils/ -lpthread -lkyotocabinet -lkyototycoon -lctemplate -Wall -Wextra -funsigned-char -m64 -march=core2 -O3 -g -O0 src/document.cc src/posting.cc src/logger.cc src/association.cc src/postline.cc src/command.cc tests/benchmark.cc gmock-gtest.a -o tests/benchmark

src/association.cc:253: warning: unused parameter ‘dict’

tests/benchmark.cc: In member function ‘virtual void BenchmarkTest_SearchMapTest_Test::TestBody()’:

tests/benchmark.cc:14: warning: unused variable ‘tally’

tests/postline-unittest; tests/document-unittest; tests/association-unittest; tests/posting-unittest; tests/benchmark;

[==========] Running 6 tests from 1 test case.

[----------] Global test environment set-up.

[----------] 6 tests from PostLineTest

[ RUN      ] PostLineTest.VarIntCodecHeaderTest

[       OK ] PostLineTest.VarIntCodecHeaderTest (0 ms)

[ RUN      ] PostLineTest.VarIntCodecSectionTest

[       OK ] PostLineTest.VarIntCodecSectionTest (0 ms)

[ RUN      ] PostLineTest.PostLineTest

[       OK ] PostLineTest.PostLineTest (52 ms)

[ RUN      ] PostLineTest.BadParametersPostLineTest

[       OK ] PostLineTest.BadParametersPostLineTest (0 ms)

[ RUN      ] PostLineTest.RealisticPostLineTest

[       OK ] PostLineTest.RealisticPostLineTest (0 ms)

[ RUN      ] PostLineTest.BigPostLineTest

[       OK ] PostLineTest.BigPostLineTest (58 ms)

[----------] 6 tests from PostLineTest (174 ms total)

[----------] Global test environment tear-down

[==========] 6 tests from 1 test case ran. (174 ms total)

[  PASSED  ] 6 tests.

[==========] Running 7 tests from 2 test cases.

[----------] Global test environment set-up.

[----------] 1 test from DocumentDeathTest

[ RUN      ] DocumentDeathTest.RemovalTest

[       OK ] DocumentDeathTest.RemovalTest (678 ms)

[----------] 1 test from DocumentDeathTest (678 ms total)

[----------] 6 tests from DocumentTest

[ RUN      ] DocumentTest.ConstructorTest

[       OK ] DocumentTest.ConstructorTest (62 ms)

[ RUN      ] DocumentTest.CleanTextTest

[       OK ] DocumentTest.CleanTextTest (24 ms)

[ RUN      ] DocumentTest.MetaTest

[       OK ] DocumentTest.MetaTest (32 ms)

[ RUN      ] DocumentTest.ManagerTest

[       OK ] DocumentTest.ManagerTest (59 ms)

[ RUN      ] DocumentTest.DuplicatePermanentTest

[       OK ] DocumentTest.DuplicatePermanentTest (26 ms)

[ RUN      ] DocumentTest.InitTest

[       OK ] DocumentTest.InitTest (51 ms)

[----------] 6 tests from DocumentTest (255 ms total)

[----------] Global test environment tear-down

[==========] 7 tests from 2 test cases ran. (933 ms total)

[  PASSED  ] 7 tests.

[==========] Running 3 tests from 1 test case.

[----------] Global test environment set-up.

[----------] 3 tests from AssociationTest

[ RUN      ] AssociationTest.ConstructorTest

[       OK ] AssociationTest.ConstructorTest (72 ms)

[ RUN      ] AssociationTest.WhitespaceTest

[       OK ] AssociationTest.WhitespaceTest (50 ms)

[ RUN      ] AssociationTest.EndingTest

[       OK ] AssociationTest.EndingTest (49 ms)

[----------] 3 tests from AssociationTest (171 ms total)

[----------] Global test environment tear-down

[==========] 3 tests from 1 test case ran. (171 ms total)

[  PASSED  ] 3 tests.

[==========] Running 1 test from 1 test case.

[----------] Global test environment set-up.

[----------] 1 test from PostingTest

[ RUN      ] PostingTest.SearchTest

[       OK ] PostingTest.SearchTest (314 ms)

[----------] 1 test from PostingTest (314 ms total)

[----------] Global test environment tear-down

[==========] 1 test from 1 test case ran. (314 ms total)

[  PASSED  ] 1 test.

[==========] Running 1 test from 1 test case.

[----------] Global test environment set-up.

[----------] 1 test from BenchmarkTest

[ RUN      ] BenchmarkTest.SearchMapTest

[       OK ] BenchmarkTest.SearchMapTest (1337 ms)

[----------] 1 test from BenchmarkTest (1338 ms total)

[----------] Global test environment tear-down

[==========] 1 test from 1 test case ran. (1339 ms total)

[  PASSED  ] 1 test.

[Donovan Hide]

Excellent! You may have just nominated yourself as the official Snow Leopard tester (I've moved onto Lion)! Try "make run" next!

Rest assured I'm working hard towards a successful demo for the 8th. Currently the most visible change should be much faster associations, and it's still running on a single thread, so the quad core speed should be very pleasant indeed! Also whitespace ignorance and trimming should give better looking and less noisy results.

The two most useful upcoming changes are ordering documents by metadata (longest association fragment,number of associated documents, congress session,etc.) and correct paging of documents and the queue items. This has required a fairly major refactor and that's what I've been busy with. The code is starting to get cleaner and there are lots more tests.

I've been visualising the association results in my mind and perhaps actually visualising them on the screen might be a useful next step! The "From Position", "To Position" and "Length" values present in the the document page and search results lend themselves to scatter plotting with:

From = x

To = y

Length = spot radius

I'm imagining that plagiarism/churn/copy will look like diagonal lines heading upper-rightwards, and it will be easy to see clusters of fragments becoming apparent. I think once we see it, it will make intuitive sense and help in navigating results. Example here (will work much like the histograms):

http://code.google.com/apis/chart/interactive/docs/gallery/scatterchart.html

Is Wednesday a good day for another conference call?

Cheers,

Donny.

[Tom Lee]

Sure, Wednesday could work -- is 11AM ET okay?

[Donovan Hide]

Good for me! Will check with Martin tomorrow too.

[Tom Lee]

Actually, is there any chance that tomorrow morning could work?  Just realizing that Wednesday is filling up fast.  If not, no big deal, Wednesday will work fine.

[Donovan Hide]

Fine with me, same time?

[Thomas Lee]

Yup. Talk to you then.