how to make full use of the cpu when running a multithreaded program on multi core

[blackli...@gmail.com]

I am testing the recognition speed with caffe on the ubuntu .there is the result.

computer or server	type	all time	average
i5 4core, 4 threads, 3.2ghz	Single thread, 100 pictures	36.5s	365ms
i5 4core, 4 threads, 3.2ghz	Single thread, 1000 pictures	305.0s	305ms
i5 4core, 4 threads, 3.2ghz	5 threads, 100 pictures/threads	77s	154ms
i5 4core, 4 threads, 3.2ghz	5 threads, 100 pictures/threads	155.8s	156ms
E2650, 20-core 40threads, 2.3ghz（server）	Single thread, 100 pictures	195.0s	2000ms
E2650, 20-core 40threads, 2.3ghz（server）	Single thread, 1000 pictures	too long
E2650, 20-core 40threads, 2.3ghz（server）	10 threads，100 pictures/threads	1132s	1132ms

Whether because the Multi core task scheduling is not good enough？if so ,what should I do to make this more  effective.