Execution time of DROID profile mode

49 views
Skip to first unread message

Jerry Kuang

unread,
May 23, 2016, 4:20:14 PM5/23/16
to droid-list
Hi Brian,

I run DROID profile mode in command line mode again 5000 files and 1 files. I find that execution time for each of them doesn't differ too much no matter how many files the DROID runs against. I just wonder what the major reason for this. Such as is it because that the initialization time of DROID profile mode takes longer time (10 seconds ? )and the identification procedure just takes 1 to 4 seconds when the working load is around 1 to 5000 files?

I test the DROID profile mode in both windows and linux. The results are as below.

In windows system, it takes 13 seconds to complete the file type identification procedure when it runs against only 1 file. When running against 5000 files, it takes 18 seconds to complete the procedure.

In linux system, it takes 22 seconds to complete the file type identification procedure when it runs against only 1 file. When running against 92 files, it also takes 22 seconds to complete the procedure.

PS: What I mean about the running procedure of droid profile mode is to get the profile, export the profile, and open the csv file to get what we want.

Thank you,
Jerry

Brian O'Reilly

unread,
Jun 17, 2016, 10:26:24 AM6/17/16
to droid-list
Hi Jerry
Firstly, apologies for not responding earlier, I've been away from the office for the last few weeks.

Performance monitoring is a challenging area and it can be difficult to pin down exactly what is going on in a given situation.  Apart from DROID itself, much will depend on various factors such as the specification of the host machine, other processes currently running, types and sizes of files, whether they are on local or networked drives, and (in the latter case) the network speed. Changing DROID options such as the number of bytes to scan and use of hashing will also play a role. That said, with file numbers around the 5000 mark I'm not surprised that you don't see a noticeable difference in processing times when compared to a single file.  As you point out, the initialisation time for the DROID profile is a major factor - this overhead will apply both in the command line profile mode and in the GUI, though not in the command line "no profile" mode.  Clearly, the percentage time overhead of initialisation will decrease with as the number of files increases (other things being equal), but you may need to process several thousand files before you see a marked difference.

Testing of the latest release included regular runs over a set of around 70,000 files, mostly using the GUI version running on Windows 7.  This did prove that the current release is intrinsically faster than the previous one.  It is also worth noting that the best performance times were achieved on repeat runs over the same set of files on a local drive.  This can be attributed to the operating system caching the files when read from disk - we did not see this behaviour when testing with network files, where the application would need to retrieve the files over the network on each repeat request.

The CSV export is also much faster in the latest version - this will be most noticeable with very large profiles consisting of hundreds of thousands of files.

Hope this helps,
Regards, Brian
Reply all
Reply to author
Forward
0 new messages