Performance results of CIF v1 RC1-03

Showing 1-9 of 9 messages
Performance results of CIF v1 RC1-03 Gabriel Iovino 10/5/12 6:44 AM
[Setup]

 - Virtual machine (4 core, 4GB ram)
 - cif_smrt, cif_router and the postgres DB are all on the same server
 - No other load / processes running on the host
 - These numbers were calculated with no post-processing (e.g. no dns resolution) 
 - This was run with 16 threads, 32 threads was a little slower due to thread overhead

[Dataset]

wc -l random.csv
    45000 random.csv

head random.csv
    http://F0S3M069MNWVB5FZJ.CN/,this is a url,2012-10-04T16:48:47
    180.24.254.1,this is an IP,2012-10-05T01:37:46
 
[CIF]
 
time ./bin/cif_smrt -r /opt/cif/etc/random.cfg -f random -d -t 16 -A root -N 1
     real    5m36.945s

[Analysis]

45,000 indicators @ 336s

45,000 / 336 = 133 records per sec
133 * 86,400 = 11,491,200 records per day
365 * 11,491,200 = 4,194,288,000 records per year

[Commentary]

A single host CIF installation could help you aggregate and normalize 4 
billion threat observations a year. No queries or feeds were being 
generated nor was post-processing (analytics) performed while generating 
these statistics. Adding additional hosts running cif_smrt should allow 
this to scale until you make the database fall over.

Gabe
Re: [ci-framework] Performance results of CIF v1 RC1-03 wes 10/5/12 7:15 AM
Gah. Still too slow.. Needs tweaking..

--
wes
Sent from my iPhone
--
http://code.google.com/p/collective-intelligence-framework/wiki/FAQ
http://code.google.com/p/collective-intelligence-framework/wiki/CommunityRules
---
You received this message because you are subscribed to the Google Groups "ci-framework" group.
Visit this group at http://groups.google.com/group/ci-framework?hl=en.
 
 
Re: [ci-framework] Performance results of CIF v1 RC1-03 mcholste 10/5/12 7:54 AM
I dare you guys to run that against cif-rest-sphinx.  It should
complete in under 10 seconds.
Re: [ci-framework] Performance results of CIF v1 RC1-03 wes 10/5/12 7:56 AM
It's on my list of things to look into, but cause we moved to pb instead of jason, might take some work.

--
wes
claimid.com/wesyoung
Sent from my iPhone

Re: [ci-framework] Performance results of CIF v1 RC1-03 Doug Burks 10/5/12 8:01 AM
I hear a triple-dog-dare coming on!  :)
Doug
Doug Burks
http://securityonion.blogspot.com
Re: [ci-framework] Performance results of CIF v1 RC1-03 Dave Dittrich 10/5/12 1:20 PM
On Fri, Oct 5, 2012 at 7:54 AM, Martin Holste <mcho...@gmail.com> wrote:
> I dare you guys to run that against cif-rest-sphinx.  It should
> complete in under 10 seconds.

Yeah, that is really the way to go.  I just finished a script that
runs through a list of IPs and uses cif-rest-sphynx to see if they
are/are not known to CIF.  Here are the results:

Number of IP addresses in list:
$ wc -l all-ips.txt
354 all-ips.txt

Script that queries CIF via perl client:

$ cat test.sh
#!/bin/bash
cat all-ips.txt | while read ip
do
        cif -n -q $ip
done

$ time bash test.sh > /dev/null
85.95user 10.34system 11:03.74elapsed 14%CPU (0avgtext+0avgdata
118416maxresident)k
4248inputs+0outputs (30major+2248630minor)pagefaults 0swaps


Now script that uses lynx to query cif-rest-sphynx, showing total
number of successful queries, how many have at least 1 record in CIF,
and how many have no records at all:

$ time ./cifcheck -v `cat all-ips.txt` | wc -l
9.39user 2.25system 0:58.53elapsed 19%CPU (0avgtext+0avgdata 41872maxresident)k
6560inputs+16outputs (30major+1388283minor)pagefaults 0swaps
349
$ ./cifcheck --min=1 `cat all-ips.txt` | wc -l
63
$ ./cifcheck -v --misses `cat all-ips.txt` | wc -l
286


--
Dave Dittrich
dave.d...@gmail.com
Re: [ci-framework] Performance results of CIF v1 RC1-03 wes 10/5/12 3:15 PM
This is insertion rate.. Fwiw.

--
wes
claimid.com/wesyoung
Sent from my iPhone

On Oct 5, 2012, at 10:54, Martin Holste <mcho...@gmail.com> wrote:

Re: [ci-framework] Performance results of CIF v1 RC1-03 wes 10/5/12 3:16 PM
Haven't done any write perf stuff yet, hence the tweaking comment...

--
wes
claimid.com/wesyoung
Sent from my iPhone

On Oct 5, 2012, at 10:54, Martin Holste <mcho...@gmail.com> wrote:

Re: Performance results of CIF v1 RC1-03 Gabriel Iovino 10/9/12 10:55 AM
On Friday, October 5, 2012 9:44:04 AM UTC-4, Gabriel Iovino wrote:
[Analysis]

45,000 indicators @ 336s

45,000 / 336 = 133 records per sec
133 * 86,400 = 11,491,200 records per day
365 * 11,491,200 = 4,194,288,000 records per year


I believe these numbers would give you _very_ rough sizing estimates on what one might expect in terms of disk space:

45,000           * 1.5 kB = 65.91 MB
11,491,200     * 1.5 kB = 16.43 GB
4,194,288,000 * 1.5 kB =   5.85 TB

In future versions when we add the capability to store malware binaries and associated sandbox reports, we'll have an entirely new disk space conversation. 

Gabe