Config files repository

Showing 1-44 of 44 messages
Config files repository Kyle Maxwell 9/11/12 2:29 PM
Thinking out loud here, and I'm sick with a cold or sinus infection or
something equally miserable, so please forgive any ambiguous
communication.

I think it might be useful to maintain a separate "bleeding edge"
repository of configuration files (/opt/cif/etc/*.cfg) that people can
pull from individually if they like. This could eventually feed into
the core cif-smrt repository
(https://github.com/collectiveintel/cif-smrt) after we've vetted
things out and then the QC team at REN-ISAC or wherever blesses it as
well.

Do people think this would work better than relying exclusively on the
mailing list for this? Our private repository is up to 49 separate
feeds now, not counting whitelists or internal feeds from our own
systems, and I'd love to share these more rapidly. Great work like
DFIR N00B and others could be submitted via pull requests (assuming
Github usage), and then over time Wes et al. can cherry-pick the bits
they like and build that into the stable distribution.

If people think this is a good idea, I'll publish out our full
repository tomorrow.

--
Kyle Maxwell [krma...@gmail.com]
http://www.xwell.org
Twitter: @kylemaxwell
Re: [ci-framework] Config files repository mcholste 9/11/12 2:36 PM
I think that's a good game plan.
> --
> http://code.google.com/p/collective-intelligence-framework/wiki/FAQ
> http://code.google.com/p/collective-intelligence-framework/wiki/CommunityRules
> ---
> You received this message because you are subscribed to the Google Groups "ci-framework" group.
> Visit this group at http://groups.google.com/group/ci-framework?hl=en.
>
>
Re: [ci-framework] Config files repository Brent Wrisley 9/11/12 3:07 PM
I think it's a great idea to publish these.

Sent from my iPhone
Re: [ci-framework] Config files repository wes 9/11/12 3:41 PM
[i thought i wrote this down somewhere, maybe not…]

ya, for sure.. it's actually on my list of things to do this week (after our last thread about it)… sort of an "ubuntu repo meets emerging threats" for rule configs, following that model, something between that and an "official bleeding edge" branch that you can point at where a team of peep's can maintain the pull req's (i'm assuming kyle is volunteering here ;)).

in addition to that, i was gonna make cif-smrt configurable to different repo's, so if you didn't like the "official repo" (or wanted to add another 3rd party, private repo) it's easier to do (we do it today internally, but it's a PITA.. needs to be cleaner). that way you can pick the approach you want.

i'll have a mock-up (and split out the current cif-smrt) by week's end and we can go from there. i hate being the bottle-kneck here…

On 2012-09-11, at 5:29 PM, Kyle Maxwell wrote:

> Do people think this would work better than relying exclusively on the
> mailing list for this? Our private repository is up to 49 separate
> feeds now, not counting whitelists or internal feeds from our own
> systems, and I'd love to share these more rapidly. Great work like
> DFIR N00B and others could be submitted via pull requests (assuming
> Github usage), and then over time Wes et al. can cherry-pick the bits
> they like and build that into the stable distribution.
>
> If people think this is a good idea, I'll publish out our full
> repository tomorrow.

--
Wes
wesyoung.me

Re: [ci-framework] Config files repository wes 9/12/12 2:32 PM
quick follow-through:

i created:

https://github.com/collectiveintel/cif-smrt-rules

merged kyles initial pull req:

https://github.com/collectiveintel/cif-smrt/pull/19

into the v0-test and master-test branches:

https://github.com/collectiveintel/cif-smrt-rules/tree/v0-test
https://github.com/collectiveintel/cif-smrt-rules/tree/master-test

i haven't yet back-ported the v0-test to:

https://github.com/collectiveintel/cif-v0/tree/master/etc

but will try to circle back with that later in the week (just wanted to get the framework laid out). i also linked cif-smrt-rules under the cif-smrt repo:

https://github.com/collectiveintel/cif-smrt

the general idea (for now, we can evolve it as it goes):

+ cif-smrt-rules will be the main repo for rules, it can be forked, modified, etc w/o having to touch the cif-smrt repo
+ master/master-test will be the 'in-dev out of the box main/test rules', we'll probably bundle both and just turn off "test" by default
+ vx/vx-test will be legacy versions of the repo

so (for example) if you wanted to pull-req something for v0:

+ fork cif-smrt-rules
+ git checkout v0-test
+ modify code, perform commit
+ submit pull req to the cif-smrt-rules repo

we'll try to merge those into v0-test and master-test quickly, allowing others to check them out more rapidly for testing

as feed configs become "QA'd" (whatever that means, community consensus of sorts) we'll merge stuff from -test to the stable branches for "out of the box functionality".

outside of that, you're obviously more than welcome to fork the entire repo internally and us it to make your own private branches that never need to be committed back. over time we'll get cif-smrt to be smrt'er at being able to auto pull the main repo for updates so the update process is less painful (or more painful if you decide to auto-pull the -test feed and it breaks something, your choice :)).

more to come, stay tuned...

(i'm sure it's all crystal clear how this is gonna work and we don't have to write any doc for it to make sense (but we will anyway, just for kicks).)
Wes
wesyoung.me

Re: [ci-framework] Config files repository kl 9/14/12 11:59 AM
AlienVault malware feed that was added.

Hi there,

When I run the following on cif v0.1 nothing is returned.

root@ip-10-2-213-153:/opt/cif/etc# /opt/cif/bin/cif_feedparser -c /opt/cif/etc/alienvault.cfg -f malware -T low -d
running 1 threads at /opt/cif/bin/cif_feedparser line 92.
root@ip-10-2-213-153:/opt/cif/etc# 

thanks
Re: [ci-framework] Config files repository wes 9/17/12 3:58 AM
right now v0 is going by the old rules:

https://github.com/collectiveintel/cif-v0/blob/master/etc
https://github.com/collectiveintel/cif-v0/blob/master/etc/alienvault.cfg

and the malware category hasn't been back-ported yet. if you want you can try this in your alienvault.cfg for the time being (it would be overwritten by an upgrade):

[av_malware]
regex = '^(\S+)\s#\sMalware IP;RBN'
regex_values = 'address'
impact = 'malware'
description = 'rbn'
confidence = 65
period = daily

see how that works.

i'm gonna try to get some of this back-ported to v0 in the following week or two, with a maint release to v0 fixing some other bugs.

On Sep 14, 2012, at 2:59 PM, kl wrote:

> When I run the following on cif v0.1 nothing is returned.
>
> root@ip-10-2-213-153:/opt/cif/etc# /opt/cif/bin/cif_feedparser -c /opt/cif/etc/alienvault.cfg -f malware -T low -d
> running 1 threads at /opt/cif/bin/cif_feedparser line 92.
> root@ip-10-2-213-153:/opt/cif/etc#

--
Wes
wesyoung.me

Re: [ci-framework] Config files repository kl 9/17/12 4:11 AM
Hi
Thanks, I'll check

Is there a chance that /opt/cif/bin/cif_feeds is having trouble running (gets killed or stuck) if I added some of the new cfg files from here  https://github.com/collectiveintel/cif-smrt-rules/tree/v0-test/etc  and confidence 85 to vabl.cfg?

Thank u!--
Best,
Shiran Kleiderman
+972  - 542380838
Skype - shirank1

Re: [ci-framework] Config files repository kl 9/17/12 4:36 AM
hi
attaching an image of where cron /opt/cif/bin/cif_feeds stops

thanks (:
Re: [ci-framework] Config files repository wes 9/17/12 4:57 AM
hrm, i wonder if you're running out of memory... killed usually means the kernel got a-hold of it for some reason and squashed it.

is there anything in /var/log/syslog or /var/log/messages ?

On Sep 17, 2012, at 7:36 AM, kl wrote:

> attaching an image of where cron /opt/cif/bin/cif_feeds stops

--
Wes
wesyoung.me

Re: [ci-framework] Config files repository kl 9/17/12 5:00 AM
Thanks
I'll check once the cron stops (I tried running it again).
Could this happen because I added the cfg files (nixspam, etc)??
They also increased the query times for feeds so I removed them in the meantime.
Re: [ci-framework] Config files repository wes 9/17/12 5:05 AM
i haven't had a chance to test some of these yet, so i can't guarantee anything at the moment. Kyle did some really good work on these, but i only had time to proof read them, haven't had the cycles to run them in a test instance yet.

if you have the cycles, you're more than welcome to test them out, to do so i'd backup /opt/cif/etc first so you have an easy back-out option.

On Sep 17, 2012, at 7:11 AM, Shiran Kleiderman wrote:

> Is there a chance that /opt/cif/bin/cif_feeds is having trouble running (gets killed or stuck) if I added some of the new cfg files from here  https://github.com/collectiveintel/cif-smrt-rules/tree/v0-test/etc  and confidence 85 to vabl.cfg?

--
Wes
wesyoung.me

Re: [ci-framework] Config files repository wes 9/17/12 5:09 AM
yes, it's very possible. part of the reason these will go to v0/v1-test first is for this very reason. if we get a rather large feed, it's possible that under v0 it'll bog your system way down to the point where it's not usable. we've had to eliminate some feeds in the past for this reason [in v0].

i haven't had a chance to look at the sizes of any of those feeds, so i couldn't say for sure which one, i know the scan feed in alienvault is a good example that we had to disable in places due to it's size for now. so if you track down which one is causing the pain, let us know and we'll add a warning to the config and disable it by default.

this will be less of an issue in v1 fwiw.

On Sep 17, 2012, at 8:00 AM, kl wrote:

> I'll check once the cron stops (I tried running it again).
> Could this happen because I added the cfg files (nixspam, etc)??
> They also increased the query times for feeds so I removed them in the meantime.

--
Wes
wesyoung.me

Re: [ci-framework] Config files repository kl 9/17/12 5:14 AM
Hi
Right now I get the killed message on the feed that u can see in the file (I removed the new cfg files and I still get Killed).
Attaching what I get in syslog (I'm using a large Amazon instance and all was good till now).

Thanks (:
--
Best,
Shiran Kleiderman
+972  - 542380838
Skype - shirank1

Re: [ci-framework] Config files repository wes 9/17/12 5:28 AM
yea, but all that data you pulled in is still in the database, so cif_feeds is still trying to process it.

what does:

/etc/sysctl.conf

look like?

how much ram is in your large-instance?

what does your [cif_feeds] section of /home/cif/.cif look like?

what does:

select count(id) from archive where created >= '2012-09-17T00:00:00Z';

give you?

chances are you might have to manually purge some data and re-run your daily cron if you're not able to add more resources to your host.

if you had to purge some data (although i *wouldn't* recommend it just yet), you'd need to comment out *all* of your crons, make sure none of the perl scripts are running (ps aux | grep perl) and do something like:

$ sudo su - cif
$ psql -U postgres -d cif
cif=# BEGIN
cif-# delete from infrastructure where created >= '2012-09-17T00:00:00Z';
cif-# delete from domain where created >= '2012-09-17T00:00:00Z';
cif-# delete from url where created >= '2012-09-17T00:00:00Z';
cif-# delete from malware where created >= '2012-09-17T00:00:00Z';
cif-# delete from asn where created >= '2012-09-17T00:00:00Z';
cif-# delete from cc where created >= '2012-09-17T00:00:00Z';
cif-# delete from email        where created >= '2012-09-17T00:00:00Z';
cif-# delete from rir where created >= '2012-09-17T00:00:00Z';
cif-# delete from archive where created >= '2012-09-17T00:00:00Z';
cif-# COMMIT;

and then re-run your daily cron.

On Sep 17, 2012, at 8:14 AM, Shiran Kleiderman wrote:

> Right now I get the killed message on the feed that u can see in the file (I removed the new cfg files and I still get Killed).
> Attaching what I get in syslog (I'm using a large Amazon instance and all was good till now).

--
Wes
wesyoung.me

Re: [ci-framework] Config files repository kl 9/17/12 5:38 AM
Hi

1)
sysctl.conf is all commented... didn't see anything active there

2)
[cif_feeds]
maxrecords = 10000
severity_feeds = high,medium
confidence_feeds = 95,85,65,50
apikeys = role_everyone_feed
max_days = 5
disabled_feeds = hash,rir,asn,countrycode,malware

[client]
apikey = 
timeout = 60
verify_tls = 0

3)
cif=# select count(id) from archive where created >= '2012-09-17T00:00:00Z';
count-------96781(1 row)
--
Best,
Shiran Kleiderman
+972  - 542380838
Skype - shirank1

Re: [ci-framework] Config files repository kl 9/17/12 7:18 AM
Hi

These are the two processes that are working hard.
Both grow in memory usage and on the sql one the idle interchanges with SELECT and then idle again.

It's always killed on the scan 85% feed attached.
Is there a specific one that I can remove?

Thanks!
Re: [ci-framework] Config files repository kl 9/17/12 7:25 AM
Hi

These are the two processes that are working hard.
Both grow in memory usage and on the sql one the idle interchanges with SELECT and then idle again.

It's always killed on the scan 85% feed attached.
Is there a specific one that I can remove?

Thanks!

Re: [ci-framework] Config files repository wes 9/17/12 8:16 AM
are you using the av_scan feed in alienvault.cfg ?

On Sep 17, 2012, at 10:25 AM, kl wrote:

> It's always killed on the scan 85% feed attached.
> Is there a specific one that I can remove?

--
Wes
wesyoung.me

Re: [ci-framework] Config files repository kl 9/17/12 8:18 AM
I erased it before the last run.

The cron is running again and stuck in the same place, eating up memory.
Will expanding the Amazon instance work?


stuck here:
[role_everyone_feed] scan infrastructure medium severity 85% confidence private feed: 13340033 created -- a17e1f46-5b27-5819-9dd4-6b6452be8ddc
removing feed: 42910 -- 6e5c97a9-c740-5ad2-89d0-d23d091e3079

--
Best,
Shiran Kleiderman
+972  - 542380838
Skype - shirank1

Re: [ci-framework] Config files repository wes 9/17/12 8:22 AM
try this:

$ sudo su - cif
$ psql -U postgres -d cif

cif=# begin;
cif=# delete from infrastructure_scan where created >= '2012-09-17T00:00:00Z';
cif=# commit;

and then try re-running cif_feeds.

this should remove all the scan entries from the index table from today. then tomorrow when your crontool runs, it'll re-populate with the other scan data.

you could also re-run:

$ cif_crontool -p hourly -d

now that the test cfg's are removed, which would get back most of the other scan data for today (drg_ssh, etc) before re-running cif_feeds. your call on that. if you do this, make sure do you a test "cif_feeds -d" after pruning the infrastructure_scan table to make sure this clears the problem up first.

On Sep 17, 2012, at 11:18 AM, Shiran Kleiderman wrote:

> The cron is running again and stuck in the same place, eating up memory.
> Will expanding the Amazon instance work?

--
Wes
wesyoung.me

Re: [ci-framework] Config files repository Kevin Benton 9/17/12 8:24 AM
If it is just running out of memory, you could also try adding a swapfile if you don't want to pay for the bigger instance. The performance will suffer but it should work.

Here is a how-to. Scroll to the part that says "Four-step Process to Add Swap File".


--
http://code.google.com/p/collective-intelligence-framework/wiki/FAQ
http://code.google.com/p/collective-intelligence-framework/wiki/CommunityRules
---
You received this message because you are subscribed to the Google Groups "ci-framework" group.
Visit this group at http://groups.google.com/group/ci-framework?hl=en.
 
 



--
Kevin Benton

Re: [ci-framework] Config files repository kl 9/17/12 8:24 AM
Hi
Thanks.

The problem started on September 14.
Can I run what you wrote from that date?

Thank u (:
--
Best,
Shiran Kleiderman
+972  - 542380838
Skype - shirank1

Re: [ci-framework] Config files repository wes 9/17/12 8:59 AM
sure, you'll just have a small gap in the overall data when doing feeds / queries. not a huge deal after a few days it'll re-populate.

On Sep 17, 2012, at 11:24 AM, Shiran Kleiderman wrote:

> The problem started on September 14.
> Can I run what you wrote from that date?

--
Wes
wesyoung.me

Re: [ci-framework] Config files repository kl 9/17/12 9:52 AM
Hi
I ran
cif=# begin;
cif=# delete from infrastructure_scan where created >= '2012-09-13T00:00:00Z';
cif=# commit;
and then cif_feeds, but the cron job gets killed.

Anyway, I'll upgrade the instance tmw.
Until now, everything ran smooth. Will the upgrade solve the problem or is there anything else I can try?

Thanks
--
Best,
Shiran Kleiderman
+972  - 542380838
Skype - shirank1

Re: [ci-framework] Config files repository wes 9/21/12 5:04 AM
(sorry, the week got away from me)

did the upgrade work?

what does your:

/etc/postgresql/8.4/main/postgresql.conf

look like? did you follow:

http://code.google.com/p/collective-intelligence-framework/wiki/PostgresInstall_v0#Performance_Tuning

specifically:

wal_buffers = 16MB
work_mem = 1GB # 25% of total ram
shared_buffers = 1GB # 25% of total ram
checkpoint_segments = 32 # For more write-heavy systems, values from 32 (checkpoint every 512MB) to 256 (every 4GB) are popular nowadays. see: http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server for more info
effective_cache_size = 1GB    # whatever you set shmmax to

(requires postgres restart)

?

let me know,

On Sep 17, 2012, at 12:52 PM, Shiran Kleiderman wrote:

> Anyway, I'll upgrade the instance tmw.
> Until now, everything ran smooth. Will the upgrade solve the problem or is there anything else I can try?

--
Wes
wesyoung.me

Re: [ci-framework] Config files repository kl 9/21/12 11:28 AM
Hi
Thanks for the response.
Attaching the file.

My initial setup was the amazon ami that u published, so the file hasn't been changed.

I upgraded to 16 gb of memory.
Should I edit the file by what u stated?



--
Best,
Shiran Kleiderman
+972  - 542380838
Skype - shirank1

Re: [ci-framework] Config files repository wes 9/21/12 11:30 AM
yes, please do and let me know if that helps at all..

On 2012-09-21, at 2:28 PM, Shiran Kleiderman wrote:

> Should I edit the file by what u stated?

--
Wes
wesyoung.me

Re: [ci-framework] Config files repository kl 9/21/12 11:39 AM
Hi
All of the steps under Performance Tuning?
Thanks
--
Best,
Shiran Kleiderman
+972  - 542380838
Skype - shirank1

Re: [ci-framework] Config files repository wes 9/21/12 11:42 AM
ya.

On 2012-09-21, at 2:39 PM, Shiran Kleiderman wrote:

> All of the steps under Performance Tuning?

--
Wes
wesyoung.me

Re: [ci-framework] Config files repository kl 9/21/12 12:12 PM
Thanks.
I'll configure it.

A question pls:
Lets say cif feeds and the daily or hourly feeds are running together,
how much memory should be free?

Currently about 1.5-3gb is free out of 16 while two are running together.
--
Best,
Shiran Kleiderman
+972  - 542380838
Skype - shirank1

Re: [ci-framework] Config files repository kl 9/21/12 12:30 PM
Hi
Another important question please (:

Which instance should I choose? By your experience, the specs of the servers...
Thanks!

Extra Large Instance

15 GB memory
8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each)
1,690 GB instance storage
64-bit platform
I/O Performance: High
EBS-Optimized Available: 1000 Mbps
API name: m1.xlarge

OR

High-Memory Extra Large Instance

17.1 GB of memory
6.5 EC2 Compute Units (2 virtual cores with 3.25 EC2 Compute Units each)
420 GB of instance storage
64-bit platform
I/O Performance: Moderate
EBS-Optimized Available: No
API name: m2.xlarge

Re: [ci-framework] Config files repository wes 9/22/12 4:57 AM

On Sep 21, 2012, at 3:12 PM, Shiran Kleiderman wrote:

> Lets say cif feeds and the daily or hourly feeds are running together,
> how much memory should be free?

hard to say, depends on how much data there is and your psql settings.

> Currently about 1.5-3gb is free out of 16 while two are running together.

the psql settings will control how much ram is chewed up, so it could be that they're just too low.

check out:

http://code.google.com/p/collective-intelligence-framework/wiki/PostgresInstall_v0#References

specifically this book:

http://www.amazon.com/PostgreSQL-High-Performance-Gregory-Smith/dp/184951030X/ref=sr_1_1?ie=UTF8&qid=1321356392&sr=8-1

which is a huge help in understanding the psql performance settings (written specifically for administrators, helped guide some of the performance doc we wrote).
--
Wes
wesyoung.me

Re: [ci-framework] Config files repository wes 9/22/12 4:59 AM
right now we're running with 16GB, which is fine.

i'd implore you to get that book and test out the different psql settings first.. might save you some $$ in the long run... the extra memory might not help unless psql knows it can use it effectively (database management is a pain :)).

hth,

On Sep 21, 2012, at 3:30 PM, Shiran Kleiderman wrote:

> http://aws.amazon.com/ec2/instance-types/
> Extra Large Instance
>
> 15 GB memory
> 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each)
> 1,690 GB instance storage
> 64-bit platform
> I/O Performance: High
> EBS-Optimized Available: 1000 Mbps
> API name: m1.xlarge
>

--
Wes
wesyoung.me

Re: [ci-framework] Config files repository kl 9/22/12 2:45 PM
Thank u very much for all of your help.
Re: [ci-framework] Config files repository kl 9/22/12 5:06 PM
Hi
I'm trying the upgrade now, thanks.

Another question,
Is it ok, that this feed doesn't have a confidence level?  https://github.com/collectiveintel/cif-v0/blob/master/etc/vabl.cfg 
How does it affect queries?

Thank u!
Re: [ci-framework] Config files repository kl 9/22/12 7:06 PM
Hi
I upgraded to 17gb of memory, The first round the crons ran fine, the second time, cif feeds ran and I got:

... ran ok and quick and then >>>>

[role_everyone_feed] botnet domain medium severity 65% confidence private feed: 13393299 created -- 1638e6bf-2c61-5cab-9f6f-267f07402058
removing feed: 42689 -- 86ce1029-f481-5820-ac69-13d9bcdc1af7
DBD::Pg::st execute failed: ERROR:  out of memory
DETAIL:  Failed on request of size 968. [for Statement "
    SELECT DISTINCT on (domain.uuid) domain.uuid, address, confidence, archive.data
    FROM domain
    LEFT JOIN apikeys_groups ON domain.guid = apikeys_groups.guid
    LEFT JOIN archive ON domain.uuid = archive.uuid
    WHERE
        detecttime >= ?
        AND domain.confidence >= ?
        AND domain.severity >= ?
        AND domain.restriction <= ?
        AND apikeys_groups.uuid = ?
        AND NOT EXISTS (
            SELECT dw.address FROM domain_whitelist dw
            WHERE
                    dw.detecttime >= ?
                    AND dw.confidence >= 25
                    AND dw.md5 = domain.md5
        )
    ORDER BY domain.uuid ASC, domain.id ASC, confidence DESC, severity DESC, domain.restriction ASC, detecttime DESC, domain.id DESC
    LIMIT ?
" with ParamValues: 1='2012-09-18T04:28:33Z', 2='50', 3='medium', 4='private', 5='4e84b32d-cbe3-41d6-a80b-1fcba757b246', 6='2012-09-18T04:28:33Z', 7='500000'] at /usr/share/perl5/DBIx/ContextualFetch.pm line 52.
Re: [ci-framework] Config files repository kl 9/22/12 7:12 PM
Hi
And I also followed the steps for tuning the DB, v0
Thanks

--
http://code.google.com/p/collective-intelligence-framework/wiki/FAQ
http://code.google.com/p/collective-intelligence-framework/wiki/CommunityRules
---
You received this message because you are subscribed to the Google Groups "ci-framework" group.
Visit this group at http://groups.google.com/group/ci-framework?hl=en.
 
 



--
Best,
Shiran Kleiderman
+972  - 542380838
Skype - shirank1

Re: [ci-framework] Config files repository kl 9/22/12 9:38 PM
Hi
Regarding the memory problem, I upgraded to 17gb and followed the performance tuning guide.

This is the memory status and errors:

root@domU-12-31-39-08-06-20:/tmp# free -m
             total       used       free     shared    buffers     cached
Mem:         17079      16833        245          0         42      14583
-/+ buffers/cache:       2207      14871
Swap:          511          0        511
root@domU-12-31-39-08-06-20:/tmp# sudo su - cif
cif@domU-12-31-39-08-06-20:~$ /opt/cif/bin/cif_feeds
DBI connect('database=cif;host=localhost','postgres',...) failed: could not fork new process for connection: Cannot allocate memory

could not fork new process for connection: Cannot allocate memory
could not fork new process for connection: Cannot allocate memory at /usr/share/perl5/Ima/DBI.pm line 328.

cif@domU-12-31-39-08-06-20:~$ /opt/cif/bin/cif_crontool -p hourly -T low
DBI connect('database=cif;host=localhost','postgres',...) failed: could not fork new process for connection: Cannot allocate memory

could not fork new process for connection: Cannot allocate memory
could not fork new process for connection: Cannot allocate memory at /usr/share/perl5/Ima/DBI.pm line 328.
No such file or directory at /opt/cif/bin/cif_crontool line 151.


THANKS
Re: [ci-framework] Config files repository kl 9/22/12 10:00 PM
Sorry for the spam (: , the idle rows here, do they affect the performance?

Thanks

postgres  2171  0.2 14.1 4419532 2471712 ?     Ss   00:14   0:36 postgres: postgres cif 127.0.0.1(58400) idle
postgres  2173  0.1  7.0 4418008 1234416 ?     Ss   00:14   0:18 postgres: postgres cif 127.0.0.1(58401) idle
postgres  2174  0.1  7.1 4492992 1250304 ?     Ss   00:14   0:26 postgres: postgres cif 127.0.0.1(58402) idle
postgres  2176  0.0  7.0 4417216 1234816 ?     Ss   00:14   0:11 postgres: postgres cif 127.0.0.1(58403) idle
postgres  2177  0.1  7.0 4418008 1232076 ?     Ss   00:14   0:18 postgres: postgres cif 127.0.0.1(58404) idle
www-data  2179  0.0  0.9 416236 171472 ?       S    00:14   0:06 /usr/sbin/apache2 -k start
www-data  2180  0.0  0.9 403832 162884 ?       S    00:14   0:03 /usr/sbin/apache2 -k start
www-data  2181  0.0  0.9 416084 171316 ?       S    00:14   0:06 /usr/sbin/apache2 -k start
www-data  2182  0.0  0.9 403608 162688 ?       S    00:14   0:03 /usr/sbin/apache2 -k start
www-data  2183  0.0  0.9 406452 165040 ?       S    00:14   0:03 /usr/sbin/apache2 -k start
postgres  2510  0.2  7.0 4483852 1233604 ?     Ss   00:34   0:35 postgres: postgres cif 127.0.0.1(58416) idle
postgres  2511  0.1  7.0 4482512 1231412 ?     Ss   00:34   0:18 postgres: postgres cif 127.0.0.1(58417) idle
postgres  2512  0.1  7.0 4419344 1235092 ?     Ss   00:34   0:18 postgres: postgres cif 127.0.0.1(58418) idle
postgres  2513  0.0  7.0 4418052 1231460 ?     Ss   00:34   0:10 postgres: postgres cif 127.0.0.1(58419) idle
postgres  2514  0.1  7.0 4482680 1239300 ?     Ss   00:34   0:18 postgres: postgres cif 127.0.0.1(58420) idle
Re: [ci-framework] Config files repository wes 9/24/12 4:53 AM
good catch. when you said that the first time, i was looking at the wrong repo, so it didn't click.

https://github.com/collectiveintel/cif-v0/commit/4ff21739cd529394588cb6caec139e897fc4aa1b#etc/vabl.cfg

it'll make it into the main release this week.

On Sep 22, 2012, at 8:06 PM, 12 wrote:

> Is it ok, that this feed doesn't have a confidence level?  https://github.com/collectiveintel/cif-v0/blob/master/etc/vabl.cfg
> How does it affect queries?

--
Wes
wesyoung.me

Re: [ci-framework] Config files repository Edwin Kairu 7/24/13 4:45 AM
Hi wes,

Quick question: what is the default 'period' for the CFGs you uploaded to  
https://github.com/collectiveintel/cif-smrt-rules/tree/master-test/etc that don't have this parameter set? Thanks.
Re: [ci-framework] Config files repository Gabriel Iovino 7/24/13 5:08 AM
On Wednesday, July 24, 2013 7:45:52 AM UTC-4, Edwin Kairu wrote:
Hi wes,

Quick question: what is the default 'period' for the CFGs you uploaded to  
https://github.com/collectiveintel/cif-smrt-rules/tree/master-test/etc that don't have this parameter set? Thanks

In v1, "daily"


Gabe 
Re: [ci-framework] Config files repository Edwin Kairu 7/24/13 7:24 AM
Thanks Gabriel!!
More topics »