Pattern Mining and memory issues

Johannes Castner

unread,

Dec 30, 2018, 8:02:39 AM12/30/18

to opencog

I'm quite interested in using the pattern miner for the last chapter of my dissertation and so I'm excited to work with it and try things out. However, it seems as though the compilation of even small test knowledge bases takes a lot of memory? For serious data sets, such as the environmental ontology (https://raw.githubusercontent.com/EnvironmentOntology/envo/master/envo.obo) it always crashes. Is there a way to compile the data in a more memory efficient way, or compile it somehow elsewhere and upload it?

Nil Geisweiller

unread,

Dec 30, 2018, 4:43:13 PM12/30/18

to ope...@googlegroups.com

Hi Johannes,

I'm not sure what you mean by compilation but it is recommended to
disable guile compilation as it makes loading large data set
impractical. For that do as follows

guile --no-auto-compile -l my-data.scm

where my-data.scm is the file containing your atomspace data.

Hope it helps, let me know otherwise.

Nil

> --
> You received this message because you are subscribed to the Google
> Groups "opencog" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to opencog+u...@googlegroups.com
> <mailto:opencog+u...@googlegroups.com>.
> To post to this group, send email to ope...@googlegroups.com
> <mailto:ope...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/opencog.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/opencog/da7f4204-8b1d-47d8-8cc1-0d05da2048e8%40googlegroups.com
> <https://groups.google.com/d/msgid/opencog/da7f4204-8b1d-47d8-8cc1-0d05da2048e8%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

Johannes Castner

unread,

Jan 1, 2019, 8:57:02 AM1/1/19

to opencog

Hi Nil,

Thank you, that did that trick. Now I run:

guile> (pm-run-patternminer)

Yet no matter what knowledge base I run this on it never seems to find any interesting patterns. What could be the cause of that?

Johannes

Johannes Castner

unread,

Jan 1, 2019, 9:00:05 AM1/1/19

to opencog

P.S:

This is the output I get when I run it on the GO.scm example ontology (I've also tried the ugly soda drinker etc. and the result is always the same):

totalLinkNum = 430254, actualProcessedLinkNum = 0
PatternMiner: mining finished!

Calculating Surprisingness for 1 gram patterns.
100% completed.PatternMiner: done (gram = 1) interestingness evaluation!0 patterns found! Outputting to file ...
Debug: PatternMiner: writing (gram = 1) interesting patterns to file Surprisingness_b_1gram.scm

Debug: PatternMiner: writing (gram = 1) pattern statics to csv file PatternStatics_1gram.csv
gram = 1: 0 patterns found!
Debug: PatternMiner: writing (gram = 1) frequent patterns to file FrequentPatterns_1gram.scm

On Sunday, December 30, 2018 at 9:43:13 PM UTC, Nil wrote:

Johannes Castner

unread,

Jan 2, 2019, 11:02:37 AM1/2/19

to opencog

I think that the problem is that it doesn't load the configuration file:

[WARN] No configuration file was loaded! Param=Pattern_mining_mode

How do I make sure that it loads the configuration file?

Johannes

On Sunday, December 30, 2018 at 9:43:13 PM UTC, Nil wrote:

Nil Geisweiller

unread,

Jan 3, 2019, 4:07:25 AM1/3/19

to ope...@googlegroups.com

Hi,

you should use the new pattern miner, see

https://github.com/opencog/opencog/blob/master/opencog/learning/miner/README.md

BTW, what doc did you follow to get to that command? I'm asking so I can
redirect the user to the new pattern miner.

Nil

> <https://raw.githubusercontent.com/EnvironmentOntology/envo/master/envo.obo>)

>
> > it always crashes. Is there a way to compile the data in a more
> memory
> > efficient way, or compile it somehow elsewhere and upload it?
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "opencog" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send

> > an email to opencog+u...@googlegroups.com <javascript:>
> > <mailto:opencog+u...@googlegroups.com <javascript:>>.

> > To post to this group, send email to ope...@googlegroups.com

> <javascript:>
> > <mailto:ope...@googlegroups.com <javascript:>>.

> > Visit this group at https://groups.google.com/group/opencog

> <https://groups.google.com/group/opencog>.

> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/opencog/da7f4204-8b1d-47d8-8cc1-0d05da2048e8%40googlegroups.com
> <https://groups.google.com/d/msgid/opencog/da7f4204-8b1d-47d8-8cc1-0d05da2048e8%40googlegroups.com>
>
> >

> <https://groups.google.com/d/msgid/opencog/da7f4204-8b1d-47d8-8cc1-0d05da2048e8%40googlegroups.com?utm_medium=email&utm_source=footer

> <https://groups.google.com/d/msgid/opencog/da7f4204-8b1d-47d8-8cc1-0d05da2048e8%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "opencog" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to opencog+u...@googlegroups.com
> <mailto:opencog+u...@googlegroups.com>.
> To post to this group, send email to ope...@googlegroups.com
> <mailto:ope...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/opencog.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/opencog/59c1cb09-8e7e-4b25-a50c-18a24ee512b4%40googlegroups.com
> <https://groups.google.com/d/msgid/opencog/59c1cb09-8e7e-4b25-a50c-18a24ee512b4%40googlegroups.com?utm_medium=email&utm_source=footer>.

Johannes Castner

unread,

Jan 3, 2019, 4:27:01 AM1/3/19

to opencog

Hi Nil,

Thank you!!! I followed this one: https://wiki.opencog.org/w/Using_The_Pattern_Miner#Hands_On

Switching now; thank you!

Johannes

Johannes Castner

unread,

Jan 3, 2019, 4:37:19 AM1/3/19

to opencog

Hi Nil,

Does the new pattern miner support surprisingness yet?

That's what I'm hoping to make use of.

Johannes

On Thursday, January 3, 2019 at 9:07:25 AM UTC, Nil wrote:

Nil Geisweiller

unread,

Jan 3, 2019, 4:52:09 AM1/3/19

to ope...@googlegroups.com

On 1/3/19 11:37 AM, Johannes Castner wrote:
> Hi Nil,
>

> Does the new pattern miner support surprisingness yet?

Nope :-(

That's the last missing bit really.

> That's what I'm hoping to make use of.

Great.

I might have recommended you to use the old pattern miner in the
meantime but I'm afraid it's gonna be too buggy. And implementing
surprisingness in the new pattern miner is pretty much my next task, so...

What is your timing? When do you need this?

If I understand correctly you want to mine patterns in

https://raw.githubusercontent.com/EnvironmentOntology/envo/master/envo.obo

and I suppose you already have some scripts to turn that into atoms.
Could you share that script? we might want to add it to

https://github.com/opencog/external-tools

and maybe turn your use-case into a pattern miner example, that would be
great.

Nil

> https://groups.google.com/d/msgid/opencog/59c1cb09-8e7e-4b25-a50c-18a24ee512b4%40googlegroups.com
> <https://groups.google.com/d/msgid/opencog/59c1cb09-8e7e-4b25-a50c-18a24ee512b4%40googlegroups.com>
>
> >
> <https://groups.google.com/d/msgid/opencog/59c1cb09-8e7e-4b25-a50c-18a24ee512b4%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/opencog/59c1cb09-8e7e-4b25-a50c-18a24ee512b4%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>

> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "opencog" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to opencog+u...@googlegroups.com
> <mailto:opencog+u...@googlegroups.com>.
> To post to this group, send email to ope...@googlegroups.com
> <mailto:ope...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/opencog.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/opencog/d5610b6d-77f9-485b-bb28-dba3f4b49fb6%40googlegroups.com
> <https://groups.google.com/d/msgid/opencog/d5610b6d-77f9-485b-bb28-dba3f4b49fb6%40googlegroups.com?utm_medium=email&utm_source=footer>.

Johannes Castner

unread,

Jan 3, 2019, 9:36:04 AM1/3/19

to opencog

Hi Nil,

I thought to adopt the one for the GO.scm as in this example? https://github.com/opencog/test-datasets/tree/master/go

I have not yet seriously worked on it as I need to first be sure that the patternminer works on my system and I haven't been able to get something very simple to work as expected yet. I'll be tinkering again soon to see if I can get one of the examples to work and then I'll try it out on the GO.scm example. If that works, I'll start modifying the script that creates the scheme GO.scm file until it works with envo.obo ...Of course I'd love to share the results and what I'm finding with you; it'd be great if I can in some ways contribute to your awesome system! Does my plan sound reasonable? In terms of timing; I really need to have a working paper using this system ready before April 1st, so that I can distribute my dissertation to my committee. This means that I'd love to have the results as soon as possible; preferably before the end of January if that is possible?

Johannes

Nil Geisweiller

unread,

Jan 3, 2019, 1:27:31 PM1/3/19

to ope...@googlegroups.com

Hi Johannes,

On 1/3/19 4:36 PM, Johannes Castner wrote:
> Hi Nil,
>

> I thought to adopt the one for the GO.scm as in this example?
> https://github.com/opencog/test-datasets/tree/master/go

It seems that code is more recent

https://github.com/opencog/agi-bio/tree/master/knowledge-import

> I have not yet seriously worked on it as I need to first be sure that
> the patternminer works on my system and I haven't been able to get

> something very simple to work as expected yet. I'll be tinkering again \

I should mention that the new pattern miner is already quite stable. If
you can install opencog on your system, it should work.

> ...Of course I'd love to share the results and what I'm finding with
> you; it'd be great if I can in some ways contribute to your awesome

It would also be of interest to the AGI-Bio team.

> system! Does my plan sound reasonable? In terms of timing; I really
> need to have a working paper using this system ready before April 1st,
> so that I can distribute my dissertation to my committee. This means
> that I'd love to have the results as soon as possible; preferably before
> the end of January if that is possible?

That seems doable. I should start implementing surprisingness next week.
The time it takes depends on how it is implemented, I'm aiming at
something as general as possible but might also start with something
more specialized, I mean I'll do my best so you have something that
works on time.

Meanwhile if you can run the pattern miner on your data just to make
sure it works so far, maybe even verify at hand that it can find
surprising patterns, etc, would be very good.

Nil

>
> Johannes
>

> https://groups.google.com/d/msgid/opencog/d5610b6d-77f9-485b-bb28-dba3f4b49fb6%40googlegroups.com
> <https://groups.google.com/d/msgid/opencog/d5610b6d-77f9-485b-bb28-dba3f4b49fb6%40googlegroups.com>
>
> >
> <https://groups.google.com/d/msgid/opencog/d5610b6d-77f9-485b-bb28-dba3f4b49fb6%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/opencog/d5610b6d-77f9-485b-bb28-dba3f4b49fb6%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>

> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "opencog" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to opencog+u...@googlegroups.com
> <mailto:opencog+u...@googlegroups.com>.
> To post to this group, send email to ope...@googlegroups.com
> <mailto:ope...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/opencog.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/opencog/2f509e44-e3b8-442f-9bfe-9c456b031e8e%40googlegroups.com
> <https://groups.google.com/d/msgid/opencog/2f509e44-e3b8-442f-9bfe-9c456b031e8e%40googlegroups.com?utm_medium=email&utm_source=footer>.

Johannes Castner

unread,

Jan 3, 2019, 1:45:03 PM1/3/19

to opencog

Hi Nil,

Fantastic, thank you for all of your help and great work; I'll be testing the new pattern miner tomorrow morning then!

Johannes

Johannes Castner

unread,

Jan 3, 2019, 2:01:56 PM1/3/19

to opencog

Hi Nil,

Actually, I couldn't wait to try it and I found that it seems to be working! Does it still save files with the mined patterns, or does it just store those mined patterns in the Atom Space?

Johannes

Nil Geisweiller

unread,

Jan 3, 2019, 3:54:11 PM1/3/19

to ope...@googlegroups.com

On 1/3/19 9:01 PM, Johannes Castner wrote:
> Actually, I couldn't wait to try it and I found that it seems to be
> working!

Cool!

> Does it still save files with the mined patterns, or does it
> just store those mined patterns in the Atom Space?

The list of discovered patterns with minimum support is directly
returned by cog-mine, and a fortiori stored in the atomspace. If that
list is empty it means no pattern with minimum support were found, which
probably indicates that you need to lower the minimum support.

Carefully read

(help cog-mine)

and if not enough the README.md which should provide you the background
knowledge you need to understand the help.

Nil

>
> Johannes
>

> https://groups.google.com/d/msgid/opencog/d5610b6d-77f9-485b-bb28-dba3f4b49fb6%40googlegroups.com

> https://groups.google.com/d/msgid/opencog/2f509e44-e3b8-442f-9bfe-9c456b031e8e%40googlegroups.com
> <https://groups.google.com/d/msgid/opencog/2f509e44-e3b8-442f-9bfe-9c456b031e8e%40googlegroups.com>
>
> >
> <https://groups.google.com/d/msgid/opencog/2f509e44-e3b8-442f-9bfe-9c456b031e8e%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/opencog/2f509e44-e3b8-442f-9bfe-9c456b031e8e%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>

> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "opencog" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to opencog+u...@googlegroups.com
> <mailto:opencog+u...@googlegroups.com>.
> To post to this group, send email to ope...@googlegroups.com
> <mailto:ope...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/opencog.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/opencog/c41f8dce-33b9-474c-b677-f63c8ddcfa32%40googlegroups.com
> <https://groups.google.com/d/msgid/opencog/c41f8dce-33b9-474c-b677-f63c8ddcfa32%40googlegroups.com?utm_medium=email&utm_source=footer>.

Johannes Castner

unread,

Jan 4, 2019, 4:05:19 AM1/4/19

to opencog

Thank you, Nil!

Johannes Castner

unread,

Jan 4, 2019, 12:23:01 PM1/4/19

to opencog

Hi Nil,

So now I have a more complete report to make:

This works:

(cog-mine (cog-atomspace))

But this doesn't:

(cog-mine (cog-atomspace) (NumberNode 4))

(cog-mine (cog-atomspace) 10)

The error messages are:

;;; <stdin>:20:0: warning: possibly wrong number of arguments to `cog-mine'
opencog/miner/miner-utils.scm:299:0: In procedure cog-mine:
Invalid keyword: (NumberNode "4.000000")

and

;;; <stdin>:21:0: warning: possibly wrong number of arguments to `cog-mine'
opencog/miner/miner-utils.scm:299:0: In procedure cog-mine:
Invalid keyword: 10

Am I understanding things wrong, or am I not up to date?

Johannes

On Thursday, January 3, 2019 at 9:07:25 AM UTC, Nil wrote:

Johannes Castner

unread,

Jan 4, 2019, 12:27:11 PM1/4/19

to opencog

Hi Nil,

Never mind the last message; this works well:

(cog-mine (cog-atomspace) #:minsup 10)

Just not used to the notation but now I figured it out by tinkering around.

Johannes

Sarah Connorh

unread,

Jan 9, 2019, 11:53:10 AM1/9/19

to ope...@googlegroups.com

Hey,

Is there any ML applied here ?

To unsubscribe from this group and stop receiving emails from it, send an email to opencog+u...@googlegroups.com.
To post to this group, send email to ope...@googlegroups.com.

Visit this group at https://groups.google.com/group/opencog.

To view this discussion on the web visit https://groups.google.com/d/msgid/opencog/75606350-65fc-f287-a153-cbce2e972cb4%40gmail.com.

Reply all

Reply to author

Forward