Fishing for Graphs in a Hadoop Data Lake
54 views
Skip to first unread message
Mark Waser
Feb 1, 2018, 11:01:24 AM2/1/18
to ope...@googlegroups.com
https://www.oreilly.com/ideas/fishing-for-graphs-in-a-hadoop-data-lake
So . . . . who wants to port MOSES?
Linas Vepstas
Feb 1, 2018, 1:51:02 PM2/1/18
to opencog
Not sure why you say "moses" .. it would be "atomspace" to me.
Atomspace has been a kind-of graph database since forever, but there was no promotion of it as such on internet forums, and so no one ever heard of it, and things like spark/haddoop have taken over 99.99% of the mindshare and actual usage. What to do in that world? The atomspace still offers some neat ideas and concepts, that these other systems can't yet do ... but its only a matter of time. So I'd say: port the atomspace to these graph db systems.On Thu, Feb 1, 2018 at 10:01 AM, Mark Waser <mark....@gmail.com> wrote:
https://www.oreilly.com/ideas/fishing-for-graphs-in-a-hadoop-data-lake
So . . . . who wants to port MOSES?
--
You received this message because you are subscribed to the Google Groups "opencog" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opencog+unsubscribe@googlegroups.com.
To post to this group, send email to ope...@googlegroups.com.
Visit this group at https://groups.google.com/group/opencog.
To view this discussion on the web visit https://groups.google.com/d/msgid/opencog/00bd01d39b75%24e5a570a0%24b0f051e0%24%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
cassette tapes - analog TV - film cameras - you
Ben Goertzel
Feb 2, 2018, 8:28:35 AM2/2/18
to opencog
We are undertaking now to port MOSES into the Atomspace (specifically
to make each MOSES deme an Atomspace, while leaving the
deme-management, feature-selection etc. code intact...)...
However as Linas notes, the intersection btw Atomspace and graph-DBs
is not about MOSES in particular....
A few of us have been thinking about potentially connecting Atomspace
to Apache Ignite, so that Ignite could serve as a sort of distributed
backing-store, and the current Atomspace code would serve as the
"local cache" Atomspace on each machine in the distributed network....
Ignite is an interesting beast which is a sort of middleware layer
between data store and application.... This is not quite the same as
porting Atomspace logic to a graph DB, but perhaps it's in a similar
spirit...
-- Ben
On Fri, Feb 2, 2018 at 2:50 AM, Linas Vepstas <linasv...@gmail.com> wrote:
> Not sure why you say "moses" .. it would be "atomspace" to me.
>
> Atomspace has been a kind-of graph database since forever, but there was no
> promotion of it as such on internet forums, and so no one ever heard of it,
> and things like spark/haddoop have taken over 99.99% of the mindshare and
> actual usage. What to do in that world? The atomspace still offers some
> neat ideas and concepts, that these other systems can't yet do ... but its
> only a matter of time. So I'd say: port the atomspace to these graph db
> systems.
>
> --linas
>
> On Thu, Feb 1, 2018 at 10:01 AM, Mark Waser <mark....@gmail.com> wrote:
>>
>> https://www.oreilly.com/ideas/fishing-for-graphs-in-a-hadoop-data-lake
>>
>>
>>
>> So . . . . who wants to port MOSES?
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "opencog" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to opencog+u...@googlegroups.com.
Ben Goertzel, PhD
http://goertzel.org
"In the province of the mind, what one believes to be true is true or
becomes true, within certain limits to be found experientially and
experimentally. These limits are further beliefs to be transcended. In
the mind, there are no limits.... In the province of connected minds,
what the network believes to be true, either is true or becomes true
within certain limits to be found experientially and experimentally.
These limits are further beliefs to be transcended. In the network's
mind there are no limits." -- John Lilly
to make each MOSES deme an Atomspace, while leaving the
deme-management, feature-selection etc. code intact...)...
However as Linas notes, the intersection btw Atomspace and graph-DBs
is not about MOSES in particular....
A few of us have been thinking about potentially connecting Atomspace
to Apache Ignite, so that Ignite could serve as a sort of distributed
backing-store, and the current Atomspace code would serve as the
"local cache" Atomspace on each machine in the distributed network....
Ignite is an interesting beast which is a sort of middleware layer
between data store and application.... This is not quite the same as
porting Atomspace logic to a graph DB, but perhaps it's in a similar
spirit...
-- Ben
On Fri, Feb 2, 2018 at 2:50 AM, Linas Vepstas <linasv...@gmail.com> wrote:
> Not sure why you say "moses" .. it would be "atomspace" to me.
>
> Atomspace has been a kind-of graph database since forever, but there was no
> promotion of it as such on internet forums, and so no one ever heard of it,
> and things like spark/haddoop have taken over 99.99% of the mindshare and
> actual usage. What to do in that world? The atomspace still offers some
> neat ideas and concepts, that these other systems can't yet do ... but its
> only a matter of time. So I'd say: port the atomspace to these graph db
> systems.
>
> --linas
>
> On Thu, Feb 1, 2018 at 10:01 AM, Mark Waser <mark....@gmail.com> wrote:
>>
>> https://www.oreilly.com/ideas/fishing-for-graphs-in-a-hadoop-data-lake
>>
>>
>>
>> So . . . . who wants to port MOSES?
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "opencog" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> To post to this group, send email to ope...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/opencog.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/opencog/00bd01d39b75%24e5a570a0%24b0f051e0%24%40gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> cassette tapes - analog TV - film cameras - you
>
>> Visit this group at https://groups.google.com/group/opencog.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/opencog/00bd01d39b75%24e5a570a0%24b0f051e0%24%40gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> cassette tapes - analog TV - film cameras - you
>
> --
> You received this message because you are subscribed to the Google Groups
> "opencog" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to opencog+u...@googlegroups.com.
> You received this message because you are subscribed to the Google Groups
> "opencog" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> To post to this group, send email to ope...@googlegroups.com.
> Visit this group at https://groups.google.com/group/opencog.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/opencog/CAHrUA36ScSdVJdgBr%3DmhPPz27-S-JYOAsLcocTEXgibrWsTpuw%40mail.gmail.com.
> Visit this group at https://groups.google.com/group/opencog.
> To view this discussion on the web visit
Ben Goertzel, PhD
http://goertzel.org
"In the province of the mind, what one believes to be true is true or
becomes true, within certain limits to be found experientially and
experimentally. These limits are further beliefs to be transcended. In
the mind, there are no limits.... In the province of connected minds,
what the network believes to be true, either is true or becomes true
within certain limits to be found experientially and experimentally.
These limits are further beliefs to be transcended. In the network's
mind there are no limits." -- John Lilly
Linas Vepstas
Feb 2, 2018, 5:55:53 PM2/2/18
to opencog
On Fri, Feb 2, 2018 at 7:28 AM, Ben Goertzel <b...@goertzel.org> wrote:
We are undertaking now to port MOSES into the Atomspace (specifically
to make each MOSES deme an Atomspace, while leaving the
deme-management, feature-selection etc. code intact...)...
I'm not sure who "we" are, in this sentence. The reason that "we", meaning "me"
did not try this in the past is that the Atomspace has a performance profile that
does not fit well with the default algorithms in MOSES. A naive port substituting
template Vertex<> by Handle could easily slow down MOSES by a factor of
10x or 100x. Clearly, that's not viable. At least, not to me.
template Vertex<> by Handle could easily slow down MOSES by a factor of
10x or 100x. Clearly, that's not viable. At least, not to me.
One can get clever in several ways. The most obvious one is to have a
copyin/copyout system to copy Atoms into moses-internal format, run MOSES
for a while, and then copyout back into the atomspace. There are other, more
subtle approaches possible.
copyin/copyout system to copy Atoms into moses-internal format, run MOSES
for a while, and then copyout back into the atomspace. There are other, more
subtle approaches possible.
However as Linas notes, the intersection btw Atomspace and graph-DBs
is not about MOSES in particular....
A few of us have been thinking about potentially connecting Atomspace
to Apache Ignite,
describes work that needs to happen prior to a redis/riak/ignite port.
so that Ignite could serve as a sort of distributed
backing-store, and the current Atomspace code would serve as the
"local cache" Atomspace on each machine in the distributed network....
Ignite is an interesting beast which is a sort of middleware layer
between data store and application.... This is not quite the same as
porting Atomspace logic to a graph DB, but perhaps it's in a similar
spirit...
The actual task to be done depends srtrongy on who is doing it, how
many people are involved, and how clever they are.
many people are involved, and how clever they are.
I fully expect other opensource groups to reinvent an atomspace-like system
on top of a graph DB. They will start with a clean slate, not reusing any atomspace
code, and probably reinventing it blisssfully unaware that the Atomspace exists.
But if you decide to start with the existing code base, then I think
https://github.com/opencog/atomspace/issues/1502
https://github.com/opencog/atomspace/issues/1502
spells out the preliminary groundwork that has to be accomplished, before
you can layer it on top of Ignite (or redis, or whatever).
-- Ben
On Fri, Feb 2, 2018 at 2:50 AM, Linas Vepstas <linasv...@gmail.com> wrote:
> Not sure why you say "moses" .. it would be "atomspace" to me.
>
> Atomspace has been a kind-of graph database since forever, but there was no
> promotion of it as such on internet forums, and so no one ever heard of it,
> and things like spark/haddoop have taken over 99.99% of the mindshare and
> actual usage. What to do in that world? The atomspace still offers some
> neat ideas and concepts, that these other systems can't yet do ... but its
> only a matter of time. So I'd say: port the atomspace to these graph db
> systems.
>
> --linas
>
> On Thu, Feb 1, 2018 at 10:01 AM, Mark Waser <mark....@gmail.com> wrote:
>>
>> https://www.oreilly.com/ideas/fishing-for-graphs-in-a-hadoop-data-lake
>>
>>
>>
>> So . . . . who wants to port MOSES?
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "opencog" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to opencog+unsubscribe@googlegroups.com.
>> To post to this group, send email to ope...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/opencog.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/opencog/00bd01d39b75%24e5a570a0%24b0f051e0%24%40gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> cassette tapes - analog TV - film cameras - you
>
> --
> You received this message because you are subscribed to the Google Groups
> "opencog" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to opencog+unsubscribe@googlegroups.com.
> To post to this group, send email to ope...@googlegroups.com.
> Visit this group at https://groups.google.com/group/opencog.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/opencog/CAHrUA36ScSdVJdgBr%3DmhPPz27-S-JYOAsLcocTEXgibrWsTpuw%40mail.gmail.com.
>
> For more options, visit https://groups.google.com/d/optout.
--
Ben Goertzel, PhD
http://goertzel.org
"In the province of the mind, what one believes to be true is true or
becomes true, within certain limits to be found experientially and
experimentally. These limits are further beliefs to be transcended. In
the mind, there are no limits.... In the province of connected minds,
what the network believes to be true, either is true or becomes true
within certain limits to be found experientially and experimentally.
These limits are further beliefs to be transcended. In the network's
mind there are no limits." -- John Lilly
--
You received this message because you are subscribed to the Google Groups "opencog" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opencog+unsubscribe@googlegroups.com.
To post to this group, send email to ope...@googlegroups.com.
Visit this group at https://groups.google.com/group/opencog.
To view this discussion on the web visit https://groups.google.com/d/msgid/opencog/CACYTDBfgoj_4vb65EtwDrBERkEh-_s8F%2BWMAk%3DcqFHwomFbbEA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Ben Goertzel
Feb 2, 2018, 11:29:22 PM2/2/18
to opencog, Nil Geisweiller
Nil, what are your thoughts on
https://github.com/opencog/atomspace/issues/1502
?
Linas about,
***
Perhaps -- I mean competing options are common in the OSS world, which
is probably
as it should be
However, we are well positioned to do something in this direction
ourselves this year
also .... At the moment Ignite seems the technology most likely to
have the flexibility
needed to make this a tractable project, but this needs to be
validated by more hands-on
exploration...
thx
ben
>> >> email to opencog+u...@googlegroups.com.
https://github.com/opencog/atomspace/issues/1502
?
Linas about,
***
I fully expect other opensource groups to reinvent an atomspace-like system
on top of a graph DB. They will start with a clean slate, not reusing
any atomspace
code, and probably reinventing it blisssfully unaware that the Atomspace exists.
***
on top of a graph DB. They will start with a clean slate, not reusing
any atomspace
code, and probably reinventing it blisssfully unaware that the Atomspace exists.
Perhaps -- I mean competing options are common in the OSS world, which
is probably
as it should be
However, we are well positioned to do something in this direction
ourselves this year
also .... At the moment Ignite seems the technology most likely to
have the flexibility
needed to make this a tractable project, but this needs to be
validated by more hands-on
exploration...
thx
ben
>> >> To post to this group, send email to ope...@googlegroups.com.
>> >> Visit this group at https://groups.google.com/group/opencog.
>> >> To view this discussion on the web visit
>> >>
>> >> https://groups.google.com/d/msgid/opencog/00bd01d39b75%24e5a570a0%24b0f051e0%24%40gmail.com.
>> >> For more options, visit https://groups.google.com/d/optout.
>> >
>> >
>> >
>> >
>> > --
>> > cassette tapes - analog TV - film cameras - you
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "opencog" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to opencog+u...@googlegroups.com.
>> >> Visit this group at https://groups.google.com/group/opencog.
>> >> To view this discussion on the web visit
>> >>
>> >> https://groups.google.com/d/msgid/opencog/00bd01d39b75%24e5a570a0%24b0f051e0%24%40gmail.com.
>> >> For more options, visit https://groups.google.com/d/optout.
>> >
>> >
>> >
>> >
>> > --
>> > cassette tapes - analog TV - film cameras - you
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "opencog" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > To post to this group, send email to ope...@googlegroups.com.
>> > Visit this group at https://groups.google.com/group/opencog.
>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/opencog/CAHrUA36ScSdVJdgBr%3DmhPPz27-S-JYOAsLcocTEXgibrWsTpuw%40mail.gmail.com.
>> >
>> > For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>> --
>> Ben Goertzel, PhD
>> http://goertzel.org
>>
>> "In the province of the mind, what one believes to be true is true or
>> becomes true, within certain limits to be found experientially and
>> experimentally. These limits are further beliefs to be transcended. In
>> the mind, there are no limits.... In the province of connected minds,
>> what the network believes to be true, either is true or becomes true
>> within certain limits to be found experientially and experimentally.
>> These limits are further beliefs to be transcended. In the network's
>> mind there are no limits." -- John Lilly
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "opencog" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to opencog+u...@googlegroups.com.
>> > Visit this group at https://groups.google.com/group/opencog.
>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/opencog/CAHrUA36ScSdVJdgBr%3DmhPPz27-S-JYOAsLcocTEXgibrWsTpuw%40mail.gmail.com.
>> >
>> > For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>> --
>> Ben Goertzel, PhD
>> http://goertzel.org
>>
>> "In the province of the mind, what one believes to be true is true or
>> becomes true, within certain limits to be found experientially and
>> experimentally. These limits are further beliefs to be transcended. In
>> the mind, there are no limits.... In the province of connected minds,
>> what the network believes to be true, either is true or becomes true
>> within certain limits to be found experientially and experimentally.
>> These limits are further beliefs to be transcended. In the network's
>> mind there are no limits." -- John Lilly
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "opencog" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> To post to this group, send email to ope...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/opencog.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/opencog/CACYTDBfgoj_4vb65EtwDrBERkEh-_s8F%2BWMAk%3DcqFHwomFbbEA%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> cassette tapes - analog TV - film cameras - you
>
>> Visit this group at https://groups.google.com/group/opencog.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/opencog/CACYTDBfgoj_4vb65EtwDrBERkEh-_s8F%2BWMAk%3DcqFHwomFbbEA%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> cassette tapes - analog TV - film cameras - you
>
> --
> You received this message because you are subscribed to the Google Groups
> "opencog" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to opencog+u...@googlegroups.com.
> You received this message because you are subscribed to the Google Groups
> "opencog" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> To post to this group, send email to ope...@googlegroups.com.
> Visit this group at https://groups.google.com/group/opencog.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/opencog/CAHrUA3796euGCfW_c0LENc7NiP7nVDAfSM07JXaHaXj%3DAc2BxA%40mail.gmail.com.
> Visit this group at https://groups.google.com/group/opencog.
> To view this discussion on the web visit
Nil Geisweiller
Feb 6, 2018, 10:47:32 PM2/6/18
to Ben Goertzel, opencog, Nil Geisweiller
On 02/03/2018 06:29 AM, Ben Goertzel wrote:
> Nil, what are your thoughts on
>
> https://github.com/opencog/atomspace/issues/1502
>
> ?
I seem to agree, however I don't think SetLink should be banished,
> Nil, what are your thoughts on
>
> https://github.com/opencog/atomspace/issues/1502
>
> ?
instead it should probably be parsimoniously used, when for instance the
user/agent wants to represent an immutable concept, probably a small one.
Although, I've been thinking why not replace, at least underneath,
ListLink by ConsLinks (and perhaps SetLink by SetConsLinks)? This would
solve the arity limit on the persistent storage side. For instance
List A B C
would be represented by
Cons A
Cons B
Cons C
List
and
Set A B C
by
SetCons A ...
not sure how to deal with the canonical ordering, I suppose the
user/agent would enter a tree of SetCons in any order, and the AtomSpace
may automatically re-order it.
Nil
Linas Vepstas
Feb 7, 2018, 10:03:28 PM2/7/18
to opencog, Ben Goertzel, Nil Geisweiller
On Tue, Feb 6, 2018 at 9:47 PM, 'Nil Geisweiller' via opencog <ope...@googlegroups.com> wrote:
On 02/03/2018 06:29 AM, Ben Goertzel wrote:
Nil, what are your thoughts on
https://github.com/opencog/atomspace/issues/1502
?
I seem to agree, however I don't think SetLink should be banished, instead it should probably be parsimoniously used, when for instance the user/agent wants to represent an immutable concept, probably a small one.
Yes.
Although, I've been thinking why not replace, at least underneath, ListLink by ConsLinks (and perhaps SetLink by SetConsLinks)? This would solve the arity limit on the persistent storage side. For instance
List A B C
would be represented by
Cons A
Cons B
Cons C
List
and
Set A B C
by
SetCons A ...
not sure how to deal with the canonical ordering, I suppose the user/agent would enter a tree of SetCons in any order, and the AtomSpace may automatically re-order it.
OK, this is actually a major unsolved problem for the atomspace. We need the ability to create ordered lists of things, and the ability to edit those ordered lists. Currently, this is extremely difficult verging on impossible. And I have no answer for it, either.
The problem with
Cons A
Cons B
Cons C
List
is that, if I want do remove C from the sequence, I'd have to first: delete Cons A then delete Cons B Then delete Cons C then recreate Cons B (without the C under it, this time) and then recreate Cons A. Ugh. Lots of work, and it's begging for a race condition.
Worse, if Cons A has a non-empty incoming set, then it cannot be deleted without clearing out the incoming set first. Which might be impossible. In essence its quick-setting concrete, and you need a jackhammer to "fix it".
Huh. Can I coin a new saying? "fix it with a jackhammer"
--linas
Reply all
Reply to author
Forward
0 new messages