Progress of importing large file

4 views
Skip to first unread message

s.ar...@geophy.com

unread,
Sep 16, 2016, 4:48:19 AM9/16/16
to Stardog
Hi all, anyway to monitor the process when importing a large file to stardog?

Best,

Zachary Whitley

unread,
Sep 16, 2016, 6:25:43 AM9/16/16
to sta...@clarkparsia.com
You can watch the logs for status updates on the various phases of the
Import but I think that's just for new database creation with bulk import not adding data to an already existing database. 

In that case you might be able to break up your data into smaller pieces and monitor the progress yourself. I don't think there would be any penalty for loading it this way except that you'll need to be careful with blank nodes and it won't be loaded in a single transaction. 

On Sep 16, 2016, at 4:48 AM, s.ar...@geophy.com wrote:

Hi all, anyway to monitor the process when importing a large file to stardog?

Best,

--
-- --
You received this message because you are subscribed to the C&P "Stardog" group.
To post to this group, send email to sta...@clarkparsia.com
To unsubscribe from this group, send email to
stardog+u...@clarkparsia.com
For more options, visit this group at
http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
---
You received this message because you are subscribed to the Google Groups "Stardog" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stardog+u...@clarkparsia.com.

Evren Sirin

unread,
Sep 16, 2016, 9:41:49 AM9/16/16
to Stardog
Zachary is right about the log file. And the progress monitor will
appear in the log even when adding data to an already existing
database but it will show up only if the task takes more than 5
seconds. So you can do tail -f on the log file to see the progress.
But splitting the transaction into smaller parts will be slower than
one big transaction.

Best,
Evren

Zachary Whitley

unread,
Sep 16, 2016, 11:03:14 AM9/16/16
to Stardog
Thanks for the clarification. I actually did some testing with loading times a little while back with the geonames data set with different compression algorithms, serializations, db options, and bulk vs add data load. It was somewhat interesting and I'd be happy to share the results if anyone would like to see it.


> For more options, visit this group at
> http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Stardog" group.
> To unsubscribe from this group and stop receiving emails from it, send an

>
> --
> -- --
> You received this message because you are subscribed to the C&P "Stardog"
> group.
> To post to this group, send email to sta...@clarkparsia.com
> To unsubscribe from this group, send email to

> For more options, visit this group at
> http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en

--
-- --
You received this message because you are subscribed to the C&P "Stardog" group.
To post to this group, send email to sta...@clarkparsia.com
To unsubscribe from this group, send email to

For more options, visit this group at
http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
---
You received this message because you are subscribed to the Google Groups "Stardog" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stardog+unsubscribe@clarkparsia.com.


Evren Sirin

unread,
Sep 16, 2016, 11:40:02 AM9/16/16
to Stardog
Sure, I'd be interested to see that.

Thanks,
Evren
>> > stardog+u...@clarkparsia.com
>> > For more options, visit this group at
>> > http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
>> > ---
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Stardog" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to stardog+u...@clarkparsia.com.
>> >
>> > --
>> > -- --
>> > You received this message because you are subscribed to the C&P
>> > "Stardog"
>> > group.
>> > To post to this group, send email to sta...@clarkparsia.com
>> > To unsubscribe from this group, send email to
>> > stardog+u...@clarkparsia.com
>> > For more options, visit this group at
>> > http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
>>
>> --
>> -- --
>> You received this message because you are subscribed to the C&P "Stardog"
>> group.
>> To post to this group, send email to sta...@clarkparsia.com
>> To unsubscribe from this group, send email to
>> stardog+u...@clarkparsia.com
>> For more options, visit this group at
>> http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Stardog" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to stardog+u...@clarkparsia.com.
>>
>
> --
> -- --
> You received this message because you are subscribed to the C&P "Stardog"
> group.
> To post to this group, send email to sta...@clarkparsia.com
> To unsubscribe from this group, send email to
> stardog+u...@clarkparsia.com

Evren Sirin

unread,
Sep 19, 2016, 10:13:27 AM9/19/16
to Stardog
Zachary,

Were you also measuring geosparql indexing time with geonames? If
that's the case, we'd be *very* interested in seeing those numbers :)

Thanks,
Evren

Zachary Whitley

unread,
Sep 19, 2016, 12:09:52 PM9/19/16
to Stardog
I did not but it shouldn't be too difficult to do. I'll need to wait until I get home from work but I'll follow up in a couple of days with those results.

The first thing I did was to download the geonames data set.

Direct download http://download.geonames.org/all-geonames-rdf.zip


File: all-geonames-rdf.zip

Date: (2016 02 26)

File size: 603M

Sha1: abb9d8e13b1cb0b659422dfa804fcfe3a675cac3

The format is a little strange with a line with the url of the main resource followed by an RDF/XML file on a single line for that resource. I extracted the RDF/XML to a single N-TRIPLES file with the following bash script


#!/bin/bash


while read file; do

   rapper --quiet <(echo $file) 2> errors

done < <(awk ‘NR % 2 == 0 { print; }’ $*)


I was a little curious about how well various compression tools performed against various serializations. (I know Stardog doesn't support xv but I included it as well). I also noticed that the documentation states that Stardog only supports GZIP and ZIP but then goes on to mention BZIP2 as well. The *.txt extension was for the original geonames format. Default compression options were used for each tool.


File Size Summary


-rw-rw-r--. 1 zcw100 zcw100  18G May 28 00:55 geonames-20160226.nt

-rw-rw-r--. 1 zcw100 zcw100 556M May 28 13:01 geonames-20160226.nt.bz2

-rw-rw-r--. 1 zcw100 zcw100 1.1G May 28 12:57 geonames-20160226.nt.gz

-rw-rw-r--. 1 zcw100 zcw100 680M Sep 18 16:03 geonames-20160226.nt.xz

-rw-rw-r--. 1 zcw100 zcw100 1.1G Sep 18 14:49 geonames-20160226.nt.zip

-rw-rw-r--. 1 zcw100 zcw100  18G May 28 14:14 geonames-20160226.rdf

-rw-rw-r--. 1 zcw100 zcw100 329M Sep 18 16:00 geonames-20160226.rdf.bz2

-rw-rw-r--. 1 zcw100 zcw100 598M Sep 18 15:06 geonames-20160226.rdf.gz

-rw-rw-r--. 1 zcw100 zcw100 467M Sep 18 17:45 geonames-20160226.rdf.xz

-rw-rw-r--. 1 zcw100 zcw100 598M Sep 18 15:05 geonames-20160226.rdf.zip

-rw-rw-r--. 1 zcw100 zcw100  12G May 28 12:41 geonames-20160226.ttl

-rw-rw-r--. 1 zcw100 zcw100 312M May 28 13:21 geonames-20160226.ttl.bz2

-rw-rw-r--. 1 zcw100 zcw100 562M May 28 12:37 geonames-20160226.ttl.gz

-rw-rw-r--. 1 zcw100 zcw100 418M May 28 13:39 geonames-20160226.ttl.xz

-rw-rw-r--. 1 zcw100 zcw100 563M May 28 12:53 geonames-20160226.ttl.zip

-rw-rw-r--. 1 zcw100 zcw100  15G Sep 18 17:03 geonames-20160226.txt

-rw-rw-r--. 1 zcw100 zcw100 346M Sep 18 17:44 geonames-20160226.txt.bz2

-rw-rw-r--. 1 zcw100 zcw100 604M Sep 18 17:07 geonames-20160226.txt.gz

-rw-rw-r--. 1 zcw100 zcw100 424M Sep 18 17:44 geonames-20160226.txt.xz

-rw-rw-r--. 1 zcw100 zcw100 604M Sep 18 17:06 geonames-20160226.txt.zip


BZIP2 did quite well and was able to deal with most of the redundancy in N-TRIPLES with a final file size of 556M compared to 312M for similarly compressed TURTLE. One advantage I can see from using N-TRIPLES is the ability to split the file into multiple files for parallelized parsing with multiple cores and I believe with the preserve b-nodes option should be safe.

I was a little surprised that RDF/XML didn't do worse and the new kid on the block xz was unremarkable. (although I didn't measure compression time or memory usage)


Bulk Loading


nt

stardog-admin db create -n geonames geonames-20160226.nt

Loaded 162,474,860 triples to geonames from 1 file(s) in 00:16:04.663 @ 168.4K triples/sec.



nt (split into 16 files)

stardog-admin db create -n geonames *.nt

Loaded 162,474,860 triples to geonames from 17 file(s) in 00:12:17.478 @ 220.3K triples/sec.


nt.gz

Stardog-admin db create -n geonames geonames-20160226.nt.gz

Loaded 162,474,860 triples to geonames from 1 file(s) in 00:14:51.904 @ 182.2K triples/sec.


ttl.gz

stardog-admin db create -n geonames geonames-20160226.ttl.gz

Loaded 162,474,860 triples to geonames from 1 file(s) in 00:12:30.565 @ 216.5K triples/sec.


rdf

Stardog-admin db create-n geonames geonames-20160226.rdf

Loaded 162,474,860 triples to geonames from 1 file(s) in 00:11:43.014 @ 231.1K triples/sec.


rdf.gz

Stardog-admin db create -n geonames geonames-20160226.rdf.gz

Loaded 162,474,860 triples to geonames from 1 file(s) in 00:13:13.577 @ 204.7K triples/sec.


ttl.gz (index triples only)

Stardog-admin db create -n geonames --index-triples-only geonames-20160226.ttl.gz

Loaded 162,474,860 triples to geonames from 1 file(s) in 00:11:21.923 @ 238.3K triples/sec.


Loading (after db creation)


Stardog-admin db create -n geonames

Stardog data add geonames geonames-20160226.nt

Added 162,474,860 triples in 00:16:08.903



System info


cpu: Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz

Memory : 16Gb

SSD drive


STARDOG_JAVA_ARGS=-Xms4g -Xmx4g -XX:MaxDirectMemorySize=4g

Stardog 4.1.2



I was a little surprised that there wasn't more variation especially between bulk load and non-bulk loading and compressed TURTLE loaded faster than compressed N-TRIPLES although loading multiple N-TRIPLE files was the fastest.

I hope you find this useful. I personally found it interesting and helpful with making future decisions about serialization format, compression choice and loading method. I'll see what I can do about getting you those geo indexing numbers as soon as I can.

I was wondering if it might be helpful to add information to the Stardog docs regarding memory allocation similar to this for ElasticSearch

https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html

I'm assuming that the memory allocation issues for Lucene will apply equally to Stardog as well as the information on 32Gb JVM memory issues and swap. 


--Zach


>>> > For more options, visit this group at
>>> > http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
>>> > ---
>>> > You received this message because you are subscribed to the Google
>>> > Groups
>>> > "Stardog" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send
>>> > an

>>> >
>>> > --
>>> > -- --
>>> > You received this message because you are subscribed to the C&P
>>> > "Stardog"
>>> > group.
>>> > To post to this group, send email to sta...@clarkparsia.com
>>> > To unsubscribe from this group, send email to

>>> > For more options, visit this group at
>>> > http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
>>>
>>> --
>>> -- --
>>> You received this message because you are subscribed to the C&P "Stardog"
>>> group.
>>> To post to this group, send email to sta...@clarkparsia.com
>>> To unsubscribe from this group, send email to

>>> For more options, visit this group at
>>> http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
>>> ---
>>> You received this message because you are subscribed to the Google Groups
>>> "Stardog" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an

>>>
>>
>> --
>> -- --
>> You received this message because you are subscribed to the C&P "Stardog"
>> group.
>> To post to this group, send email to sta...@clarkparsia.com
>> To unsubscribe from this group, send email to

>> For more options, visit this group at
>> http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en

--
-- --
You received this message because you are subscribed to the C&P "Stardog" group.
To post to this group, send email to sta...@clarkparsia.com
To unsubscribe from this group, send email to

For more options, visit this group at
http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
---
You received this message because you are subscribed to the Google Groups "Stardog" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stardog+unsubscribe@clarkparsia.com.


Zachary Whitley

unread,
Sep 19, 2016, 12:11:59 PM9/19/16
to Stardog
I should add that I was also surprised that that fastest load was uncompressed RDF/XML.

Zachary Whitley

unread,
Sep 19, 2016, 9:07:02 PM9/19/16
to sta...@clarkparsia.com
ttl.gz with spatial enabled

stardog-admin db create -o spatial.enabled=true -n geonames geonames-20160226.ttl.gz

Spatial indexing slowed until running out of heap space at approximately 94% with OOM exception

Increased head space from 4G to 8G

Loaded 162,474,860 triples to geonames from 1 file(s) in 00:12:27.715 @ 217.3K triples/sec.
Building Spatial Index finished in 00:06:06.819




On Mon, Sep 19, 2016 at 12:11 PM, Zachary Whitley <zachary...@wavestrike.com> wrote:
I should add that I was also surprised that that fastest load was uncompressed RDF/XML.

Evren Sirin

unread,
Sep 20, 2016, 1:35:04 PM9/20/16
to Stardog
Thanks a lot for sharing these results. These results mostly agree
with what we see in our tests as well. Some comments:

* RDF/XML is not very slow as you observe though it is never fastest
in our tests and typically takes more space. We don't use it ever
unless there is some legacy data we have to deal with.

* The .bz2 compression can result significantly smaller files but
parsing (decompression on-the-fly) is very slow which is not
surprising. If the file is already in .bz2 format then it makes sense
to load it as is. It also makes sense to use it if you are loading
over a slow network connection. But if pre loading speed is important
then it is not suitable.

* Loading uncompressed vs compressed files are not much different
speed-wise. Compressed files reduce I/O but then decompression slows
things down so it is a tie. But .gz saves a lot of disk space so
that's what we use for the data files we deal with.

* Ntriples vs Turtle is also close speed-wise. Ntriples files are more
verbose but simpler to parse. In our tests Turtle comes along a little
faster and Turtle is generally more pleasant to work with.

The other factors that affect loading speed noticeably is the
strict.parsing=false setting (up to %5-%10 depending on the frequency
of literals in your dataset) and using multiple files (if you have
many cores on your machine). Once you get to the billion triples scale
memory limit (especially off-heap) become quite important for the
speed.

Best,
Evren
>>>> >>> > stardog+u...@clarkparsia.com
>>>> >>> > For more options, visit this group at
>>>> >>> > http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
>>>> >>> > ---
>>>> >>> > You received this message because you are subscribed to the Google
>>>> >>> > Groups
>>>> >>> > "Stardog" group.
>>>> >>> > To unsubscribe from this group and stop receiving emails from it,
>>>> >>> > send
>>>> >>> > an
>>>> >>> > email to stardog+u...@clarkparsia.com.
>>>> >>> >
>>>> >>> > --
>>>> >>> > -- --
>>>> >>> > You received this message because you are subscribed to the C&P
>>>> >>> > "Stardog"
>>>> >>> > group.
>>>> >>> > To post to this group, send email to sta...@clarkparsia.com
>>>> >>> > To unsubscribe from this group, send email to
>>>> >>> > stardog+u...@clarkparsia.com
>>>> >>> > For more options, visit this group at
>>>> >>> > http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
>>>> >>>
>>>> >>> --
>>>> >>> -- --
>>>> >>> You received this message because you are subscribed to the C&P
>>>> >>> "Stardog"
>>>> >>> group.
>>>> >>> To post to this group, send email to sta...@clarkparsia.com
>>>> >>> To unsubscribe from this group, send email to
>>>> >>> stardog+u...@clarkparsia.com
>>>> >>> For more options, visit this group at
>>>> >>> http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
>>>> >>> ---
>>>> >>> You received this message because you are subscribed to the Google
>>>> >>> Groups
>>>> >>> "Stardog" group.
>>>> >>> To unsubscribe from this group and stop receiving emails from it,
>>>> >>> send an
>>>> >>> email to stardog+u...@clarkparsia.com.
>>>> >>>
>>>> >>
>>>> >> --
>>>> >> -- --
>>>> >> You received this message because you are subscribed to the C&P
>>>> >> "Stardog"
>>>> >> group.
>>>> >> To post to this group, send email to sta...@clarkparsia.com
>>>> >> To unsubscribe from this group, send email to
>>>> >> stardog+u...@clarkparsia.com
>>>> >> For more options, visit this group at
>>>> >> http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
>>>>
>>>> --
>>>> -- --
>>>> You received this message because you are subscribed to the C&P
>>>> "Stardog" group.
>>>> To post to this group, send email to sta...@clarkparsia.com
>>>> To unsubscribe from this group, send email to
>>>> stardog+u...@clarkparsia.com
>>>> For more options, visit this group at
>>>> http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Stardog" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to stardog+u...@clarkparsia.com.
>>>>
>>>
>>
>> --
>> -- --
>> You received this message because you are subscribed to the C&P "Stardog"
>> group.
>> To post to this group, send email to sta...@clarkparsia.com
>> To unsubscribe from this group, send email to
>> stardog+u...@clarkparsia.com
>> For more options, visit this group at
>> http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Stardog" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to stardog+u...@clarkparsia.com.
>
>
> --
> -- --
> You received this message because you are subscribed to the C&P "Stardog"
> group.
> To post to this group, send email to sta...@clarkparsia.com
> To unsubscribe from this group, send email to
> stardog+u...@clarkparsia.com

Zachary Whitley

unread,
Sep 20, 2016, 2:29:12 PM9/20/16
to Stardog
I'm glad that I could help and your responses are very helpful as well in understanding the trade offs. I was hoping to test the strict parsing but didn't have a good data set to use. I prefer ttl myself but the one thing I could think of about gz compressed n-triples is I believe that you can split a gz file into multiple smaller compressed files without needing to decompress and re-compress and take advantage of the speed up with multiple files.

Since we're on the topic of understanding Stardog performance tuning, I've always thought that it would be nice to have a reasoning test query. Something like a -rt flag that would issue a query with and without reasoning and return some stats like "Query with reasoning (SL) took 312ms and returned 23 triples, query without reasoning took 103ms and returned 5 triples. Cost 11.6ms per inferred triple". I know this is something I can do myself when I find the time to get around to it but I thought I'd share.





>>>> >>> > For more options, visit this group at
>>>> >>> > http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
>>>> >>> > ---
>>>> >>> > You received this message because you are subscribed to the Google
>>>> >>> > Groups
>>>> >>> > "Stardog" group.
>>>> >>> > To unsubscribe from this group and stop receiving emails from it,
>>>> >>> > send
>>>> >>> > an

>>>> >>> >
>>>> >>> > --
>>>> >>> > -- --
>>>> >>> > You received this message because you are subscribed to the C&P
>>>> >>> > "Stardog"
>>>> >>> > group.
>>>> >>> > To post to this group, send email to sta...@clarkparsia.com
>>>> >>> > To unsubscribe from this group, send email to

>>>> >>> > For more options, visit this group at
>>>> >>> > http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
>>>> >>>
>>>> >>> --
>>>> >>> -- --
>>>> >>> You received this message because you are subscribed to the C&P
>>>> >>> "Stardog"
>>>> >>> group.
>>>> >>> To post to this group, send email to sta...@clarkparsia.com
>>>> >>> To unsubscribe from this group, send email to

>>>> >>> For more options, visit this group at
>>>> >>> http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
>>>> >>> ---
>>>> >>> You received this message because you are subscribed to the Google
>>>> >>> Groups
>>>> >>> "Stardog" group.
>>>> >>> To unsubscribe from this group and stop receiving emails from it,
>>>> >>> send an

>>>> >>>
>>>> >>
>>>> >> --
>>>> >> -- --
>>>> >> You received this message because you are subscribed to the C&P
>>>> >> "Stardog"
>>>> >> group.
>>>> >> To post to this group, send email to sta...@clarkparsia.com
>>>> >> To unsubscribe from this group, send email to

>>>> >> For more options, visit this group at
>>>> >> http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
>>>>
>>>> --
>>>> -- --
>>>> You received this message because you are subscribed to the C&P
>>>> "Stardog" group.
>>>> To post to this group, send email to sta...@clarkparsia.com
>>>> To unsubscribe from this group, send email to

>>>> For more options, visit this group at
>>>> http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Stardog" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send

>>>>
>>>
>>
>> --
>> -- --
>> You received this message because you are subscribed to the C&P "Stardog"
>> group.
>> To post to this group, send email to sta...@clarkparsia.com
>> To unsubscribe from this group, send email to

>> For more options, visit this group at
>> http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Stardog" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>
>
> --
> -- --
> You received this message because you are subscribed to the C&P "Stardog"
> group.
> To post to this group, send email to sta...@clarkparsia.com
> To unsubscribe from this group, send email to

> For more options, visit this group at
> http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en

--
-- --
You received this message because you are subscribed to the C&P "Stardog" group.
To post to this group, send email to sta...@clarkparsia.com
To unsubscribe from this group, send email to

For more options, visit this group at
http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
---
You received this message because you are subscribed to the Google Groups "Stardog" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stardog+unsubscribe@clarkparsia.com.


Zachary Whitley

unread,
Sep 21, 2016, 2:40:15 PM9/21/16
to Stardog
Any chance you've looked into LZ4 compression? Compression ratios aren't the highest but it's supposed to have very fast decompression rates. I'd be happy to trade compression time and disk space for a faster load. There's an Apache 2 licensed implementation at https://github.com/jpountz/lz4-java

On Tue, Sep 20, 2016 at 1:34 PM, Evren Sirin <ev...@stardog.com> wrote:

>>>> >>> > For more options, visit this group at
>>>> >>> > http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
>>>> >>> > ---
>>>> >>> > You received this message because you are subscribed to the Google
>>>> >>> > Groups
>>>> >>> > "Stardog" group.
>>>> >>> > To unsubscribe from this group and stop receiving emails from it,
>>>> >>> > send
>>>> >>> > an

>>>> >>> >
>>>> >>> > --
>>>> >>> > -- --
>>>> >>> > You received this message because you are subscribed to the C&P
>>>> >>> > "Stardog"
>>>> >>> > group.
>>>> >>> > To post to this group, send email to sta...@clarkparsia.com
>>>> >>> > To unsubscribe from this group, send email to

>>>> >>> > For more options, visit this group at
>>>> >>> > http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
>>>> >>>
>>>> >>> --
>>>> >>> -- --
>>>> >>> You received this message because you are subscribed to the C&P
>>>> >>> "Stardog"
>>>> >>> group.
>>>> >>> To post to this group, send email to sta...@clarkparsia.com
>>>> >>> To unsubscribe from this group, send email to

>>>> >>> For more options, visit this group at
>>>> >>> http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
>>>> >>> ---
>>>> >>> You received this message because you are subscribed to the Google
>>>> >>> Groups
>>>> >>> "Stardog" group.
>>>> >>> To unsubscribe from this group and stop receiving emails from it,
>>>> >>> send an

>>>> >>>
>>>> >>
>>>> >> --
>>>> >> -- --
>>>> >> You received this message because you are subscribed to the C&P
>>>> >> "Stardog"
>>>> >> group.
>>>> >> To post to this group, send email to sta...@clarkparsia.com
>>>> >> To unsubscribe from this group, send email to

>>>> >> For more options, visit this group at
>>>> >> http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
>>>>
>>>> --
>>>> -- --
>>>> You received this message because you are subscribed to the C&P
>>>> "Stardog" group.
>>>> To post to this group, send email to sta...@clarkparsia.com
>>>> To unsubscribe from this group, send email to

>>>> For more options, visit this group at
>>>> http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Stardog" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send

>>>>
>>>
>>
>> --
>> -- --
>> You received this message because you are subscribed to the C&P "Stardog"
>> group.
>> To post to this group, send email to sta...@clarkparsia.com
>> To unsubscribe from this group, send email to

>> For more options, visit this group at
>> http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Stardog" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>
>
> --
> -- --
> You received this message because you are subscribed to the C&P "Stardog"
> group.
> To post to this group, send email to sta...@clarkparsia.com
> To unsubscribe from this group, send email to

> For more options, visit this group at
> http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en

--
-- --
You received this message because you are subscribed to the C&P "Stardog" group.
To post to this group, send email to sta...@clarkparsia.com
To unsubscribe from this group, send email to

For more options, visit this group at
http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
---
You received this message because you are subscribed to the Google Groups "Stardog" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stardog+unsubscribe@clarkparsia.com.


Reply all
Reply to author
Forward
0 new messages