Vote release 0.8

5 views

Skip to first unread message

Julien Nioche

unread,

Jun 6, 2017, 3:44:38 AM6/6/17

to crawler...@googlegroups.com

Hi,

All issues tagged for 0.8 have been addressed. Any objections to releasing 0.8? Am happy to do the release.

Julien

On 23 March 2017 at 15:38, Lewis John Mcgibbney <lewis.m...@gmail.com> wrote:

Hi Julien,
I've included all of the CHANGES for current development under the 0.8 milestone.
Pending issues can be found at https://github.com/crawler-commons/crawler-commons/milestone/3
Lewis

On Wed, Mar 22, 2017 at 2:39 AM, Julien Nioche <lists.dig...@gmail.com> wrote:
Hi,

There's been some great work done since the last release [1], notably the SAX-based sitemap parsing, it would be good to release 0.8 soon.

Could you please look at existing issues and mark them as milestone crawler-commons-0.8?

Thanks

Julien

[1] https://github.com/crawler-commons/crawler-commons/blob/master/CHANGES.txt

--

Open Source Solutions for Text Engineering

http://www.digitalpebble.com
http://digitalpebble.blogspot.com/
#digitalpebble

--
You received this message because you are subscribed to the Google Groups "crawler-commons" group.
To unsubscribe from this group and stop receiving emails from it, send an email to crawler-commons+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/crawler-commons.
For more options, visit https://groups.google.com/d/optout.

--
Lewis
Dr. Lewis J. McGibbney Ph.D, B.Sc
Director, MCMA Associates
Phone: +1(626)498-3090
Skype: lewis.john.mcgibbney
Email: lewis.m...@gmail.com

--
You received this message because you are subscribed to the Google Groups "crawler-commons" group.
To unsubscribe from this group and stop receiving emails from it, send an email to crawler-commons+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/crawler-commons.
For more options, visit https://groups.google.com/d/optout.

Open Source Solutions for Text Engineering

http://www.digitalpebble.com
http://digitalpebble.blogspot.com/
#digitalpebble

Ken Krugler

unread,

Jun 6, 2017, 9:43:25 AM6/6/17

to crawler...@googlegroups.com

Hi Julien,

On Jun 6, 2017, at 12:44am, Julien Nioche <lists.dig...@gmail.com> wrote:

Hi,

All issues tagged for 0.8 have been addressed. Any objections to releasing 0.8? Am happy to do the release.

Sounds good - and thanks for being the release master!

— Ken

To unsubscribe from this group and stop receiving emails from it, send an email to crawler-commo...@googlegroups.com.

Visit this group at https://groups.google.com/group/crawler-commons.
For more options, visit https://groups.google.com/d/optout.

--------------------------

Ken Krugler

+1 530-210-6378

http://www.scaleunlimited.com

custom big data solutions & training

Hadoop, Cascading, Cassandra & Solr

Sebastian Nagel

unread,

Jun 6, 2017, 11:59:14 AM6/6/17

to crawler...@googlegroups.com

Hi Julien,

I'm running the sitemap performance test [1] on the two mentioned test files.

The SAX parser seems to be significantly slower than the last time I've run the tests
+40% for sitemap-test-2017-03-03.warc.gz
39s -> 57s
+10% for sitemap-test-2017-03-04.warc.gz
40s -> 45s

I'll double check whether the measurements are correct, and if yes, try to run a profile
to find the reason.

Sebastian

[1] https://groups.google.com/d/topic/crawler-commons/pOLsCVwRsxY/discussion

On 06/06/2017 09:44 AM, Julien Nioche wrote:
> Hi,
>
> All issues tagged for 0.8 have been addressed. Any objections to releasing 0.8? Am happy to do the
> release.
>
> Julien
>
> On 23 March 2017 at 15:38, Lewis John Mcgibbney <lewis.m...@gmail.com

> <mailto:lewis.m...@gmail.com>> wrote:
>
> Hi Julien,
> I've included all of the CHANGES for current development under the 0.8 milestone.
> Pending issues can be found at https://github.com/crawler-commons/crawler-commons/milestone/3
> <https://github.com/crawler-commons/crawler-commons/milestone/3>
> Lewis
>
> On Wed, Mar 22, 2017 at 2:39 AM, Julien Nioche <lists.dig...@gmail.com
> <mailto:lists.dig...@gmail.com>> wrote:
>
> Hi,
>
> There's been some great work done since the last release [1], notably the SAX-based sitemap
> parsing, it would be good to release 0.8 soon.
>
> Could you please look at existing issues

> <https://github.com/crawler-commons/crawler-commons/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aopen%20>

> and mark them as milestone crawler-commons-0.8?
>
> Thanks
>
> Julien
>
> [1] https://github.com/crawler-commons/crawler-commons/blob/master/CHANGES.txt

> <https://github.com/crawler-commons/crawler-commons/blob/master/CHANGES.txt>
>
> --
> *
> */Open Source Solutions for Text Engineering/
> /
> /http://www.digitalpebble.com <http://www.digitalpebble.com/>
> http://digitalpebble.blogspot.com/ <http://digitalpebble.blogspot.com/>
> #digitalpebble <http://twitter.com/digitalpebble>

>
> --
> You received this message because you are subscribed to the Google Groups "crawler-commons"
> group.
> To unsubscribe from this group and stop receiving emails from it, send an email to

> crawler-commo...@googlegroups.com
> <mailto:crawler-commo...@googlegroups.com>.

> Visit this group at https://groups.google.com/group/crawler-commons

> <https://groups.google.com/group/crawler-commons>.
> For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
>
>
>
>
> --
> /Lewis
> /

> Dr. Lewis J. McGibbney Ph.D, B.Sc
> Director, MCMA Associates

> Phone: +1(626)498-3090 <tel:(626)%20498-3090>
> Skype: lewis.john.mcgibbney
> Email: lewis.m...@gmail.com <mailto:lewis.m...@gmail.com>

>
>
> --
> You received this message because you are subscribed to the Google Groups "crawler-commons" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to

> crawler-commo...@googlegroups.com <mailto:crawler-commo...@googlegroups.com>.

> Visit this group at https://groups.google.com/group/crawler-commons

> <https://groups.google.com/group/crawler-commons>.
> For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
>
>
>
>
> --
> *
> */Open Source Solutions for Text Engineering/
> /
> /http://www.digitalpebble.com <http://www.digitalpebble.com/>
> http://digitalpebble.blogspot.com/
> #digitalpebble <http://twitter.com/digitalpebble>

>
> --
> You received this message because you are subscribed to the Google Groups "crawler-commons" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to

> crawler-commo...@googlegroups.com <mailto:crawler-commo...@googlegroups.com>.

Julien Nioche

unread,

Jun 6, 2017, 12:11:35 PM6/6/17

to crawler...@googlegroups.com

Great, thanks Sebastian!

On 6 June 2017 at 16:59, 'Sebastian Nagel' via crawler-commons <crawler...@googlegroups.com> wrote:

Hi Julien,

I'm running the sitemap performance test [1] on the two mentioned test files.

The SAX parser seems to be significantly slower than the last time I've run the tests
+40% for sitemap-test-2017-03-03.warc.gz
39s -> 57s
+10% for sitemap-test-2017-03-04.warc.gz
40s -> 45s

I'll double check whether the measurements are correct, and if yes, try to run a profile
to find the reason.

Sebastian

[1] https://groups.google.com/d/topic/crawler-commons/pOLsCVwRsxY/discussion

On 06/06/2017 09:44 AM, Julien Nioche wrote:
> Hi,
>
> All issues tagged for 0.8 have been addressed. Any objections to releasing 0.8? Am happy to do the
> release.
>
> Julien
>
> On 23 March 2017 at 15:38, Lewis John Mcgibbney <lewis.m...@gmail.com

> <mailto:lewis.mcgibbney@gmail.com>> wrote:
>
> Hi Julien,
> I've included all of the CHANGES for current development under the 0.8 milestone.
> Pending issues can be found at https://github.com/crawler-commons/crawler-commons/milestone/3
> <https://github.com/crawler-commons/crawler-commons/milestone/3>
> Lewis
>
> On Wed, Mar 22, 2017 at 2:39 AM, Julien Nioche <lists.dig...@gmail.com

> <mailto:lists.digitalpebble@gmail.com>> wrote:
>
> Hi,
>
> There's been some great work done since the last release [1], notably the SAX-based sitemap
> parsing, it would be good to release 0.8 soon.
>
> Could you please look at existing issues
> <https://github.com/crawler-commons/crawler-commons/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aopen%20>
> and mark them as milestone crawler-commons-0.8?
>
> Thanks
>
> Julien
>
> [1] https://github.com/crawler-commons/crawler-commons/blob/master/CHANGES.txt
> <https://github.com/crawler-commons/crawler-commons/blob/master/CHANGES.txt>
>
> --
> *
> */Open Source Solutions for Text Engineering/
> /
> /http://www.digitalpebble.com <http://www.digitalpebble.com/>
> http://digitalpebble.blogspot.com/ <http://digitalpebble.blogspot.com/>
> #digitalpebble <http://twitter.com/digitalpebble>

>
> --
> You received this message because you are subscribed to the Google Groups "crawler-commons"
> group.
> To unsubscribe from this group and stop receiving emails from it, send an email to

> crawler-commons+unsubscribe@googlegroups.com
> <mailto:crawler-commons+unsub...@googlegroups.com>.

> Visit this group at https://groups.google.com/group/crawler-commons
> <https://groups.google.com/group/crawler-commons>.
> For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
>
>
>
>
> --
> /Lewis
> /
> Dr. Lewis J. McGibbney Ph.D, B.Sc
> Director, MCMA Associates
> Phone: +1(626)498-3090 <tel:(626)%20498-3090>
> Skype: lewis.john.mcgibbney

> Email: lewis.m...@gmail.com <mailto:lewis.mcgibbney@gmail.com>

>
>
> --
> You received this message because you are subscribed to the Google Groups "crawler-commons" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to

> crawler-commons+unsubscribe@googlegroups.com <mailto:crawler-commons+unsub...@googlegroups.com>.

> Visit this group at https://groups.google.com/group/crawler-commons
> <https://groups.google.com/group/crawler-commons>.
> For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
>
>
>
>
> --
> *
> */Open Source Solutions for Text Engineering/
> /
> /http://www.digitalpebble.com <http://www.digitalpebble.com/>
> http://digitalpebble.blogspot.com/
> #digitalpebble <http://twitter.com/digitalpebble>
>
> --
> You received this message because you are subscribed to the Google Groups "crawler-commons" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to

> crawler-commons+unsubscribe@googlegroups.com <mailto:crawler-commons+unsub...@googlegroups.com>.

> Visit this group at https://groups.google.com/group/crawler-commons.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "crawler-commons" group.

To unsubscribe from this group and stop receiving emails from it, send an email to crawler-commons+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/crawler-commons.
For more options, visit https://groups.google.com/d/optout.

Sebastian Nagel

unread,

Jun 7, 2017, 8:23:35 AM6/7/17

to crawler...@googlegroups.com

+1

- the performance problem is fixed now, see
https://github.com/crawler-commons/crawler-commons/issues/153

I've run the performance tests again. Similar results than before.

Thanks!

On 06/06/2017 06:11 PM, Julien Nioche wrote:
> Great, thanks Sebastian!
>
> On 6 June 2017 at 16:59, 'Sebastian Nagel' via crawler-commons <crawler...@googlegroups.com

> <mailto:crawler...@googlegroups.com>> wrote:
>
> Hi Julien,
>
> I'm running the sitemap performance test [1] on the two mentioned test files.
>
> The SAX parser seems to be significantly slower than the last time I've run the tests
> +40% for sitemap-test-2017-03-03.warc.gz
> 39s -> 57s
> +10% for sitemap-test-2017-03-04.warc.gz
> 40s -> 45s
>
> I'll double check whether the measurements are correct, and if yes, try to run a profile
> to find the reason.
>
> Sebastian
>
> [1] https://groups.google.com/d/topic/crawler-commons/pOLsCVwRsxY/discussion
> <https://groups.google.com/d/topic/crawler-commons/pOLsCVwRsxY/discussion>
>
> On 06/06/2017 09:44 AM, Julien Nioche wrote:
> > Hi,
> >
> > All issues tagged for 0.8 have been addressed. Any objections to releasing 0.8? Am happy to do the
> > release.
> >
> > Julien
> >

> > On 23 March 2017 at 15:38, Lewis John Mcgibbney <lewis.m...@gmail.com <mailto:lewis.m...@gmail.com>

> > #digitalpebble <http://twitter.com/digitalpebble <http://twitter.com/digitalpebble>>

> >
> > --
> > You received this message because you are subscribed to the Google Groups "crawler-commons"
> > group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to

> > crawler-commo...@googlegroups.com <mailto:crawler-commons%2Bunsu...@googlegroups.com>
> > <mailto:crawler-commo...@googlegroups.com
> <mailto:crawler-commons%2Bunsu...@googlegroups.com>>.

> > Visit this group at https://groups.google.com/group/crawler-commons <https://groups.google.com/group/crawler-commons>
> > <https://groups.google.com/group/crawler-commons
> <https://groups.google.com/group/crawler-commons>>.
> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout> <https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>>.
> >
> >
> >
> >
> > --
> > /Lewis
> > /
> > Dr. Lewis J. McGibbney Ph.D, B.Sc
> > Director, MCMA Associates

> > Phone: +1(626)498-3090 <tel:%2B1%28626%29498-3090> <tel:(626)%20498-3090>
> > Skype: lewis.john.mcgibbney
> > Email: lewis.m...@gmail.com <mailto:lewis.m...@gmail.com>
> <mailto:lewis.m...@gmail.com <mailto:lewis.m...@gmail.com>>

> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups "crawler-commons" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to

> > crawler-commo...@googlegroups.com
> <mailto:crawler-commons%2Bunsu...@googlegroups.com>
> <mailto:crawler-commo...@googlegroups.com
> <mailto:crawler-commons%2Bunsu...@googlegroups.com>>.

> <https://groups.google.com/d/optout> <https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>>.
> >
> >
> >
> >
> > --
> > *
> > */Open Source Solutions for Text Engineering/
> > /
> > /http://www.digitalpebble.com <http://www.digitalpebble.com/>

> > http://digitalpebble.blogspot.com/ <http://digitalpebble.blogspot.com/>
> > #digitalpebble <http://twitter.com/digitalpebble <http://twitter.com/digitalpebble>>

> >
> > --
> > You received this message because you are subscribed to the Google Groups "crawler-commons" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to

> > crawler-commo...@googlegroups.com
> <mailto:crawler-commons%2Bunsu...@googlegroups.com>
> <mailto:crawler-commo...@googlegroups.com
> <mailto:crawler-commons%2Bunsu...@googlegroups.com>>.

> > Visit this group at https://groups.google.com/group/crawler-commons
> <https://groups.google.com/group/crawler-commons>.
> > For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
>
> --

> You received this message because you are subscribed to the Google Groups "crawler-commons" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to

> crawler-commo...@googlegroups.com
> <mailto:crawler-commons%2Bunsu...@googlegroups.com>.

> Visit this group at https://groups.google.com/group/crawler-commons
> <https://groups.google.com/group/crawler-commons>.
> For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
>
>
>
>
> --
> *
> */Open Source Solutions for Text Engineering/
> /
> /http://www.digitalpebble.com <http://www.digitalpebble.com/>
> http://digitalpebble.blogspot.com/
> #digitalpebble <http://twitter.com/digitalpebble>
>
> --
> You received this message because you are subscribed to the Google Groups "crawler-commons" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to

> crawler-commo...@googlegroups.com <mailto:crawler-commo...@googlegroups.com>.

Julien Nioche

unread,

Jun 7, 2017, 11:17:31 AM6/7/17

to crawler...@googlegroups.com

Thanks everyone! I have pushed a RC for 0.8

Artefacts => https://oss.sonatype.org/content/repositories/comgithubcrawler-commons-1004

Tag => https://github.com/crawler-commons/crawler-commons/releases/tag/crawler-commons-0.8

You can test it with Maven by inserting the following into your pom.xml

<name>your custom repo</name>

<url>https://oss.sonatype.org/content/repositories/comgithubcrawler-commons-1004/</url>

</repository>

</repositories>

I'll update the JAVADOCs, README etc... once the vote is finished.

Thanks

Julien

On 7 June 2017 at 13:23, 'Sebastian Nagel' via crawler-commons <crawler...@googlegroups.com> wrote:

+1

- the performance problem is fixed now, see
https://github.com/crawler-commons/crawler-commons/issues/153

I've run the performance tests again. Similar results than before.

Thanks!

On 06/06/2017 06:11 PM, Julien Nioche wrote:
> Great, thanks Sebastian!
>

> On 6 June 2017 at 16:59, 'Sebastian Nagel' via crawler-commons <crawler-commons@googlegroups.com

> <mailto:crawler-commons@googlegroups.com>> wrote:
>
> Hi Julien,
>
> I'm running the sitemap performance test [1] on the two mentioned test files.
>
> The SAX parser seems to be significantly slower than the last time I've run the tests
> +40% for sitemap-test-2017-03-03.warc.gz
> 39s -> 57s
> +10% for sitemap-test-2017-03-04.warc.gz
> 40s -> 45s
>
> I'll double check whether the measurements are correct, and if yes, try to run a profile
> to find the reason.
>
> Sebastian
>
> [1] https://groups.google.com/d/topic/crawler-commons/pOLsCVwRsxY/discussion
> <https://groups.google.com/d/topic/crawler-commons/pOLsCVwRsxY/discussion>
>
> On 06/06/2017 09:44 AM, Julien Nioche wrote:
> > Hi,
> >
> > All issues tagged for 0.8 have been addressed. Any objections to releasing 0.8? Am happy to do the
> > release.
> >
> > Julien
> >

> > On 23 March 2017 at 15:38, Lewis John Mcgibbney <lewis.m...@gmail.com <mailto:lewis.mcgibbney@gmail.com>

> > <mailto:lewis.mcgibbney@gmail.com <mailto:lewis.mcgibbney@gmail.com>>> wrote:
> >
> > Hi Julien,
> > I've included all of the CHANGES for current development under the 0.8 milestone.
> > Pending issues can be found at https://github.com/crawler-commons/crawler-commons/milestone/3
> <https://github.com/crawler-commons/crawler-commons/milestone/3>
> > <https://github.com/crawler-commons/crawler-commons/milestone/3
> <https://github.com/crawler-commons/crawler-commons/milestone/3>>
> > Lewis
> >

> > On Wed, Mar 22, 2017 at 2:39 AM, Julien Nioche <lists.dig...@gmail.com <mailto:lists.digitalpebble@gmail.com>

> > crawler-commons+unsubscribe@googlegroups.com <mailto:crawler-commons%2Bunsu...@googlegroups.com>
> > <mailto:crawler-commons+unsub...@googlegroups.com

> <mailto:crawler-commons%2Bunsu...@googlegroups.com>>.
> > Visit this group at https://groups.google.com/group/crawler-commons <https://groups.google.com/group/crawler-commons>
> > <https://groups.google.com/group/crawler-commons
> <https://groups.google.com/group/crawler-commons>>.
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout> <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
> >
> >
> >
> >
> > --
> > /Lewis
> > /
> > Dr. Lewis J. McGibbney Ph.D, B.Sc
> > Director, MCMA Associates
> > Phone: +1(626)498-3090 <tel:%2B1%28626%29498-3090> <tel:(626)%20498-3090>

> > Skype: lewis.john.mcgibbney
> > Email: lewis.m...@gmail.com <mailto:lewis.mcgibbney@gmail.com>
> <mailto:lewis.mcgibbney@gmail.com <mailto:lewis.mcgibbney@gmail.com>>

> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups "crawler-commons" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to

> > crawler-commons+unsubscribe@googlegroups.com
> <mailto:crawler-commons%2Bunsu...@googlegroups.com>
> <mailto:crawler-commons+unsub...@googlegroups.com

> <mailto:crawler-commons%2Bunsu...@googlegroups.com>>.
> > Visit this group at https://groups.google.com/group/crawler-commons <https://groups.google.com/group/crawler-commons>
> > <https://groups.google.com/group/crawler-commons
> <https://groups.google.com/group/crawler-commons>>.
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout> <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
> >
> >
> >
> >
> > --
> > *
> > */Open Source Solutions for Text Engineering/
> > /
> > /http://www.digitalpebble.com <http://www.digitalpebble.com/>
> > http://digitalpebble.blogspot.com/ <http://digitalpebble.blogspot.com/>
> > #digitalpebble <http://twitter.com/digitalpebble <http://twitter.com/digitalpebble>>

> >
> > --
> > You received this message because you are subscribed to the Google Groups "crawler-commons" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to

> > crawler-commons+unsubscribe@googlegroups.com
> <mailto:crawler-commons%2Bunsu...@googlegroups.com>
> <mailto:crawler-commons+unsub...@googlegroups.com
> <mailto:crawler-commons%2Bunsu...@googlegroups.com>>.

> > Visit this group at https://groups.google.com/group/crawler-commons
> <https://groups.google.com/group/crawler-commons>.
> > For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google Groups "crawler-commons" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to

> crawler-commons+unsubscribe@googlegroups.com
> <mailto:crawler-commons%2Bunsu...@googlegroups.com>.

> Visit this group at https://groups.google.com/group/crawler-commons
> <https://groups.google.com/group/crawler-commons>.
> For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
>
>
>
>
> --
> *
> */Open Source Solutions for Text Engineering/
> /
> /http://www.digitalpebble.com <http://www.digitalpebble.com/>
> http://digitalpebble.blogspot.com/
> #digitalpebble <http://twitter.com/digitalpebble>
>
> --
> You received this message because you are subscribed to the Google Groups "crawler-commons" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to

> crawler-commons+unsubscribe@googlegroups.com <mailto:crawler-commons+unsub...@googlegroups.com>.

> Visit this group at https://groups.google.com/group/crawler-commons.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "crawler-commons" group.

To unsubscribe from this group and stop receiving emails from it, send an email to crawler-commons+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/crawler-commons.
For more options, visit https://groups.google.com/d/optout.

Sebastian Nagel

unread,

Jun 8, 2017, 4:53:47 AM6/8/17

to crawler...@googlegroups.com

+1

Successfully upgraded a project to use 0.8.

On 06/07/2017 05:17 PM, Julien Nioche wrote:
> Hi
>
> Thanks everyone! I have pushed a RC for 0.8
>
> Artefacts => https://oss.sonatype.org/content/repositories/comgithubcrawler-commons-1004
>
>
> Tag => https://github.com/crawler-commons/crawler-commons/releases/tag/crawler-commons-0.8
>
> You can test it with Maven by inserting the following into your pom.xml
>
> <repositories>
> <repository>
> <id>my-repo2</id>
> <name>your custom repo</name>
> <url>https://oss.sonatype.org/content/repositories/comgithubcrawler-commons-1004/</url>
> </repository>
> </repositories>
>
> I'll update the JAVADOCs, README etc... once the vote is finished.
>
> Thanks
>
> Julien
>
>
> On 7 June 2017 at 13:23, 'Sebastian Nagel' via crawler-commons <crawler...@googlegroups.com

> <mailto:crawler...@googlegroups.com>> wrote:
>
> +1
>
> - the performance problem is fixed now, see
> https://github.com/crawler-commons/crawler-commons/issues/153
> <https://github.com/crawler-commons/crawler-commons/issues/153>
>
> I've run the performance tests again. Similar results than before.
>
> Thanks!
>
>
> On 06/06/2017 06:11 PM, Julien Nioche wrote:
> > Great, thanks Sebastian!
> >

> > On 6 June 2017 at 16:59, 'Sebastian Nagel' via crawler-commons <crawler...@googlegroups.com <mailto:crawler...@googlegroups.com>

> > <mailto:crawler...@googlegroups.com <mailto:crawler...@googlegroups.com>>> wrote:
> >
> > Hi Julien,
> >
> > I'm running the sitemap performance test [1] on the two mentioned test files.
> >
> > The SAX parser seems to be significantly slower than the last time I've run the tests
> > +40% for sitemap-test-2017-03-03.warc.gz
> > 39s -> 57s
> > +10% for sitemap-test-2017-03-04.warc.gz
> > 40s -> 45s
> >
> > I'll double check whether the measurements are correct, and if yes, try to run a profile
> > to find the reason.
> >
> > Sebastian
> >
> > [1] https://groups.google.com/d/topic/crawler-commons/pOLsCVwRsxY/discussion
> <https://groups.google.com/d/topic/crawler-commons/pOLsCVwRsxY/discussion>
> > <https://groups.google.com/d/topic/crawler-commons/pOLsCVwRsxY/discussion
> <https://groups.google.com/d/topic/crawler-commons/pOLsCVwRsxY/discussion>>
> >
> > On 06/06/2017 09:44 AM, Julien Nioche wrote:
> > > Hi,
> > >
> > > All issues tagged for 0.8 have been addressed. Any objections to releasing 0.8? Am happy to do the
> > > release.
> > >
> > > Julien
> > >

> > > On 23 March 2017 at 15:38, Lewis John Mcgibbney <lewis.m...@gmail.com <mailto:lewis.m...@gmail.com> <mailto:lewis.m...@gmail.com
> <mailto:lewis.m...@gmail.com>>
> > > <mailto:lewis.m...@gmail.com <mailto:lewis.m...@gmail.com> <mailto:lewis.m...@gmail.com

> <mailto:lewis.m...@gmail.com>>>> wrote:
> > >
> > > Hi Julien,
> > > I've included all of the CHANGES for current development under the 0.8 milestone.
> > > Pending issues can be found at https://github.com/crawler-commons/crawler-commons/milestone/3
> <https://github.com/crawler-commons/crawler-commons/milestone/3>
> > <https://github.com/crawler-commons/crawler-commons/milestone/3
> <https://github.com/crawler-commons/crawler-commons/milestone/3>>
> > > <https://github.com/crawler-commons/crawler-commons/milestone/3
> <https://github.com/crawler-commons/crawler-commons/milestone/3>
> > <https://github.com/crawler-commons/crawler-commons/milestone/3
> <https://github.com/crawler-commons/crawler-commons/milestone/3>>>
> > > Lewis
> > >

> > > On Wed, Mar 22, 2017 at 2:39 AM, Julien Nioche <lists.dig...@gmail.com <mailto:lists.dig...@gmail.com>
> <mailto:lists.dig...@gmail.com <mailto:lists.dig...@gmail.com>>
> > > <mailto:lists.dig...@gmail.com <mailto:lists.dig...@gmail.com>

> <http://twitter.com/digitalpebble> <http://twitter.com/digitalpebble

> <http://twitter.com/digitalpebble>>>
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups "crawler-commons"
> > > group.
> > > To unsubscribe from this group and stop receiving emails from it, send an email to

> > > crawler-commo...@googlegroups.com
> <mailto:crawler-commons%2Bunsu...@googlegroups.com>
> <mailto:crawler-commons%2Bunsu...@googlegroups.com
> <mailto:crawler-commons%252Buns...@googlegroups.com>>
> > > <mailto:crawler-commo...@googlegroups.com
> <mailto:crawler-commons%2Bunsu...@googlegroups.com>
> > <mailto:crawler-commons%2Bunsu...@googlegroups.com
> <mailto:crawler-commons%252Buns...@googlegroups.com>>>.

> > > Phone: +1(626)498-3090 <tel:%2B1%28626%29498-3090> <tel:%2B1%28626%29498-3090>
> <tel:(626)%20498-3090>
> > > Skype: lewis.john.mcgibbney
> > > Email: lewis.m...@gmail.com <mailto:lewis.m...@gmail.com> <mailto:lewis.m...@gmail.com
> <mailto:lewis.m...@gmail.com>>
> > <mailto:lewis.m...@gmail.com <mailto:lewis.m...@gmail.com>
> <mailto:lewis.m...@gmail.com <mailto:lewis.m...@gmail.com>>>
> > >
> > >

> > > --
> > > You received this message because you are subscribed to the Google Groups "crawler-commons" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an email to

> > > crawler-commo...@googlegroups.com <mailto:crawler-commons%2Bunsu...@googlegroups.com>
> > <mailto:crawler-commons%2Bunsu...@googlegroups.com
> <mailto:crawler-commons%252Buns...@googlegroups.com>>
> > <mailto:crawler-commo...@googlegroups.com
> <mailto:crawler-commons%2Bunsu...@googlegroups.com>
> > <mailto:crawler-commons%2Bunsu...@googlegroups.com
> <mailto:crawler-commons%252Buns...@googlegroups.com>>>.

> <https://groups.google.com/d/optout <https://groups.google.com/d/optout>
> > <https://groups.google.com/d/optout <https://groups.google.com/d/optout>>>.
> > >
> > >
> > >
> > >
> > > --
> > > *
> > > */Open Source Solutions for Text Engineering/
> > > /
> > > /http://www.digitalpebble.com <http://www.digitalpebble.com/>
> > > http://digitalpebble.blogspot.com/ <http://digitalpebble.blogspot.com/>

> <http://digitalpebble.blogspot.com/ <http://digitalpebble.blogspot.com/>>
> > > #digitalpebble <http://twitter.com/digitalpebble <http://twitter.com/digitalpebble>
> <http://twitter.com/digitalpebble <http://twitter.com/digitalpebble>>>
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups "crawler-commons" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an email to

> > > Visit this group at https://groups.google.com/group/crawler-commons <https://groups.google.com/group/crawler-commons>
> > <https://groups.google.com/group/crawler-commons <https://groups.google.com/group/crawler-commons>>.
> > > For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>
> <https://groups.google.com/d/optout <https://groups.google.com/d/optout>>.
> >
> > --

> > You received this message because you are subscribed to the Google Groups "crawler-commons" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to

> > crawler-commo...@googlegroups.com <mailto:crawler-commons%2Bunsu...@googlegroups.com>
> > <mailto:crawler-commons%2Bunsu...@googlegroups.com
> <mailto:crawler-commons%252Buns...@googlegroups.com>>.

> > Visit this group at https://groups.google.com/group/crawler-commons
> <https://groups.google.com/group/crawler-commons>
> > <https://groups.google.com/group/crawler-commons
> <https://groups.google.com/group/crawler-commons>>.
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout> <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
> >
> >
> >
> >
> > --
> > *
> > */Open Source Solutions for Text Engineering/
> > /
> > /http://www.digitalpebble.com <http://www.digitalpebble.com/>
> > http://digitalpebble.blogspot.com/ <http://digitalpebble.blogspot.com/>
> > #digitalpebble <http://twitter.com/digitalpebble <http://twitter.com/digitalpebble>>
> >
> > --
> > You received this message because you are subscribed to the Google Groups "crawler-commons" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to

> > crawler-commo...@googlegroups.com
> <mailto:crawler-commons%2Bunsu...@googlegroups.com>
> <mailto:crawler-commo...@googlegroups.com

> <mailto:crawler-commons%2Bunsu...@googlegroups.com>>.
> > Visit this group at https://groups.google.com/group/crawler-commons
> <https://groups.google.com/group/crawler-commons>.
> > For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google Groups "crawler-commons" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to

> crawler-commo...@googlegroups.com

> <mailto:crawler-commons%2Bunsu...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/crawler-commons
> <https://groups.google.com/group/crawler-commons>.
> For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
>
>
>
>
> --
> *
> */Open Source Solutions for Text Engineering/
> /
> /http://www.digitalpebble.com <http://www.digitalpebble.com/>
> http://digitalpebble.blogspot.com/
> #digitalpebble <http://twitter.com/digitalpebble>
>
> --
> You received this message because you are subscribed to the Google Groups "crawler-commons" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to

> crawler-commo...@googlegroups.com <mailto:crawler-commo...@googlegroups.com>.

Julien Nioche

unread,

Jun 8, 2017, 9:09:25 AM6/8/17

to crawler...@googlegroups.com

Thanks Sebastian. I'll proceed with the rest of the release tomorrow morning UK time unless we hear a -1 from anyone else in the meantime.

On 8 June 2017 at 09:53, 'Sebastian Nagel' via crawler-commons <crawler...@googlegroups.com> wrote:

+1

Successfully upgraded a project to use 0.8.

On 06/07/2017 05:17 PM, Julien Nioche wrote:
> Hi
>
> Thanks everyone! I have pushed a RC for 0.8
>
> Artefacts => https://oss.sonatype.org/content/repositories/comgithubcrawler-commons-1004
>
>
> Tag => https://github.com/crawler-commons/crawler-commons/releases/tag/crawler-commons-0.8
>
> You can test it with Maven by inserting the following into your pom.xml
>
> <repositories>
> <repository>
> <id>my-repo2</id>
> <name>your custom repo</name>
> <url>https://oss.sonatype.org/content/repositories/comgithubcrawler-commons-1004/</url>
> </repository>
> </repositories>
>
> I'll update the JAVADOCs, README etc... once the vote is finished.
>
> Thanks
>
> Julien
>
>

> On 7 June 2017 at 13:23, 'Sebastian Nagel' via crawler-commons <crawler-commons@googlegroups.com

> <mailto:crawler-commons@googlegroups.com>> wrote:
>
> +1
>
> - the performance problem is fixed now, see
> https://github.com/crawler-commons/crawler-commons/issues/153
> <https://github.com/crawler-commons/crawler-commons/issues/153>
>
> I've run the performance tests again. Similar results than before.
>
> Thanks!
>
>
> On 06/06/2017 06:11 PM, Julien Nioche wrote:
> > Great, thanks Sebastian!
> >

> > On 6 June 2017 at 16:59, 'Sebastian Nagel' via crawler-commons <crawler-commons@googlegroups.com <mailto:crawler-commons@googlegroups.com>

> > <mailto:crawler-commons@googlegroups.com <mailto:crawler-commons@googlegroups.com>>> wrote:
> >
> > Hi Julien,
> >
> > I'm running the sitemap performance test [1] on the two mentioned test files.
> >
> > The SAX parser seems to be significantly slower than the last time I've run the tests
> > +40% for sitemap-test-2017-03-03.warc.gz
> > 39s -> 57s
> > +10% for sitemap-test-2017-03-04.warc.gz
> > 40s -> 45s
> >
> > I'll double check whether the measurements are correct, and if yes, try to run a profile
> > to find the reason.
> >
> > Sebastian
> >
> > [1] https://groups.google.com/d/topic/crawler-commons/pOLsCVwRsxY/discussion
> <https://groups.google.com/d/topic/crawler-commons/pOLsCVwRsxY/discussion>
> > <https://groups.google.com/d/topic/crawler-commons/pOLsCVwRsxY/discussion
> <https://groups.google.com/d/topic/crawler-commons/pOLsCVwRsxY/discussion>>
> >
> > On 06/06/2017 09:44 AM, Julien Nioche wrote:
> > > Hi,
> > >
> > > All issues tagged for 0.8 have been addressed. Any objections to releasing 0.8? Am happy to do the
> > > release.
> > >
> > > Julien
> > >

> > > On 23 March 2017 at 15:38, Lewis John Mcgibbney <lewis.m...@gmail.com <mailto:lewis.mcgibbney@gmail.com> <mailto:lewis.mcgibbney@gmail.com
> <mailto:lewis.mcgibbney@gmail.com>>
> > > <mailto:lewis.mcgibbney@gmail.com <mailto:lewis.mcgibbney@gmail.com> <mailto:lewis.mcgibbney@gmail.com

> <mailto:lewis.mcgibbney@gmail.com>>>> wrote:
> > >
> > > Hi Julien,
> > > I've included all of the CHANGES for current development under the 0.8 milestone.
> > > Pending issues can be found at https://github.com/crawler-commons/crawler-commons/milestone/3
> <https://github.com/crawler-commons/crawler-commons/milestone/3>
> > <https://github.com/crawler-commons/crawler-commons/milestone/3
> <https://github.com/crawler-commons/crawler-commons/milestone/3>>
> > > <https://github.com/crawler-commons/crawler-commons/milestone/3
> <https://github.com/crawler-commons/crawler-commons/milestone/3>
> > <https://github.com/crawler-commons/crawler-commons/milestone/3
> <https://github.com/crawler-commons/crawler-commons/milestone/3>>>
> > > Lewis
> > >

> > > On Wed, Mar 22, 2017 at 2:39 AM, Julien Nioche <lists.dig...@gmail.com <mailto:lists.digitalpebble@gmail.com>
> <mailto:lists.digitalpebble@gmail.com <mailto:lists.digitalpebble@gmail.com>>
> > > <mailto:lists.digitalpebble@gmail.com <mailto:lists.digitalpebble@gmail.com>

> <http://twitter.com/digitalpebble> <http://twitter.com/digitalpebble

> > > crawler-commons+unsubscribe@googlegroups.com
> <mailto:crawler-commons%2Bunsu...@googlegroups.com>
> <mailto:crawler-commons%2Bunsu...@googlegroups.com
> <mailto:crawler-commons%252Bunsubscribe@googlegroups.com>>
> > > <mailto:crawler-commons+unsub...@googlegroups.com
> <mailto:crawler-commons%2Bunsu...@googlegroups.com>
> > <mailto:crawler-commons%2Bunsu...@googlegroups.com
> <mailto:crawler-commons%252Bunsubscribe@googlegroups.com>>>.

> > > Visit this group at https://groups.google.com/group/crawler-commons
> <https://groups.google.com/group/crawler-commons>
> <https://groups.google.com/group/crawler-commons <https://groups.google.com/group/crawler-commons>>
> > > <https://groups.google.com/group/crawler-commons <https://groups.google.com/group/crawler-commons>
> > <https://groups.google.com/group/crawler-commons <https://groups.google.com/group/crawler-commons>>>.
> > > For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>
> > <https://groups.google.com/d/optout <https://groups.google.com/d/optout>>
> <https://groups.google.com/d/optout <https://groups.google.com/d/optout>
> > <https://groups.google.com/d/optout <https://groups.google.com/d/optout>>>.
> > >
> > >
> > >
> > >
> > > --
> > > /Lewis
> > > /
> > > Dr. Lewis J. McGibbney Ph.D, B.Sc
> > > Director, MCMA Associates
> > > Phone: +1(626)498-3090 <tel:%2B1%28626%29498-3090> <tel:%2B1%28626%29498-3090>

> <tel:(626)%20498-3090>
> > > Skype: lewis.john.mcgibbney
> > > Email: lewis.m...@gmail.com <mailto:lewis.mcgibbney@gmail.com> <mailto:lewis.mcgibbney@gmail.com
> <mailto:lewis.mcgibbney@gmail.com>>
> > <mailto:lewis.mcgibbney@gmail.com <mailto:lewis.mcgibbney@gmail.com>
> <mailto:lewis.mcgibbney@gmail.com <mailto:lewis.mcgibbney@gmail.com>>>

> > >
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups "crawler-commons" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an email to

> > > crawler-commons+unsubscribe@googlegroups.com <mailto:crawler-commons%2Bunsu...@googlegroups.com>
> > <mailto:crawler-commons%2Bunsu...@googlegroups.com
> <mailto:crawler-commons%252Bunsubscribe@googlegroups.com>>
> > <mailto:crawler-commons+unsub...@googlegroups.com
> <mailto:crawler-commons%2Bunsu...@googlegroups.com>
> > <mailto:crawler-commons%2Bunsu...@googlegroups.com
> <mailto:crawler-commons%252Bunsubscribe@googlegroups.com>>>.

> > > Visit this group at https://groups.google.com/group/crawler-commons
> <https://groups.google.com/group/crawler-commons>
> <https://groups.google.com/group/crawler-commons <https://groups.google.com/group/crawler-commons>>
> > > <https://groups.google.com/group/crawler-commons <https://groups.google.com/group/crawler-commons>
> > <https://groups.google.com/group/crawler-commons <https://groups.google.com/group/crawler-commons>>>.
> > > For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>
> > <https://groups.google.com/d/optout <https://groups.google.com/d/optout>>
> <https://groups.google.com/d/optout <https://groups.google.com/d/optout>
> > <https://groups.google.com/d/optout <https://groups.google.com/d/optout>>>.
> > >
> > >
> > >
> > >
> > > --
> > > *
> > > */Open Source Solutions for Text Engineering/
> > > /
> > > /http://www.digitalpebble.com <http://www.digitalpebble.com/>
> > > http://digitalpebble.blogspot.com/ <http://digitalpebble.blogspot.com/>

> <http://digitalpebble.blogspot.com/ <http://digitalpebble.blogspot.com/>>
> > > #digitalpebble <http://twitter.com/digitalpebble <http://twitter.com/digitalpebble>
> <http://twitter.com/digitalpebble <http://twitter.com/digitalpebble>>>
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups "crawler-commons" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an email to

> > > crawler-commons+unsubscribe@googlegroups.com <mailto:crawler-commons%2Bunsu...@googlegroups.com>
> > <mailto:crawler-commons%2Bunsu...@googlegroups.com
> <mailto:crawler-commons%252Bunsubscribe@googlegroups.com>>
> > <mailto:crawler-commons+unsub...@googlegroups.com
> <mailto:crawler-commons%2Bunsu...@googlegroups.com>
> > <mailto:crawler-commons%2Bunsu...@googlegroups.com
> <mailto:crawler-commons%252Bunsubscribe@googlegroups.com>>>.

> > > Visit this group at https://groups.google.com/group/crawler-commons <https://groups.google.com/group/crawler-commons>
> > <https://groups.google.com/group/crawler-commons <https://groups.google.com/group/crawler-commons>>.
> > > For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>
> <https://groups.google.com/d/optout <https://groups.google.com/d/optout>>.
> >
> > --

> > You received this message because you are subscribed to the Google Groups "crawler-commons" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to

> > crawler-commons+unsubscribe@googlegroups.com <mailto:crawler-commons%2Bunsu...@googlegroups.com>
> > <mailto:crawler-commons%2Bunsu...@googlegroups.com
> <mailto:crawler-commons%252Bunsubscribe@googlegroups.com>>.

> > Visit this group at https://groups.google.com/group/crawler-commons
> <https://groups.google.com/group/crawler-commons>
> > <https://groups.google.com/group/crawler-commons
> <https://groups.google.com/group/crawler-commons>>.
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout> <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
> >
> >
> >
> >
> > --
> > *
> > */Open Source Solutions for Text Engineering/
> > /
> > /http://www.digitalpebble.com <http://www.digitalpebble.com/>
> > http://digitalpebble.blogspot.com/ <http://digitalpebble.blogspot.com/>
> > #digitalpebble <http://twitter.com/digitalpebble <http://twitter.com/digitalpebble>>
> >
> > --
> > You received this message because you are subscribed to the Google Groups "crawler-commons" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to

> > crawler-commons+unsubscribe@googlegroups.com
> <mailto:crawler-commons%2Bunsu...@googlegroups.com>
> <mailto:crawler-commons+unsub...@googlegroups.com

> <mailto:crawler-commons%2Bunsu...@googlegroups.com>>.
> > Visit this group at https://groups.google.com/group/crawler-commons
> <https://groups.google.com/group/crawler-commons>.
> > For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google Groups "crawler-commons" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to

> crawler-commons+unsubscribe@googlegroups.com

> <mailto:crawler-commons%2Bunsu...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/crawler-commons
> <https://groups.google.com/group/crawler-commons>.
> For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
>
>
>
>
> --
> *
> */Open Source Solutions for Text Engineering/
> /
> /http://www.digitalpebble.com <http://www.digitalpebble.com/>
> http://digitalpebble.blogspot.com/
> #digitalpebble <http://twitter.com/digitalpebble>
>
> --
> You received this message because you are subscribed to the Google Groups "crawler-commons" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to

> crawler-commons+unsubscribe@googlegroups.com <mailto:crawler-commons+unsub...@googlegroups.com>.

> Visit this group at https://groups.google.com/group/crawler-commons.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "crawler-commons" group.

To unsubscribe from this group and stop receiving emails from it, send an email to crawler-commons+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/crawler-commons.
For more options, visit https://groups.google.com/d/optout.

Ken Krugler

unread,

Jun 8, 2017, 8:54:25 PM6/8/17

to crawler...@googlegroups.com

Hi Julien,

+1 from me, tried with a project and worked fine.

One comment for the release note would be the move of the PaidLevelDomain class in our package hierarchy:

- import crawlercommons.url.PaidLevelDomain;
+ import crawlercommons.domains.PaidLevelDomain;

— Ken

To unsubscribe from this group and stop receiving emails from it, send an email to crawler-commo...@googlegroups.com.

Visit this group at https://groups.google.com/group/crawler-commons.
For more options, visit https://groups.google.com/d/optout.

Lewis John Mcgibbney

unread,

Jun 9, 2017, 12:17:14 AM6/9/17

to crawler...@googlegroups.com

Didn't get a chance to upgrade any project so far, I've pulled the tag, built and tested it. All looks good Julien.

Thank you for cutting. +1

Lewis

> On 6 June 2017 at 16:59, 'Sebastian Nagel' via crawler-commons <crawler...@googlegroups.com

> <mailto:crawler...@googlegroups.com>> wrote:
>
> Hi Julien,
>
> I'm running the sitemap performance test [1] on the two mentioned test files.
>
> The SAX parser seems to be significantly slower than the last time I've run the tests
> +40% for sitemap-test-2017-03-03.warc.gz
> 39s -> 57s
> +10% for sitemap-test-2017-03-04.warc.gz
> 40s -> 45s
>
> I'll double check whether the measurements are correct, and if yes, try to run a profile
> to find the reason.
>
> Sebastian
>
> [1] https://groups.google.com/d/topic/crawler-commons/pOLsCVwRsxY/discussion
> <https://groups.google.com/d/topic/crawler-commons/pOLsCVwRsxY/discussion>
>
> On 06/06/2017 09:44 AM, Julien Nioche wrote:
> > Hi,
> >
> > All issues tagged for 0.8 have been addressed. Any objections to releasing 0.8? Am happy to do the
> > release.
> >
> > Julien
> >

> > On 23 March 2017 at 15:38, Lewis John Mcgibbney <lewis.m...@gmail.com <mailto:lewis.m...@gmail.com>

> > <mailto:lewis.m...@gmail.com <mailto:lewis.m...@gmail.com>>> wrote:
> >
> > Hi Julien,
> > I've included all of the CHANGES for current development under the 0.8 milestone.
> > Pending issues can be found at https://github.com/crawler-commons/crawler-commons/milestone/3
> <https://github.com/crawler-commons/crawler-commons/milestone/3>
> > <https://github.com/crawler-commons/crawler-commons/milestone/3
> <https://github.com/crawler-commons/crawler-commons/milestone/3>>
> > Lewis
> >

> > On Wed, Mar 22, 2017 at 2:39 AM, Julien Nioche <lists.dig...@gmail.com <mailto:lists.dig...@gmail.com>

> > crawler-commo...@googlegroups.com <mailto:crawler-commons%2Bunsu...@googlegroups.com>
> > <mailto:crawler-commo...@googlegroups.com

> <mailto:crawler-commons%2Bunsu...@googlegroups.com>>.
> > Visit this group at https://groups.google.com/group/crawler-commons <https://groups.google.com/group/crawler-commons>
> > <https://groups.google.com/group/crawler-commons
> <https://groups.google.com/group/crawler-commons>>.
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout> <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
> >
> >
> >
> >
> > --
> > /Lewis
> > /
> > Dr. Lewis J. McGibbney Ph.D, B.Sc
> > Director, MCMA Associates
> > Phone: +1(626)498-3090 <tel:%2B1%28626%29498-3090> <tel:(626)%20498-3090>

> > Skype: lewis.john.mcgibbney
> > Email: lewis.m...@gmail.com <mailto:lewis.m...@gmail.com>
> <mailto:lewis.m...@gmail.com <mailto:lewis.m...@gmail.com>>
> >
> >

> > --
> > You received this message because you are subscribed to the Google Groups "crawler-commons" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to

> > crawler-commo...@googlegroups.com
> <mailto:crawler-commons%2Bunsu...@googlegroups.com>
> <mailto:crawler-commo...@googlegroups.com

> <mailto:crawler-commons%2Bunsu...@googlegroups.com>>.
> > Visit this group at https://groups.google.com/group/crawler-commons <https://groups.google.com/group/crawler-commons>
> > <https://groups.google.com/group/crawler-commons
> <https://groups.google.com/group/crawler-commons>>.
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout> <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
> >
> >
> >
> >
> > --
> > *
> > */Open Source Solutions for Text Engineering/
> > /
> > /http://www.digitalpebble.com <http://www.digitalpebble.com/>
> > http://digitalpebble.blogspot.com/ <http://digitalpebble.blogspot.com/>
> > #digitalpebble <http://twitter.com/digitalpebble <http://twitter.com/digitalpebble>>

> >
> > --
> > You received this message because you are subscribed to the Google Groups "crawler-commons" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to

> > crawler-commo...@googlegroups.com
> <mailto:crawler-commons%2Bunsu...@googlegroups.com>
> <mailto:crawler-commo...@googlegroups.com
> <mailto:crawler-commons%2Bunsu...@googlegroups.com>>.

> > Visit this group at https://groups.google.com/group/crawler-commons
> <https://groups.google.com/group/crawler-commons>.
> > For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google Groups "crawler-commons" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to

> crawler-commo...@googlegroups.com
> <mailto:crawler-commons%2Bunsu...@googlegroups.com>.

> Visit this group at https://groups.google.com/group/crawler-commons
> <https://groups.google.com/group/crawler-commons>.
> For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
>
>
>
>
> --
> *
> */Open Source Solutions for Text Engineering/
> /
> /http://www.digitalpebble.com <http://www.digitalpebble.com/>
> http://digitalpebble.blogspot.com/
> #digitalpebble <http://twitter.com/digitalpebble>
>
> --
> You received this message because you are subscribed to the Google Groups "crawler-commons" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to

> crawler-commo...@googlegroups.com <mailto:crawler-commo...@googlegroups.com>.

> Visit this group at https://groups.google.com/group/crawler-commons.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "crawler-commons" group.

Lewis

Dr. Lewis J. McGibbney Ph.D, B.Sc

Director, MCMA Associates

Phone: +1(626)498-3090

Julien Nioche

unread,

Jun 9, 2017, 5:14:56 AM6/9/17

to crawler...@googlegroups.com

Hi Ken

This was done in the previous release and documented in the announcement at the time => "the package crawlercommons.url has been replaced with crawlercommons.domains"

Thanks

Julien

> > crawler-commons+unsubscribe@googlegroups.com <mailto:crawler-commons%2Bunsubsc...@googlegroups.com>
> > <mailto:crawler-commons+unsubscr...@googlegroups.com
> <mailto:crawler-commons%2Bunsubsc...@googlegroups.com>>.

> > Visit this group at https://groups.google.com/group/crawler-commons <https://groups.google.com/group/crawler-commons>
> > <https://groups.google.com/group/crawler-commons
> <https://groups.google.com/group/crawler-commons>>.
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout> <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
> >
> >
> >
> >
> > --
> > /Lewis
> > /
> > Dr. Lewis J. McGibbney Ph.D, B.Sc
> > Director, MCMA Associates
> > Phone: +1(626)498-3090 <tel:%2B1%28626%29498-3090> <tel:(626)%20498-3090>
> > Skype: lewis.john.mcgibbney
> > Email: lewis.m...@gmail.com <mailto:lewis.mcgibbney@gmail.com>
> <mailto:lewis.mcgibbney@gmail.com <mailto:lewis.mcgibbney@gmail.com>>
> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups "crawler-commons" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to
> > crawler-commons+unsubscribe@googlegroups.com

> <mailto:crawler-commons%2Bunsubsc...@googlegroups.com>
> <mailto:crawler-commons+unsubscr...@googlegroups.com
> <mailto:crawler-commons%2Bunsubsc...@googlegroups.com>>.

> > Visit this group at https://groups.google.com/group/crawler-commons <https://groups.google.com/group/crawler-commons>
> > <https://groups.google.com/group/crawler-commons
> <https://groups.google.com/group/crawler-commons>>.
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout> <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
> >
> >
> >
> >
> > --
> > *
> > */Open Source Solutions for Text Engineering/
> > /
> > /