Unload hg.tryton.org

[Cédric Krier]

Hi,

The hg.tryton.org site has served 66GB for June. But mainly to only 3
hosts:

* ns332279.ip-91-121-118.eu: 25.86 GB
* static-160-102-225-77.ipcom.comunitel.net: 24.04 GB
* cli-???.wholesale.adamo.es: 13.26 GB

Those 3 hosts consume more than 90% of the bandwidth of the site.
I suspect those are testing machines that clone (from scratch)
repositories on each builds.
I would like to request to the owner of those hosts, a fare use of our
resources. For that there are many options:

* Use a local mirror of the Tryton repositories (like
drone.tryton.org does with hgwebcachingproxy [1])

* Use mirrors instead of hg.tryton.org [2]

* Use PyPI packages if you do not build on development branch



Rem: I disclosed those two first host names because I pretty sure the
two first are servers and last one I hide some part because it looks
like a dynamic customer IP.


[1] https://bitbucket.org/Unity-Technologies/hgwebcachingproxy
[2] https://bugs.tryton.org/issue7536


Thanks to cooperate,
--
Cédric Krier - B2CK SPRL
Email/Jabber: cedric...@b2ck.com
Tel: +32 472 54 46 59
Website: http://www.b2ck.com/

[Cédric Krier]

On 2018-07-06 10:16, Cédric Krier wrote:
> Hi,
>
> The hg.tryton.org site has served 66GB for June. But mainly to only 3
> hosts:
>
> * ns332279.ip-91-121-118.eu: 25.86 GB
> * static-160-102-225-77.ipcom.comunitel.net: 24.04 GB
> * cli-???.wholesale.adamo.es: 13.26 GB
>
> Those 3 hosts consume more than 90% of the bandwidth of the site.
> I suspect those are testing machines that clone (from scratch)
> repositories on each builds.
> I would like to request to the owner of those hosts, a fare use of our
> resources. For that there are many options:
>
> * Use a local mirror of the Tryton repositories (like
> drone.tryton.org does with hgwebcachingproxy [1])
>
> * Use mirrors instead of hg.tryton.org [2]
>
> * Use PyPI packages if you do not build on development branch

We found that the issue may be in tox.ini the install_command contains
'--find-links https://trydevpi.tryton.org/' but it should be only:

install_command = pip install --pre --process-dependency-links {opts} {packages}

[Albert Cervera i Areny]

2018-07-06 10:16 GMT+02:00 Cédric Krier <cedric...@b2ck.com>:

Hi,

The hg.tryton.org site has served 66GB for June. But mainly to only 3
hosts:

    * ns332279.ip-91-121-118.eu: 25.86 GB

NaN-tic responsible for this one and we're going to use the bitbucket mirror soon.

 

    * static-160-102-225-77.ipcom.comunitel.net: 24.04 GB
    * cli-???.wholesale.adamo.es: 13.26 GB

Those 3 hosts consume more than 90% of the bandwidth of the site.
I suspect those are testing machines that clone (from scratch)
repositories on each builds.
I would like to request to the owner of those hosts, a fare use of our
resources. For that there are many options:

    * Use a local mirror of the Tryton repositories (like
      drone.tryton.org does with hgwebcachingproxy [1])

    * Use mirrors instead of hg.tryton.org [2]

    * Use PyPI packages if you do not build on development branch



Rem: I disclosed those two first host names because I pretty sure the
two first are servers and last one I hide some part because it looks
like a dynamic customer IP.


[1] https://bitbucket.org/Unity-Technologies/hgwebcachingproxy
[2] https://bugs.tryton.org/issue7536


Thanks to cooperate,

--
Cédric Krier - B2CK SPRL
Email/Jabber: cedric...@b2ck.com
Tel: +32 472 54 46 59
Website: http://www.b2ck.com/

--
You received this message because you are subscribed to the Google Groups "tryton" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/tryton/20180706081618.GL4715%40kei.

--

Albert Cervera i Areny
http://www.NaN-tic.com
Tel. 93 553 18 03

[Sergi Almacellas Abellana]

El 06/07/18 a les 10:16, Cédric Krier ha escrit:

> * Use a local mirror of the Tryton repositories (like
> drone.tryton.org does with hgwebcachingproxy [1])

We've setup an hgwebcachingproxy and setup dnsmasq to use the proxy ip
for cloning the repositories on our testing environment.

Cloning the repositories using the http protocol works well but when
using the https server we've get a certificate error. Which is expected
as we do not have any valid certificate for hg.tryton.org

Could you explain how do you solved this issue on drone.tryton.org?




--
Sergi Almacellas Abellana
www.koolpi.com
Twitter: @pokoli_srk

[Cédric Krier]

On 2018-07-09 17:22, Sergi Almacellas Abellana wrote:
> El 06/07/18 a les 10:16, Cédric Krier ha escrit:
> > * Use a local mirror of the Tryton repositories (like
> > drone.tryton.org does with hgwebcachingproxy [1])
>
> We've setup an hgwebcachingproxy and setup dnsmasq to use the proxy ip
> for cloning the repositories on our testing environment.
>
> Cloning the repositories using the http protocol works well but when
> using the https server we've get a certificate error. Which is expected
> as we do not have any valid certificate for hg.tryton.org
>
> Could you explain how do you solved this issue on drone.tryton.org?

Well we have a valid certificate for hg.tryton.org
So probably the best is to have a custom trydevpy or we could provide
another parameter to define a custom domain name.

[Cédric Krier]

On 2018-07-09 17:55, Cédric Krier wrote:
> On 2018-07-09 17:22, Sergi Almacellas Abellana wrote:
> > El 06/07/18 a les 10:16, Cédric Krier ha escrit:
> > > * Use a local mirror of the Tryton repositories (like
> > > drone.tryton.org does with hgwebcachingproxy [1])
> >
> > We've setup an hgwebcachingproxy and setup dnsmasq to use the proxy ip
> > for cloning the repositories on our testing environment.
> >
> > Cloning the repositories using the http protocol works well but when
> > using the https server we've get a certificate error. Which is expected
> > as we do not have any valid certificate for hg.tryton.org
> >
> > Could you explain how do you solved this issue on drone.tryton.org?
>
> Well we have a valid certificate for hg.tryton.org
> So probably the best is to have a custom trydevpy or we could provide
> another parameter to define a custom domain name.

For the record, I just stopped to use hgwebcachingproxy because it does
not work on latest mercurial version (and it is too much work to upgrade
it). So now, we run a simple `hg serve` on the machine which a
preoutgoing hook to keep repository up to date:
https://stackoverflow.com/questions/3925007/creating-a-local-transparent-cache-of-a-mercurial-repository

The only disadvantage is that new repositories must be created manually
on the server. But now, we rely only on standard mercurial tools.

[Cédric Krier]

On 2018-07-06 10:16, Cédric Krier wrote:

> Hi,
>
> The hg.tryton.org site has served 66GB for June. But mainly to only 3
> hosts:
>
> * ns332279.ip-91-121-118.eu: 25.86 GB
> * static-160-102-225-77.ipcom.comunitel.net: 24.04 GB
> * cli-???.wholesale.adamo.es: 13.26 GB

The statistics for July shows:

* ns332279.ip-91-121-118.eu: 27.62 GB
* static-160-102-225-77.ipcom.comunitel.net: 25.26 GB
* cli-???.wholesale.adamo.es: 1.98 GB

So thanks to cli-???.wholesale.adamo.es to have reduced by 10 times its
consumption.

The others please to try to reduce your consumption by using the
bitbucket or github mirrors.

Thanks,