PGXN-Tester: build failed on Multicorn

[Ronan Dunklau]

Hello,

I'm the Multicorn project maintainer, and have seen that the pgxn-tester
reports Multicorn as not compiling. From the available logs, it seems python
2.6 is installed on the buildfarm members.

How can I help to ensure Multicorn builds on the buildfarm ? The results look
bad on the website, even though I know its tests pass successfully from 9.2
to 9.4, with several python versions (2.7 / 3.4 / 3.3).

Thank you !

--
Ronan Dunklau

[Tomas Vondra]

Hi Ronan,

On 06/08/15 10:45, Ronan Dunklau wrote:
> Hello,
>
> I'm the Multicorn project maintainer, and have seen that the
> pgxn-tester reports Multicorn as not compiling. From the available
> logs, it seems python 2.6 is installed on the buildfarm members.

Yeah, it's a CentOS 6.6 box IIRC, and that only provides python 2.6 by
default. I'll try to upgrade that to a newer python version (probably
2.7) somehow - probably by compiling that.

> How can I help to ensure Multicorn builds on the buildfarm ?

Is there anything else that should be done?

> The results look bad on the website, even though I know its tests
> pass successfully from 9.2 to 9.4, with several python versions (2.7
> / 3.4 / 3.3).

I know - handling packages with external dependencies is one of the weak
spots of pgxn-tester. multicorn is not the only package impacted by this :-/

Sorry about that.

regards
Tomas

[Jim Nasby]

On 6/9/15 11:53 AM, Tomas Vondra wrote:
> I know - handling packages with external dependencies is one of the weak
> spots of pgxn-tester. multicorn is not the only package impacted by this
> :-/

FWIW, it's more a problem of PGXN itself; there should be some way to at
least provide references to external dependencies. Sadly that won't help
much with integrating with OS packaging systems, but it could at least
provide a URL if not a script that would look for required deps.
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Data in Trouble? Get it in Treble! http://BlueTreble.com

[David E. Wheeler]

On Jun 9, 2015, at 10:30 AM, Jim Nasby <Jim....@BlueTreble.com> wrote:

> FWIW, it's more a problem of PGXN itself; there should be some way to at least provide references to external dependencies. Sadly that won't help much with integrating with OS packaging systems, but it could at least provide a URL if not a script that would look for required deps.

I’m happy to entertain proposals for how to handle this sort of thing. Some META.yml section. I’ve pinged the Perl toolchain folks, but they haven’t done it yet, either. Requirements, to my mind, would be:

* Platform-independent way of specifying dependencies.
* Provide enough information for a client to determine and install dependencies.

I have no idea what this would look like.

Best,

David

[David E. Wheeler]

On Jun 9, 2015, at 11:13 AM, David E. Wheeler <da...@justatheory.com> wrote:

> I’ve pinged the Perl toolchain folks, but they haven’t done it yet, either.

They’ve had a little discussion here:

https://github.com/Perl-Toolchain-Gang/CPAN-Meta/issues/82

No good solutions, though.

Best,

David

[Jim Nasby]

Yeah. The ideas aren't bad, but the potential for feature-creep is enormous.

Something I think we can minimally do, that's not "dangerous" is add a
field to META that specifies a script that will test for necessary deps
and spit out a message for the user for anything that's missing. That's
the ultimate in flexibility, lets PGXN know there's at least some kind
of external dep, as well as what to run to tell the user what's missing
(if anything). If the script correctly exits non-zero on a problem pgxn
clients would know to stop.

We could possibly go a bit further though, without painting ourselves in
a corner. If you think about what a 'dependency script' would spit out
at a user, it would be:

- some common name for the package
- version restrictions (if any)
- a website to find the package at

If that data was machine-parsable, it wouldn't be hard to create
something that took the package name and/or URL and searched the
packaging system for a specific OS looking for it. Between those two
pieces of data you could probably find a unique package hit 90% of the
time or more.

Obviously this wouldn't be perfect, but it would probably suffice most
of the time.

[Ronan Dunklau]

Le mardi 9 juin 2015 18:53:02 Tomas Vondra a écrit :
> Hi Ronan,
>
> On 06/08/15 10:45, Ronan Dunklau wrote:
> > Hello,
> >
> > I'm the Multicorn project maintainer, and have seen that the
> > pgxn-tester reports Multicorn as not compiling. From the available
> > logs, it seems python 2.6 is installed on the buildfarm members.
>
> Yeah, it's a CentOS 6.6 box IIRC, and that only provides python 2.6 by
> default. I'll try to upgrade that to a newer python version (probably
> 2.7) somehow - probably by compiling that.
>
> > How can I help to ensure Multicorn builds on the buildfarm ?
>
> Is there anything else that should be done?

This will let most of the test pass. There are some tests specific to the
sqlalchemy_fdw, since it is the most used one as far as I can tell, so those
tests will fail if the sqlalchemy python library is not installed. But
honestly, that is something I could probably take care of myself by excluding
those tests if sqlalchemy is not installed, since it is an optional
dependency.

>
> > The results look bad on the website, even though I know its tests
> > pass successfully from 9.2 to 9.4, with several python versions (2.7
> > / 3.4 / 3.3).
>
> I know - handling packages with external dependencies is one of the weak
> spots of pgxn-tester. multicorn is not the only package impacted by this :-/

>
> Sorry about that.

No need to ! Such a platform is a fantastic idea, and the automated part of it
gives incentive to automate the whole build and install process, which will
ultimately benefit end-users too.

Thank you for your work on that.

>
> regards
> Tomas

[Tomas Vondra]

On 06/10/15 11:47, Ronan Dunklau wrote:
> Le mardi 9 juin 2015 18:53:02 Tomas Vondra a écrit :
>> Hi Ronan,
>>
>> On 06/08/15 10:45, Ronan Dunklau wrote:
>>> Hello,
>>>
>>> I'm the Multicorn project maintainer, and have seen that the
>>> pgxn-tester reports Multicorn as not compiling. From the available
>>> logs, it seems python 2.6 is installed on the buildfarm members.
>>
>> Yeah, it's a CentOS 6.6 box IIRC, and that only provides python 2.6 by
>> default. I'll try to upgrade that to a newer python version (probably
>> 2.7) somehow - probably by compiling that.
>>
>>> How can I help to ensure Multicorn builds on the buildfarm ?
>>
>> Is there anything else that should be done?
>
> This will let most of the test pass. There are some tests specific to the
> sqlalchemy_fdw, since it is the most used one as far as I can tell, so those
> tests will fail if the sqlalchemy python library is not installed. But
> honestly, that is something I could probably take care of myself by excluding
> those tests if sqlalchemy is not installed, since it is an optional
> dependency.

I've built a local Python 2.7.6 - for now only on the 'hactar' machine,
and forced retesting of the multicorn extension (by removing the
original benchmarks). It still fails, but with different error messages
at compile time. Can you take a look?

The best to look at the recent results is http://pgxn-tester.org/results

regards
Tomas

[Ronan Dunklau]

Thank you. I took a look at the results for 1.2.0 and it appears:

- libpq seems to be missing for the test including a local connection to the
backend: ! ERROR: Error in python: ImportError
! DETAIL: libpq.so.5: cannot open shared object file: No such file or
directory

It seems there is something wrong with the psycopg2 installation. How did you
install it ?

- several errors stems from the fact that the database used for the tests are
created with the SQL_ASCII encoding. I must admit I always ran my tests on
UTF-8 enabled databases. Some of those errors are bugs in Multicorn, some are
"normal", in the sense that it won't be possible to convert some unicode
characters to ASCII.

For other versions, it seems there is a problem with the CFLAGS used, in
particular fpic. I'll take a closer look at it when possible.

Again, thank you for all of this.



>
> regards
> Tomas

[Tomas Vondra]

I wonder whether this happens because of the multiple PostgreSQL
versions. Will look into that later today.

> - several errors stems from the fact that the database used for the tests are
> created with the SQL_ASCII encoding. I must admit I always ran my tests on
> UTF-8 enabled databases. Some of those errors are bugs in Multicorn, some are
> "normal", in the sense that it won't be possible to convert some unicode
> characters to ASCII.
>
> For other versions, it seems there is a problem with the CFLAGS used, in
> particular fpic. I'll take a closer look at it when possible.

OK, let me know if I need to tweak any of those (but I have not changed
anything, IIRC). I had to build the Python with -fPIC, but that's all.

Tomas

[Ronan Dunklau]

So, I've released a new distribution of Multicorn, specifically to correct the
bugs on my end regarding encoding, and to make the sqlalchemy test run
depending on whether or not sqlalchemy and psycopg2 are correctly installed.

Results are here:

http://pgxn-tester.org/distributions/multicorn/1.2.2

Thank you very much for your cooperation on this issue !


>
> Tomas

[Tomas Vondra]

On 06/19/15 08:50, Ronan Dunklau wrote:
>
> So, I've released a new distribution of Multicorn, specifically to correct the
> bugs on my end regarding encoding, and to make the sqlalchemy test run
> depending on whether or not sqlalchemy and psycopg2 are correctly installed.
>
> Results are here:
>
> http://pgxn-tester.org/distributions/multicorn/1.2.2
>
> Thank you very much for your cooperation on this issue !

Cool! I'll fix the other build machines once I get back home. I'll have
to remember what I did on hactar, though ;-)

Tomas