PGXN Client commands dispatch

[Daniele Varrazzo]

Hello,

I've refactored the PGXN Client code into an internal implementation
script and a generic dispatch script called pgxn. This way it is
possible to add commands implemented into a program or script written
in whatever language. The mechanism is not dissimilar from the way git
is extended.

The code is implemented in the devel branch of the pgxn client
repository <https://github.com/dvarrazzo/pgxnclient/>. There is not a
tag for the feature as there are still other features I want to
implement for the release 0.3. You can check out the code and run it
from the working copy, without the need of installing it.

I'm looking forward for a real user for this feature: it can be for
instance the PGXN Utils. I'm available for any help/feedback about
interfacing the "utils" with the "client". I am also positive in
merging the utils source repository and distribute everything under
the "client" flag.

Further informations about how to extend the client are available at
<http://pgxnclient.projects.postgresql.org/ext.html> and in the source
code. I may have missed some point as the project is a little bit
"cold": I couldn't have worked on it for quite a long time. Please
write here/me for any help request. In the next days I will use my
Copious Spare Time™ to work at the next PGXN Client release.

Cheers,

-- Daniele

[David E. Wheeler]

On Jul 6, 2011, at 12:53 PM, Daniele Varrazzo wrote:

> I've refactored the PGXN Client code into an internal implementation
> script and a generic dispatch script called pgxn. This way it is
> possible to add commands implemented into a program or script written
> in whatever language. The mechanism is not dissimilar from the way git
> is extended.

This is awesome, thanks Daniele!

> I'm looking forward for a real user for this feature: it can be for
> instance the PGXN Utils. I'm available for any help/feedback about
> interfacing the "utils" with the "client". I am also positive in
> merging the utils source repository and distribute everything under
> the "client" flag.

I'll try to put aside some time to get pgxn_meta_validator to work with it, so that if it's installed, the user can just do

pgxn validate-meta META.json

> Further informations about how to extend the client are available at
> <http://pgxnclient.projects.postgresql.org/ext.html> and in the source
> code. I may have missed some point as the project is a little bit
> "cold": I couldn't have worked on it for quite a long time. Please
> write here/me for any help request. In the next days I will use my
> Copious Spare Time™ to work at the next PGXN Client release.

Very much appreciated, Daniele. Personally I'm overcommitted these days, too, but I think it's important to keep pushing this stuff forward, so I'll carve out some time for it next week.

Best,

David

[Dickson S. Guedes]

2011/7/6 Daniele Varrazzo <daniele....@gmail.com>:

> I've refactored the PGXN Client code into an internal implementation
> script and a generic dispatch script called pgxn. This way it is
> possible to add commands implemented into a program or script written
> in whatever language. The mechanism is not dissimilar from the way git
> is extended.

Very cool! Thanks!

> The code is implemented in the devel branch of the pgxn client
> repository <https://github.com/dvarrazzo/pgxnclient/>. There is not a
> tag for the feature as there are still other features I want to
> implement for the release 0.3. You can check out the code and run it
> from the working copy, without the need of installing it.

I did have to install to make it work, cause just typing ./bin/pgxn
was throwing an error. But a 'python seput.py build' followed by a
'sudo python setup.py install' works!

> I'm looking forward for a real user for this feature: it can be for
> instance the PGXN Utils. I'm available for any help/feedback about
> interfacing the "utils" with the "client".

I have some alternatives:

1) create a task on PGXN Utils to "register" it to PGXN Client, that
creates a symlink on 'libexec' dir. This could be dangerous in some
situations when a new version of (in this case) 'pgxn_utils' is
installed and the link are point to the old version, so the user must
call 'register' on update;

2) change the actual PGXN Utils executable name from 'pgxn_utils' to
'pgxn-utils', that sounds intrusive, but how it will be in the PATH,
the symlink ins't necessary. The next pgxn_utils version could be
changed to the provide the new executable name and everything will
works as expected;

3) provide a libexec script packaged with PGXN Client named
'pgxn-utils' that, when called, checks for the actual path of
'pgxn_utils' and dispatch it;

I feel that 2nd and 3rd are more O.S. portable than 1rst. I'm more
inclined to option 2 (two) but I want to know your opinion guys.

Best.
--
Dickson S. Guedes
e-mail/gtalk: gue...@guedesof.net  - skype: guediz
http://guedesoft.net - http://www.postgresql.org.br

[David E. Wheeler]

On Jul 16, 2011, at 6:12 PM, Dickson S. Guedes wrote:

> I have some alternatives:
>
> 1) create a task on PGXN Utils to "register" it to PGXN Client, that
> creates a symlink on 'libexec' dir. This could be dangerous in some
> situations when a new version of (in this case) 'pgxn_utils' is
> installed and the link are point to the old version, so the user must
> call 'register' on update;
>
> 2) change the actual PGXN Utils executable name from 'pgxn_utils' to
> 'pgxn-utils', that sounds intrusive, but how it will be in the PATH,
> the symlink ins't necessary. The next pgxn_utils version could be
> changed to the provide the new executable name and everything will
> works as expected;
>
> 3) provide a libexec script packaged with PGXN Client named
> 'pgxn-utils' that, when called, checks for the actual path of
> 'pgxn_utils' and dispatch it;
>
> I feel that 2nd and 3rd are more O.S. portable than 1rst. I'm more
> inclined to option 2 (two) but I want to know your opinion guys.

I think I prefer #2 of these three. It would certainly be the simplest thing to do. However, I can see an advantage to requiring stuff to be in libexec, so that all the pgxn-* stuff is in one place. So I'm okay with that, too. For pgxn_validate_meta, I was planning to update my installer to:

* See if `pgxn` is in the path
* If it is, call `pgxn --libexec`
* If that outputs a value, install a symlink there

Not sure how portable #3 is, though. Still, seems pretty straight-forward to me, and I prefer this approach to either #1 or #3 above.

Best,

David

[Dickson S. Guedes]

2011/7/16 David E. Wheeler <da...@kineticode.com>:

> On Jul 16, 2011, at 6:12 PM, Dickson S. Guedes wrote:
>
>> I have some alternatives:
>>
>> 1) create a task on PGXN Utils to "register" it to PGXN Client, that
>> creates a symlink on 'libexec' dir. This could be dangerous in some
>> situations when a new version of (in this case) 'pgxn_utils' is
>> installed and the link are point to the old version, so the user must
>> call 'register' on update;
>>
>> 2) change the actual PGXN Utils executable name from 'pgxn_utils' to
>> 'pgxn-utils', that sounds intrusive, but how it will be in the PATH,
>> the symlink ins't necessary. The next pgxn_utils version could be
>> changed to the provide the new executable name and everything will
>> works as expected;
>>
>> 3) provide a libexec script packaged with PGXN Client named
>> 'pgxn-utils' that, when called, checks for the actual path of
>> 'pgxn_utils' and dispatch it;
>>
>> I feel that 2nd and 3rd are more O.S. portable than 1rst. I'm more
>> inclined to option 2 (two) but I want to know your opinion guys.
>
> I think I prefer #2 of these three. It would certainly be the simplest thing to do. However, I can see an advantage to requiring stuff to be in libexec, so that all the pgxn-* stuff is in one place. So I'm okay with that, too.

I agree. But, since the executable's name will follow the pattern
expected by PGXN Client's dispatcher, I think there is no need to put
a link in libexec really. It is just a change from a '_' to a '-'. By
the other hand, I think that will be different to others utilities
that not be prefixed with 'pgxn', in this case, a symlink will be the
best way.

> For pgxn_validate_meta, I was planning to update my installer to:
>
> * See if `pgxn` is in the path
> * If it is, call `pgxn --libexec`
> * If that outputs a value, install a symlink there

Ah, just a fix, it is `pgxn help --libexec`.

Thanks for your opinion!

Best.

---

[David E. Wheeler]

On Jul 16, 2011, at 8:03 PM, Dickson S. Guedes wrote:

> I agree. But, since the executable's name will follow the pattern
> expected by PGXN Client's dispatcher, I think there is no need to put
> a link in libexec really. It is just a change from a '_' to a '-'. By
> the other hand, I think that will be different to others utilities
> that not be prefixed with 'pgxn', in this case, a symlink will be the
> best way.

Oh, you mean they should go into libexec but be allowed to be named pgxn-* or pgxn_*?

> Ah, just a fix, it is `pgxn help --libexec`.

Oh, missed that, thanks!

David

[Dickson S. Guedes]

2011/7/17 David E. Wheeler <da...@kineticode.com>:

> On Jul 16, 2011, at 8:03 PM, Dickson S. Guedes wrote:
>
>> I agree. But, since the executable's name will follow the pattern
>> expected by PGXN Client's dispatcher, I think there is no need to put
>> a link in libexec really. It is just a change from a '_' to a '-'. By
>> the other hand, I think that will be different to others utilities
>> that not be prefixed with 'pgxn', in this case, a symlink will be the
>> best way.
>
> Oh, you mean they should go into libexec but be allowed to be named pgxn-* or pgxn_*?

No. I saying that in the specific case of pgxn_utils, it is better to
just change the executable name from 'pgxn_utils' to 'pgxn-utils',
since it is in the PATH, there is nothing else to do.

For that future script, created in other language than Python, that
will extend the pgxn client but has some names like something
(hypothetical) 'convert_archive_to_pgxn_zip', would be better to use
symlinks (as 'pgxn-convert-to-zip', for example) in libexec, sure.

--

[David E. Wheeler]

On Jul 16, 2011, at 11:32 PM, Dickson S. Guedes wrote:

> No. I saying that in the specific case of pgxn_utils, it is better to
> just change the executable name from 'pgxn_utils' to 'pgxn-utils',
> since it is in the PATH, there is nothing else to do.

Ah, I see. Name change makes sense, but I don't know if Daniele is going to go for allowing a program to be anywhere in the PATH.

> For that future script, created in other language than Python, that
> will extend the pgxn client but has some names like something
> (hypothetical) 'convert_archive_to_pgxn_zip', would be better to use
> symlinks (as 'pgxn-convert-to-zip', for example) in libexec, sure.

Gotcha.

Best,

David

[Dickson S. Guedes]

2011/7/17 David E. Wheeler <da...@kineticode.com>:

> On Jul 16, 2011, at 11:32 PM, Dickson S. Guedes wrote:
>
>> No. I saying that in the specific case of pgxn_utils, it is better to
>> just change the executable name from 'pgxn_utils' to 'pgxn-utils',
>> since it is in the PATH, there is nothing else to do.
>
> Ah, I see. Name change makes sense, but I don't know if Daniele is going to go for allowing a program to be anywhere in the PATH.
>

Yes, it works very well! I tested it. I will rename the executable in
the next version of PGXN Utils.

Best.

[David E. Wheeler]

On Jul 7, 2011, at 10:48 AM, David E. Wheeler wrote:

>> I'm looking forward for a real user for this feature: it can be for
>> instance the PGXN Utils. I'm available for any help/feedback about
>> interfacing the "utils" with the "client". I am also positive in
>> merging the utils source repository and distribute everything under
>> the "client" flag.
>
> I'll try to put aside some time to get pgxn_meta_validator to work with it, so that if it's installed, the user can just do
>
> pgxn validate-meta META.json

I finally did this. If you install PGXN::Meta::Validator v0.14.0 or higher from CPAN, you should thereafter be able to do this:

pgxn validat-meta META.json

And get validation of your META.json before you release. W00t!

So Daniele, any chance of a final release of pgxnclient 0.3.0 so that it can be installed via easy_install?

Thanks,

David

[Dickson S. Guedes]

2011/8/31 David E. Wheeler <da...@kineticode.com>:

> On Jul 7, 2011, at 10:48 AM, David E. Wheeler wrote:
>
>>> I'm looking forward for a real user for this feature: it can be for
>>> instance the PGXN Utils. I'm available for any help/feedback about
>>> interfacing the "utils" with the "client". I am also positive in
>>> merging the utils source repository and distribute everything under
>>> the "client" flag.
>>
>> I'll try to put aside some time to get pgxn_meta_validator to work with it, so that if it's installed, the user can just do
>>
>>    pgxn validate-meta META.json
>
> I finally did this. If you install PGXN::Meta::Validator v0.14.0 or higher from CPAN, you should thereafter be able to do this:
>
>    pgxn validat-meta META.json

Nice! The pgxn-utils master branch are working in that way too...

Just a doubt, it will be 'validat-meta' without 'e'?

> And get validation of your META.json before you release. W00t!

Yeah! So, I'll use that validation in next pgxn-utils's version when
'bundle' and 'release' task are called...

Thanks David!

[Daniele Varrazzo]

On Wed, Aug 31, 2011 at 4:49 AM, David E. Wheeler <da...@kineticode.com> wrote:

> So Daniele, any chance of a final release of pgxnclient 0.3.0 so that it can be installed via easy_install?

Sorry but I haven't managed yet to finish the $(grep -r TODO) in the
library. August has been an eventful month resulting in almost zero
LoC written.

I've reviewed the points and I think the most important thing to fix
is to assume extensions unique and thus accept them in commands such
as install etc. Should be easy to fix. A big one is to finish the
i18n, but we currently have nobody asking for l10n, so I'm not going
to waste time hunting for missing translation hooks. As a minimum I
want to close the points in
https://github.com/dvarrazzo/pgxnclient/issues

So, I'm asking:

- do we have a plan? Is there a timeline you would like to meet?
- we have pgxnutils and now the validator: do we want to integrate
them into a single package or shall we leave this point to the
distros?

I will work on the client points in these evenings. Please, if there's
any open point I've forgotten about, add a ticket to the above
tracker.

Cheers,

-- Daniele

[Dickson S. Guedes]

2011/8/31 Daniele Varrazzo <daniele....@gmail.com>:

> On Wed, Aug 31, 2011 at 4:49 AM, David E. Wheeler <da...@kineticode.com> wrote:
>> So Daniele, any chance of a final release of pgxnclient 0.3.0 so that it can be installed via easy_install?

[...]

> So, I'm asking:
>
> - do we have a plan? Is there a timeline you would like to meet?
> - we have pgxnutils and now the validator: do we want to integrate
> them into a single package or shall we leave this point to the
> distros?

I think that we not need it, yet. One thing that I can do is add a
test in installing step of pgxn-utils to see if there is a pgxn
validate and/or a pgxn client already installed and recommends its
installation, for example.

IMHO, "plugables modules" make more sense to me, like we are doing
with pgxn validate-meta ..., pgxn utils ... using the actual pgxn's
dispatcher capability, but the final user or developer must be careful
about version compatibility between each one.

[]s

[Daniele Varrazzo]

On Wed, Aug 31, 2011 at 5:15 PM, Dickson S. Guedes <gue...@guedesoft.net> wrote:
> 2011/8/31 Daniele Varrazzo <daniele....@gmail.com>:
>> On Wed, Aug 31, 2011 at 4:49 AM, David E. Wheeler <da...@kineticode.com> wrote:
>>> So Daniele, any chance of a final release of pgxnclient 0.3.0 so that it can be installed via easy_install?
> [...]
>> So, I'm asking:
>>
>> - do we have a plan? Is there a timeline you would like to meet?
>> - we have pgxnutils and now the validator: do we want to integrate
>> them into a single package or shall we leave this point to the
>> distros?
>
> I think that we not need it, yet. One thing that I can do is add a
> test in installing step of pgxn-utils to see if there is a pgxn
> validate and/or a pgxn client already installed and recommends its
> installation, for example.
>
> IMHO, "plugables modules" make more sense to me, like we are doing
> with pgxn validate-meta ..., pgxn utils ... using the actual pgxn's
> dispatcher capability, but the final user or developer must be careful
> about version compatibility between each one.

Yes, makes sense for me. Furthermore, trying to install
python+ruby+perl stuff in the right locations is something that looks
less than trivial (Python's distutils is intelligent about Python
code, but other code should be treated as dumb data, which would
probably make a poor installation).

-- Daniele

[David E. Wheeler]

On Aug 30, 2011, at 9:12 PM, Dickson S. Guedes wrote:

> Nice! The pgxn-utils master branch are working in that way too...
>
> Just a doubt, it will be 'validat-meta' without 'e'?

No, with the "e". Sorry.

>> And get validation of your META.json before you release. W00t!
>
> Yeah! So, I'll use that validation in next pgxn-utils's version when
> 'bundle' and 'release' task are called...

If it's installed, of course. Very good idea.

Best,

David

[David E. Wheeler]

On Aug 31, 2011, at 2:50 AM, Daniele Varrazzo wrote:

> Sorry but I haven't managed yet to finish the $(grep -r TODO) in the
> library. August has been an eventful month resulting in almost zero
> LoC written.

No worries, I've had the same issue. Been too busy with $work to hack much code. ONly been working on it lately because I'm preparing to move PGXN to a community-supplied server and so am doing some cleanup.

> I've reviewed the points and I think the most important thing to fix
> is to assume extensions unique and thus accept them in commands such
> as install etc. Should be easy to fix.

That would be nice.

> A big one is to finish the
> i18n, but we currently have nobody asking for l10n, so I'm not going
> to waste time hunting for missing translation hooks.

Agreed.

> As a minimum I
> want to close the points in
> https://github.com/dvarrazzo/pgxnclient/issues

I left a couple of comments on those issues.

> So, I'm asking:
>
> - do we have a plan? Is there a timeline you would like to meet?

No timeline really. I'm giving a talk at Postgres Open in two weeks, but it's not directly PGXN-related. But I would love to see this finalized and publicized. September is a great month to ramp up that sort of thing, especially with the release of 9.1 expected.

> - we have pgxnutils and now the validator: do we want to integrate
> them into a single package or shall we leave this point to the
> distros?

I have no opinion on this. If you want to bundle PGXN::Meta::Validator, it's fine with me. It does, however, have a few CPAN dependencies, notably the JSON and SemVer libraries. Might make it a PITA t bundle.

> I will work on the client points in these evenings. Please, if there's
> any open point I've forgotten about, add a ticket to the above
> tracker.

Fantastic, thank you Daniele!

Best,

David

[Dickson S. Guedes]

2011/8/31 David E. Wheeler <da...@kineticode.com>:

> On Aug 31, 2011, at 2:50 AM, Daniele Varrazzo wrote:
>> - do we have a plan? Is there a timeline you would like to meet?
>
> No timeline really. I'm giving a talk at Postgres Open in two weeks, but it's not directly PGXN-related. But I would love to see this finalized and publicized. September is a great month to ramp up that sort of thing, especially with the release of 9.1 expected.

Hum, I'm expect to release the pgxn next version next week, with some
improvements and better covering tests about 'bundle' and 'release',
could I have web access to that "test server" again David?

Thanks.

[Dickson S. Guedes]

2011/8/31 Dickson S. Guedes <gue...@guedesoft.net>:
> Hum, I'm expect to release the pgxn next version ...

oops..

s/pgxn/pgxn-utils/

Sorry...

[David E. Wheeler]

On Aug 31, 2011, at 10:06 AM, Dickson S. Guedes wrote:

>> No timeline really. I'm giving a talk at Postgres Open in two weeks, but it's not directly PGXN-related. But I would love to see this finalized and publicized. September is a great month to ramp up that sort of thing, especially with the release of 9.1 expected.
>
> Hum, I'm expect to release the pgxn next version next week, with some
> improvements and better covering tests about 'bundle' and 'release',
> could I have web access to that "test server" again David?

Sure. What URL did I give you for that before?

David

[Dickson S. Guedes]

2011/8/31 David E. Wheeler <da...@kineticode.com>:

Was http://somewhere_i_dont_remenber.kineticode.com:some_port

Sorry, I couldn't find it anywhere in my logs... :/