tests: indicating a particular version of psycopg2 is required?

[Chris Withers]

Hi Mike,

How do I indicate in a test that a specific version of psycopg2 is required?

I'm already doing:

class _RangeTypeMixin(object):
__requires__ = 'range_types',
__dialect__ = 'postgresql+psycopg2'

...where:

@property
def range_types(self):
def check_range_types():
if not against("postgresql+psycopg2"):
return False
try:
self.db.execute("select '[1,2)'::int4range;")
return True
except:
return False

return only_if(check_range_types)

...but range type support only arrived in psycopg 2.5.

Chris

--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk

[Michael Bayer]

On May 18, 2013, at 12:17 PM, Chris Withers <ch...@simplistix.co.uk> wrote:

> Hi Mike,
>
> How do I indicate in a test that a specific version of psycopg2 is required?
>
> I'm already doing:
>
> class _RangeTypeMixin(object):
> __requires__ = 'range_types',
> __dialect__ = 'postgresql+psycopg2'
>
> ...where:
>
> @property
> def range_types(self):
> def check_range_types():
> if not against("postgresql+psycopg2"):
> return False
> try:
> self.db.execute("select '[1,2)'::int4range;")
> return True
> except:
> return False
>
> return only_if(check_range_types)
>
> ...but range type support only arrived in psycopg 2.5.

hm that's recent, no chance the range type can work prior to 2.5 and/or with other DBAPIs?

I'd implement "_has_native_range" on the psycopg2 dialect, then:

@testing.skip_if(lambda: not testing.db.dialect._has_native_range)
def test_my_range(self):
#...

>
> Chris
>
> --
> Simplistix - Content Management, Batch Processing & Python Consulting
> - http://www.simplistix.co.uk
>

> --
> You received this message because you are subscribed to the Google Groups "sqlalchemy-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy-dev...@googlegroups.com.
> To post to this group, send email to sqlalche...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sqlalchemy-devel?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Chris Withers]

On 19/05/2013 16:13, Michael Bayer wrote:
>
>> @property
>> def range_types(self):
>> def check_range_types():
>> if not against("postgresql+psycopg2"):
>> return False
>> try:
>> self.db.execute("select '[1,2)'::int4range;")
>> return True
>> except:
>> return False
>>
>> return only_if(check_range_types)
>>
>> ...but range type support only arrived in psycopg 2.5.
>
> hm that's recent, no chance the range type can work prior to 2.5 and/or with other DBAPIs?
>
> I'd implement "_has_native_range" on the psycopg2 dialect, then:

It panned out as above but with an addition to check psycopg was
compatible too:

https://github.com/cjw296/sqlalchemy/blob/f1f2f4511864e3f1fc9f5343af47fbf86b045c2f/test/requirements.py#L580

Is this okay or do I need to do something before I send in the pull request?

[Michael Bayer]

On May 19, 2013, at 2:32 PM, Chris Withers <ch...@simplistix.co.uk> wrote:

> On 19/05/2013 16:13, Michael Bayer wrote:
>>
>>> @property
>>> def range_types(self):
>>> def check_range_types():
>>> if not against("postgresql+psycopg2"):
>>> return False
>>> try:
>>> self.db.execute("select '[1,2)'::int4range;")
>>> return True
>>> except:
>>> return False
>>>
>>> return only_if(check_range_types)
>>>
>>> ...but range type support only arrived in psycopg 2.5.
>>
>> hm that's recent, no chance the range type can work prior to 2.5 and/or with other DBAPIs?
>>
>> I'd implement "_has_native_range" on the psycopg2 dialect, then:
>
> It panned out as above but with an addition to check psycopg was compatible too:
>
> https://github.com/cjw296/sqlalchemy/blob/f1f2f4511864e3f1fc9f5343af47fbf86b045c2f/test/requirements.py#L580
>
> Is this okay or do I need to do something before I send in the pull request?

don't you want me to put SQLAlchemy on git first? Just as soon as I have full mastery over wtf the difference is between --bare and --mirror and fully understand all that. I've got Mako on git now but it's a little bumpy.

[Chris Withers]

On 19/05/2013 23:54, Michael Bayer wrote:
>
>> It panned out as above but with an addition to check psycopg was compatible too:
>>
>> https://github.com/cjw296/sqlalchemy/blob/f1f2f4511864e3f1fc9f5343af47fbf86b045c2f/test/requirements.py#L580
>>
>> Is this okay or do I need to do something before I send in the pull request?
>
> don't you want me to put SQLAlchemy on git first?

Can't tell if you're joking or not ;-)

Assuming you're not, hg-git has come on a long way. I'll do my work in
git (just for familiarity) and then when a branch is ready to merge I'll
pull it back into my hg repo and push it up to bitbucket as a bookmark.

> Just as soon as I have full mastery over wtf the difference is between --bare and --mirror and fully understand all that.

--bare basically means no working copy. Git will prevent you from
pushing into a remote with a working copy, since that will screw it up.

Basically, bare repos end up being directories with .git on the end that
you put on servers that you want to push to. If the server is GitHub or
BitBucket, you don't need to worry about this :-)

Not had any experience with --mirror, but from the man page it looks
like it aims to mirror only the branches you want on a mirror in
addition to producing a bare repo.

> I've got Mako on git now but it's a little bumpy.

If you can let me know what the bumps are, I'll see if I can help...

[Michael Bayer]

basically if you want to have a repo that looks "just like" the one on bitbucket, you pretty much need to use "--mirror". To update the mirror, you do "git remote update". These aren't hard things, but compared to mercurial, it's very hard to learn - mercurial has none of these options and variants. In the mercurial case, all repos are the same (whether or not they have a working copy changes nothing), you can then "hg push" or "hg pull" between any of these repos, and it does exactly the same thing in all cases.

With git, now we have:

1. git pull - if you have a working copy
2. git fetch - above, but also if you *dont* have a working copy
3. git fetch --update-head-ok / --tags / --prune - hey, git can't assume you want this, or that, or the other thing, who knows! please set your many, many flags based on the subtle use cases linus torvalds had or did not have one day.
4. git remote update - better if you have --mirror, and this is different from fetch! Just surf stackoverflow for three hours to slowly sort of understand why

bonus! your various "push" hooks don't work if you're doing "fetch"! because taking the data from another repo and updating the local one is entirely different from taking the data from another repo and updating the local one.

But! i can edit the damn history finally ! (not only delete branches [FINALLY] but I learned rebase -i too, which is also why I had to get really good with --mirror if you want revision deletes and stuff to actually transfer over automatically). So yes, because of that the Mako migration is basically a rehearsal for the SQLAlchemy one, right now my only remaining concern is bumping the userbase that may be currently dependent on pulling from our mercurial tree.

[Chris Withers]

On 20/05/2013 15:00, Michael Bayer wrote:
>
>> If you can let me know what the bumps are, I'll see if I can help...
>
> basically if you want to have a repo that looks "just like" the one on bitbucket, you pretty much need to use "--mirror".

I have to admit I've never used --mirror. That may be because I've
always migrated repos from svn rather than mercurial.

If you do a hg-git push into a bare repo, what's missing?
Where does --mirror come in?

Out of interest, have you looked at felipec's git-remote-hg? Tweeted
about it recently and looked like an interesting take on
get-git-from-hg-server.

> To update the mirror, you do "git remote update".

Never used that, I generally either "git fetch" or, more common, "get
fetch".

> These aren't hard things, but compared to mercurial, it's very hard to learn - mercurial has none of these options and variants.

Haha, that's all perspective. I find Hg impenetrable.. branches? no
bookmarks, not these other things. What do you mean you forgot to enable
the extension that should never be disabled on any repo ever? ;-)

> In the mercurial case, all repos are the same (whether or not they have a working copy changes nothing), you can then "hg push" or "hg pull" between any of these repos, and it does exactly the same thing in all cases.

Yeah, but that's part of what makes git rock: the index and working copy
interaction. I don't know why you can't push into a repo with a working
copy (ie: non-bare) but I'll forgive the developers of git in return of
having an index/staging area at all...

> With git, now we have:
>
> 1. git pull - if you have a working copy

I think this is like "hg pull -u", it's what I used most.

> 2. git fetch - above, but also if you *dont* have a working copy

Unfortunately, this is like "hg pull", made worse by the fact that the
deprecated "hg fetch" was much more like "git pull"... Anyway, "git
fetch" just brings all the revisions from a remote repo into the local
repo, but leaves the current branch, working copy and index alone.

> 3. git fetch --update-head-ok / --tags / --prune - hey, git can't assume you want this, or that, or the other thing, who knows! please set your many, many flags based on the subtle use cases linus torvalds had or did not have one day.

I don't generally bother passing options to "get fetch" when I do use it.

My work cycle is mainly:

"git checkout -b something" - create a branch to work in and check it out
"git add -p somefiles" - add bits of my changes in a coherent commit
"git commit" - make the commit
"git push origin something" - push my new branch back to the server for
backup
*rinse and repeat before issuing pull request from github*

> 4. git remote update - better if you have --mirror, and this is different from fetch!

I think Hg is tripping you up, I've never used "git remote update" and I
rarely use "git fetch"...

> bonus! your various "push" hooks don't work if you're doing "fetch"!

Where are the hooks that aren't working? If in the local repo, well
yeah, would it really make sense to have push hooks fire when you're
doing a pull? ;-)

> because taking the data from another repo and updating the local one is entirely different from taking the data from another repo and updating the local one.

*chuckles* I do remember this feeling, it does pass, and you end up
realising that while often misnamed and confusing, some of the things
git does are really useful...

> But! i can edit the damn history finally ! (not only delete branches [FINALLY] but I learned rebase -i too, which is also why I had to get really good with --mirror if you want revision deletes and stuff to actually transfer over automatically).

Yep, that and "git add -p" are my favourite friends... Split a revision
into bits using rebase -i yet? Then you feel special :-)

> So yes, because of that the Mako migration is basically a rehearsal for the SQLAlchemy one, right now my only remaining concern is bumping the userbase that may be currently dependent on pulling from our mercurial tree.

How would you get stats on how many people that is? If there are
regulars, see if they have an active github account ;-)

Anyway, I did have some sqlalchemy-related questions a message or two
back ;-) - Is it legit to end up with specialist functions and
constraints in the dialect module (ie: sqlalchemy.dialects.postgresql)
or should they be in the main modules even though they'll only work with pg?

cheers,

[Michael Bayer]

On May 20, 2013, at 1:07 PM, Chris Withers <ch...@simplistix.co.uk> wrote:

> On 20/05/2013 15:00, Michael Bayer wrote:
>>
>>> If you can let me know what the bumps are, I'll see if I can help...
>>
>> basically if you want to have a repo that looks "just like" the one on bitbucket, you pretty much need to use "--mirror".
>
> I have to admit I've never used --mirror. That may be because I've always migrated repos from svn rather than mercurial.

the mercurial migration is easy, that's done. --mirror is because I'd like the public "push" repo to be bitbucket (showing Mako here because it is working fully):

https://bitbucket.org/zzzeek/mako

but I also need a mirror of it within my own hosting:

http://www.makotemplates.org/trac/log/

http://git.makotemplates.org/mako.git

When a push is received by bitbucket, a POST request is sent to my own server which I intercept and use to update my local mirrors:

https://confluence.atlassian.com/display/BITBUCKET/POST+Service+Management

https://bitbucket.org/zzzeek/publishthing/src/0ac95234ca3f25ef9655f818ce64d8a4d9d9ec14/publishthing/dvcs_hooks.py?at=master (my script)

Only using --mirror do you get *everything* to be mirrored fully, that's why it's there.

http://stackoverflow.com/questions/3959924/whats-the-difference-between-git-clone-mirror-and-git-clone-bare

I guess git is "easy" for everyone because they lean on github to do all the tricky bits. Overall, I like that git is very open and already I understand half of the details of the repository format, whereas mercurial it's entirely a black box. I'm just amazed at how everyone else seems to have embraced this much more nuts and bolts model too, seeing how impatient users typically are when being exposed to implementation details.

> Out of interest, have you looked at felipec's git-remote-hg? Tweeted about it recently and looked like an interesting take on get-git-from-hg-server.

I'm slightly debating whether to try to at least temporarily push a mirror from the git repo back to hg.sqlalchemy.org but I probably wont bother.

>
>> To update the mirror, you do "git remote update".
>
> Never used that, I generally either "git fetch" or, more common, "get fetch".

its a mirroring thing.

>
>> These aren't hard things, but compared to mercurial, it's very hard to learn - mercurial has none of these options and variants.
>
> Haha, that's all perspective. I find Hg impenetrable.. branches? no bookmarks, not these other things. What do you mean you forgot to enable the extension that should never be disabled on any repo ever? ;-)

I've never bothered with hg bookmarks - they aren't a standard, and I get pull requests with hg branches anyway. The branching model built in to hg is very limited, but it's pretty easy to use (though i have gotten into trouble with it).

>
>> In the mercurial case, all repos are the same (whether or not they have a working copy changes nothing), you can then "hg push" or "hg pull" between any of these repos, and it does exactly the same thing in all cases.
>
> Yeah, but that's part of what makes git rock: the index and working copy interaction. I don't know why you can't push into a repo with a working copy (ie: non-bare) but I'll forgive the developers of git in return of having an index/staging area at all...

it's because git doesn't have the same distinction between "repo" and "working copy" as hg - a "push" into a git repo implies the actual working copy has to change, and I totally see why they don't want to do that.

>
>> With git, now we have:
>>
>> 1. git pull - if you have a working copy
>
> I think this is like "hg pull -u", it's what I used most.
>
>> 2. git fetch - above, but also if you *dont* have a working copy
>
> Unfortunately, this is like "hg pull", made worse by the fact that the deprecated "hg fetch" was much more like "git pull"... Anyway, "git fetch" just brings all the revisions from a remote repo into the local repo, but leaves the current branch, working copy and index alone.
>
>> 3. git fetch --update-head-ok / --tags / --prune - hey, git can't assume you want this, or that, or the other thing, who knows! please set your many, many flags based on the subtle use cases linus torvalds had or did not have one day.
>
> I don't generally bother passing options to "get fetch" when I do use it.

yeah I guess when you're trying to mirror, all the requirements are different - I need everything. --mirror with remote update solves all that, I guess it's not that big a deal, it just took me weeks to figure that out.

>
>> So yes, because of that the Mako migration is basically a rehearsal for the SQLAlchemy one, right now my only remaining concern is bumping the userbase that may be currently dependent on pulling from our mercurial tree.
>
> How would you get stats on how many people that is? If there are regulars, see if they have an active github account ;-)

so far I'm balking at a github mirror because I'm queasy about having ssh keys on my project server. will see if i can figure something out. There's no way to get github/bitbucket servers to update each other automatically without hosting a server in the middle (is there? anyone know of some service out there?)

>
> Anyway, I did have some sqlalchemy-related questions a message or two back ;-) - Is it legit to end up with specialist functions and constraints in the dialect module (ie: sqlalchemy.dialects.postgresql) or should they be in the main modules even though they'll only work with pg?

if the construct is not part of the SQL standard and only works on postgresql, then it stays part of postgresql. If it is a SQL standard then it usually becomes part of sqlalchemy.sql.

[Chris Withers]

On 20/05/2013 18:57, Michael Bayer wrote:
>
> On May 20, 2013, at 1:07 PM, Chris Withers<ch...@simplistix.co.uk> wrote:
>>
>> I have to admit I've never used --mirror. That may be because I've always migrated repos from svn rather than mercurial.
>
> the mercurial migration is easy, that's done. --mirror is because I'd like the public "push" repo to be bitbucket (showing Mako here because it is working fully):

Hah, okay, well now you're just showing off ;-)

> but I also need a mirror of it within my own hosting:
>
> http://www.makotemplates.org/trac/log/

Ah, yes, trac...

> When a push is received by bitbucket, a POST request is sent to my own server which I intercept and use to update my local mirrors:
>
> https://confluence.atlassian.com/display/BITBUCKET/POST+Service+Management
>
> https://bitbucket.org/zzzeek/publishthing/src/0ac95234ca3f25ef9655f818ce64d8a4d9d9ec14/publishthing/dvcs_hooks.py?at=master (my script)

That's pretty cool, sorry it was a pig to learn, but cool that it works
at all :-)

> I guess git is "easy" for everyone because they lean on github to do all the tricky bits.

I guess the rest of us are just a bit more slack. For my stuff, having a
maintained issue tracker and public code review system as well as all
the github social stuff was so far ahead of my own crappy svn server and
ancient roundup tracker instance that it was a no brainer...

> I'm just amazed at how everyone else seems to have embraced this much more nuts and bolts model too, seeing how impatient users typically are when being exposed to implementation details.

I guess git users and orm users are a bit different ;-)
Myself, I love that git is built for failure; I've had git-svn blow up
on people at work, or people have accidentally deleted branches, rebased
things away, etc; provided there's no git gc involved, the reflog can
heal most ills...

> I'm slightly debating whether to try to at least temporarily push a mirror from the git repo back to hg.sqlalchemy.org but I probably wont bother.

No complaints from me if you don't bother ;-)

>>> These aren't hard things, but compared to mercurial, it's very hard to learn - mercurial has none of these options and variants.
>>
>> Haha, that's all perspective. I find Hg impenetrable.. branches? no bookmarks, not these other things. What do you mean you forgot to enable the extension that should never be disabled on any repo ever? ;-)
>
> I've never bothered with hg bookmarks - they aren't a standard,

If the pg-ranges pull request you get from me is Hg, it'll be a bookmark :-P

> yeah I guess when you're trying to mirror, all the requirements are different - I need everything.

Yep.

>>> So yes, because of that the Mako migration is basically a rehearsal for the SQLAlchemy one, right now my only remaining concern is bumping the userbase that may be currently dependent on pulling from our mercurial tree.
>>
>> How would you get stats on how many people that is? If there are regulars, see if they have an active github account ;-)
>
> so far I'm balking at a github mirror because I'm queasy about having ssh keys on my project server.

Why would they be needed?
If it's to do the push by ssh, what about pushing over https?
If you do want to push by ssh, just have a private/public key pair that
you only use for pushing commits to github...

> will see if i can figure something out. There's no way to get github/bitbucket servers to update each other automatically without hosting a server in the middle (is there? anyone know of some service out there?)

Not that I know of, hope for another GitHub April 1st announcement that
turns out to really be true...

...that said, I'm surprised bitbucket has grown the ability to pull one
of their repos by either hg or git...

[Michael Bayer]

On May 21, 2013, at 4:33 PM, Chris Withers <ch...@simplistix.co.uk> wrote:

>>
>> so far I'm balking at a github mirror because I'm queasy about having ssh keys on my project server.
>
> Why would they be needed?
> If it's to do the push by ssh, what about pushing over https?
> If you do want to push by ssh, just have a private/public key pair that you only use for pushing commits to github...

pushing over https means I have a cleartext password on the server. so yeah I'd have an ssh key on there just for pushing.

>
>> will see if i can figure something out. There's no way to get github/bitbucket servers to update each other automatically without hosting a server in the middle (is there? anyone know of some service out there?)
>
> Not that I know of, hope for another GitHub April 1st announcement that turns out to really be true...
>
> ...that said, I'm surprised bitbucket has grown the ability to pull one of their repos by either hg or git...

its terrific, just wish I could switch the repo of a project in place.