Bitbucket Private Repos Redux

[Sean DeNigris]

I'm trying to set up a private Bitbucket project with SmalltalkCI. 

The last time I tried to load a private Bitbucket project locally via
Metacello, I found out [1] that, because Metacello downloads a zip file
manually, SSH credentials are not considered, causing failure to load
private projects ([digression] IIRC because even supplying credentials was
broken [2] [/digression]. 

The workaround, which was fine for local development, was to specify a local
filetree repo in the baseline. However, in a Bitbucket CI pipeline, the
local repo is not there (and the absolute file URL can not be recreated).

Has/will anything change to allow remote repos to be loaded via SSH keys?
Any other workarounds that I'm not seeing?

Thanks!

[1] http://forum.world.st/Loading-a-Baseline-of-a-Private-Project-tt4854306.html
[2] https://github.com/dalehenrich/metacello-work/pull/368

[Sean DeNigris]

However, in a Bitbucket CI pipeline, the
local repo is not there (and the absolute file URL can not be recreated).

After a long battle with GitLab, I was able to configure the CI to clone private GL repos into suitable locations for my filetree:// urls. Unfortunately, this method is a drag because I now have to manually clone every dependent project in a CI configuration file. Any ideas are welcome. The only thing I can think of is maybe wrapping the load in a handler which tries to do a `LibC system: 'git clone…'`. Is there anything in the API where I can pass some meta-info into the baseline spec to specify the canonical repo? I'm also not sure this will work because IIUC the CI would have to run Pharo in the same shell as the SSH key setup (e.g. not in a sub-shell)…

[Sean DeNigris]

The only thing I can think of is maybe wrapping the load in a handler which tries to do a `LibC system: 'git clone…'`. Is there anything in the API where I can pass some meta-info into the baseline spec to specify the canonical repo?

Bump ;)

Can anyone point me to a suitable hook (if one exists) where I can try to insert a fallback where, if a local clone doesn't exist, I can try to clone it and then continue.

Less urgently, has there ever been any discussion about inserting arbitrary metadata to the Spec API? For example, in this case, even though we are loading dependencies via FT urls (due to the internal limitation of MetaC that SSH key authentication is not supported*), it would be useful to denote the canonical (remote) repo.

* Could this limitation be overcome now that we have (at least in Pharo, not sure about other platforms) libgit support? E.g. Iceberg allows push/pull via ssh-agent keys

[Dale Henrichs]

On Oct 15, 2017, at 6:47 AM, Sean DeNigris <sean.p....@gmail.com> wrote:

The only thing I can think of is maybe wrapping the load in a handler which tries to do a `LibC system: 'git clone…'`. Is there anything in the API where I can pass some meta-info into the baseline spec to specify the canonical repo?

Bump ;)

Can anyone point me to a suitable hook (if one exists) where I can try to insert a fallback where, if a local clone doesn't exist, I can try to clone it and then continue.

The intention is to specify your fallback repo in the baseline and then use a Metacello lock to point to the local clone when it becomes available — a platform specific tool’s responsibility

Less urgently, has there ever been any discussion about inserting arbitrary metadata to the Spec API? For example, in this case, even though we are loading dependencies via FT urls (due to the internal limitation of MetaC that SSH key authentication is not supported*), it would be useful to denote the canonical (remote) repo.

For SSH I think that platform specific solutions are the right approach and are outside the realm of Metacello itself.  The developers are expected to make decisions about repos using platform tools (like iceberg) and then use the lock command to inform Metacello where to find it.

For the broader problems I am in favor of Metacello load specs like [1].

[1] https://github.com/GsDevKit/GsDevKit_home/blob/gh-pages/Seaside31.ston

* Could this limitation be overcome now that we have (at least in Pharo, not sure about other platforms) libgit support? E.g. Iceberg allows push/pull via ssh-agent keys

 

--
You received this message because you are subscribed to the Google Groups "Metacello" group.
To unsubscribe from this group and stop receiving emails from it, send an email to metacello+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sean DeNigris]

The intention is to specify your fallback repo in the baseline and then use a Metacello lock to point to the local clone when it becomes available — a platform specific tool’s responsibility

IIUC you're saying to specify the remote in the baseline, but AFAICT this will not work because it's a private gitlab.com repo.
Firstly, are gitlab:// URLs supported? Even if they are, I don't think my user/password authentication was ever integrated, but anyway I'm unwilling to store plain text usernames and passwords in my image. MetaC doesn't yet support ssh-agent keys, right? Although as a workaround I guess I could just override with a lock in every case and at least the canonical remote would be declared in the spec. 

[Dale Henrichs]

On 10/18/2017 03:47 PM, Sean DeNigris wrote:

The intention is to specify your fallback repo in the baseline and then use a Metacello lock to point to the local clone when it becomes available — a platform specific tool’s responsibility

IIUC you're saying to specify the remote in the baseline, but AFAICT this will not work because it's a private gitlab.com repo.

So why is it not possible for you to simply work with local clones? I don't quite understand why you need to have URLs for installing projects kept in private repositories ... doesn't Iceberg support cloning of repositories from github, bitbucket and gitlab ... if they did, there would be no reason to be asking me for support ...

Firstly, are gitlab:// URLs supported? Even if they are, I don't think my user/password authentication was ever integrated, but anyway I'm unwilling to store plain text usernames and passwords in my image.

That seems to be the standard case, which makes supporting URLs requiring passwords a bad idea ...

MetaC doesn't yet support ssh-agent keys, right?

I'd really rather not get involved in the vagaries of cross-platform support for SSH operations ... the `github:` URL support was invented to provide a smooth transition for folks to start using GitHub without requiring that all users install git on their local machines ... doesn't iceberg provide a service for creating local clones from URLs?

As far as I can tell, it will not be practical for me to invent URLs that don't play well with the local tools and given the rate of change of the local tools I don't want to get into the business of keeping up with the changes made by local tools builders ... It has been my experience that local tool builders are able to adapt their tools to Metacello much easier than it is for me to try to adapt Metacello to each of the local tools ... first it was GitFileTree and Theirry invented a custom set of urls that he supported and now I believe that Iceberg has custom urls as well ... and all of the tools know how to translate the public urls for github and bitbucket into a local clone of some sort ... Gitlab/gittorious was always a problem for adapting to Metacello, because their download fo zip files never have provided the kind of information that I needed to support the same features available from bitbucket and github ... perhaps that has changed ...

But now I think that all developers should do work with local clones --- which brings us back to local tool support being the important factor ...

Although as a workaround I guess I could just override with a lock in every case and at least the canonical remote would be declared in the spec.

Yes .. BTW, I don't consider `Metacello lock` a workaround ... it is as it supposed to be ... the external url is translated (by local tools) into a local clone if one does not already exist (i.e., no lock) and then the lock command makes sure that all other refences refer to the locked local clone ...

There is an issue when you have multiple images that want to share the same local clones, but I've been talking about solutions for this for several years[1][2] .. perhaps you only need to wait a couple more years before that capability shows up in your local tools:)

Dale

[1] https://www.youtube.com/watch?v=QshDlH1ADZQ&list=PLCGAAdUizzH05oEXf-KfKt-F2YqfNHSJq&index=8
[2] https://www.youtube.com/watch?v=Ejmqs0xLvSk&index=25&list=PLCGAAdUizzH06AkHg6_UxZ6QZBgz84yAc

[Sean DeNigris]

So why is it not possible for you to simply work with local clones?

Just for CI on e.g. GitLab. The original reason I wanted to trap the errors was to clone the repo and try again (after standardizing my naming convention).

 

That seems to be the standard case, which makes supporting URLs requiring passwords a bad idea ...

Agreed. Although for anyone using sqs, ss3, or sthub, aren't all their repo communications handled that way already?! Once I realized this, I ported all my projects to GH, but it breaks down eventually for dependencies of dependencies of...

doesn't iceberg provide a service for creating local clones from URLs?

Hmm, that's an interesting idea. Since I'm not too concerned with backward compatibility for these projects, I might be able to leverage Iceberg, but the question remains: where/how can I hook into MetaC to "clone and retry"? Can/should I just wrap the whole `Metacello new…` with an #on:do: block?

 

Yes .. BTW, I don't consider `Metacello lock` a workaround ... it is as it supposed to be ... the external url is translated (by local tools) into a local clone if one does not already exist (i.e., no lock) and then the lock command makes sure that all other refences refer to the locked local clone ...

Ah, okay. Good to know. So the only question in this scenario is: if I have all local URLS in the spec, in a CI setting how do I know the remote for dependencies without hand rolling the lock for each dependency for every project? That was the other part of my question about attaching arbitrary metadata to parts of the baseline.

 

perhaps you only need to wait a couple more years before that capability shows up in your local tools:) 

 :)

[Dale Henrichs]

On 10/20/2017 07:50 AM, Sean DeNigris wrote:

So why is it not possible for you to simply work with local clones?

Just for CI on e.g. GitLab. The original reason I wanted to trap the errors was to clone the repo and try again (after standardizing my naming convention).

okay for CI, local cloning should not be necessary ... and isn't for non-private repos ... travisCI has mechanisms[1] for encrypting secret data in .travis.yml files that can then be used to encode your user/pass combo for use in your application (setting the user/pass in the image, for downloading zip files adding username/pass support to Metacello[2]...

[1] https://docs.travis-ci.com/user/encryption-keys/
[2] https://github.com/Metacello/metacello/pull/368

 

That seems to be the standard case, which makes supporting URLs requiring passwords a bad idea ...

Agreed. Although for anyone using sqs, ss3, or sthub, aren't all their repo communications handled that way already?! Once I realized this, I ported all my projects to GH, but it breaks down eventually for dependencies of dependencies of...

doesn't iceberg provide a service for creating local clones from URLs?

Hmm, that's an interesting idea. Since I'm not too concerned with backward compatibility for these projects, I might be able to leverage Iceberg, but the question remains: where/how can I hook into MetaC to "clone and retry"? Can/should I just wrap the whole `Metacello new…` with an #on:do: block?

Okay, the preferred hook is to have a platform/tool-specific url that Metacello passes through to the platform for interpretation ...

I am definitely willing to add better support for handling custom URLs and perhaps this is the route that we can go ... What I'm imagining is a scheme registration where Metacello allows users to register handlers for a scheme and the contract is that the handler will return a valid filetree url, that Metacello will then use for loading the packages ... This should be relatively straightforward to implement and will make it possible for platform/tools/users to invent their own urls without having to modify Metacello which is the case today ...

 

Yes .. BTW, I don't consider `Metacello lock` a workaround ... it is as it supposed to be ... the external url is translated (by local tools) into a local clone if one does not already exist (i.e., no lock) and then the lock command makes sure that all other refences refer to the locked local clone ...

Ah, okay. Good to know. So the only question in this scenario is: if I have all local URLS in the spec, in a CI setting how do I know the remote for dependencies without hand rolling the lock for each dependency for every project? That was the other part of my question about attaching arbitrary metadata to parts of the baseline.

I think that custom URL handlers would address this as long as we make sure that the project can be locked as part of the handling process ... The secret data could be isolated to the handler logic  and use travis-style encryption ..

If you think that the URL scheme might make sense, let's get an issue opened for this against Metacello where we can hammer out the details. I've got Metacello work in my near term future, since I am part way through a port of Metacello to the base for GemStone so that commercial customers can use git for their development ...

Dale

[3] https://github.com/Metacello/metacello

[Sean DeNigris]

Dale Henrichs-3 wrote

If you think that the URL scheme might make sense, let's get an issue 
opened for this against Metacello

IIUC you're suggesting that there would be a hook so that I could handle
seansprivategitlab:// however I wanted behind the scenes. That sounds
intriguing, but how would I ensure that the seansprivategitlab:// support
was loaded in the image to be used by MetaC? Also, when you say that cloning
isn't necessary, last I checked SSH keys, gitlab, and bitbucket did not seem
well-supported by MetaC, so I'm not sure how I would avoid it (or whether I
want to; I still don't like the idea of unencrypted credentials in the
image, even if they're secure on disk; finally I've been using SmalltalkCI -
primarily because I thought Travis was GH-only).

Custom URL Support https://github.com/Metacello/metacello/issues/475