I'm increasingly unhappy with repository declaration format

[Esteban Lorenzano]

Hi, 

thing is, while the notation 

provider://user/project:commitish/subdir

… is cool, it does not cover a wide area, which is all the cases where standard procedure does not applies. 

I was thinking in a backward compatible notation, that can be both extended to Metacello and to baselines, and I think using a builder pattern can fix all the problems.

We could have something like: 

spec 

baseline: ‘Blah’ 

with: [

spec repository: [ :builder |

builder 

url: ‘git@blahblah’;

subdirectory: ’src’;

format: #tonel ] ].

then we can have something like #asMetacelloRepositorySpec, that would apply to string and to blockclosure… this way we will be able to provide a flexible way for non-common cases. 

What do you think? 

Esteban

ps: of course, following the “rules of engagements”, proposing something like this means I’m willing to work on it, at least for the general case + pharo :)

[ph...@highoctane.be]

Whatever makes the spec stuff clearer and easier to deal with gets my vote.

Phil

--
You received this message because you are subscribed to the Google Groups "Metacello" group.
To unsubscribe from this group and stop receiving emails from it, send an email to metacello+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dale Henrichs]

I am not completely opposed to your proposal, but having a single string specification for the repo is awfully convenient ....

What are the specific urls and problems that you are having ... it may be that we can keep the strings and find other ways to solve the problems?

Like I don't think that it is necessary to have the disk format included in the url ... perhaps we should have a "filetree format decoder" that allows the the platform to figure out which repository reader to use ... the `.filetree` properties file was always conceived of as the place where information could stashed that allowed readers to figure out what disk format the repository is ... we could change the name to something like `.repository` and simply include the format field ... no .repository file implies filetree ... and so one ...

Are there other issues that you are thinking about?

Dale

--
You received this message because you are subscribed to the Google Groups "Metacello" group.

To unsubscribe from this group and stop receiving emails from it, send an email to metacello+...@googlegroups.com.

[Dale Henrichs]

Phil,

I will ask you the same question. What is "not clear and hard" that you are thinking about ...

I get the sense that there must be a proliferation of urls in the pharo universe that I am not aware, so I'd like to see some examples of these different urls.

I know that in some of my current development work I have a similar proliferation of urls and I am personally leaning to NOT using an url to denote either the type of repository and/or the platform implementation being used to manage the git repos ...

my preference is to retain the github:// style --- using the provider to distinguish where to download the url and then to use platform-specific url handlers (which can be customized by developers) to determine which tool to use to manage the repository ... a discussion along these lines has been started here[1] ...

I think it is a mistake to encode the disk format in the Metacello spec ... the disk format can change from commit to commit and the Metcello spec should not have to change if the format of the repo chenages especially when the format can change from commit to commit ...

I would like to see specific issues described so that we have a chance to be clear on exactly which problem(s) we are solving BEFORE starting to come up with solutions...

Dale

[1] https://github.com/Metacello/metacello/issues/474

To unsubscribe from this group and stop receiving emails from it, send an email to metacello+...@googlegroups.com.

[Dale Henrichs]

Here is an almost complete list of urls that have come up during the cypress/cypress2 (alternate filetree reader/loader/writer to Monticello):

'tonel:<dirpath>'

'cypress:<dirpath>'

'cypressft:<dirpath>'

'cypressfiletree:<dirpath>

'gitcypress:<dirpath>

'gitfiletree:<dirpath>'

'cypresslax:<dirpath>'

'chunk:<dirpath>'

If we settle on the simple expediency of embedding a `.repository.ston` file with a #format field, then all of these could be collapsed to:

   'filetree:<dirpath>'

simplifying things for developers immensely and eliminating any need to change the metacello specification ...

I understand that the in-image repository technology is also involed ... gitfiletree and iceberg ... but I am not really familiar with the ruls (if any) being used here ... my pref is to have tools register in the image for handling the github:// family of urls and if there is no handler, then the tarball is downloaded and `filetree://` is used to read the repository (with the expectation that there is a .repository.ston file to denote disk format ...

I would like to see examples of the proliferation of url styles that have appeared in Pharo over the years --- perhaps there are other issues that I am not aware of that are lurking in the corners?

Dale

On 12/22/17 3:12 AM, Esteban Lorenzano wrote:

[Dale Henrichs]

Okay this would be an interesting url to support ... and this is a case where branch and directory info is missing:

  g...@git.gemtalksystems.com:GsDevKit_sys_local.git

Thierry, did you have a solution for the branch directory issue for you gitfiletree urls?

Dale

[Cyril Ferlicot]

On Friday, December 22, 2017 at 6:42:16 PM UTC+1, Dale wrote:

Phil,

I will ask you the same question. What is "not clear and hard" that you are thinking about ...

I get the sense that there must be a proliferation of urls in the pharo universe that I am not aware, so I'd like to see some examples of these different urls.

I know that in some of my current development work I have a similar proliferation of urls and I am personally leaning to NOT using an url to denote either the type of repository and/or the platform implementation being used to manage the git repos ...

my preference is to retain the github:// style --- using the provider to distinguish where to download the url and then to use platform-specific url handlers (which can be customized by developers) to determine which tool to use to manage the repository ... a discussion along these lines has been started here[1] ...

I think it is a mistake to encode the disk format in the Metacello spec ... the disk format can change from commit to commit and the Metcello spec should not have to change if the format of the repo chenages especially when the format can change from commit to commit ...

I would like to see specific issues described so that we have a chance to be clear on exactly which problem(s) we are solving BEFORE starting to come up with solutions...

Dale

[1] https://github.com/Metacello/metacello/issues/474

I agree with the fact that ideally it would be cool to not have the format in the baseline. I think we don't care that version 2.4.7 is filetree and 3.4.1 is tonel for a dependency.

For me, as I said in the issues on Metacello repository I would like to:

- Load a project from a private repository through Metacello API (`Metacello new...`)

- Load a project from a repository on a private server through Metacello API (`Metacello new...`)

- Be able to depend from a private project

- Be able to depend from a project on a private server

By project no a private server I mean a git repository on a personal server (via gitlab, github enterprise, just git...).

This includes to manage the fact that the server might not have a default ssh port.

This includes also that it can be protected by a password, for example in the case of gitlab. (For this I see the solution of SSH and maybe, if someone has the courage to do it, an integration of the main hosting services as github, gitlab, bitbucket)

For example, I have a server with a non default ssh port and a private gitlab at gitlab.ferlicot.fr. I would like to have two projects on it and to make one depend on the other. Then, in Pharo, I would like to be able to load the projects via a classic Metacello command. (Metacello new [...] load). It does not seems to be possible for now.

I think those needs cover most of the "missing" parts to make Metacello cool to use by companies. I know that it is a lot of work. I don't have the knowledge and time to help much so I hope I can help to make this subject move forward via those discussions. :)

[Dale Henrichs]

On 12/22/17 3:52 PM, Cyril Ferlicot wrote:
>
>
> On Friday, December 22, 2017 at 6:42:16 PM UTC+1, Dale wrote:

... snip

Cyril,

How are these projects cloned when using the git command line? Examples
of the command line would be good enough ...

Dale

[Esteban Lorenzano]

On 22 Dec 2017, at 19:37, Dale Henrichs <dale.h...@gemtalksystems.com> wrote:

Okay this would be an interesting url to support ... and this is a case where branch and directory info is missing:

  g...@git.gemtalksystems.com:GsDevKit_sys_local.git

here is another: 

g...@git.afi-sa.fr:afi/mbm.git

also, this goes through a non-standard port, then is has to be expressed like this:

ssh://g...@git.afi-sa.fr:PORT/afi/mbm.git

but then, what I was proposing is a way where we can keep this simple strings for most of cases, then people with non-standard urls (like private servers, etc.) they can express the repository in a generic way. 

[Cyril Ferlicot]

On Saturday, December 23, 2017 at 1:10:08 AM UTC+1, Dale wrote:

Cyril,

How are these projects cloned when using the git command line? Examples
of the command line would be good enough ...

Via SSH I use a remote URL like this one:

ssh://g...@domain.name.com:PORT/Path/To/The/Repo/RepoName.git

For example:

ssh://g...@gitlab.ferlicot.fr:3730/ProjectTeam/MyProject.git

where 3730 is my custom SSH port.

Also, If I do not use SSH but Https I directly use an URL like https://gitlab.ferlicot.fr/cyril/myProject.git and I get asked for my username and password.

 

Dale

[Dale Henrichs]

Esteban,

Cyril has started a discussion on the pharo list about extending ZnUrl to permit custom url parsing.

I have heard that you guys are using regex for parsing iceberg repository descriptions and I am curious which urls you are using regex for?

At these the urls that you are trying to handle:

  g...@git.afi-sa.fr:afi/mbm.git
  sh://g...@git.afi-sa.fr:PORT/afi/mbm.git

or are there other examples?

As I mention in that thread I am partial to Thierry's approach for handling custom urls:

 'gitfiletree://gitlab.ferlicot.fr/Projet/Bazard:dev/src' asUrl.
 'gitfiletree://user:pa...@gitlab.ferlicot.fr:3456/Projet/Bazard:dev/src' asUrl.
 'gitfiletree://us...@gitlab.ferlicot.fr:3456/Projet/Bazard:dev/src' asUrl.
 'gitfiletree://gitlab.ferlicot.fr/Projet?dir=src&branch=dev&protocol=git' asUrl.

I would choose a different scheme name[1] ...

The first case would be the standard custom repository description for accessing public repositories for which have not defined a zip download scheme.

The second and third examples should cover the cases where developers need to use alternate ports and/or specify username/password ... the protocol query (see fourth option) allows developers to choose either https: or git@ modes of accessing the repo and would play into platform speicific support for ssh agents during the cloning process..

I'm not sure I'm a fan of the fourth option other than being able to choose the protocol to use (https or git) ...

I think that Thierry's gitfiletree scheme covers all of the requirements that we've seen so far, right?

so it seems that we have the need for two new schemes:

  - custom:// (ala gitfiletree)
  - path:/<filepath> as the generic path to a filetree/cypress/tonel/??? repository (scheme name up for debate)

I also mention that I am inclined to have developers register a tool to handle different schemes ... so that it we can arrange to have a generic set of repository descriptions that can be used with a variety of tools ... @asMetacelloRepository would be expected to match a scheme with a tool and this would be a platform-specific imlementation ... the code in MetacelloPlatform>>createRepository: aRepositorySpec would be reduced to:

  aRepositorySpec description asUrl asMetacelloRepository

If Sven is agreeable to extending ZnUrl to allow for custom parsing of urls, then I think that we will have enough flexibility to be able to support the existing set of urls and define the common custom url formats and then I think we'd be in business.

Dale

[1] https://github.com/dalehenrich/filetree/blob/734eed46ea57ebf5e24e5d935768bd49727fc22f/repository/MonticelloFileTree-Git.package/MCFileTreeGitRepository.class/class/basicFromUrl..st#L26