Question about Variant revisions

[Tobias Roeser]

Hello,

I have some problem in my understanding how variants are versioned.

Imagine the _module_ "org.scalatest.scalatest". The concrete version of that
module including its dependency constraints are the _variant_. As the
contraint for the same module with the same version might change over time
(e.g. some incompatibilities are discovered), a new variant needs to be
created. Those variants are referred by their git hash, right?

So, imagine you have another module you need, e.g. "org.slf4j.slf4j-api". If
both "latest versions" are referred to by a git-hash, how can I specify older
versions of them, assuming the required versions are not part of the same git
commit/hash? The tricky part for me to understand is, how do you manage to
have two different versions of the git tree at the same time?

I believe there is some fundamentaly wrong understanding by me. (Maybe, the
way the hashes are used to vesion the modules/variants do not map to my mental
concept of package revisions.)

Can you explain or alternatively point me to an example git/package
repository?

Best regards,
Tobias

[Tobias Roeser]

Hi Fredrik,

thanks for publishing your adept-repo-example to Github.

Example:
metadata/
org.slf4j/
slf4j-api/
24aa28cb021e7d8a7e742452fe24241d0ad05ff855dc1ae334d6349601901097.json

Now, I can see that each module ends up in a file with a name containing a
hash. Assuming, that name will be calculated based on its content, I can see
how all revisions can be seen simultaneously. But, how is the name of the file
calculated? Obviously, there is some tooling needed editing such files, if the
name depends on the content. Also, any changes would de facto bump the hash
and thus not just the variant, which was modified.

I'll stop my wild guessing here... Need to read the spec again.

Regards,
Tobias

[Fredrik Ekholdt]

Ok, so the first issue you are referring to is how to safely replace a variant with a new variant in the case the meta-data changes (let us say you update the binary versions) - right?

The idea is that you basically refer to the git hashes/commits on the repositories. If a user doesn’t care he will stay at the previous version of the repo. If he does he will update the repo and get the new variant.

The build tool will be able to create a  file (like the .lock file in Bundler) which contains the repositories and the exact commits that you used at the given point, so if the user decides to update the meta-data this file will change (the hash will be updated).

So the answer to your second question is that you do not have 2 different versions. 

Regarding repositories, each repo will be self-contained in the sense that it will all the meta-data you need to resolve the variants in it. (I still haven’t implemented this yet - starting on adding git functionality today providing I do not find any show stoppers). 

That means that the repo where org.scalatest.scalatest, which requires some variant of org.slf4j.slf4j-api will have this very variant (including a pointer to where it came from). 

Now if you also depend on org.slf4j.slf4j-api, you will also use this repository. 

You will not have a problem as long as the org.slf4j.slf4j-api variants are exactly the same.

If they are a bit different, you will be under-constrained on org.slf4j.slf4j-api, because there can be 2 variants (or more) that both matches all the constraints you specified. BTW: the actual variant meta-data won’t contain information about which repo it comes from, but Adept will know it because it knows where it found the file.

The way resolution works now is that if you are under-constrained, it will suggest the paths that are compatible with what your constraints. It is now up to the build tool to choose which of the variants are the ‘best’ (I am thinking it will pick the ‘official’ variant first, then pick the highest versions if there is ambiguity between those). This will be in the extensions and I haven’t built this yet.

Can you explain or alternatively point me to an example git/package
repository?

I do have an example of data i have imported directly from Ivy:

https://github.com/freekh/adept-repo-example

I haven’t implemented the git repo part yet and also there is no binary version information there, so although it doesn’t show the solution of your issue I thought it would be interesting to have a look at it.

As I am adding functionality, I am building out adepthub.com, which does resolution online and gives you a file which contains all of the artifacts (jars) that you need.  There is also a link that links back to the dependencies you initially specified  (hasn’t been implemented yet) so you can easily the dependencies online. This way you can easily find meta-data in multiple repos, easily publish (you can create repos) and the clients (sbt/gradle) only need a very thin wrapper (really only needs to download the artifacts). BTW: I will also make the meta-data is available though so you can still use regular adept and offline resolution if you want to. 

It also imports automatically from Ivy (using Adept’s Ivy helper). Note that there is some functionality I still want to add such as converting from the weird _2.10 format to mean that it is binary compatible with scala etc etc. 

Also, I haven’t “launched” yet, so expect some warts! When I do have end-to-end functionality I will of course put an announcement here :)

You can try it out by executing this:

curl -L -H "Content-Type: application/json"  -d "[{\"org\": \"joda-time\", \"name\": \"joda-time\", \"version\": \"2.3\"},{\"org\": \"com.fasterxml.jackson.core\", \"name\": \"jackson-databind\", \"version\": \"2.2.3\"},{\"org\": \"org.slf4j\", \"name\": \"slf4j-api\", \"version\": \"1.7.5\"},{\"org\": \"org.apache.httpcomponents\", \"name\": \"httpclient\", \"version\": \"4.3.1\"},{\"org\": \"commons-io\", \"name\": \"commons-io\", \"version\": \"2.4\"},{\"org\": \"commons-codec\", \"name\": \"commons-codec\", \"version\": \"1.8\"},{\"org\": \"com.amazonaws\", \"name\": \"aws-java-sdk\", \"version\": \"1.5.6\"},{\"org\": \"com.typesafe.akka\", \"name\": \"akka-actor_2.10\", \"version\": \"2.2.1\"}]"  http://adepthub.com/adf/ivy

If I can get git functionality up today and do not find any resolution issues, I will start on a sbt plugin for _adepthub_ (the _plugin_ will be open source).

The reasons I am starting on the adepthub plugin, and not an adept plugin, are as follows: it will require less work for me to do, it is easier to manage issues because I can exactly what is going on with the online service, it enables me to push out features without finalising the Adept API before I know it works well, I can manage how the meta-data is used, I can manage the import from Ivy (clean up the meta-data for imported modules) and lastly because I think the user experience will be better because it will be faster (no meta-data downloads) and because I can enforce some conventions on the meta-data. 

Note that you only need to interface with Adepthub if you want to update/change your dependencies. Artifacts will either be downloaded from where ever they where found (if they where imported) or hosted on bintray or, at a later stage, seeded as bittorrent files.

Below you have the commands/tasks that I _think_ will be provided:

- adepthub init

Initialise with your dependencies from Ivy: uploads if you have new/different artifacts than what is in any of the repositories online. When it is done you will have the files that has all the artifacts that you need (one per configuration).

- adepthub install com.typesafe.akka/akka_actor 2.2

Adds akka actor to your dependencies and downloads a new file (with all the artifacts) this. Gives you an error explaining what is wrong, if it is not possible to add this dependency. In the beginning the error messages will be a bit technical, but as I progress I hope to have “did-you-mean” error messages and alternatives to actions you can take to fix the issue.

- adepthub rm com.typesafe.akka/akka_actor

Removes all dependencies with the id com.typesaef.akka/akka_actor

- adethub info

List the information in your artifact file. Prints out a dependency tree.

- adept search play 

Returns all the ids with the binary versions of the variants matching the search criteria

- adept list com.typesafe.play/play

Lists all the variants of Play with the most useful meta-information. As I progress I want to have have tab completion

- compile 

Will depend on a task that downloads all the artifacts, copies them to a directory and renames them to something which is human readable and constructs a classpath based on the files. It will warn (or perhaps throw an error) you if you have something in libraryDependencies. 

- adepthub publish foo-repo

Publishes to _your_ adepthub repo ‘foo-repo'. It requires that your ssh key bound to a adepthub account.

- adepthub show 

Open the url so you can inspect and/change your dependencies in the web UI.

The adept plugin (which will be a separate plugin from adepthub) will behave similarly, but you would have to define the repositories that you need and download them.

Best regards,
Tobias

[Fredrik Ekholdt]

Ah, sorry. I was writing you a (long) response :)

On 12 Dec 2013, at 14:54, Tobias Roeser <le.pet...@web.de> wrote:

Hi Fredrik,

thanks for publishing your adept-repo-example to Github.

Example:
metadata/
 org.slf4j/
   slf4j-api/
     24aa28cb021e7d8a7e742452fe24241d0ad05ff855dc1ae334d6349601901097.json

Now, I can see that each module ends up in a file with a name containing a
hash. Assuming, that name will be calculated based on its content, I can see
how all revisions can be seen simultaneously. But, how is the name of the file
calculated? Obviously, there is some tooling needed editing such files, if the
name depends on the content. Also, any changes would de facto bump the hash  
and thus not just the variant, which was modified.

Yep, as you say the name is calculated depending on the content. Right now it is not the most beautiful way of doing it so I am open to changing it: 

https://github.com/adept-dm/adept/blob/master/src/main/scala/adept/core/models/Hash.scala#L67

Maybe I should just calculate the hash based on the actual JSON? This way it should be a trivial case of just using any SHA-256 implementation to hash it? WDYT?

I'll stop my wild guessing here... Need to read the spec again.

No worries! A discussion is probably just as well :) Don’t hesititate to comment in the spec if something is unclear and I will update it! The spec is still work in progress!

[Tobias Roeser]

Am Donnerstag, 12. Dezember 2013, 14:55:26 schrieb Fredrik Ekholdt:
| On 12 Dec 2013, at 12:37, Tobias Roeser <le.pet...@web.de> wrote:
| > Hello,
| >
| > I have some problem in my understanding how variants are versioned.
| >
| > Imagine the _module_ "org.scalatest.scalatest". The concrete version of
| > that module including its dependency constraints are the _variant_. As
| > the contraint for the same module with the same version might change
| > over time (e.g. some incompatibilities are discovered), a new variant
| > needs to be created. Those variants are referred by their git hash,
| > right?
| >
| >
| > So, imagine you have another module you need, e.g. "org.slf4j.slf4j-api".
| > If both "latest versions" are referred to by a git-hash, how can I
| > specify older versions of them, assuming the required versions are not
| > part of the same git commit/hash? The tricky part for me to understand
| > is, how do you manage to have two different versions of the git tree at
| > the same time?
| >
| > I believe there is some fundamentaly wrong understanding by me. (Maybe,
| > the way the hashes are used to vesion the modules/variants do not map to
| > my mental concept of package revisions.)
|
| Ok, so the first issue you are referring to is how to safely replace a
| variant with a new variant in the case the meta-data changes (let us say
| you update the binary versions) - right?

Not exactly. I thought of a case, where the binary stays the same but you
need to update it's metadata because you need to tighten the constraints.
(Example: You have an old proprietary closed-source db driver and just
discovered that the cool db-log facility it uses has a critical bug in version
1.3.2 and is not compatible, but 1.3.1 was. Of course, you want to share that
with others.)

If I understand you correctly, you can't have the original metadata and the
changed one I just described above in the same repo at the same time?

Ok, now I understand the direction of the development. I will try it out, as
soon as possible. For now, I will focus on my understanding of the metadata.

Regards,
Tobias

[Fredrik Ekholdt]

Ah, right now I get it now - sorry for being slow. No, you would not want to have 2 variants, because you do not want anybody to use the first one - right? The artifact is the same, but you want to avoid that people use your artifact with the wrong db log thingy.
That being said, it is possible to have 2 similar variants. If a users then updates to the latest meta-data and if there is nothing else that helps Adept choose between the 2 (in the case where the user doesn’t use have any other constraints from any other dependency, transitive or not, on the version of the awesome db-log thingy) the graph will now be under-constrained. This means that the user will be prompted to choose either one or the other.

Yep, if you have any other questions please do ask :) In the mean time I will continue with the impl, will give updates the further I get so that you can see how it works with your own eyes. :)
>
> Regards,
> Tobias