working POC of adept with sbt

[Fredrik Ekholdt]

Hey everybody and happy weekend :)

While it has been quite here on this forum, I have been continuing to hack a bit away at adept these last weeks. 

The design is (of course) built around distributed versioned metadata, i.e. I store all information about org, name, version, locations of artifacts, ... locally. Currently I using a DB to store all the information, which actually is working out quite well. The only thing which where it gets complicated is to create the queries for fetching the last version of a module efficiently. Except for the occasional hard-to-write query, versioning is pretty simple since all write operations are idempotent. On the other hand, it is easier to deal with the actual data (storing and fetching, making sure it is reliable and consistent) than with a file based approach. Since it is built from the ground, there are more room for optimizing on cloning and pull performance and make sure the semantics works well for dependency management. I think performance is very important - nobody likes something that wastes perf....

On my POC I can now clone and pull from a repository and I can download modules. More interestingly I have created a link to Ivy which makes it possible to add dependencies using Ivy into an Adept repository. Having a bridge from Ivy/Maven is also important if this is to be adapted :P at all.

The code is here: https://github.com/freekh/adept

My goal is to make sure the code here is not too unfamiliar to a java dev and to make sure incrementale compilation is efficient (ergo all the files :)

Although there might not be that many pure java devs who are interested, something which is familiar to them should take away a bit of the magic :)

The adept-sbt plugin can be found here: https://github.com/freekh/adept-sbt 

It only works for 0.13 and is very much only a POC so building it can still be a bit tricky.

The _current_ design (might change) is that the first time you use it, you have to clone a repository and add the dependencies in a setting, adeptDependencies. The plugin will now create a file, dependencies.adept where it stores all the modules, and repositories and the last version in which the modules were found. If somebody else starts off fresh with the project, adept will use this file to check if you have the repositories required and the correct version. If you have nothing it will clone, if you have an old version it pulls the repository to make sure you are at the correct version. 

My goal going forward is (expect fixing some of the many TODOs :) is to add pruning of modules (i.e. you have a version A of something.foo  and version B, what to you do...)

I have a demo here: http://www.youtube.com/watch?v=uJyRCA9-0Ho&feature=youtube_gdata

I have created a presentation for some general ideas of adept here: http://www.youtube.com/watch?v=MVltyw24TaY&feature=youtube_gdata  if you are interested.

Feedback! Ideas! Praise! are all welcome :)

[Evan Chan]

Fredrik,

Looking forward to trying this out, but things have been pretty busy on a personal level.

On Friday, April 12, 2013 7:12:35 AM UTC-7, Fredrik Ekholdt wrote:

Hey everybody and happy weekend :)

While it has been quite here on this forum, I have been continuing to hack a bit away at adept these last weeks. 

The design is (of course) built around distributed versioned metadata, i.e. I store all information about org, name, version, locations of artifacts, ... locally. Currently I using a DB to store all the information, which actually is working out quite well. The only thing which where it gets complicated is to create the queries for fetching the last version of a module efficiently. Except for the occasional hard-to-write query, versioning is pretty simple since all write operations are idempotent. On the other hand, it is easier to deal with the actual data (storing and fetching, making sure it is reliable and consistent) than with a file based approach. Since it is built from the ground, there are more room for optimizing on cloning and pull performance and make sure the semantics works well for dependency management. I think performance is very important - nobody likes something that wastes perf....

Agreed, though I would think that dependency resolution time should in general be much smaller than the network pull time. 

On my POC I can now clone and pull from a repository and I can download modules. More interestingly I have created a link to Ivy which makes it possible to add dependencies using Ivy into an Adept repository. Having a bridge from Ivy/Maven is also important if this is to be adapted :P at all.

The code is here: https://github.com/freekh/adept

My goal is to make sure the code here is not too unfamiliar to a java dev and to make sure incrementale compilation is efficient (ergo all the files :)

+1.   I know as Scala devs we love the advanced features and functional features, but if we want more people to play with and help develop it, keeping it simple is really important.  

 

Although there might not be that many pure java devs who are interested, something which is familiar to them should take away a bit of the magic :)

We'll have to get Java devs interested if this is to go anywhere.

 

The adept-sbt plugin can be found here: https://github.com/freekh/adept-sbt 

It only works for 0.13 and is very much only a POC so building it can still be a bit tricky.

How hard would it be to use adept as a command line utility, together with scalac?

 

The _current_ design (might change) is that the first time you use it, you have to clone a repository and add the dependencies in a setting, adeptDependencies. The plugin will now create a file, dependencies.adept where it stores all the modules, and repositories and the last version in which the modules were found. If somebody else starts off fresh with the project, adept will use this file to check if you have the repositories required and the correct version. If you have nothing it will clone, if you have an old version it pulls the repository to make sure you are at the correct version. 

My goal going forward is (expect fixing some of the many TODOs :) is to add pruning of modules (i.e. you have a version A of something.foo  and version B, what to you do...)

I have a demo here: http://www.youtube.com/watch?v=uJyRCA9-0Ho&feature=youtube_gdata

I have created a presentation for some general ideas of adept here: http://www.youtube.com/watch?v=MVltyw24TaY&feature=youtube_gdata  if you are interested.

Feedback! Ideas! Praise! are all welcome :)

So at this point,  it seems building another POC from scratch would be a waste of time, but maybe to build on what you've done and make it work with git instead of a DB?

 

[Fredrik Ekholdt]

On Apr 13, 2013, at 8:24 AM, Evan Chan wrote:

Fredrik,

Looking forward to trying this out, but things have been pretty busy on a personal level.

Great! :)

On Friday, April 12, 2013 7:12:35 AM UTC-7, Fredrik Ekholdt wrote:

Hey everybody and happy weekend :)

While it has been quite here on this forum, I have been continuing to hack a bit away at adept these last weeks. 

The design is (of course) built around distributed versioned metadata, i.e. I store all information about org, name, version, locations of artifacts, ... locally. Currently I using a DB to store all the information, which actually is working out quite well. The only thing which where it gets complicated is to create the queries for fetching the last version of a module efficiently. Except for the occasional hard-to-write query, versioning is pretty simple since all write operations are idempotent. On the other hand, it is easier to deal with the actual data (storing and fetching, making sure it is reliable and consistent) than with a file based approach. Since it is built from the ground, there are more room for optimizing on cloning and pull performance and make sure the semantics works well for dependency management. I think performance is very important - nobody likes something that wastes perf....

Agreed, though I would think that dependency resolution time should in general be much smaller than the network pull time. 

Yes, dependency resolution is quite quick now. I want to make it even faster though - if we can make so fast that SBT would not have to cache the results I am happy :)

On my POC I can now clone and pull from a repository and I can download modules. More interestingly I have created a link to Ivy which makes it possible to add dependencies using Ivy into an Adept repository. Having a bridge from Ivy/Maven is also important if this is to be adapted :P at all.

The code is here: https://github.com/freekh/adept

My goal is to make sure the code here is not too unfamiliar to a java dev and to make sure incrementale compilation is efficient (ergo all the files :)

+1.   I know as Scala devs we love the advanced features and functional features, but if we want more people to play with and help develop it, keeping it simple is really important.  

:) I am not sure the code I have is quite there yet though, but if we have this as a general goal I think it will be helpful.

 

Although there might not be that many pure java devs who are interested, something which is familiar to them should take away a bit of the magic :)

We'll have to get Java devs interested if this is to go anywhere.

Yes, you are right there of course!

 

The adept-sbt plugin can be found here: https://github.com/freekh/adept-sbt 

It only works for 0.13 and is very much only a POC so building it can still be a bit tricky.

How hard would it be to use adept as a command line utility, together with scalac?

I do not think it would be hard, but it would be "harder"/more unfamiliar to somebody who is used to have this as part of the build tool.  Therefore, I think having plugins for the popular build systems (sbt, gradle and ant) will be necessary for this to be familiar. 

 

The _current_ design (might change) is that the first time you use it, you have to clone a repository and add the dependencies in a setting, adeptDependencies. The plugin will now create a file, dependencies.adept where it stores all the modules, and repositories and the last version in which the modules were found. If somebody else starts off fresh with the project, adept will use this file to check if you have the repositories required and the correct version. If you have nothing it will clone, if you have an old version it pulls the repository to make sure you are at the correct version. 

My goal going forward is (expect fixing some of the many TODOs :) is to add pruning of modules (i.e. you have a version A of something.foo  and version B, what to you do...)

I have a demo here: http://www.youtube.com/watch?v=uJyRCA9-0Ho&feature=youtube_gdata

I have created a presentation for some general ideas of adept here: http://www.youtube.com/watch?v=MVltyw24TaY&feature=youtube_gdata  if you are interested.

Feedback! Ideas! Praise! are all welcome :)

So at this point,  it seems building another POC from scratch would be a waste of time, but maybe to build on what you've done and make it work with git instead of a DB?

I would love it if you were to help out! If git is a pre-requisite to make this happen I do not mind starting on this myself :) 

That being said, I have been thinking more about it and my big worry about using git is that we lock ourselves down to the constraints it has. Don't get me wrong, Git is awesome and very fast as a vcs, but I worry about the overall complexity that we will end up having if we were to build adept on it.

I agree it is very easy to build a POC on it though.

Git will help us to do the versioning and pulling and pushing, but will it be possible to retrieve and add data quick enough if we use only files? Perhaps the Blueprints TinkerGraph DB you suggested  could solve this for us though, I briefly tried to find more info about how to use it with Git, but could not. 

If we are only using files  I think that might be an issue, because now we have to put the data in an index or a DB and we have 2 sets of data that needs to be managed. I think making sure the system is consistent is very important, therefore I worry about how much code we will end up having to write to make sure that everything is consistent compared to using a DB which can do a lot for us here.

Also I have noticed that with Homebrew,  cloning and pulling from it is really slow. Last time it took me about 3 minutes to clone. There are about 2500 modules on about 12 megs of data. The commit tree is not very deep. I am not sure if it is representative though.

I think it would be nice if somebody (I will try it myself when I have time) could try to bench git performance to be sure?

If git can handle 50 000 different modules in 50 000 files (or in a Graph DB) spread out over, let us say, 25 000 commits without being slow and if we can keep the total amount of data to download to be around 10 megabytes (or around there) - I would be sold! That being said, if we are planning on indexing data, the time spent on doing this after a clone should also be thrown into the equation. 

The DB handled 50K random modules (random org, random name, random version, random artifact hashes) over 50K commits without any problems when testing it on an earlier version, in fact I could not notice any difference in how long the operations (adding, committing, reading) took except cloning which took longer because there where more data. Again, the total amount of data to clone here was about 9.2 megs compressed from 30 megs. I guess this is normal since a DB normally handles much higher volumes than this though. I am confident that it will require quite a lot of work to make something that is as efficient as a DB in retrieving structured data - I mean this is what we use DBs for :)

Perhaps there will be no central repository when using adept and we will never see this amount of data, but I still think it is important that adept scales well if we want it to make an impact. 

Anybody else worried about performance or is it just me?

 

[Fredrik Ekholdt]

BTW a note on Git scalability from Linus: http://osdir.com/ml/git/2009-05/msg00051.html

"And git obviously doesn't have that kind of model at all. Git 
fundamnetally never really looks at less than the whole repo. Even if you 
limit things a bit (ie check out just a portion, or have the history go 
back just a bit), git ends up still always caring about the whole thing, 
and carrying the knowledge around.

So git scales really badly if you force it to look at everything as one 
_huge_ repository. I don't think that part is really fixable, although we 
can probably improve on it.

"

I think it would still be very interesting to have some benchmarks though (if we care about perf that is). I think that is the only way to discuss performance....

On Friday, 12 April 2013 16:12:35 UTC+2, Fredrik Ekholdt wrote:

Hey everybody and happy weekend :)

While it has been quite here on this forum, I have been continuing to hack a bit away at adept these last weeks. 

The design is (of course) built around distributed versioned metadata, i.e. I store all information about org, name, version, locations of artifacts, ... locally. Currently I using a DB to store all the information, which actually is working out quite well. The only thing which where it gets complicated is to create the queries for fetching the last version of a module efficiently. Except for the occasional hard-to-write query, versioning is pretty simple since all write operations are idempotent. On the other hand, it is easier to deal with the actual data (storing and fetching, making sure it is reliable and consistent) than with a file based approach. Since it is built from the ground, there are more room for optimizing on cloning and pull performance and make sure the semantics works well for dependency management. I think performance is very important - nobody likes something that wastes perf....

On my POC I can now clone and pull from a repository and I can download modules. More interestingly I have created a link to Ivy which makes it possible to add dependencies using Ivy into an Adept repository. Having a bridge from Ivy/Maven is also important if this is to be adapted :P at all.

The code is here: https://github.com/freekh/adept

My goal is to make sure the code here is not too unfamiliar to a java dev and to make sure incrementale compilation is efficient (ergo all the files :)

Although there might not be that many pure java devs who are interested, something which is familiar to them should take away a bit of the magic :)

The adept-sbt plugin can be found here: https://github.com/freekh/adept-sbt 

It only works for 0.13 and is very much only a POC so building it can still be a bit tricky.

[Evan Chan]

By the way, we can significantly reduce the number of files git has to manage by having one file per org/module, instead of one file per version.  It would also make version scanning much faster.  Any downsides u guys see?

-Evan

Carry your candle, run to the darkness

Seek out the helpless, deceived and poor

Hold out your candle for all to see it

Take your candle, and go light your world

 

[Fredrik Ekholdt]

On Saturday, April 13, 2013, Evan Chan wrote:

By the way, we can significantly reduce the number of files git has to manage by having one file per org/module, instead of one file per version.  It would also make version scanning much faster.  Any downsides u guys see?

Yeah, that is a very good idea I think! The only drawback i can see is that it would be a bit slower to parse, but i cannot imagine that would lead to any perf issues. Still think it would be good to see how git scales on a large repo with a lot of different modules though...

[Fredrik Ekholdt]

I did some git benchmarking myself this morning and actually it seems like git holds out quite well, even with 20K (didn't have the patience to wait for 50K)

It becomes slower to add new modules as it goes along but not that much. (started at around 200ms to add + commit, ended at 1.3 secs - I think a git gc could clear that up though I didn't think about trying before git did it automatically when I added something later)

Cloning and pulling are slower than on the DB, but not that much. 

So, despite what I have been worried about it does seem like git is scaling quite well...

Therefore and since everybody seems to want adept to use Git, I have started rewriting everything to see how it will look with Git. 

Took me some hours but I have now almost gotten feature parity using git.

You can find it here: https://github.com/freekh/adept/tree/git-impl

It can init, clone, add from ivy and download artifacts, just need a bit more time to fix the sbt plugin. It also compiles for 2.9.3 and I have made sure it should not be too hard to make it compile on 2.10 (just some deps that must be changed), which I could not before.

I added an example of the generated metadata here: https://github.com/freekh/adept-central

There are a number of open challenges though. I have to:

- add metadata (should be easy)

- determine a better json format (now it is not really optimal)

- decide on how pruning of versions should work (while keeping it compatible with modules that are imported from ivy)

- figure out how to make downloading from multiple locations work in a more efficient way. Now it just takes the one that completes first - I was thinking it would be cool if it could figure out which host it should bet on or something like that

- add pulling and figure out how to check if it needs to pull or not

- figure out how to handle merge conflicts

- add lot of verification to make sure data is kept consistent

- add multi repo support

- figure out if there should be branches

- more I am certain :) 

[Fredrik Ekholdt]

I have also updated the adept-sbt plugin and it should work with 0.12.2 now.

I removed the adept-sbt git repo, to make it easier to work with it. 

Now you can: 

- go to adept

- then cd into adept-sbt then 

- test-project and then

- run "sbt compile"

sbt will now use adept to build the classpath. It does not work with multiple conflicting versions yet though.

I have changed the way the sbt plugin so it ressembles more to ivy: now you set the adept repositories and it uses those... 

I think this is more familiar though it loses the benefit of always checking the latest version the repositories must be in.

I have also started to think about how to get the bulk of the metadata from maven-central to an adept repository. I will post some updates as get further!

F

On Friday, 12 April 2013 16:12:35 UTC+2, Fredrik Ekholdt wrote:

Hey everybody and happy weekend :)

While it has been quite here on this forum, I have been continuing to hack a bit away at adept these last weeks. 

The design is (of course) built around distributed versioned metadata, i.e. I store all information about org, name, version, locations of artifacts, ... locally. Currently I using a DB to store all the information, which actually is working out quite well. The only thing which where it gets complicated is to create the queries for fetching the last version of a module efficiently. Except for the occasional hard-to-write query, versioning is pretty simple since all write operations are idempotent. On the other hand, it is easier to deal with the actual data (storing and fetching, making sure it is reliable and consistent) than with a file based approach. Since it is built from the ground, there are more room for optimizing on cloning and pull performance and make sure the semantics works well for dependency management. I think performance is very important - nobody likes something that wastes perf....

On my POC I can now clone and pull from a repository and I can download modules. More interestingly I have created a link to Ivy which makes it possible to add dependencies using Ivy into an Adept repository. Having a bridge from Ivy/Maven is also important if this is to be adapted :P at all.

The code is here: https://github.com/freekh/adept

My goal is to make sure the code here is not too unfamiliar to a java dev and to make sure incrementale compilation is efficient (ergo all the files :)

Although there might not be that many pure java devs who are interested, something which is familiar to them should take away a bit of the magic :)

The adept-sbt plugin can be found here: https://github.com/freekh/adept-sbt 

It only works for 0.13 and is very much only a POC so building it can still be a bit tricky.

[Evan Chan]

Fredrik,

I cloned your repo and started playing with it.   Didn't have to patience to watch the video, sorry  :)   One quick comment.

An advantage of the separate-metadata approach that you have not mentioned:

- Ability to foresee dependency issues BEFORE jars are downloaded

This is a pretty big time saver in my opinion.

Also, can you post the other links again to the sample adept repo that you have again?

thanks,

Evan

--

Because the people who are crazy enough to think they can change the world,

are the ones who do.     -- Steve Jobs

[Fredrik Ekholdt]

Hi!

On Apr 22, 2013, at 2:31 AM, Evan Chan wrote:

Fredrik,

I cloned your repo and started playing with it.  

That is great! 

Didn't have to patience to watch the video, sorry  :)  

Don't worry :) I should make a new which is shorter and more interesting when I have more time :)

One quick comment.

An advantage of the separate-metadata approach that you have not mentioned:

- Ability to foresee dependency issues BEFORE jars are downloaded

This is a pretty big time saver in my opinion.

Hmm… Right, I guess it is in particular the build tools that can do a lot of things before _anything_ is downloaded. 

It would be great if you had a specific example of how this has been a time sink for you!?

Another thing that I do not mention is that you never need an internet connection if you already have downloaded stuff :)

Sometimes with Ivy and Maven, it invalidates its caches and starts to try to download stuff although I already have everything it needs!

Mostly annoying if you travel a lot (like me :)

Also, can you post the other links again to the sample adept repo that you have again?

The sample adept repo can be found here: https://github.com/freekh/adept-central

You can clone it by doing:

adept clone g...@github.com:freekh/adept-central.git

then you should be able to inspect the json files under .adept

Note that I am planning on changing the json format a bit. Currently there are more than one module in each file (one per hash), which is good I am repeating the org, name, version several places and the automatic deserialization also add empty lists which I think I want to remove.

[Mark Harrah]

After watching the videos and browsing adept-central, here are my thoughts:

Perhaps keep your code in a branch on sbt/adept for visibility? We could list all existing POCs on the master README.

I'd suggest keeping the dependency declarations separate from how to get the dependencies. It is very tempting to include a URL in the metadata, but at a minimum there would have to be a way to ignore these explicit URLs. I think I'd prefer to have a "legacy" or "direct" artifact repository that maps a hash to a URL. I'm undecided on whether this is a good idea, but my experiences so far indicate that separating "what" from "how" is a good idea. The execution just needs to be better to give a better user experience in common cases.

I like the dependencies.adept file idea. It seems to me that it is close to an ideal balance of a project providing full information to build with the principle that repositories should be configured per-machine. It sounds like the main work on a user's part right now is if a dependencies.adept file has repositories you don't want to use? (I also think the name should be more like resolve.adept or something that indicates it is an output and not an input. This is a minor issue right now though.)

I like the idea of having all versions for a module in one file. This solves the problem of where to put the metadata when using multiple version attributes (see the other thread on Pruning modules).

I like the download logging with the count, amount to download, speed, and progress. I expect there will be annoying interactions with sbt logging, but I think that is a detail for later.

I've mentioned this before, but I think that as much as possible, the expected behavior should be specified in a separate, English document once that behavior is decided on. This is the easiest way for a contributor/user to know what _should_ be happening and helps identify what is just not implemented yet. Certainly POCs don't need to be fully specified, but I think making this part of the process early is important. Once behaviors are reasonably well agreed on, I'd say specify them. I can be the one to extract this information and maintain a specification (used somewhat loosely), but only if everyone else thinks this is a good idea and cooperates ;).

-Mark

[Fredrik Ekholdt]

Yep, that is a good idea. I have pushed my branch now.

>
> I'd suggest keeping the dependency declarations separate from how to get the dependencies. It is very tempting to include a URL in the metadata, but at a minimum there would have to be a way to ignore these explicit URLs. I think I'd prefer to have a "legacy" or "direct" artifact repository that maps a hash to a URL. I'm undecided on whether this is a good idea, but my experiences so far indicate that separating "what" from "how" is a good idea. The execution just needs to be better to give a better user experience in common cases.

OK. It would be great if you could think a bit about the pros/cons for this?
The cons for me with this approach is that: 1) sharing jar files will become more complicated 2) it is harder to debug 3) cannot reuse maven central directly
One pro would be that you could quite safely download one artifact from several repositories at the same time.

>
> I like the dependencies.adept file idea. It seems to me that it is close to an ideal balance of a project providing full information to build with the principle that repositories should be configured per-machine. It sounds like the main work on a user's part right now is if a dependencies.adept file has repositories you don't want to use? (I also think the name should be more like resolve.adept or something that indicates it is an output and not an input. This is a minor issue right now though.)

Great that you like the idea! After rewriting the SBT plugin I did not include this, but since you think it sounds like a good idea I will put it back in!

Yes, I guess if you do not want to use certain repositories with this approach you would have to delete them from the resolve.adept file (and the local adept if you have it there as well).
Perhaps it could be solved in a simpler manner and have a SBT setting that lets you prune out the repositories you do not want to use?

>
> I like the idea of having all versions for a module in one file. This solves the problem of where to put the metadata when using multiple version attributes (see the other thread on Pruning modules).

Yes, Evan had a great idea there! :)

>
> I like the download logging with the count, amount to download, speed, and progress. I expect there will be annoying interactions with sbt logging, but I think that is a detail for later.

Yep. I also want to add an ETA as well. I think it is very important to have a good user experience (even for a cmd line tool), and that means it is important to be responsive.
My hope is to make the downloader much better.
Currently it downloads as fast as possible from all locations. If there is a file that has multiple locations it chooses the one that finishes first.
My dream is to have a downloader that downloads multiple parts of a file, then figures out what is fastest (download everything from the same host or from another). There will have to be a mechanism for corrupt data handling as well.. Again, I was thinking that torrents might help us do this, but the issue is that we would need support for it. I guess this goes back to how we get artifacts/dependencies.

>
> I've mentioned this before, but I think that as much as possible, the expected behavior should be specified in a separate, English document once that behavior is decided on. This is the easiest way for a contributor/user to know what _should_ be happening and helps identify what is just not implemented yet. Certainly POCs don't need to be fully specified, but I think making this part of the process early is important. Once behaviors are reasonably well agreed on, I'd say specify them. I can be the one to extract this information and maintain a specification (used somewhat loosely), but only if everyone else thinks this is a good idea and cooperates ;).

Yes, I agree that this is very very important. I think it is important that adept has a culture where every feature means changing documentation.
For me, it would be great if you want to do this now!
At least we should have one person coordinating this to make sure the doc is as crisp, useful and up-to-date as it can be. I want to emphasize on "crisp", it is important that we do not have more doc than needed either. If there is too much doc, it is harder to find what you need and the maintenance burden is heavier.
>
> -Mark

[Mark Harrah]

On Tue, 23 Apr 2013 13:41:57 +0200

Can you elaborate on 2) ?

For 3), you could still use Maven Central directly. I agree that being able to specify things as a URL somewhere is one approach to compatibility with non-adept artifact repositories. The proposal was to have a separate type of artifact repository that maps a hash to a URL instead of a hash to content. (That repository might be a single file or many files each containing a URL or list of URLs.) This contains the problem to a specific repository instead of spreading it out through the metadata.

For me, URLs are mainly for compatibility. Yes, they are convenient, but an important aspect of a dependency manager is to ensure that artifacts are still around years later. Allowing a URL discourages this unless it is only used to point at stable URLs, such as a legacy repository.

A URL is also back to the problems with caching artifacts in Maven/Ivy. You don't locally know whether the content at the other end has changed. You have to contact the remote server to find out. So, it shouldn't be encouraged.

In terms of evolving, a hash->URL repository gives more flexibility. The user can swap out that hash->URL repository if their dependencies become available in a normal hash->content repository. They don't have to be in charge of the metadata (or write some transformation on the metadata) to do this.

> > I like the dependencies.adept file idea. It seems to me that it is close to an ideal balance of a project providing full information to build with the principle that repositories should be configured per-machine. It sounds like the main work on a user's part right now is if a dependencies.adept file has repositories you don't want to use? (I also think the name should be more like resolve.adept or something that indicates it is an output and not an input. This is a minor issue right now though.)
> Great that you like the idea! After rewriting the SBT plugin I did not include this, but since you think it sounds like a good idea I will put it back in!

Oh, was there a reason? What sounded like a good idea to me was that it was an explicit, human readable output that you could check into your project as

a) documentation of a set of repositories that would successfully resolve the dependencies
b) the exact commits from those repositories
c) a way to lock down a project

So, perhaps by default:

a) the .adept file is generated by a resolve with the configured repositories and used commits
b) the user can lock it (adds lock: true to the file or whatever). This means that adept will use only the listed repositories at the specific commits.
c) on a failed resolution, if the .adept file contains repositories that the user does not have, adept generates a warning with the missing repositories.

> Yes, I guess if you do not want to use certain repositories with this approach you would have to delete them from the resolve.adept file (and the local adept if you have it there as well).
> Perhaps it could be solved in a simpler manner and have a SBT setting that lets you prune out the repositories you do not want to use?



> > I like the idea of having all versions for a module in one file. This solves the problem of where to put the metadata when using multiple version attributes (see the other thread on Pruning modules).
> Yes, Evan had a great idea there! :)
> >
> > I like the download logging with the count, amount to download, speed, and progress. I expect there will be annoying interactions with sbt logging, but I think that is a detail for later.
> Yep. I also want to add an ETA as well. I think it is very important to have a good user experience (even for a cmd line tool), and that means it is important to be responsive.
> My hope is to make the downloader much better.
> Currently it downloads as fast as possible from all locations. If there is a file that has multiple locations it chooses the one that finishes first.
> My dream is to have a downloader that downloads multiple parts of a file, then figures out what is fastest (download everything from the same host or from another). There will have to be a mechanism for corrupt data handling as well.. Again, I was thinking that torrents might help us do this, but the issue is that we would need support for it. I guess this goes back to how we get artifacts/dependencies.
> >
> > I've mentioned this before, but I think that as much as possible, the expected behavior should be specified in a separate, English document once that behavior is decided on. This is the easiest way for a contributor/user to know what _should_ be happening and helps identify what is just not implemented yet. Certainly POCs don't need to be fully specified, but I think making this part of the process early is important. Once behaviors are reasonably well agreed on, I'd say specify them. I can be the one to extract this information and maintain a specification (used somewhat loosely), but only if everyone else thinks this is a good idea and cooperates ;).
> Yes, I agree that this is very very important. I think it is important that adept has a culture where every feature means changing documentation.
> For me, it would be great if you want to do this now!
> At least we should have one person coordinating this to make sure the doc is as crisp, useful and up-to-date as it can be. I want to emphasize on "crisp", it is important that we do not have more doc than needed either. If there is too much doc, it is harder to find what you need and the maintenance burden is heavier.

For me, this is separate from user documentation. This is a specification, with ideally enough detail/precision that you could write a clean implementation from it. (In practice, that wouldn't happen without someone actually trying to do it.)

-Mark

> > -Mark
>

[Fredrik Ekholdt]

In Ivy/Maven when something fails I can relatively simply figure out what went wrong (i.e. the resolver is wrong, …) because it is just an URL. When you can see exactly where stuff comes from this can be a good way to debug. of course, it would be better if the users never had to come into this type of situation :)

>
> For 3), you could still use Maven Central directly. I agree that being able to specify things as a URL somewhere is one approach to compatibility with non-adept artifact repositories. The proposal was to have a separate type of artifact repository that maps a hash to a URL instead of a hash to content. (That repository might be a single file or many files each containing a URL or list of URLs.) This contains the problem to a specific repository instead of spreading it out through the metadata.

Aha, ok I see what you mean. I agree that that sounds like a good idea!

>
> For me, URLs are mainly for compatibility. Yes, they are convenient, but an important aspect of a dependency manager is to ensure that artifacts are still around years later. Allowing a URL discourages this unless it is only used to point at stable URLs, such as a legacy repository.

Right, I see where you are getting at. As long as we have URLs for compatibility, I think this is a good idea.

>
> A URL is also back to the problems with caching artifacts in Maven/Ivy. You don't locally know whether the content at the other end has changed. You have to contact the remote server to find out. So, it shouldn't be encouraged.

Currently, the sha-1 is stored in information for adept. When an artifact is downloaded the sha-1 is checked and you get an error if it does not check out. If a sha-1 is changed it will download the new artifact. In this setting I think there should not be any caching issues as such?

>
> In terms of evolving, a hash->URL repository gives more flexibility. The user can swap out that hash->URL repository if their dependencies become available in a normal hash->content repository. They don't have to be in charge of the metadata (or write some transformation on the metadata) to do this.

Yes, this is true. We will have to have an adept artifact-server or something then to make it possible to host this? With this approach, it would also be possible to use torrents though so that is a enormous plus :)

>
>
>>> I like the dependencies.adept file idea. It seems to me that it is close to an ideal balance of a project providing full information to build with the principle that repositories should be configured per-machine. It sounds like the main work on a user's part right now is if a dependencies.adept file has repositories you don't want to use? (I also think the name should be more like resolve.adept or something that indicates it is an output and not an input. This is a minor issue right now though.)
>> Great that you like the idea! After rewriting the SBT plugin I did not include this, but since you think it sounds like a good idea I will put it back in!
>
> Oh, was there a reason? What sounded like a good idea to me was that it was an explicit, human readable output that you could check into your project as

I had some difficulty in figuring out how to check which commit a module is in, and therefore I did not add it. I am planning on putting it back in though!

>
> a) documentation of a set of repositories that would successfully resolve the dependencies
> b) the exact commits from those repositories
> c) a way to lock down a project
>

Yep this is what I was thinking was nice with it! :)

> So, perhaps by default:
>
> a) the .adept file is generated by a resolve with the configured repositories and used commits

Yep, that is the way it worked

> b) the user can lock it (adds lock: true to the file or whatever). This means that adept will use only the listed repositories at the specific commits.

Yes, that is an interesting idea. You could also simply check it into you VCS, so that you would be able to easily spot changes.

> c) on a failed resolution, if the .adept file contains repositories that the user does not have, adept generates a warning with the missing repositories.

I was thinking that adept would pull and update itself to the last version specified in the resolve.adept file.

>
>> Yes, I guess if you do not want to use certain repositories with this approach you would have to delete them from the resolve.adept file (and the local adept if you have it there as well).
>> Perhaps it could be solved in a simpler manner and have a SBT setting that lets you prune out the repositories you do not want to use?
>
>
>
>>> I like the idea of having all versions for a module in one file. This solves the problem of where to put the metadata when using multiple version attributes (see the other thread on Pruning modules).
>> Yes, Evan had a great idea there! :)
>>>
>>> I like the download logging with the count, amount to download, speed, and progress. I expect there will be annoying interactions with sbt logging, but I think that is a detail for later.
>> Yep. I also want to add an ETA as well. I think it is very important to have a good user experience (even for a cmd line tool), and that means it is important to be responsive.
>> My hope is to make the downloader much better.
>> Currently it downloads as fast as possible from all locations. If there is a file that has multiple locations it chooses the one that finishes first.
>> My dream is to have a downloader that downloads multiple parts of a file, then figures out what is fastest (download everything from the same host or from another). There will have to be a mechanism for corrupt data handling as well.. Again, I was thinking that torrents might help us do this, but the issue is that we would need support for it. I guess this goes back to how we get artifacts/dependencies.
>>>
>>> I've mentioned this before, but I think that as much as possible, the expected behavior should be specified in a separate, English document once that behavior is decided on. This is the easiest way for a contributor/user to know what _should_ be happening and helps identify what is just not implemented yet. Certainly POCs don't need to be fully specified, but I think making this part of the process early is important. Once behaviors are reasonably well agreed on, I'd say specify them. I can be the one to extract this information and maintain a specification (used somewhat loosely), but only if everyone else thinks this is a good idea and cooperates ;).
>> Yes, I agree that this is very very important. I think it is important that adept has a culture where every feature means changing documentation.
>> For me, it would be great if you want to do this now!
>> At least we should have one person coordinating this to make sure the doc is as crisp, useful and up-to-date as it can be. I want to emphasize on "crisp", it is important that we do not have more doc than needed either. If there is too much doc, it is harder to find what you need and the maintenance burden is heavier.
>
> For me, this is separate from user documentation. This is a specification, with ideally enough detail/precision that you could write a clean implementation from it. (In practice, that wouldn't happen without someone actually trying to do it.)

Aha, I see what you mean then. I think specifications are a lot of work to maintain and might be buggy/detached from the actual implementation(s). If we were writing a protocol I would agree that this is vital.
My goal with adept, to be blunt, is simply to make sure that the JVM ecosystem has a dependency management system that does not give me grey hairs and makes me waste my time on waiting for it to finish...
I am not sure if spending time on a specification that enables you to write a new implementation is the way.
How about if we concentrate all our efforts on making a clean, well documented implementation that makes it possible to extract whatever you want and build on top of it?
>
> -Mark
>
>>> -Mark
>>
>

[Mark Harrah]

> >>>>> I have a demo here: http://www.youtube.com/watch?v=uJyRCA9-0Ho&feature=youtube_gdata
> >>>>> I have created a presentation for some general ideas of adept here: http://www.youtube.com/watch?v=MVltyw24TaY&feature=youtube_gdata if you are interested.
> >>>>>
> >>>>> Feedback! Ideas! Praise! are all welcome :)
> >>>
> >>> After watching the videos and browsing adept-central, here are my thoughts:
> >>>
> >>> Perhaps keep your code in a branch on sbt/adept for visibility? We could list all existing POCs on the master README.
> >> Yep, that is a good idea. I have pushed my branch now.
> >>>
> >>> I'd suggest keeping the dependency declarations separate from how to get the dependencies. It is very tempting to include a URL in the metadata, but at a minimum there would have to be a way to ignore these explicit URLs. I think I'd prefer to have a "legacy" or "direct" artifact repository that maps a hash to a URL. I'm undecided on whether this is a good idea, but my experiences so far indicate that separating "what" from "how" is a good idea. The execution just needs to be better to give a better user experience in common cases.
> >> OK. It would be great if you could think a bit about the pros/cons for this?
> >> The cons for me with this approach is that: 1) sharing jar files will become more complicated 2) it is harder to debug 3) cannot reuse maven central directly
> >> One pro would be that you could quite safely download one artifact from several repositories at the same time.
> >
> >
> > Can you elaborate on 2) ?
> In Ivy/Maven when something fails I can relatively simply figure out what went wrong (i.e. the resolver is wrong, …) because it is just an URL. When you can see exactly where stuff comes from this can be a good way to debug. of course, it would be better if the users never had to come into this type of situation :)

I'm not sure how this changes. There is still a URL that gets formed and downloaded from. If there are multiple types of repositories, you can still indicate what types were tried.

[...]

> > A URL is also back to the problems with caching artifacts in Maven/Ivy. You don't locally know whether the content at the other end has changed. You have to contact the remote server to find out. So, it shouldn't be encouraged.
> Currently, the sha-1 is stored in information for adept. When an artifact is downloaded the sha-1 is checked and you get an error if it does not check out. If a sha-1 is changed it will download the new artifact. In this setting I think there should not be any caching issues as such?

Yes, you're right.

> > In terms of evolving, a hash->URL repository gives more flexibility. The user can swap out that hash->URL repository if their dependencies become available in a normal hash->content repository. They don't have to be in charge of the metadata (or write some transformation on the metadata) to do this.
> Yes, this is true. We will have to have an adept artifact-server or something then to make it possible to host this? With this approach, it would also be possible to use torrents though so that is a enormous plus :)

No, it can just be a file based repository. You can either have a single file with the mappings or map the path at example.org/repo/a6/349e8d to return a URL or redirect to it or whatever.

Overall, this only considers existing URLs. How will adept publish artifacts itself? A scheme that automatically creates a maven-like path will be tricky with adept's more powerful metadata. Or, does the user manually specify URLs for each artifact? That's tedious.

How are mirroring and proxying supported? The URLs would need to be ignored or rewritten and in the end there'd have some other scheme backing it anyway. Having a specific artifact repository and making the path the hash solves these kinds of problems.

https://github.com/sbt/adept/wiki/NEScala-Proposal#artifact-repository

> >>> I like the dependencies.adept file idea. It seems to me that it is close to an ideal balance of a project providing full information to build with the principle that repositories should be configured per-machine. It sounds like the main work on a user's part right now is if a dependencies.adept file has repositories you don't want to use? (I also think the name should be more like resolve.adept or something that indicates it is an output and not an input. This is a minor issue right now though.)
> >> Great that you like the idea! After rewriting the SBT plugin I did not include this, but since you think it sounds like a good idea I will put it back in!
> >
> > Oh, was there a reason? What sounded like a good idea to me was that it was an explicit, human readable output that you could check into your project as
> I had some difficulty in figuring out how to check which commit a module is in, and therefore I did not add it. I am planning on putting it back in though!

Can you elaborate? Why do you need to lookup the commit? Either the user explicitly specifies it (so you know it trivially) or it is HEAD or a branch, and you ask git to resolve it.

> > a) documentation of a set of repositories that would successfully resolve the dependencies
> > b) the exact commits from those repositories
> > c) a way to lock down a project
> >
> Yep this is what I was thinking was nice with it! :)
> > So, perhaps by default:
> >
> > a) the .adept file is generated by a resolve with the configured repositories and used commits
> Yep, that is the way it worked
> > b) the user can lock it (adds lock: true to the file or whatever). This means that adept will use only the listed repositories at the specific commits.
> Yes, that is an interesting idea. You could also simply check it into you VCS, so that you would be able to easily spot changes.
> > c) on a failed resolution, if the .adept file contains repositories that the user does not have, adept generates a warning with the missing repositories.
> I was thinking that adept would pull and update itself to the last version specified in the resolve.adept file.

What do you mean here? Are you now suggesting that adept reads from it even when unlocked?

-Mark

[Fredrik Ekholdt]

On Apr 24, 2013, at 8:53 PM, Mark Harrah wrote:

>>>>>>> I have a demo here: http://www.youtube.com/watch?v=uJyRCA9-0Ho&feature=youtube_gdata
>>>>>>> I have created a presentation for some general ideas of adept here: http://www.youtube.com/watch?v=MVltyw24TaY&feature=youtube_gdata if you are interested.
>>>>>>>
>>>>>>> Feedback! Ideas! Praise! are all welcome :)
>>>>>
>>>>> After watching the videos and browsing adept-central, here are my thoughts:
>>>>>
>>>>> Perhaps keep your code in a branch on sbt/adept for visibility? We could list all existing POCs on the master README.
>>>> Yep, that is a good idea. I have pushed my branch now.
>>>>>
>>>>> I'd suggest keeping the dependency declarations separate from how to get the dependencies. It is very tempting to include a URL in the metadata, but at a minimum there would have to be a way to ignore these explicit URLs. I think I'd prefer to have a "legacy" or "direct" artifact repository that maps a hash to a URL. I'm undecided on whether this is a good idea, but my experiences so far indicate that separating "what" from "how" is a good idea. The execution just needs to be better to give a better user experience in common cases.
>>>> OK. It would be great if you could think a bit about the pros/cons for this?
>>>> The cons for me with this approach is that: 1) sharing jar files will become more complicated 2) it is harder to debug 3) cannot reuse maven central directly
>>>> One pro would be that you could quite safely download one artifact from several repositories at the same time.
>>>
>>>
>>> Can you elaborate on 2) ?
>> In Ivy/Maven when something fails I can relatively simply figure out what went wrong (i.e. the resolver is wrong, …) because it is just an URL. When you can see exactly where stuff comes from this can be a good way to debug. of course, it would be better if the users never had to come into this type of situation :)
>
> I'm not sure how this changes. There is still a URL that gets formed and downloaded from. If there are multiple types of repositories, you can still indicate what types were tried.

Yes, you are right.

>
> [...]
>
>>> A URL is also back to the problems with caching artifacts in Maven/Ivy. You don't locally know whether the content at the other end has changed. You have to contact the remote server to find out. So, it shouldn't be encouraged.
>> Currently, the sha-1 is stored in information for adept. When an artifact is downloaded the sha-1 is checked and you get an error if it does not check out. If a sha-1 is changed it will download the new artifact. In this setting I think there should not be any caching issues as such?
>
> Yes, you're right.
>
>>> In terms of evolving, a hash->URL repository gives more flexibility. The user can swap out that hash->URL repository if their dependencies become available in a normal hash->content repository. They don't have to be in charge of the metadata (or write some transformation on the metadata) to do this.
>> Yes, this is true. We will have to have an adept artifact-server or something then to make it possible to host this? With this approach, it would also be possible to use torrents though so that is a enormous plus :)
>
> No, it can just be a file based repository. You can either have a single file with the mappings or map the path at example.org/repo/a6/349e8d to return a URL or redirect to it or whatever.

Good point :)

>
> Overall, this only considers existing URLs. How will adept publish artifacts itself? A scheme that automatically creates a maven-like path will be tricky with adept's more powerful metadata. Or, does the user manually specify URLs for each artifact? That's tedious.

Yes, I agree. For files hosted by adept itself, this type of scheme sounds good.

>
> How are mirroring and proxying supported? The URLs would need to be ignored or rewritten and in the end there'd have some other scheme backing it anyway. Having a specific artifact repository and making the path the hash solves these kinds of problems.

Just so we are clear, there are 2 things you would want to support with mirroring/proxying AFAIK:
1) more bandwidth (or _some_ bandwitdh in case something goes down :)
2) support repositories which are hosted on closed network (i.e. where companies does not allow direct internet connections)

If we take the approach which is to both have adept artifact repositories and support for URLs, I think both of these use cases can be supported.
I think we can make the adept artifact repos will give you the flexibility to support both use cases.
URLs will give more bandwidth easily. For only URLs I guess supporting use case 2) is a bit harder.

>
> https://github.com/sbt/adept/wiki/NEScala-Proposal#artifact-repository
>
>>>>> I like the dependencies.adept file idea. It seems to me that it is close to an ideal balance of a project providing full information to build with the principle that repositories should be configured per-machine. It sounds like the main work on a user's part right now is if a dependencies.adept file has repositories you don't want to use? (I also think the name should be more like resolve.adept or something that indicates it is an output and not an input. This is a minor issue right now though.)
>>>> Great that you like the idea! After rewriting the SBT plugin I did not include this, but since you think it sounds like a good idea I will put it back in!
>>>
>>> Oh, was there a reason? What sounded like a good idea to me was that it was an explicit, human readable output that you could check into your project as
>> I had some difficulty in figuring out how to check which commit a module is in, and therefore I did not add it. I am planning on putting it back in though!
>
> Can you elaborate? Why do you need to lookup the commit? Either the user explicitly specifies it (so you know it trivially) or it is HEAD or a branch, and you ask git to resolve it.

Perhaps this is not needed but I wanted to figure out the last commit you have to have to have the exact same repositories as the one that created the resolve.adept file, and not the last commit that person was using.
User story:
I constantly keep myself updated on the latest commits from all repos I am using because of my pet project. I use only certain modules which are not updated often in my main project though.
For people working with me on the main project, should only be required to update their metadata to the latest commit that I am actually using in that project.
If they constantly would have to pull (because I use the HEAD reference I was on), adept would seem slower than it really needs to be.

>
>>> a) documentation of a set of repositories that would successfully resolve the dependencies
>>> b) the exact commits from those repositories
>>> c) a way to lock down a project
>>>
>> Yep this is what I was thinking was nice with it! :)
>>> So, perhaps by default:
>>>
>>> a) the .adept file is generated by a resolve with the configured repositories and used commits
>> Yep, that is the way it worked
>>> b) the user can lock it (adds lock: true to the file or whatever). This means that adept will use only the listed repositories at the specific commits.
>> Yes, that is an interesting idea. You could also simply check it into you VCS, so that you would be able to easily spot changes.
>>> c) on a failed resolution, if the .adept file contains repositories that the user does not have, adept generates a warning with the missing repositories.
>> I was thinking that adept would pull and update itself to the last version specified in the resolve.adept file.
>
> What do you mean here? Are you now suggesting that adept reads from it even when unlocked?

Hmm.. I think I am misunderstanding you.
Perhaps you could elaborate on what the point of the lock would be? The way I understood it was the point of locking down a project was to carefully manage what dependencies are being used in the project. Thus, if you had a file which lists all of the modules, repositories etc etc under version control it would be easy to spot a change or similar during a code review. Sorry for being slow, I guess I am just struggling to combine your truth with the one I had :)


>
> -Mark

[Evan Chan]

>>>
>>>
>>> Can you elaborate on 2) ?
>> In Ivy/Maven when something fails I can relatively simply figure out what went wrong (i.e. the resolver is wrong, …) because it is just an URL. When you can see exactly where stuff comes from this can be a good way to debug. of course, it would be better if the users never had to come into this type of situation :)
>
> I'm not sure how this changes.  There is still a URL that gets formed and downloaded from.  If there are multiple types of repositories, you can still indicate what types were tried.

Yes, you are right.

>
> [...]
>
>>> A URL is also back to the problems with caching artifacts in Maven/Ivy.  You don't locally know whether the content at the other end has changed.  You have to contact the remote server to find out.  So, it shouldn't be encouraged.
>> Currently, the sha-1 is stored in information for adept. When an artifact is downloaded the sha-1 is checked and you get an error if it does not check out. If a sha-1 is changed it will download the new artifact. In this setting I think there should not be any caching issues as such?
>
> Yes, you're right.
>
>>> In terms of evolving, a hash->URL repository gives more flexibility.  The user can swap out that hash->URL repository if their dependencies become available in a normal hash->content repository.  They don't have to be in charge of the metadata (or write some transformation on the metadata) to do this.
>> Yes, this is true. We will have to have an adept artifact-server or something then to make it possible to host this? With this approach, it would also be possible to use torrents though so that is a enormous plus :)
>
> No, it can just be a file based repository.  You can either have a single file with the mappings or map the path at example.org/repo/a6/349e8d to return a URL or redirect to it or whatever.

Good point :)

An extra hash -> URL mapping is kinda inefficient and means extra state to keep track of, no?   How about storing the Maven _relative_ URL as an extra, optional attribute?    It gets stored along with the package metadata.

The reason why I like relative URLs is it makes dealing with proxies and multiple backups of repos very easy.

Actually, once you know the org, module, and version, in theory you can create the relative Maven URL, or is that not always true?

My guess is that the ability to just have adept metadata pointing at existing Maven jars/repos will be a huge compatibility win.    It will probably make it much much easier to bootstrap people.   You can then easily let people see concrete benefits from having better metadata and adept......  gee what if the standard Scala libs, Play, Akka etc. all had this extra compatibility info..

-Evan

 

[Mark Harrah]

I think it is pretty easy/cheap to check for an update to a single file containing those mappings at the same time that the git metadata repositories are updated.

A relative URL might work, but there also needs to be a list of servers to resolve that relative URL against. That server list is probably not tied to the specific artifact, which means all servers have to get checked for all artifacts, which is slow.

> The reason why I like relative URLs is it makes dealing with proxies and
> multiple backups of repos very easy.

Yes, it is an improvement over a hard-code URL in some respects.

> Actually, once you know the org, module, and version, in theory you can
> create the relative Maven URL, or is that not always true?

There are other things to consider, like the classifier, extension, and turning a -SNAPSHOT into a timestamp.

> My guess is that the ability to just have adept metadata pointing at
> existing Maven jars/repos will be a huge compatibility win. It will
> probably make it much much easier to bootstrap people. You can then
> easily let people see concrete benefits from having better metadata and
> adept...... gee what if the standard Scala libs, Play, Akka etc. all had
> this extra compatibility info..

Yes, to some extent, although in my experience doing resolution in a compatible way is rather involved.

-Mark

[Mark Harrah]

On Fri, 26 Apr 2013 11:08:22 +0200
Fredrik Ekholdt <fre...@gmail.com> wrote:
> On Apr 24, 2013, at 8:53 PM, Mark Harrah wrote:

[...]

> > How are mirroring and proxying supported? The URLs would need to be ignored or rewritten and in the end there'd have some other scheme backing it anyway. Having a specific artifact repository and making the path the hash solves these kinds of problems.
> Just so we are clear, there are 2 things you would want to support with mirroring/proxying AFAIK:
> 1) more bandwidth (or _some_ bandwitdh in case something goes down :)
> 2) support repositories which are hosted on closed network (i.e. where companies does not allow direct internet connections)
>
> If we take the approach which is to both have adept artifact repositories and support for URLs, I think both of these use cases can be supported.
> I think we can make the adept artifact repos will give you the flexibility to support both use cases.
> URLs will give more bandwidth easily. For only URLs I guess supporting use case 2) is a bit harder.

We are talking about existing URLs, though. How do you get more bandwidth from URLs that you don't control? You have to mirror the content and then rewrite the URLs to point at servers you do control.

> > https://github.com/sbt/adept/wiki/NEScala-Proposal#artifact-repository
> >
> >>>>> I like the dependencies.adept file idea. It seems to me that it is close to an ideal balance of a project providing full information to build with the principle that repositories should be configured per-machine. It sounds like the main work on a user's part right now is if a dependencies.adept file has repositories you don't want to use? (I also think the name should be more like resolve.adept or something that indicates it is an output and not an input. This is a minor issue right now though.)
> >>>> Great that you like the idea! After rewriting the SBT plugin I did not include this, but since you think it sounds like a good idea I will put it back in!
> >>>
> >>> Oh, was there a reason? What sounded like a good idea to me was that it was an explicit, human readable output that you could check into your project as
> >> I had some difficulty in figuring out how to check which commit a module is in, and therefore I did not add it. I am planning on putting it back in though!
> >
> > Can you elaborate? Why do you need to lookup the commit? Either the user explicitly specifies it (so you know it trivially) or it is HEAD or a branch, and you ask git to resolve it.
> Perhaps this is not needed but I wanted to figure out the last commit you have to have to have the exact same repositories as the one that created the resolve.adept file, and not the last commit that person was using.
> User story:
> I constantly keep myself updated on the latest commits from all repos I am using because of my pet project. I use only certain modules which are not updated often in my main project though.
> For people working with me on the main project, should only be required to update their metadata to the latest commit that I am actually using in that project.
> If they constantly would have to pull (because I use the HEAD reference I was on), adept would seem slower than it really needs to be.

This sounds like an automatic advisory lock as an optimization and is different from the mandatory lock role, but its behavior isn't clear. What happens when the original user adds a dependency? when someone else adds a dependency? Those dependencies might not be available in the commit in the advisory file. Adept would now need to record the original configuration for the resolution so that it could detect if it changed, right?

> >>> a) documentation of a set of repositories that would successfully resolve the dependencies
> >>> b) the exact commits from those repositories
> >>> c) a way to lock down a project
> >>>
> >> Yep this is what I was thinking was nice with it! :)
> >>> So, perhaps by default:
> >>>
> >>> a) the .adept file is generated by a resolve with the configured repositories and used commits
> >> Yep, that is the way it worked
> >>> b) the user can lock it (adds lock: true to the file or whatever). This means that adept will use only the listed repositories at the specific commits.
> >> Yes, that is an interesting idea. You could also simply check it into you VCS, so that you would be able to easily spot changes.
> >>> c) on a failed resolution, if the .adept file contains repositories that the user does not have, adept generates a warning with the missing repositories.
> >> I was thinking that adept would pull and update itself to the last version specified in the resolve.adept file.
> >
> > What do you mean here? Are you now suggesting that adept reads from it even when unlocked?
>
> Hmm.. I think I am misunderstanding you.
> Perhaps you could elaborate on what the point of the lock would be? The way I understood it was the point of locking down a project was to carefully manage what dependencies are being used in the project. Thus, if you had a file which lists all of the modules, repositories etc etc under version control it would be easy to spot a change or similar during a code review. Sorry for being slow, I guess I am just struggling to combine your truth with the one I had :)

Yes, that was my understanding as well. However, as above, it sounds like there is now a second, separate type of locking that is handled automatically instead of a user manually locking it down.

-Mark