Scaling Fake with a rich helper library

[Nicolas Rolland]

Hi,

 

As I will be using Fake for handling task automation, and Tomas's Formatting library as well, I wanted to add a helper to integrate both.

That lead me to look at how dependencies are managed on the task level.

 

If we look at the MSIHelper we have  :

 

let Install setParams setup =  

    if not (execProcess3 (fun info -> 

        info.FileName <- parameters.ToolPath  // "msiexec " by default

        info.WorkingDirectory <- parameters.WorkingDir

        info.Arguments <- sprintf "/qb /l* %s /i %s" parameters.LogFile setup) parameters.TimeOut) && parameters.ThrowIfSetupFails

 

That is, we just assume "msiexec" command to be there, or wherever the caller decided it should be.

It is entirely up to the outside environment's responsibility to resolve the dependencies beforehand and to supply them to the Fake layer independently.

 

Now, for the specific part of delivering a versioned binary, nuget is here to help.

But the beauty of nuget is that though its packages.config file allows others (fsproj) to make assumptions on what is here or not.

And those project reference exactly this reference.

 

In Fake, I can write a helper, which will conform to Formatting1.x version, declaring and fulfilling my dependency by other means.

Then someone else tries to use this helper with another version of the nugget, and it will fail.

It could be because of the signature, or because the gem moved its binary and I obviously hard coded both.

Fake Helper does not use versioning for its dependencies, and relegate the problem outside.

 

 

What would be the solution ?

 

- Pushing the binaries inside Fake, that is remove any reference to the outside world. This is good at a small scale, like "./tools/ILMerge/ilmerge.exe" but unmanageable more widely 

 

- Having a mapping file that would specify among different Helper which target different versions of the same external program, the one to use

A type provider would then populate a unversioned namespace with the functions and type provided. This would allow to let the type information flow to build.fsx while keeping an extensibility point which only does namespace redirection between.  (Fake.DocuHelper and module Fake.DocuHelper1.2.3 for instance)

 

- For nugets that would want to be fake-compatible, having a generic FakeHelperAttribute(defaultarg) on top of the functions that it wants to expose.

Then a type provider would inspect the chosen nugets for the specific project and make them accessible

 

let nugetHelpers = FakeNugetHelper<"package.config", ".packages">

 

and expose them for consumption in the script  build.fsx

 

 

If we look at how this is solved in homebrew, we see that they integrated both aspects of nugets (dependency management) and install steps, which makes it manageable to have no type check on the argument I guess. And it works amazingly well.

 

What is your take on dependency management ?

 

Any thought on Fake VS Brew in your opinion ?

It is very easy to contribute new formulas to their system, and it would be massively good to have this scaling capacity

 

 

 

Regards,

NR

[Nicolas Rolland]

ps : their library is so big it slows down browsing to look at it.

https://github.com/mxcl/homebrew/tree/master/Library/Formula

 

 

pps : how do you know which browser I use ?

[Nicolas Rolland]

In the same area, vim uses git-submodules which are source equivalent to nuget mechanism.

being directly source code, there is need for type providers.

 

then each helper makes its own assumption in its own space about where bin are / how to call it.

by convention a common conventional nuget repository could be given in  solutiondir/packages

 

 

On Sunday, January 27, 2013 3:42:21 PM UTC, Nicolas Rolland wrote:

[Steffen Forkmann]

Hi,

 

the idea that we split the FakeLib.dll into smaller packages and include them in the builds scripts (as fsx files – not compiled) is actually very old.

Didn’t find the time to do this.

 

With nuget and a type provider this might be very interesting. Would love to see a prototype.

 

Cheers,

Steffen

--
 
 

[Nicolas Rolland]

How would that work ?

 

- We'd have a bunch of helperv1.0.2.fsx that checks wether v1.0.2 is installed and conditionally opens the module

 

- We'd have some file fake.config.fsx that contains the files to be loaded for this project

#load "xxhelperv1.0.1.fsx"

#load "helperv1.0.3.fsx"

#load "helperv1.0.4.fsx"

 

-We'd load that fake.config.fsx

 

 

That would be very cool to find the proper mechanism to extend this.

[Nicolas Rolland]

I wonder what the type provider would bring on the table over the script files.

 

Basically, what a TP does is based on a dynamic input, it connects to a foreign system and provide native types.

 

If we don't need anything outside of fsx files, then the #load instruction will be our type provider :

you give him a script file, and it provide new native types.

 

Now, what is it that we want need on top of script file themselves ?

Well, we need the actual binaries of the correct version that our helper depends on.

But this problem could be solved by convention, where we say that for each helperxxx.v1.fsx we have a file helperfsxxv1.config which contains all the nugets that the file's code depends on.

 

Each build.fsx file would then pick from the set of helperxxx.vn.fsx it needs

#load "helperxxxvn.fsx"

#load "helperyyyvm.fsx"

 

 

Then we would supply a 'faker' script (equivalent to : fake.exe fakesetup.fsx) which would enable to

faker build.fsx -> recursively install all nugets the build.fsx depends on

 

 

Where could we get more advice on this ?

 

 

 

On Sunday, January 27, 2013 4:00:51 PM UTC, forki23 wrote:

[Matt Jackson]

Basically it sounds like recreating a package manager, apt, yast etc... not an easy thing. And I have to ask - Is binary versioning really a problem? How many of our 3rd party libs are fast moving and/or have API's that change regularly? Most of them are tied to MS products, FxCop, msiexec, msbuild, ilmerge and these only change every 2 years. Only specflow, nunit, xunit, wix follow a more fluid process but these are mature projects and mostly stable these days. I would suggest simplifying the problem and just having some sort of enforcement mechanism for each task that fails (or warns) when using a binary outside the recommended version. 

[Nicolas Rolland]

I don't think we should recreate a package system, but instead rely on the existing (nuget in that case)

The problem here is easing new task creation rather than error reporting.

 

 

The advantages of having the fsx based system are quite clear :

- Making explicit the assumptions that your code makes implicitly is a good thing in general

- Nothing prevents you from writing hard coded implicit stuff, where that can be indeed a reasonable case (like people have either xbuild or msbuild as in the msbuilder task)

 

- Removing build breaks of the library => easier/less fear to contribute

- For this reason, the maintainer can integrate pull request in Fake.Helpers.Experimental without checking anything and promote it to Fake.Helpers when documentation is ok for instance.

 

I wonder if there are better alternatives though.

 

 

=> Whichever way, we would need some guidelines for potential contributors for this precise point of binaries dependencies. I spent some time to try to integrate Formatting but I got a bit lost among the different options, wondering whether I should sort out stuff in Fake first to handle nugets and what are the alternatives. 

Even if the optimal solution is to not do anything, it would make sense to have some contributor guide for this, as this is a real issue. (I'll submit some doctrine file on this to explain the state + where we could go, so that contributor know where we are)

 

 

Are there any other build system with easy strong typing for launching tasks ?

If not, why not build upon this and make it a flagship product, which involves having a very rich (hence bullet proof) library.

 

 

--

 

I'll try to make an example of the fsx version sometimes to see how it goes... when I have time.... as everyone !