Nuget, TeamCity, and "ripple"

[Jeremy Miller]

Hey everybody,

I know I'm bouncing all around the place this past week (and it hasn't
been a good thing), but here's a new/old problem for us to consider.
*I* made the executive decision to start breaking projects out of the
FubuMVC repository because it was getting huge and the builds were
becoming slow, plus some of these libraries like HtmlTags, FubuCore,
and Bottles are starting to be used independently of FubuMVC. Great,
awesome, total PITA in terms of logistics when it comes time to
migrate code changes from one place to another. At the same time,
we're using Nuget to distribute FubuMVC. That's great, we get more
people trying it out, but we're looking bad the past couple weeks
because changes to our external dependencies behind our backs have
made the FubuMVC nuget non-functional.

Josh Flanagan has done some preliminary work on something we call
"ripple" to try to eliminate the friction we're incurring by splitting
up the repositories, but I'd like to take it further and remove some
more friction.

So, I see these problems / challenges I want to solve by a ripple
phase #2


1.) Cascade changes from upstream to downstream projects.
a.) I want a "local mode" where you can just push binaries around
on your own box fast, and have this cycle from FubuCore to Bottles to
FubuMVC to our own application minimal work on my part
b.) Automatic propagation of changes on the CI builds that would
create and publish new Nuget files on successful builds that would
trigger downstream builds.

2.) Per Dru's advice (and I think other platforms do this anyway), I
think that normal CI Nuget pushes will be published as [Nuget Name]-
EDGE, unless some yet to be determined manual process flips a switch
to "this is an official release"

3.) Our Git repositories are crazy bloated because of all the binaries
that are getting checked in. From my research yesterday, it should be
relatively easy to forgo checking the binaries into Git and using
Nuget to "restore" the proper libraries and dependencies before
building the code.

4.) #3 sounds awesome, but Software Engineering 101 tells us that
builds need to be completely repeatable and you need to be able to
trace that build #11045 of FubuMVC was built against build #231 of
FubuCore. I think we'll get much tighter with versions in the nuget
file and try to use the nuget package.config files and/or nuspec files
to be the master set of dependency information. I *think* that we
could do the CI build propagation by reaching into the downstream
package.config files and incrementing the dependency versions and
committing those files to trigger a normal CI build.

5.) An automated way to make "release" versions of all our Nuget files

6.) Nuget isn't very CLI tool friendly and I don't know about you, but
I don't want to have to use the PowerShell console inside of Visual
Studio to use fubu.exe and bottles.exe -- especially as these tools
become more important and robust. I think we'll do just a little post-
nuget install work to expose the fubu.exe and bottles.exe tools to the
command line with a forwarding file (fubu.cmd, bottles.cmd) in the
root of your repository.

-- I wanted to get rid of the git submodule stuff for buildsupport
because I think it's hokey and clumsy, but now we're using it for
sphinx support so it stays.


I'll have more updates as I go, but I'm happy to have some feedback.
I *think* that we might want to have our own dedicated TeamCity
installation / environment so that we can get more control over this
stuff. I think I'll do the TeamCity work in isolation on my box
before hand. I will NOT release any of this into the wild without
some big announcements as it will likely destabilize development for a
bit.





------------------------------------------
The Ongoing Nuget Problems
------------------------------------------

I don't know if any of you have seen the issues but the FubuMVC nuget
has gotten knocked around the past couple weeks like a slow
quarterback behind a terrible offensive line. A couple different
dependencies of FubuMVC changed underneath us (thanks Josh F.) and
broke the FubuMVC nuget installation on clean projects. Some of the
assembly redirect issues are fixed in newer versions of Nuget, but the
DotNetZip/Ionic.Zip.dll problems were a little bit worse and we're
just going to ilmerge those DLL's into the Bottles.dll to stop that
permanently.

To fix it permanently:

* we've got to make it faster to publish / upload new nuget releases.
* I think we need to get a nightly, scheduled build that creates a new
project, adds the FubuMVC nuget, and just verifies that the Getting
Started home page comes up correctly just to spot these problems in
the future
* See if we can do something in the getting started page that can trap
assembly version problems and display a friendly page with "How do I
fix this?" instructions. I.e., update Nuget on your box

[ahjohannessen]

I think this sounds really nice :) One thing that I am a bit skeptical about is 3) and Mono because I truly don't believe that NuGet team cares enough about Mono.

[ahjohannessen]

Would be crazy or even relevant to reconsider OpenWrap?

[Ian Battersby]

I'd love to +1 for OpenWrap, but I fear to encourage adoption we'd need to follow the NuGet mainstream :(

--
You received this message because you are subscribed to the Google Groups "FubuMVC Development Group" group.
To view this discussion on the web visit https://groups.google.com/d/msg/fubumvc-devel/-/8zptcps5BagJ.
To post to this group, send email to fubumv...@googlegroups.com.
To unsubscribe from this group, send email to fubumvc-deve...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/fubumvc-devel?hl=en.

[Jeremy D. Miller]

*OpenWrap*interoperates with Nuget, so there's a happy possibility of still publishing via Nuget for the masses but using OpenWrap for the ripple glue
 

Jeremy D. Miller
The Shade Tree Developer
jeremy...@yahoo.com

---

From: Ian Battersby <ian.ba...@gmail.com>
To: fubumv...@googlegroups.com
Sent: Sat, September 3, 2011 11:14:38 AM
Subject: Re: [fubumvc] Re: Nuget, TeamCity, and "ripple"

[Jeremy Miller]

Guys,

What do you think about this? http://blogs.jetbrains.com/teamcity/2011/07/20/nuget-plugin/

We might have to abuse Hadi for some extra work on it, but I think
that might cut down some of the work -- at least in windows world

[Sam Merrell]

On Sat, Sep 3, 2011 at 10:58, Jeremy Miller <jeremy...@yahoo.com> wrote:
> Guys,
>
> What do you think about this?  http://blogs.jetbrains.com/teamcity/2011/07/20/nuget-plugin/
>
> We might have to abuse Hadi for some extra work on it, but I think
> that might cut down some of the work -- at least in windows world

I was actually going to mention that JetBrains had a NuGet plugin :)

[Sam Merrell]

On Fri, Sep 2, 2011 at 08:32, Jeremy Miller <jeremy...@yahoo.com> wrote:
> -- I wanted to get rid of the git submodule stuff for buildsupport
> because I think it's hokey and clumsy, but now we're using it for
> sphinx support so it stays.

If the submodules are a problem, it shouldn't be all that hard to
remove the sphinx.rb from it. I had actually just moved it there since
it looked like most Fubu projects were using it so that meant I only
had to do one pull request to get that in there. :)

That and the sphinx.rb really is just a wrapper to the sphinx-build
executable so you can run it through rake. If you want me to help out
with any of that just let me know.

[Adam Dymitruk]

I'm available to help out with git submodule woes.

> --
> You received this message because you are subscribed to the Google Groups "FubuMVC Development Group" group.
> To post to this group, send email to fubumv...@googlegroups.com.
> To unsubscribe from this group, send email to fubumvc-deve...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/fubumvc-devel?hl=en.
>
>

--
Adam

http://adventuresinagile.blogspot.com/
http://twitter.com/adymitruk/
http://www.agilevancouver.net/
http://altnetvan.grou.ps/

[SerialSeb]

As a side-note, support for OpenWrap in TeamCity is being worked on,
with cross-package dependencies and build triggers, auto-build and
auto-publish from TeamCity. There's also an upcoming TeamCity
repository plugin for OpenWrap that lets any teamcity install turn
itself into an OpenWrap-friendly remote repository, but that's very
much alpha at this stage. Current plan is to target a combined release
for both.

[SerialSeb]

Another thought. One of the things we use a lot for cross-project
updates (i think it's your local mode here) is to simply rebuild local
copies of another project and update the local packages.

For example, openwrap depends on openfilesystem, so an update would
be:
c:\src\openwrap> o build-wrap -from c:\src\openfilesystem -quiet -
incremental; o update-wrap openfilesystem -project;

This updates the local repository with new binaries, and as the
references in ReSharper get updated dynamically whenever a new package
changes, changes are reflected immediately.

One of the things I want to add is to have a local override, so
instead of typing the command, you'd simply do a junction between /
wraps/ext/openfilesystem and your source code, and whenever you'd do
an update-wrap openfilesystem, it'd find out if something has changed
and rebuild the package, updating it in one go. Any suggestions as to
how to make that change detection efficient is more than welcome.

[Dru]

One thought: Love everything said. Gimmie more, and uh, I want it now. ;)

-d