[devel] Effectively "Standalonifying" a Rucola app.

[Eloy Duran]

Hi,

Well it's inevitable that in the not so distant future I will have to
deal with standalonifying an application again.
However this time around it's a Rucola application and so it's time to
deal with this important matter.

Now although I was very delighted that Jonathan Paisley took the
effort to write standalonify.rb for RubyCocoa
a while back, it does have it's problems and thing that can be done
better IMO.

I have some ideas about what should be done differently, but I thought
I'd throw it out here for (possible) discussion first.
First of all for those of you who do not know how standalonify works,
a simplified rundown:

- standalonify.rb first overrides OSX.NSApplicationMain(), so that any
call to that function will not really start your application.

- Then it loads your rb_main.rb file, which resides in the source root
in a regular RubyCocoa app, which in it's turn
will "require" all the libraries needed by your application. And
eventually halt the evaluation when it reaches the call to
OSX.NSApplicationMain()

- Then it goes through all the required libraries (loaded features)
and checks whether or not they're gems and where they are on the
filesystem.

- It will then copy gems into MyApp.app/Contents/Resources/Gems and
non-gem libraries into MyApp.app/Contents/Resources/ThirdParty
and alter the rb_main.rb file to reset the load paths at the top of
the file so that a regular require will first look in the ThirdParty
directory
and RubyGems will look in the Gems directory.

Now this works most of the times, but in my opinion it's not clean
enough.

First of all, all the source is evaluated, which *could* lead to weird
side effects.
Second, if I've bundled everything I actually would like it to not use
RubyGems at all.
Not that I don't think RubyGems is a great system, but it does add a
lot of overhead
which is simply not needed in a Release environment.

So configuration-wise my idea would look more like Merb does it.
Eg, in one file config/environment.rb (maybe change the filename to
dependencies or something along that lines?)
and not scattering "require" statements all over the place (other
files):

Rucola::Initializer.run do |config|
# regular libraries
config.dependency 'rexml'

# gems
config.dependency 'git'
config.dependency 'active_support', '> 2.0.1'
end

Then we would have a rake task like: build:bundle,
maybe even :all / :gems tasks depending on if you want to be somewhat
ruby version independent.
This task will only have to run the configuration for Initializer (the
file above)
and it will then know all of it's dependencies, so no more evaluating
all the source.

It should then still load all those libraries to be able to figure out
which libraries they in turn rely on,
but that should not give any problems since they are libraries and are
not supposed to actually run any code.

Then however I would like to copy all the needed libraries to eg
vendor/external (suggestions for a better name?).
And not make any distinctions anymore between gem and non-gem libraries.

Now I have no idea if this is even possible, but in my naive idea this
would be ideal.
Since then we don't have to start RubyGems anymore for loading libs
that are bundled,
thus improving startup time and no more messing with gem paths anymore.

So to recap:
In the Debug environment, the config.dependency method will simply
delegate to "require" and rubygems's "require".
In Release mode however config.dependency will first look in vendor/
externals and after that optionally use rubygems's require.

I hope that my email was clear enough to understand.
If not please do let me know any points that are unclear,
because this is IMO one of the most challenging parts of deploying
your app
and something every RubyCocoa developer will run into sooner or later
so I want to do this as well informed as possible.

Cheers,
Eloy

[benjackson]

Hey Eloy,

As one of the first devs to use (and be abused by) standaloneify.rb in
a release environment, I'd like to chime in here.

One of the most frustrating limitations of Jonathan's approach is that
it will only pick up dependencies which are loaded when it loads up
the app's code. In other words, if one of the libs I use requires
another lib in a conditional branch which is not picked up by the
shallow loading approach, I and my code are effectively fucked, and
I'm stuck explaining to my users that their app is hanging in a thread
because open-uri lazily loads tempfile.

I don't see any way to get around this other than an explicit file
scan for require statements, but it is a minefield for anyone wishing
to deploy RC apps, and IMHO needs to be addressed. Take care,

Ben

[eloy.d...@gmail.com]

Hey Benjamin,

Yes those are nasty problems.
But I wonder if we can completely eliminate that.

The thing is that, sure we can go through all the files
run them through rubynode (or maybe even simple regex)
and check for require statements. But maybe sometimes
libraries are lazily loaded because they're not always needed.
So you might end up with more libraries being bundled than needed.

This may not be a problem at all, because most of the times ruby
libraries
are very small on disk.

Maybe we should create a sort of test-your-bundled-release app rake
task,
which doesn't allow config.dependency to go out to other locations
such as rubygems.
Than you would be able to determine if everything needed for your app
has been bundled
because it should give a LoadError.

Than you could simply add it to the config dependecies.

Eloy

[benjackson]

Hey Eloy,

> Yes those are nasty problems.
> But I wonder if we can completely eliminate that.
>
> The thing is that, sure we can go through all the files
> run them through rubynode (or maybe even simple regex)
> and check for require statements. But maybe sometimes
> libraries are lazily loaded because they're not always needed.
> So you might end up with more libraries being bundled than needed.
>
> This may not be a problem at all, because most of the times ruby
> libraries
> are very small on disk.
>
> Maybe we should create a sort of test-your-bundled-release app rake
> task,
> which doesn't allow config.dependency to go out to other locations
> such as rubygems.
> Than you would be able to determine if everything needed for your app
> has been bundled
> because it should give a LoadError.
>

I'm not sure that this would solve the issue, as libs which are lazily
loaded will often depend on some kind of use action to trigger the
loading... in the case I cited, the bug only surfaced once the open-
uri call was made, which was triggered by UI actions. I don't see any
way to test for this other than a manual run-through of the program.

I agree that the extra file size is undesirable (even if negligable,
we never know for sure how it will affect each app) but I think that a
more conservative approach is necessary to avoid this as IMHO it's a
clear blocker. We might add an option to the config to opt-out of
libraries which are definitely not used but which are weighing down
the app.

- Ben

[eloy.d...@gmail.com]

Hi Benjamin,

I agree that you would have to test run your application to be able to
know if it's got everything it needs.
But first of all you would probably do that anyway before sending it
out to the world... right? ;)

And in my case I guess that's double checked. Because atm I write my
code in a test first way,
so running the tests in a sandbox environment where nothing else is
loaded but the libraries that rucola thinks you need, should reveal
those problems as well.

I understand that there will always be edge cases, so giving the
option to scan through all the files to see if they require other
files anywhere would be indeed be nice to have. But it would need a
real project that exhibits this behaviour to be able to write it, as I
don't want to write YAGNI code.

So I'll put it on the TODO as nice to have.
Of course anyone is free to implement this :)

Eloy

[benjackson]

> I agree that you would have to test run your application to be able to
> know if it's got everything it needs.
> But first of all you would probably do that anyway before sending it
> out to the world... right? ;)

Of course :P

> And in my case I guess that's double checked. Because atm I write my
> code in a test first way,
> so running the tests in a sandbox environment where nothing else is
> loaded but the libraries that rucola thinks you need, should reveal
> those problems as well.
>
> I understand that there will always be edge cases, so giving the
> option to scan through all the files to see if they require other
> files anywhere would be indeed be nice to have. But it would need a
> real project that exhibits this behaviour to be able to write it, as I
> don't want to write YAGNI code.

What does GNI stand for? :)

> So I'll put it on the TODO as nice to have.
> Of course anyone is free to implement this :)

Will think about whether it's really necessary.

[Eloy Duran]

>>
>> I understand that there will always be edge cases, so giving the
>> option to scan through all the files to see if they require other
>> files anywhere would be indeed be nice to have. But it would need a
>> real project that exhibits this behaviour to be able to write it,
>> as I
>> don't want to write YAGNI code.
>
> What does GNI stand for? :)

You Ain't Gonna Need It. :)

>
>
>> So I'll put it on the TODO as nice to have.
>> Of course anyone is free to implement this :)
>
> Will think about whether it's really necessary.
>

Cool.

Eloy