Running mix compiles programmatically

[Dave Thomas]

I’m writing a mix task that acts as a watcher, running other tasks when files change.

I have most of it working, but I don’t seem to be able to run the compile task—it seems to be silently ignored.

The function that does the running looks like this:

  defp run_tasks(tasks) do
    Logger.info "watch is running: #{Enum.join(tasks, ", ")}"
    for task <- tasks do
      Logger.info task
      change_env(task)
      Mix.Task.run(task, (if task == "compile", do: ["--force"], else: []))
      Mix.Task.reenable(task)
    end
  end

(I hacked the —force option in there to see if it would make a difference—it doesn’t)

I see output like this

[Play/watcher] mix watch                                                            
Compiled lib/watcher.ex
Compiled lib/watch/notifier.ex
Generated watcher.app

23:47:54.118 [info]  watch is running: compile, test

23:47:54.118 [info]  compile

23:47:54.119 [info]  test
.

Finished in 0.03 seconds (0.03s on load, 0.00s on tests)
1 tests, 0 failures

The compile task generates nothing, and doesn’t actually compile (I add syntax errors to files and save them).

So, is there some magic to get this to work?

Cheers

Dave

​

[José Valim]

That's because they are tasks all the way down. So compile just calls other tasks that actually perform the compilation but, since you have only reenabled the compile one, the other ones will simply be skipped.

We are having a similar discussion on another issue. What should be the best course of action: clear up how tasks (there is a private API that clears all tasks on the Mix.TasksServer) or manually clean up what you want?

Also, it is worth pointing out that simply recompiling in the same process will likely not work when you are changing a GenServer or the messages between process because the old ones will continue running. You likely want to stop the application, compile it and start it again.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dave Thomas]

We are having a similar discussion on another issue. What should be the best course of action: clear up how tasks (there is a private API that clears all tasks on the Mix.TasksServer) or manually clean up what you want?

In this case, I'm watching for changes, and then running mix tasks, so it is logically equivalent to running (say) Guard externally. So what I _really_ want to do it to tell mix to clear everything, effectively restarting. The problem is that my code is itself a mix task, and I want to keep running.

It seems ironic that he problems here are all to do with state :)

I'll try doing the global reenable and see what happens. 

Also, it is worth pointing out that simply recompiling in the same process will likely not work when you are changing a GenServer or the messages between process because the old ones will continue running. You likely want to stop the application, compile it and start it again.

Is there a clean way to do that inside mix? 

[Dave Thomas]

So, to add to the confusion…

I changed the runner to call Mix.Task.clear before running the test task.

The good news—it now compiles before it runs the tests.

The bad news—it crashes the second time I alter a file:

[Play/watcher] mix watch                                                   •[master]
Compiled lib/watch.ex
Compiled lib/watch/notifier.ex
Generated watch.app

#### at this point I touch a file.... ####

11:13:57.038 [info]  watch is running: test

11:13:57.038 [info]  test
lib/watch.ex:1: warning: redefining module Mix.Tasks.Watch
lib/watch/notifier.ex:1: warning: redefining module Mix.Tasks.Watch.Notifier
Compiled lib/watch.ex
Compiled lib/watch/notifier.ex
Generated watch.app
.

Finished in 0.03 seconds (0.03s on load, 0.00s on tests)
1 tests, 0 failures

Randomized with seed 189048

### and now I touch another file... ###

11:14:00.733 [info]  watch is running: test

11:14:00.733 [info]  test
lib/watch.ex:1: warning: redefining module Mix.Tasks.Watch
** (EXIT from #PID<0.47.0>) killed

I feel like I’m fighting the wrong battle here. Is there a better way to do this?

Dave

​

[Dave Thomas]

Ahhh.

This only happens when I watch the actual watcher—running it as a dependency in another app seems to work.

But, now it can’t find the test files the second time around:

[tmp/wobble] mix watch                                                              
watch is running: test
lib/watch.ex:1: warning: redefining module Mix.Tasks.Watch
lib/watch/notifier.ex:1: warning: redefining module Mix.Tasks.Watch.Notifier
==> watch
Compiled lib/watch.ex
Compiled lib/watch/notifier.ex
Generated watch.app
lib/wobble.ex:1: warning: redefining module Wobble
==> wobble
Compiled lib/wobble.ex
Generated wobble.app

  1) test the truth (WobbleTest)
     test/wobble_test.exs:4
     Assertion with == failed
     code: 1 + 1 == 99
     lhs:  2
     rhs:  99
     stacktrace:
       test/wobble_test.exs:5

Finished in 0.03 seconds (0.03s on load, 0.00s on tests)
1 tests, 1 failures

Randomized with seed 938036
watch is running: test

Finished in 0.00 seconds
0 tests, 0 failures

Randomized with seed 820046

​

[José Valim]

> The bad news—it crashes the second time I alter a file:

When bad things are happening on the second time is because you reloaded some code twice, causing the processes running the first version to crash. This is going to happen a lot in actual applications, when you change a GenServer code twice, the ones running initially are going to crash later.

So moving to a clean project allowed you to get rid of the issue, but that is just temporarily. If you start a GenServer in this new app, and change it twice, you are going to see the crashes again.

> But, now it can’t find the test files the second time around:

That's because the test files were already required. You can use Code.loaded_files and Code.unload_files to clean up the require cache but at this point you are really looking for trouble and already defined warnings. :)

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer

[Dave Thomas]

>
> That's because the test files were already required. You can use Code.loaded_files and Code.unload_files to clean up the require cache but at this point you are really looking for trouble and already defined warnings. :)
>

so is it basically impossible to run a watch task from within mix?

[José Valim]

I'd say that yes. Under some situations, it may be worth to have some sort of watch functionality, but you need to draw a line on what requires clean up and what doesn't. For example, Phoenix keeps the reloadable stuff in the web directory. In other cases, you need to be so careful with the cleaning up that the 0.5s for booting up / shelling out is worth the cost.


--

[Dave Thomas]

I was coming to that conclusion.

So... is this worrying. Should we be more functional than this. It seems to me that overall system state is not different to any other state—we should be able to isolate it and discard it when necessary. But right now we have a global context which we seem unable to manage.

Dave

[José Valim]

Sorry, maybe I was not clear. You can definitely do it but it takes work. You need to do the reverse of the initialization process which is to stop all applications (with Application.stop), remove all loaded modules (with :code.all_loaded + :code.purge + :code.delete) and unloaded all files (with Code.unload_files or by also stopping the Elixir+Mix application and starting it again). 

So it is possible but, at least in my opinion, it is not worth it because starting everything new is considerably cheap and easier. So for your use case of watch, I would go with shelling out, rather than do this whole dance. It will be *simpler*. However, if someone thinks this is a problem worth solving generally, please do!