Improving Arduino commandline interface
174 views
Skip to first unread message
Matthijs Kooijman
Oct 25, 2013, 12:11:19 PM10/25/13
to devel...@arduino.cc
Hi folks,
today I found out that Arduino supports some commandline options to do
compiling and uploading. I decided to explore this further, since the
Arduino-Makefile doesn't properly support 1.5 yet, and because it seems
that the amount of things to configure might be getting too big for
properly supporting from a Makefile without duplicating a lot of code
(e.g., platform.txt options).
In any case, due to some badly-named sketches (that worked fine with
Arduino-Makefile, but not with the IDE) and poor documentation reading,
I quickly ran into null pointer and index out of range exceptions.
Before fixing my code and commandline options, I decided to improve the
error handling instead.
Most of these commits should be fairly straightforward and non-invasive,
but this was my first time around the Arduino java sources, so I might
have missed some conventions or made wrong assumptions...
https://github.com/arduino/Arduino/pull/1639
Okay, regarding the commandline. Here's some more thoughts.
Right now, the GUI opens during compilation / upload, which isn't really
useful when trying to automate things. I could imagine a --no-gui
option, but it might be even more useful to just make the the default
(and only) option with --verify or --upload. Showing a GUI during the
compilation doesn't really seem useful and is potentially confusing
(you can for example open the preferences dialog, which is then forcibly
shut down after compilation is complete). I can see a few options:
1. Never show a GUI with --verify or --upload. Just do what is asked,
and report all output on stdout / stderr.
2. Show a minimal GUI with --verify or --upload, that shows the status
and/or output, but does not allow any other interaction (perhaps not
even shows the code).
3. Add a --no-gui (or --batch) option. If this option is passed, behave
like 1. If this option is no passed, show the GUI as normal and
start verification or upload. After this is complete, don't quit but
just keep the GUI open. This effectively means --verify just means
"pretend the verify button is clicked after startup".
Option 1 or possibly 3 seem sane to me, option 2 seems nice but is
probably too much work for little gain.
When running --verify or --upload, some preferences are changed. At
least the verbose preferences are always changed, the board name and
serial port only when the --board or --port options are passed. However,
it seems that normally, these options are only changed temporarily and
not save to preferences.txt. I suspect this is because with --verify and
--upload, a hard System.exit(0) is done, skipping all regular quit
handlers that call Preferences.save(). However, it seems there is a
corner case: if the user quits the GUI halfway through the compile, the
preferences _are_ saved.
This would automatically be solved if we go for option 1 above. For
option 3, the semantics would change to "Preferences are saved if and
only if --no-gui is not passed", which also seems fine to me.
Furthermore, passing --board, --port and --serial only actually does
something when --verify or --upload is actually passed (options are
silently ignored otherwise). I'd say it would make sense to always
process these options, so you can also create shortcuts to start the gui
for a specific board. Perhaps even a generic --set-preference would make
sense?
Error message handling should also be slightly adapted. My patches
already use showError wherever possible, which should be adapted to
print to stdout/stderr when no gui is shown (there is already some code
for this, but it seems unused now). It might even be useful to do this
even when there _is_ a GUI.
Additionally, it might be useful to write warnings, error messages and
stack traces to stderr instead of stdout (I think I noticed stacktraces
going to stdout. Since Throwable::printStacktrace should AFAIK print to
stderr, I suspect there is some redirection in place to print both
stderr and stdout into the GUI, which also squats everything together in
the real stdout? Not 100% sure, though). In general, getting logging a
bit more cleaned up might be useful, since there is now
Base::showMessage/Warning/Error, System.out/err.print, and
EditorConsole::message, all of which potentially end up in the same
places...
Gr.
Matthijs
today I found out that Arduino supports some commandline options to do
compiling and uploading. I decided to explore this further, since the
Arduino-Makefile doesn't properly support 1.5 yet, and because it seems
that the amount of things to configure might be getting too big for
properly supporting from a Makefile without duplicating a lot of code
(e.g., platform.txt options).
In any case, due to some badly-named sketches (that worked fine with
Arduino-Makefile, but not with the IDE) and poor documentation reading,
I quickly ran into null pointer and index out of range exceptions.
Before fixing my code and commandline options, I decided to improve the
error handling instead.
Most of these commits should be fairly straightforward and non-invasive,
but this was my first time around the Arduino java sources, so I might
have missed some conventions or made wrong assumptions...
https://github.com/arduino/Arduino/pull/1639
Okay, regarding the commandline. Here's some more thoughts.
Right now, the GUI opens during compilation / upload, which isn't really
useful when trying to automate things. I could imagine a --no-gui
option, but it might be even more useful to just make the the default
(and only) option with --verify or --upload. Showing a GUI during the
compilation doesn't really seem useful and is potentially confusing
(you can for example open the preferences dialog, which is then forcibly
shut down after compilation is complete). I can see a few options:
1. Never show a GUI with --verify or --upload. Just do what is asked,
and report all output on stdout / stderr.
2. Show a minimal GUI with --verify or --upload, that shows the status
and/or output, but does not allow any other interaction (perhaps not
even shows the code).
3. Add a --no-gui (or --batch) option. If this option is passed, behave
like 1. If this option is no passed, show the GUI as normal and
start verification or upload. After this is complete, don't quit but
just keep the GUI open. This effectively means --verify just means
"pretend the verify button is clicked after startup".
Option 1 or possibly 3 seem sane to me, option 2 seems nice but is
probably too much work for little gain.
When running --verify or --upload, some preferences are changed. At
least the verbose preferences are always changed, the board name and
serial port only when the --board or --port options are passed. However,
it seems that normally, these options are only changed temporarily and
not save to preferences.txt. I suspect this is because with --verify and
--upload, a hard System.exit(0) is done, skipping all regular quit
handlers that call Preferences.save(). However, it seems there is a
corner case: if the user quits the GUI halfway through the compile, the
preferences _are_ saved.
This would automatically be solved if we go for option 1 above. For
option 3, the semantics would change to "Preferences are saved if and
only if --no-gui is not passed", which also seems fine to me.
Furthermore, passing --board, --port and --serial only actually does
something when --verify or --upload is actually passed (options are
silently ignored otherwise). I'd say it would make sense to always
process these options, so you can also create shortcuts to start the gui
for a specific board. Perhaps even a generic --set-preference would make
sense?
Error message handling should also be slightly adapted. My patches
already use showError wherever possible, which should be adapted to
print to stdout/stderr when no gui is shown (there is already some code
for this, but it seems unused now). It might even be useful to do this
even when there _is_ a GUI.
Additionally, it might be useful to write warnings, error messages and
stack traces to stderr instead of stdout (I think I noticed stacktraces
going to stdout. Since Throwable::printStacktrace should AFAIK print to
stderr, I suspect there is some redirection in place to print both
stderr and stdout into the GUI, which also squats everything together in
the real stdout? Not 100% sure, though). In general, getting logging a
bit more cleaned up might be useful, since there is now
Base::showMessage/Warning/Error, System.out/err.print, and
EditorConsole::message, all of which potentially end up in the same
places...
Gr.
Matthijs
Matthijs Kooijman
Oct 25, 2013, 12:29:36 PM10/25/13
to devel...@arduino.cc
Hey folks,
one more thing I just remembered: Arduino seems to do its compilation in
a directory in /tmp and creates a new one on every invocation. However,
when compiling from the commandline, it would be useful to keep previous
compilation results, so it doesn't have to recompile all of the core and
libraries every time. Perhaps something like a --workdir option?
Gr.
Matthijs
one more thing I just remembered: Arduino seems to do its compilation in
a directory in /tmp and creates a new one on every invocation. However,
when compiling from the commandline, it would be useful to keep previous
compilation results, so it doesn't have to recompile all of the core and
libraries every time. Perhaps something like a --workdir option?
Gr.
Matthijs
Nils Springob
Oct 25, 2013, 1:17:01 PM10/25/13
to devel...@arduino.cc
Hi Matthijs,
I would really appreciate this feature! At the moment I am working with a self-made Makefile.
The --workdir option would also be very helpfull!
Regards,
Nils
--
Nils Springob
Am Denkmal 8, 52223 Stolberg
Tel: +49-2402-997-8391
Mobil: +49-163-6556110
I would really appreciate this feature! At the moment I am working with a self-made Makefile.
The --workdir option would also be very helpfull!
Regards,
Nils
Nils Springob
Am Denkmal 8, 52223 Stolberg
Tel: +49-2402-997-8391
Mobil: +49-163-6556110
Matthijs Kooijman
Oct 29, 2013, 2:30:01 PM10/29/13
to devel...@arduino.cc
Hey folks,
since my mail last week, I've been wanting to actually use the CLI, so I
went ahead and codes some things. I've (force) updated the existing
pullrequest:
https://github.com/arduino/Arduino/pull/1639
> one more thing I just remembered: Arduino seems to do its compilation in
> a directory in /tmp and creates a new one on every invocation. However,
> when compiling from the commandline, it would be useful to keep previous
> compilation results, so it doesn't have to recompile all of the core and
> libraries every time. Perhaps something like a --workdir option?
I've found that there was already a "build.path" preferences to
determine the workdir, so I added a --pref option so you can set it for
a single --verify or --upload run.
There were some other issues with the IDE clearing out the workdir on
every start in case the build options changed, so I have implemented
a more explicit check for changed build options instead.
> > Right now, the GUI opens during compilation / upload, which isn't really
> > useful when trying to automate things. I could imagine a --no-gui
> > option, but it might be even more useful to just make the the default
> > (and only) option with --verify or --upload. Showing a GUI during the
> > compilation doesn't really seem useful and is potentially confusing
> > (you can for example open the preferences dialog, which is then forcibly
> > shut down after compilation is complete). I can see a few options:
> >
> > 1. Never show a GUI with --verify or --upload. Just do what is asked,
> > and report all output on stdout / stderr.
> > 2. Show a minimal GUI with --verify or --upload, that shows the status
> > and/or output, but does not allow any other interaction (perhaps not
> > even shows the code).
> > 3. Add a --no-gui (or --batch) option. If this option is passed, behave
> > like 1. If this option is no passed, show the GUI as normal and
> > start verification or upload. After this is complete, don't quit but
> > just keep the GUI open. This effectively means --verify just means
> > "pretend the verify button is clicked after startup".
I added a (not so perfect) patch for option 1 now. Now, it just prevents
the GUI from being displayed, everything is still created in memory.
Ideally, this wouldn't happen either, but that requires some more
refactoring.
> > When running --verify or --upload, some preferences are changed. At
> > least the verbose preferences are always changed, the board name and
> > serial port only when the --board or --port options are passed. However,
> > it seems that normally, these options are only changed temporarily and
> > not save to preferences.txt. I suspect this is because with --verify and
> > --upload, a hard System.exit(0) is done, skipping all regular quit
> > handlers that call Preferences.save(). However, it seems there is a
> > corner case: if the user quits the GUI halfway through the compile, the
> > preferences _are_ saved.
> >
> > This would automatically be solved if we go for option 1 above. For
> > option 3, the semantics would change to "Preferences are saved if and
> > only if --no-gui is not passed", which also seems fine to me.
I implemented option 1, but because it still creates an Editor and
Editor::handleOpenInternal() saves the preferences, this is still not
fixed properly (due to the ordering, anything set with --pref is saved,
but other options are not).
Gr.
Matthijs
since my mail last week, I've been wanting to actually use the CLI, so I
went ahead and codes some things. I've (force) updated the existing
pullrequest:
https://github.com/arduino/Arduino/pull/1639
> one more thing I just remembered: Arduino seems to do its compilation in
> a directory in /tmp and creates a new one on every invocation. However,
> when compiling from the commandline, it would be useful to keep previous
> compilation results, so it doesn't have to recompile all of the core and
> libraries every time. Perhaps something like a --workdir option?
determine the workdir, so I added a --pref option so you can set it for
a single --verify or --upload run.
There were some other issues with the IDE clearing out the workdir on
every start in case the build options changed, so I have implemented
a more explicit check for changed build options instead.
> > Right now, the GUI opens during compilation / upload, which isn't really
> > useful when trying to automate things. I could imagine a --no-gui
> > option, but it might be even more useful to just make the the default
> > (and only) option with --verify or --upload. Showing a GUI during the
> > compilation doesn't really seem useful and is potentially confusing
> > (you can for example open the preferences dialog, which is then forcibly
> > shut down after compilation is complete). I can see a few options:
> >
> > 1. Never show a GUI with --verify or --upload. Just do what is asked,
> > and report all output on stdout / stderr.
> > 2. Show a minimal GUI with --verify or --upload, that shows the status
> > and/or output, but does not allow any other interaction (perhaps not
> > even shows the code).
> > 3. Add a --no-gui (or --batch) option. If this option is passed, behave
> > like 1. If this option is no passed, show the GUI as normal and
> > start verification or upload. After this is complete, don't quit but
> > just keep the GUI open. This effectively means --verify just means
> > "pretend the verify button is clicked after startup".
the GUI from being displayed, everything is still created in memory.
Ideally, this wouldn't happen either, but that requires some more
refactoring.
> > When running --verify or --upload, some preferences are changed. At
> > least the verbose preferences are always changed, the board name and
> > serial port only when the --board or --port options are passed. However,
> > it seems that normally, these options are only changed temporarily and
> > not save to preferences.txt. I suspect this is because with --verify and
> > --upload, a hard System.exit(0) is done, skipping all regular quit
> > handlers that call Preferences.save(). However, it seems there is a
> > corner case: if the user quits the GUI halfway through the compile, the
> > preferences _are_ saved.
> >
> > This would automatically be solved if we go for option 1 above. For
> > option 3, the semantics would change to "Preferences are saved if and
> > only if --no-gui is not passed", which also seems fine to me.
Editor::handleOpenInternal() saves the preferences, this is still not
fixed properly (due to the ordering, anything set with --pref is saved,
but other options are not).
Gr.
Matthijs
Matthijs Kooijman
Nov 29, 2013, 6:45:45 AM11/29/13
to devel...@arduino.cc
Hi folks,
I've updated my pullrequest once more with some minor changes:
https://github.com/arduino/Arduino/pull/1639
AFAICS it is ready for merging now. Things still aren't perfect, but at
least the code in the pullrequest is a good start for further
improvements.
Any other thoughts on this subject?
Gr.
Matthijs
I've updated my pullrequest once more with some minor changes:
https://github.com/arduino/Arduino/pull/1639
AFAICS it is ready for merging now. Things still aren't perfect, but at
least the code in the pullrequest is a good start for further
improvements.
Any other thoughts on this subject?
Gr.
Matthijs
Cristian Maglie
Dec 3, 2013, 11:31:53 AM12/3/13
to devel...@arduino.cc
I like your "incremental" approach (more commits with small changes instead of
one big commit) it helps a lot to understand your changes.
The only one that not fully convince me is this one:
Change the logic deciding when to do a full rebuild
https://github.com/matthijskooijman/Arduino/commit/4592acc213d01bc7ff118bcd49ff137e72de0566
anyway I can't think on a better solution, and from my first tests it seems to
be working properly, so it's ok for now. If anyone has any concern/suggestion
on that, I'm happy to see it.
C
Matthijs Kooijman
Dec 3, 2013, 11:36:24 AM12/3/13
to Cristian Maglie, devel...@arduino.cc
Hey Christian,
> I've looked at the patch, it seems well taken, I'm gonna merge it.
Cool!
> I like your "incremental" approach (more commits with small changes instead of
> one big commit) it helps a lot to understand your changes.
Thanks!
> The only one that not fully convince me is this one:
>
> Change the logic deciding when to do a full rebuild
> https://github.com/matthijskooijman/Arduino/commit/4592acc213d01bc7ff118bcd49ff137e72de0566
>
> anyway I can't think on a better solution, and from my first tests it seems to
> be working properly, so it's ok for now. If anyone has any concern/suggestion
> on that, I'm happy to see it.
Yeah, I agree that the solution is a bit dodgy. However, there is no
sensible separation between preferences that influence the builds and
preferences that don't, so I guess this is the best we can do. Perhaps
the preferences system could be somewhat restructured, but that'd get
use into a whole new can of (compatibility) worms...
Gr.
Matthijs
> I've looked at the patch, it seems well taken, I'm gonna merge it.
> I like your "incremental" approach (more commits with small changes instead of
> one big commit) it helps a lot to understand your changes.
> The only one that not fully convince me is this one:
>
> Change the logic deciding when to do a full rebuild
> https://github.com/matthijskooijman/Arduino/commit/4592acc213d01bc7ff118bcd49ff137e72de0566
>
> anyway I can't think on a better solution, and from my first tests it seems to
> be working properly, so it's ok for now. If anyone has any concern/suggestion
> on that, I'm happy to see it.
sensible separation between preferences that influence the builds and
preferences that don't, so I guess this is the best we can do. Perhaps
the preferences system could be somewhat restructured, but that'd get
use into a whole new can of (compatibility) worms...
Gr.
Matthijs
Reply all
Reply to author
Forward
0 new messages