LUA Scripting support (revisited)

["Sztupák Sz. Zsolt"]

Hi!

Managed to compile a working ML with seemingly stable (my simple scripts
don't crash them, and the ML stability tests run fine) and working LUA
scripting enabled. Scripts should be put inside the scripts directory,
and can be run from the "Tweaks" menu (It has a submenu, where you can
change the script that needs to be run). Only one script can run at a
time, it runs in the background, and can be killed from the menu.

Some specifications:
* LUA: version 5.2.0 (from scripting view should be the same as 5.1.x)
* Number format: signed 32 bit integer (that means no decimal values)
* Built in libraries: all, except: math, io, os
* Stability: No problems found so far. Standard ML stability tests run.
**Only tested on 60D!**

The current API is in no way complete (even less complete than alex's
previous API), and will probably change a lot, but it's already useful
for some things.
Current commands can be found at the Wiki:
http://magiclantern.wikia.com/wiki/Lua#Current_API:

Three sample scripts are (the only useful from them is probably
wb_shoot.lua):
* dumpcfg.lua: writes the current config values to the console
* print.lua: simple bmpprint testing script
* wb_shoot.lua: takes photographs with WB changing from 3000 to 6000 in
200 increments

Downloads and source: https://bitbucket.org/sztupy/magic-lantern/downloads
The download also includes the USB PTP code too.

SztupY

[eggnot]

no luck to run on 550d, camera won't start. just led blink once, over.

вторник, 6 марта 2012 г. 14:01:04 UTC+7 пользователь SztupY написал:

вторник, 6 марта 2012 г. 14:01:04 UTC+7 пользователь SztupY написал:

[Alex]

This is awesome!

Works great on 5D2. The 550D does not have enough free RAM for this.

--
http://magiclantern.wikia.com/
 
To post to this group, send email to ml-d...@googlegroups.com
To unsubscribe from this group, send email to ml-devel+u...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/ml-devel?hl=en

[Danne]

Hi!
Exactly what does the script do? Is it related to the usb controlled
functions through android devices??
I,m a little lost, don,t wanna miss out on a goood thing :).
//D

["Sztupák Sz. Zsolt"]

If the camera doesn't even start doesn't mean that the autoexec.bin is too large? Lua doesn't allocate much static variables.
(Should I give try implementing some kind of dynamically loading of modules?)

 SztupY

[Malcolm Debono]

Great work! Still need to try it out.

@Danne:

LUA scripting opens up a whole new world of possibilities! It allows you to create your own scripts (or those written by others) and run them on the camera. The CHDK forum has some great examples of what scripting can do.

["Sztupák Sz. Zsolt"]

Scripting means you can control some functions of the camera via script
files, that can be written by anyone (no ML recompiling needed)
USB control means you can control some functions of the camera through
the USB port.

So both are used to externally control the camera (not surprisingly I am
interested in both of them :) ), but the way they do it is different.

SztupY

> Hi!
> Exactly what does the script do? Is it related to the usb controlled
> functions through android devices??
> I,m a little lost, don,t wanna miss out on a goood thing :).
> //D
>

[eggnot]

if you make build to track the problem(with logging or so) i can test it on 550d. pin it here or mail me, will glad to help.

вторник, 6 марта 2012 г. 18:32:05 UTC+7 пользователь SztupY написал:

[Danne]

aaaa, Thanks Sztupy and Malcolm for clarification! Now I,m on the road
again! I use scripts all the time on my canon s90, was wondering if
this could work with magic lantern. Perfect!
//D

[Alex]

> If the camera doesn't even start doesn't mean that the autoexec.bin is too large? Lua doesn't allocate much static variables.
> (Should I give try implementing some kind of dynamically loading of modules?)

Yes. Loading Lua dynamically as a module would fix the problem.

If my understanding is correct, there are two memory pools: one used by AllocateMemory, and another one used by malloc. ML code is injected in the heap used by malloc (by changing the constant that indicates the end of BSS, which is also the start of the heap, see http://groups.google.com/group/ml-devel/browse_thread/thread/26cb46acd262b953 ).

The 550D has little available memory in the malloc pool, and that's why it can't run large binaries. But it has enough free memory in the AllocateMemory pool.

60D is the opposite: it has little free memory in AllocateMemory pool (and that's why we have trouble with uncompressed cromparks) but it can run large binaries without problems.

I have some partial success with dynamic module loading (basically compiling with fPIC/fPIE), and I could load a 2 MB binary on 550D. I had some trouble in calling firmware functions (as those calls need to be absolute) vs calling internal functions (those calls need to be relative). If you think my source code for this experiment can be helpful, I'll post it.

["Sztupák Sz. Zsolt"]

>
> Yes. Loading Lua dynamically as a module would fix the problem.
>
> If my understanding is correct, there are two memory pools: one used
> by AllocateMemory, and another one used by malloc. ML code is injected
> in the heap used by malloc (by changing the constant that indicates
> the end of BSS, which is also the start of the heap, see
> http://groups.google.com/group/ml-devel/browse_thread/thread/26cb46acd262b953
> ).
>
> The 550D has little available memory in the malloc pool, and that's
> why it can't run large binaries. But it has enough free memory in the
> AllocateMemory pool.
>
> 60D is the opposite: it has little free memory in AllocateMemory pool
> (and that's why we have trouble with uncompressed cromparks) but it
> can run large binaries without problems.

Well... this is great :) Is there actually a known reason for two memory
pools? ML seem to prefer AllocateMemory.

>
> I have some partial success with dynamic module loading (basically
> compiling with fPIC/fPIE), and I could load a 2 MB binary on 550D. I
> had some trouble in calling firmware functions (as those calls need to
> be absolute) vs calling internal functions (those calls need to be
> relative). If you think my source code for this experiment can be
> helpful, I'll post it.
>
>

Please do. I did made some small progress today, but any bootstrapping
code would be helpful.

SztupY

[Francis Danforth]

This is fantastic. I think this can be extremely powerful for users/developers who are interested in adding their own features without having to keep up with the changes made in the daily builds. Now I just need to learn how to write LUA scripts. 

For the user who recently wrote about celestial math calculations in the search for a flicker free day-to-night/night-to-day timelapse this could be there answer (since they seemed confident/competent in their preset calculations in exposure change over time).

[MagicBrick]

This might be a stupid question - but on the other hand, it might not, so here it goes: Is this safe? Since there is no eos vm or software simulator out there, can I fry my camera with a lua script or are there some boundaries and safeguards (planned)?

["Sztupák Sz. Zsolt"]

Actually by using ML you have the possibility to get the camera fried, right? It all depends on the code that you want to run. The difference is that creating lua scripts is much easier than recompiling the whole ML binary. We aren't using EOS VM's or software simulators to build ML either.

 SztupY

This might be a stupid question - but on the other hand, it might not, so here it goes: Is this safe? Since there is no eos vm or software simulator out there, can I fry my camera with a lua script or are there some boundaries and safeguards (planned)?

Am Dienstag, 6. März 2012 19:52:22 UTC+1 schrieb Francis Danforth:

This is fantastic. I think this can be extremely powerful for users/developers who are interested in adding their own features without having to keep up with the changes made in the daily builds. Now I just need to learn how to write LUA scripts. 

--

[eggnot]

1. it's clear from your description how ml functions are exposed to plugins.
but what about lua? should you add them to api first, so one can call them?

2. setprop(), is it just function to change value at address? are this addresses (like PROP_WB_KELVIN_LV you using) can be found in source code somewhere?

3. i am interested to write scripts to test all the qscale -things. can i access mvr_struct (from mvr.h) variables via script or should i dig for plugin to do this? also maybe need some functions or logic from bitrate.c
still not sure is it worth investigation, as nobody from devs confirm\disconfirm about gop length possibilities: https://groups.google.com/forum/?fromgroups#!topic/ml-devel/4YsgXvZZNyU what do you think?

thanks again for great work!

[Alex]

> Is there actually a known reason for two memory pools?

No (that's how Canon firmware is designed). Actually there are more memory pools (zlib_malloc, svg_malloc, DevelopPath_malloc, FC_malloc, PD_malloc) but they seem pretty small. So a possible reason is to reduce the coupling between software modules (a memory leak in zlib should not affect the svg module, for example).

To find them, look in AllocateMemory: the first function called is actually a common "malloc" used for all these pools; the rest of the code is for diagnostic (to get a meaningful error message). Find the references to the common "malloc" function.

> Is this safe?

Taking pictures should be safe. Same for doing some math or sleeping.

Changing properties (setprop) is not safe. The values you set will be stored into NVRAM by Canon firmware - that is, they are persistent even if you remove ML card (and start with a fresh card). Setting invalid values may result in ERR70 at every startup (camera will no longer start normally, but ML will still load, so recovery should be possible). The recovery process might take a few hours: sometimes it's not enough to just set the correct value back (the invalid value might propagate to other properties - this happened on my 5D). See http://magiclantern.wikia.com/wiki/Unbricking (but don't rely on it).

Writing to RAM is, generally speaking, not safe. It is probably safe as long as you know where to write (for example, in a structure which you know what it's used for - for example, mvr_struct).

Calling eventprocs (eoscall?) is not safe. There are functions that can write to ROM. Some of them are safe, so probably it's a good idea to document the ones which we have actually tried.

My proposal is to enable unsafe functions from the config file, without exposing those options in the menu. CHDK goes even further: they require you to compile a custom version in order to use unsafe calls (see http://chdk.wikia.com/wiki/Lua/Lua_Reference/Native_Function_Calls ).

> 2. setprop(), is it just function to change value at address? are this addresses (like PROP_WB_KELVIN_LV you using) can be found in source code somewhere?

See here:
https://bitbucket.org/hudson/magic-lantern/src/tip/src/property.h
http://magiclantern.wikia.com/wiki/Properties

and also see the warnings above.

I'd go for exposing functions which change well-known settings (ISO, shutter, white balance, drive mode etc) and also do some validation of the arguments. The generic setprop should not be enabled by default IMO.

--

["Sztupák Sz. Zsolt"]

> My proposal is to enable unsafe functions from the config file,
> without exposing those options in the menu. CHDK goes even further:
> they require you to compile a custom version in order to use unsafe
> calls (see
> http://chdk.wikia.com/wiki/Lua/Lua_Reference/Native_Function_Calls ).

Since lua is now a completely separated plugin
(https://bitbucket.org/sztupy/magic-lantern/changeset/22d551a0725c),
there is a third option: create two plugins, one with support for unsafe
calls and one without it. Then everyone can decide which one he wants to
use. (and user built scripts can assert which version they need)

It can be combined with the CHDK model: there are no precompiled
variants for the unsafe lua. But I don't think this much safety is
actually needed. (there are lesser variants, like simply not including
the .bin in official builds)

SztupY

[Giovanni Di Cello]

Hey Sztupy, have you experienced problems when formatting the card while keeping ML intact in camera?
I used to have no problems with A1ex builds but i ran into error twice with yours.
The format process stops with a full progress bar and then the CF looks like corrupted, can't create folders, can't format again.
If i put my CF card into the card reader i can still see ML files but i don't know what's wrong.
Can you please check it out? :)

["Sztupák Sz. Zsolt"]

2012.03.10. 17:43 keltez�ssel, Giovanni Di Cello �rta:

> --
> http://magiclantern.wikia.com/
>
> To post to this group, send email to ml-d...@googlegroups.com
> To unsubscribe from this group, send email to
> ml-devel+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/ml-devel?hl=en

Haven't tried this feature but will check. Btw. which camera?

SztupY

[Giovanni Di Cello]

50D.
I just did it once again and after the progress bar is full i get this message:
Cannot format.
Change card
If i try to access the files on my laptop i get an error as well.

[Lauris Heijdemann]

Hi  "Sztupák Sz. Zsolt",

I have just upgraded to the 2012Mar23.550d106.alex but ML freezes when I select lua.bin or testplug.bin
 Due to of 500d having to less mem?

But it worked using your initial release.
Is there any way you could fix this?

Greetings Lauris Heijdemann
(bas lichtjaar)

[Lauris Heijdemann]

?a lite plugin for cams with less mem?

["Sztupák Sz. Zsolt"]

Hi!

Haven't checked Alex's latest updates so I can't comment on this right now.

 SztupY

[William Miao]

Sweet Script SztupY

I have tried the magiclantern-2012Mar8.usb.lua.plugin.sztupy.zip.zip built on my Canon EOS 550D (Rebel T2i), and it worked.

I ran the WB_SHOOT.LUA script, and it started taking pictures. However, after using the script the camera doesn't focus anymore. Despite restarting it or turning it back and forward between manual and automatic. At the end, I was able to get the camera to focus again by Clear setting on my camera. :)

If there is anything I can do to contribute to this built and this community please let me know :)

wmaiouiru

On Friday, March 23, 2012 7:43:07 PM UTC-4, SztupY wrote:
> Hi!
>
>
>
> Haven't checked Alex's latest updates so I can't comment on this
> right now.
>
>
>
>  SztupY
>
> <blockquote type="cite">Hi  <img name="4F6D0A8B...@sztupy.hu_:0" origsrc="c/photos/public/AIbEiAIAAABECJi7iJCx4IGLvQEiC3ZjYXJkX3Bob3RvKig3NTA5MGU0ZmE0ZGVkZmJlNjBkMzM0ZDBmY2QxZmNiYmFjNTQxZjRkMAHZPGENznOg-oI_6-6UhT0eLr4vdQ?sz=32"> <span>&quot;Sztupák Sz. Zsolt&quot;,

>
>
>
> I have just upgraded to the 2012Mar23.550d106.alex but ML
> freezes when I select lua.bin or testplug.bin
>
>  Due to of 500d having to less mem?
>
>
>
> But it worked using your initial release.
>
> Is there any way you could fix this?
>
>
>
> Greetings Lauris Heijdemann
>
> (bas lichtjaar)
>
>
>

> </span>
> --
>
> <a href="http://magiclantern.wikia.com/" target="_blank">http://magiclantern.wikia.com/</a>
>
>  
>
> To post to this group, send email to <a href="mailto:ml-d...@googlegroups.com" target="_blank">ml-d...@googlegroups.com</a>

>
> To unsubscribe from this group, send email to

> <a href="mailto:ml-devel+u...@googlegroups.com" target="_blank">ml-devel+unsubscribe@<WBR>googlegroups.com</a>
>
> For more options, visit this group at <a href="http://groups.google.com/group/ml-devel?hl=en" target="_blank">http://groups.google.com/<WBR>group/ml-devel?hl=en</a>
> </blockquote>
>
>
> </div>



On Friday, March 23, 2012 7:43:07 PM UTC-4, SztupY wrote:
> Hi!
>
>
>
> Haven&#39;t checked Alex&#39;s latest updates so I can&#39;t comment on this
> right now.
>
>
>
>  SztupY
>
> <blockquote type="cite">Hi  <img name="4F6D0A8B...@sztupy.hu_:0" origsrc="c/photos/public/AIbEiAIAAABECJi7iJCx4IGLvQEiC3ZjYXJkX3Bob3RvKig3NTA5MGU0ZmE0ZGVkZmJlNjBkMzM0ZDBmY2QxZmNiYmFjNTQxZjRkMAHZPGENznOg-oI_6-6UhT0eLr4vdQ?sz=32"> <span>&quot;Sztupák Sz. Zsolt&quot;,

>
>
>
> I have just upgraded to the 2012Mar23.550d106.alex but ML
> freezes when I select lua.bin or testplug.bin
>
>  Due to of 500d having to less mem?
>
>
>
> But it worked using your initial release.
>
> Is there any way you could fix this?
>
>
>
> Greetings Lauris Heijdemann
>
> (bas lichtjaar)
>
>
>

> </span>
> --
>
> <a href="http://magiclantern.wikia.com/" target="_blank">http://magiclantern.wikia.com/</a>
>
>  
>
> To post to this group, send email to <a href="mailto:ml-d...@googlegroups.com" target="_blank">ml-d...@googlegroups.com</a>

>
> To unsubscribe from this group, send email to

> <a href="mailto:ml-devel+u...@googlegroups.com" target="_blank">ml-devel+unsubscribe@<WBR>googlegroups.com</a>
>
> For more options, visit this group at <a href="http://groups.google.com/group/ml-devel?hl=en" target="_blank">http://groups.google.com/<WBR>group/ml-devel?hl=en</a>
> </blockquote>
>
>
> </div>