rdm sensor ranges in rdm_collector

51 views
Skip to first unread message

Petr Vanek

unread,
Jun 27, 2017, 2:21:48 AM6/27/17
to open-lighting
Hi guys,

i presume if someone wanted to write data consumer for ola collected rdm data (rdm.openlighting.org), it would be beneficial to know some more details like sensor ranges and optimal values even prior to being connected to the fixture, hence the collector could add ranges into the log. Will rdm.openlighting.org be able to cope with more details?

Essentially:


with more more data from:


Like this:

  def _HandleSensorData(self, data):
    """Called when we get a SENSOR_DEFINITION response."""
    this_version = self._GetVersion()
    this_version['sensors'].append({
      'index': data['sensor_number'],
      'description': data['name'],
      'type': data['type'],
      'supports_recording': data['supports_recording'],
      'range_min':data['range_min'],
      'range_max':data['range_max'],
      'normal_min':data['normal_min'],
      'normal_max':data['normal_max'],
      'unit':data['unit'],

    })
    self._FetchNextSensor()

if yes, i can try to make pr for this small change,

thank you
P.

Peter Newman

unread,
Jun 27, 2017, 11:35:28 AM6/27/17
to open-lighting
Hi Petr,

Yes, I'm rather surprised we don't log units already, but the other bits would probably make sense too.

I'd previously added some other information on personalities, but I'm not sure I'd plumbed it into the rdm-app end.

In terms of adding your features, you'd need to update these places I think:
https://github.com/OpenLightingProject/rdm-app/blob/196208792e36156b26299642be25fe8e4f1d70ac/js_src/app.js#L162

https://github.com/OpenLightingProject/rdm-app/blob/196208792e36156b26299642be25fe8e4f1d70ac/admin.py#L804

https://github.com/OpenLightingProject/rdm-app/blob/196208792e36156b26299642be25fe8e4f1d70ac/model_handler.py#L284

https://github.com/OpenLightingProject/rdm-app/blob/196208792e36156b26299642be25fe8e4f1d70ac/model_loader.py#L283

But the main thing is to get it into the model collector, the other stuff can follow later, otherwise its a chicken and egg situation.

Petr Vanek

unread,
Jun 28, 2017, 4:42:48 AM6/28/17
to open-lighting
Thank you Peter.

I have updated the ModelCollector but am quite afraid to touch the server side as i cannot test it here and am afraid of breaking something, especially as it would need to be tested against older data without the new fields and newly collected stuff... or is there a way to test it?

I have added example collection log with new data into to PR comment,

cheers

Petr

Peter Newman

unread,
Jun 28, 2017, 9:03:23 PM6/28/17
to open-lighting
Thanks Petr.

You can test the app server fairly easily standalone, our code is here if you want to have a play:

Then grab the dev app engine from Google (I use the original App Engine SDK for Python):

I think you're right though, adding an old one, adding an old one, then updating with new data and adding new on it's own will all need to be tested before it's deployed.

Petr Vanek

unread,
Jun 29, 2017, 4:39:24 AM6/29/17
to open-lighting
Thank you Peter.

I am trying this but am missing few points i guess. If you have a minute on irc, i am there...

thank you

Petr

Petr Vanek

unread,
Jun 29, 2017, 7:27:44 AM6/29/17
to open-lighting
Thank you very much Peter for the irc assistance.

Local app running, imported some (older) data, product is seen but sensors are not showing, nor are DMX Personalities, seems like PIDs not initialized. Do i have to import them from somewhere?

I do not keep persistent irc presence, please let me know through here,

thank you

Petr

Peter Newman

unread,
Jun 29, 2017, 3:08:04 PM6/29/17
to open-lighting
That's good progress.

If you go to /admin then run
Update Manufacturers
Update Categories
Load ESTA PIDs
Load Manufacturer PIDs
Build PID Index

Then it should start working. You can also press the other update buttons to add more other data.

Petr Vanek

unread,
Jun 29, 2017, 4:32:59 PM6/29/17
to open-lighting
Hi Peter,

i did try. I also cleaned the datastore but no luck, no dmx personalities and no sensors. PIDs are now initialized correctly.Any other idea? The data is in the log...

thank you

Petr

Peter Newman

unread,
Jun 30, 2017, 6:08:14 AM6/30/17
to open-lighting
Did you go through the moderation queue on the admin pages after you'd added them?

I imported it and it worked fine for me after moderating it.

Do the other two fixtures that were previously in the system work for you?

Petr Vanek

unread,
Jul 9, 2017, 4:55:36 PM7/9/17
to open-lighting
Hi Peter,

sorry, i was away for the week. Yes, i did go through moderation que. I only have one default device on the site, the LED BAR. As i import other data, devices do show up on the site, but are missing DMX personalities and other RDM info, please see screenshots, one taken on http://rdm.openlighting.org/model/display?manufacturer=21075&model=165 and one locally.

thank you very much

Petr
Screen Shot 2017-07-09 at 22.50.57-fullpage.png
Screenshot from 2017-07-09 22-50-24.png

Peter Newman

unread,
Jul 9, 2017, 7:12:47 PM7/9/17
to open-lighting
Hi Petr,

Yes sorry, I realised the other device I have is another I imported. Does the existing one work for you and show personalities etc?

Do you see anything in the dev_appserver.py console logs? Can you share those?

Do other bits of the site, like the PID list work locally?

Petr Vanek

unread,
Jul 10, 2017, 4:56:13 PM7/10/17
to open-lighting
Hi Peter,

i have taken full debug logs, seems that other levels show no useful info, see gzipped txt loghere: https://github.com/vanous/mnm/files/1137046/gae_debug.log.gz
the debug log is way too verbose, but perhaps you can see something...

>Does the existing one work for you and show personalities etc?

Sadly no :(

thank you

Petr

Peter Newman

unread,
Jul 10, 2017, 7:22:31 PM7/10/17
to open-lighting
Thanks. The fact they're both broken is a good thing in the sense that it's just the page not working properly I suspect, rather than the underlying appengine code.

Can you also send view source for the faulty page, for the existing fixture. Or if it's there, there should be a long line "app.setSoftwareVersions" that's the bit I'm particularly interested in. If your browser has some developer tools (they nearly all do), are there any Javascript errors logged?

Petr Vanek

unread,
Jul 11, 2017, 1:55:08 AM7/11/17
to Open-Lighting
Hi Peter,

thank you.Full console log here: https://pastebin.com/wERFJMHH

perhaps see line 68: ReferenceError: app is not defined

the data seems to be there.

app.setSoftwareVersions([{"sensors": [{"type_str": "Temperature",
"supports_recording": false, "index": 0, "supports_min_max": false,
"description": "Temperature", "type": 0}], "supported_parameters":
[{"manufacturer_id": 0, "id": 128, "name":
"DEVICE_MODEL_DESCRIPTION"}, {"manufacturer_id": 0, "id": 129, "name":
"MANUFACTURER_LABEL"}, {"manufacturer_id": 0, "id": 130, "name":
"DEVICE_LABEL"}, {"manufacturer_id": 0, "id": 224, "name":
"DMX_PERSONALITY"}, {"manufacturer_id": 0, "id": 225, "name":
"DMX_PERSONALITY_DESCRIPTION"}, {"manufacturer_id": 0, "id": 512,
"name": "SENSOR_DEFINITION"}, {"manufacturer_id": 0, "id": 513,
"name": "SENSOR_VALUE"}, {"manufacturer_id": 0, "id": 1024, "name":
"DEVICE_HOURS"}], "version_id": 1, "label": "V1.02 \u0000LED
BAR\u0000BRITEQ", "personalities": [{"index": 1, "slot_count": 3,
"description": " \u0000\b"}, {"index": 2, "slot_count": 4,
"description": " \u0000\b"}, {"index": 3, "slot_count": 5,
"description": " \u0000\b"}, {"index": 4, "slot_count": 6,
"description": " \u0000\b"}, {"index": 5, "slot_count": 12,
"description": " \u0000\b"}, {"index": 6, "slot_count": 15,
"description": " \u0000\b"}, {"index": 7, "slot_count": 16,
"description": " \u00000"}]}]);

Full page source: https://pastebin.com/2wXSZYpM

Development SDK 1.9.55
runtime python27

pprint.pprint(os.environ.copy()):

{'APPENGINE_RUNTIME': 'python27',
'APPLICATION_ID': 'dev~olp-rdm-staging',
'AUTH_DOMAIN': 'gmail.com',
'CURRENT_MODULE_ID': 'default',
'CURRENT_VERSION_ID': '1.105377020582861770',
'DATACENTER': 'us1',
'DEFAULT_VERSION_HOSTNAME': 'localhost:8080',
'HTTPS': 'off',
'HTTP_HOST': 'localhost:8080',
'HTTP_X_APPENGINE_COUNTRY': 'ZZ',
'INSTANCE_ID': 'd8ac57d1379e296ee5d580cbe9761b7ddbec',
'REMOTE_ADDR': '192.0.2.0',
'REQUEST_ID_HASH': '4FF7BEFA',
'REQUEST_LOG_ID':
'6d6f15457daab7aebac80fe2af99a67e2f1155bacdabaa64aa00adbed3ad74fb1acb1',
'REQUEST_METHOD': 'GET',
'SCRIPT_NAME': '',
'SERVER_NAME': 'localhost',
'SERVER_PORT': '8080',
'SERVER_PROTOCOL': 'HTTP/1.1',
'SERVER_SOFTWARE': 'Development/2.0',
'TZ': 'UTC',
'USER_EMAIL': '',
'USER_ID': '',
'USER_IS_ADMIN': '0',
'USER_NICKNAME': '',
'USER_ORGANIZATION': '',
'wsgi.multithread': False}

thank you
Petr
> --
> The Open Lighting Project: open-l...@googlegroups.com, #openlighting
> (irc.freenode.org)
> To unsubscribe from this group, send email to
> open-lightin...@googlegroups.com
> For more options, visit https://groups.google.com/groups/opt_out?hl=en

Peter Newman

unread,
Jul 11, 2017, 9:40:44 AM7/11/17
to open-lighting
Which browser are you using?

The app error will indeed be the cause, but I'm not clear why it's failing to load that JS and hence isn't working.

Can we have the same JS console log and page source for your Robe fixture on that page please? There are a lot of nulls in the other LED bar, so I wonder if they're upsetting it or your browser.

I take it http://localhost:8080/js/app.js shows a page of minified JS on your browser?


On Tuesday, 11 July 2017 06:55:08 UTC+1, Petr Vanek wrote:
Hi Peter,

Peter Newman

unread,
Jul 11, 2017, 9:45:07 AM7/11/17
to open-lighting
Also does it render correctly externally, if you browse to here:
http://rdm.openlighting.org/model/display?manufacturer=8482&model=38

Petr Vanek

unread,
Jul 12, 2017, 2:10:32 AM7/12/17
to Open-Lighting
now we are onto something. latest firefox, chrome, chromium on debian.

http://localhost:8080/js/app.js response 200, size 0
http://localhost:8080/js/rdm.js is there (minified)
http://localhost:8080/js/pid_display.js is 404

seems like ./js_src files are not built with closurecompiler into
static/js (app.js indeed zero size there).

installed closure-tools via npm, and tried to run from my path rather
then what is in BUILD

please note missing one --root parameter in my line, vs --root=trunk/ in yours:

#!/bin/sh
#trunk/closure/bin/build/closurebuilder.py --root=trunk/ --root=js_src \
python ./node_modules/closure-tools/closure-bin/build/closurebuilder.py
--root=js_src \
--namespace="app.setup" --output_mode=compiled --compiler_jar=compiler.jar \
--compiler_flags="--compilation_level=ADVANCED_OPTIMIZATIONS" \
> static/js/app.js



vanous@tc-vanek:~/bin/projects/ola-web-app/rdm-app$ ./BUILD
./node_modules/closure-tools/closure-bin/build/closurebuilder.py:
Scanning paths...
./node_modules/closure-tools/closure-bin/build/closurebuilder.py: 3
sources scanned.
./node_modules/closure-tools/closure-bin/build/closurebuilder.py:
Building dependency tree..
Traceback (most recent call last):
File "./node_modules/closure-tools/closure-bin/build/closurebuilder.py",
line 262, in <module>
main()
File "./node_modules/closure-tools/closure-bin/build/closurebuilder.py",
line 211, in main
tree = depstree.DepsTree(sources)
File "/home/vanous/bin/projects/ola-web-app/rdm-app/node_modules/closure-tools/closure-bin/build/depstree.py",
line 56, in __init__
raise NamespaceNotFoundError(require, source)
depstree.NamespaceNotFoundError: Namespace "goog.ui.TableSorter" never
provided. Required in Source js_src/app.js


probably i am missing some namespace import...?

thank you very much!

Petr
>> > open-lightin...@googlegroups.com
>> > For more options, visit https://groups.google.com/groups/opt_out?hl=en
>
> --
> The Open Lighting Project: open-l...@googlegroups.com, #openlighting
> (irc.freenode.org)
> To unsubscribe from this group, send email to
> open-lightin...@googlegroups.com

Petr Vanek

unread,
Jul 12, 2017, 3:36:41 AM7/12/17
to Open-Lighting
:) stealing app.js from http://rdm.openlighting.org/js/app.js and my
local instance now runs ok :))

still i presume i need to figure out the closure compile stuff

P.

Petr Vanek

unread,
Jul 12, 2017, 5:25:11 AM7/12/17
to Open-Lighting
Latest progress, probably i am stuck here now:

1) can compile/build closure (installed google-closure-compiler,
closure-library, closure-tools)
2) my build line with paths set to closure libraries and closure tools
(for compiler.jar)

python ./node_modules/closure-tools/closure-bin/build/closurebuilder.py
--root=./node_modules/closure-library/ --root=js_src \
--namespace="app.setup" --output_mode=compiled
--compiler_jar=./node_modules/google-closure-compiler/compiler.jar \
--compiler_flags="--compilation_level=ADVANCED_OPTIMIZATIONS" \
> static/js/app.js

3) get few errors + many warnings (full log here:
https://pastebin.com/KsTWpXJg )

js_src/app.js:28: ERROR - Variable app declared more than once. First
occurence: js_src/pid_display.js
var app = app || {}
^^^^^^^^^^^^^^^

js_src/pid_display.js:58: ERROR - variable max is undeclared
max = this._field_info['max_size'];
^^^

js_src/pid_display.js:59: ERROR - variable min is undeclared
min = this._field_info['min_size'];

Which surprises me as i would expect that it builds cleanly, so
perhaps further config required on my side?

Thank you

Petr

Peter Newman

unread,
Jul 12, 2017, 7:30:01 AM7/12/17
to open-lighting
This is odd as that file isn't empty on git:

I'm guessing you'd tried, and failed, to rebuild before?

We've currently got two workflows, the newer stuff is npm/grunt:

The older stuff is closure:

Although the OLA build notes for closure are better:

You need the trunk root for Google's library stuff.

Depending on what you want to do, you may not need to rebuild the closure JS?

More later, it may be it's broken in more recent, stricter, versions of closure.
>>>> > For more options, visit https://groups.google.com/groups/opt_out?hl=en
>>>
>>> --
>>> The Open Lighting Project: open-l...@googlegroups.com, #openlighting
>>> (irc.freenode.org)
>>> To unsubscribe from this group, send email to

Petr Vanek

unread,
Jul 12, 2017, 9:23:20 AM7/12/17
to Open-Lighting
Hi Peter,

thank you. As for closure, the building works but fails with the
mentioned errors... i believe i have all the required stuff (tools.
libs, compiler).

I have tried grunt, but the Gruntfile.js completely omits app.js , it
only lists rdm.js , is there anything to do there? I had tried to just
replace rdm.js with app.js but it doesn't work :)

Thank you, let me know when you know more. In order to modify app.js,
i will need to be able to re-compile it...

cheers
Petr
>> >>>> > open-lightin...@googlegroups.com
>> >>>> > For more options, visit
>> >>>> > https://groups.google.com/groups/opt_out?hl=en
>> >>>
>> >>> --
>> >>> The Open Lighting Project: open-l...@googlegroups.com,
>> >>> #openlighting
>> >>> (irc.freenode.org)
>> >>> To unsubscribe from this group, send email to
>> >>> open-lightin...@googlegroups.com
>> >>> For more options, visit https://groups.google.com/groups/opt_out?hl=en
>
> --
> The Open Lighting Project: open-l...@googlegroups.com, #openlighting
> (irc.freenode.org)
> To unsubscribe from this group, send email to
> open-lightin...@googlegroups.com

Peter Newman

unread,
Jul 12, 2017, 11:04:19 AM7/12/17
to open-lighting
Hi,

The grunt stuff does the new tools code, e.g.:

Ah yes, I forgot you want to modify stuff in the GUI too!

It looks like a lot of this may be because it's loaded the files in the wrong order (--js js_src/pid_display.js --js js_src/app.js).

Want to try this command instead:
python ./node_modules/closure-tools/closure-bin/build/closurebuilder.py 
--root=./node_modules/closure-library/ --js js_src/app.js --js js_src/pid_display.js \ 
--namespace="app.setup" --output_mode=compiled 
--compiler_jar=./node_modules/google-closure-compiler/compiler.jar \ 
--compiler_flags="--compilation_level=ADVANCED_OPTIMIZATIONS" \ 
> static/js/app.js 

The other errors seem to be down to Google's code not liking their compiler, and do you want to try these fixes too:

Hopefully that will fix it?


On Wednesday, 12 July 2017 14:23:20 UTC+1, Petr Vanek wrote:
Hi Peter,

thank you. As for closure, the building works but fails with the
mentioned errors... i believe i have all the required stuff (tools.
libs, compiler).

I have tried grunt, but the Gruntfile.js completely omits app.js , it
only lists rdm.js , is there anything to do there? I had tried to just
replace rdm.js with app.js but it doesn't work :)

Thank you, let me know when you know more. In order to modify app.js,
i will need to be able to re-compile it...

cheers
Petr


>> >>>> > For more options, visit
>> >>>> > https://groups.google.com/groups/opt_out?hl=en
>> >>>
>> >>> --
>> >>> The Open Lighting Project: open-l...@googlegroups.com,
>> >>> #openlighting
>> >>> (irc.freenode.org)
>> >>> To unsubscribe from this group, send email to
>> >>> For more options, visit https://groups.google.com/groups/opt_out?hl=en
>
> --
> The Open Lighting Project: open-l...@googlegroups.com, #openlighting
> (irc.freenode.org)
> To unsubscribe from this group, send email to

Peter Newman

unread,
Jul 12, 2017, 11:13:59 AM7/12/17
to open-lighting
Actually this may still fail, even with a forced order of dependencies. It looks like we may have a circular dependency currently.

We need this from app.js first, to define the app variable:

But app.js requires app.MessageStructure

Which is provided in pid_display.js:

I guess we may need to pull some generic stuff out into another file to work around this nicely.

Petr Vanek

unread,
Jul 12, 2017, 11:42:33 AM7/12/17
to Open-Lighting
hmm, no luck.

i got the pull request, but compilation still fails with

./js_src/app.js:28: ERROR - Variable app declared more than once.
First occurence: ./js_src/pid_display.js
var app = app || {}
^^^^^^^^^^^^^^^


--js doesn't seem to be valid argument for closurebuilder.py but it
seems that i can specify files just by closurebuilder.py [options]
[file1.js file2.js ...]

but same error happens :/

what am i doing wrong?

:(
P.
>>> >> >>>> > open-lightin...@googlegroups.com
>>> >> >>>> > For more options, visit
>>> >> >>>> > https://groups.google.com/groups/opt_out?hl=en
>>> >> >>>
>>> >> >>> --
>>> >> >>> The Open Lighting Project: open-l...@googlegroups.com,
>>> >> >>> #openlighting
>>> >> >>> (irc.freenode.org)
>>> >> >>> To unsubscribe from this group, send email to
>>> >> >>> open-lightin...@googlegroups.com
>>> >> >>> For more options, visit
>>> >> >>> https://groups.google.com/groups/opt_out?hl=en
>>> >
>>> > --
>>> > The Open Lighting Project: open-l...@googlegroups.com,
>>> > #openlighting
>>> > (irc.freenode.org)
>>> > To unsubscribe from this group, send email to
>>> > open-lightin...@googlegroups.com
>>> > For more options, visit https://groups.google.com/groups/opt_out?hl=en
>
> --
> The Open Lighting Project: open-l...@googlegroups.com, #openlighting
> (irc.freenode.org)
> To unsubscribe from this group, send email to
> open-lightin...@googlegroups.com

Petr Vanek

unread,
Jul 12, 2017, 11:48:30 AM7/12/17
to Open-Lighting
perhaps related: https://github.com/google/closure-compiler/issues/2454

P.

Petr Vanek

unread,
Jul 12, 2017, 11:52:22 AM7/12/17
to Open-Lighting
oh, with --compilation_level=WHITESPACE_ONLY , it builds :)

Peter Newman

unread,
Jul 12, 2017, 12:49:12 PM7/12/17
to open-lighting
Excellent, you can carry on like that then, and I'll try and fix the outstanding issue. :)

Sorry, I saw the documentation, then tried to be lazy and copy your code; --js is used by the underlying Java compiler I think.

I take it the PR makes this better at least in that you don't get those other warnings?


On Wednesday, 12 July 2017 16:52:22 UTC+1, Petr Vanek wrote:
oh, with --compilation_level=WHITESPACE_ONLY , it builds :)

>>>>> >> >>>> > For more options, visit
>>>>> >> >>>> > https://groups.google.com/groups/opt_out?hl=en
>>>>> >> >>>
>>>>> >> >>> --
>>>>> >> >>> The Open Lighting Project: open-l...@googlegroups.com,
>>>>> >> >>> #openlighting
>>>>> >> >>> (irc.freenode.org)
>>>>> >> >>> To unsubscribe from this group, send email to
>>>>> >> >>> For more options, visit
>>>>> >> >>> https://groups.google.com/groups/opt_out?hl=en
>>>>> >
>>>>> > --
>>>>> > The Open Lighting Project: open-l...@googlegroups.com,
>>>>> > #openlighting
>>>>> > (irc.freenode.org)
>>>>> > To unsubscribe from this group, send email to
>>>>> > For more options, visit https://groups.google.com/groups/opt_out?hl=en
>>>
>>> --
>>> The Open Lighting Project: open-l...@googlegroups.com, #openlighting
>>> (irc.freenode.org)
>>> To unsubscribe from this group, send email to

Petr Vanek

unread,
Jul 12, 2017, 1:17:43 PM7/12/17
to open-lighting
zero warnings :)

./node_modules/closure-tools/closure-bin/build/closurebuilder.py: JavaScript compilation succeeded.

but i just tested and it is unrelated to the commit.

thank you very much

Petr


Peter Newman

unread,
Jul 12, 2017, 1:36:46 PM7/12/17
to open-lighting
Ah, do you want to continue discussion of the commit here:

Given you also had this before, or are you saying those errors don't exist when you change the other optimising level?
    1. js_src/pid_display.js:58: ERROR - variable max is undeclared
    2.     max = this._field_info['max_size'];
    3.     ^^^
    4.  
    5. js_src/pid_display.js:59: ERROR - variable min is undeclared
    6.     min = this._field_info['min_size'];
    1.     ^^^
    >>>>> >> >>>> >>&gt
    Reply all
    Reply to author
    Forward
    0 new messages