Webcam and Dynamic Graphics

[Daniel Maia]

Hi everyone!

I'm from Brazil, and my university is trying to use the WebLab Deusto system.

I created the server and add the lab, but i'm just don't know how to input the Webcam system. Does anyone know how to do it? Maybe, some tutorial...

The other problem is the Dynamic Graphics. Is there some tutorial to create a dynamic one? I couldn't find it in the documention section in WebLab's site.

Sorry for the language, and thx for the help!

Daniel Maia

[Pablo Orduña]

Hi Daniel!

Regarding the cameras: what type of webcams do you have? Do you have USB cameras or network cameras? Are you on a Linux system? If so, you might want to try motion ( http://www.lavrsen.dk/foswiki/bin/view/Motion/WebHome ). It supports several types of cameras, and might include yours. Internally in Deusto, we use D-Link DCS-192L cameras and provide access to them through Apache using it as a proxy (connecting to the final camera on its IP address).

Regarding Dynamic Graphics, what do you mean exactly? Like HTML5 canvas or similar?

Best,

--
You received this message because you are subscribed to the Google Groups "weblabdeusto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weblabdeusto...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Johanna Roussel]

Hey,

I setup a webcam streaming example using python, flask and opencv (I was first doing something with WebRTC - but one advantage is that it's not supported by every browser).
Now I want to integrate it in the WeblabDeusto framework. The idea is that multiple people can access the webcam stream but only one user can access the corresponding experiment which really interacts with a server validating some values. What would be the way to do it?
Creating two different experiments? One experiment only showing the webcam stream (derived from "ConcurrentExperiment"?) and the other one for example having two iframes (e.g. webcamstream on the left and some textfields in another iframe on the right) (derived from "Experiment.Experiment"?). Both interacting with one experiment server while the second one is using more interfaces then the webcamstream-experiment?

Which type of server would I use? I looked at your robot_movement.py example because it also contains a webcam access. But the webcam streaming server seems to be somewhere else. (somehow json is used)
Should I also integrate my webcam like in the robot_movement.py?

My first idea was to use my simple python flask server (isolated from weblabdeusto) and simply show it in the html file which will be integrated in weblabdeusto. This way the webcam wouldn't be integrated in the experiment server. So the experiment server would only handle the text field inputs.
But I think the way you do it is the better one, right? But at the moment I don't understand how it works.

Best,
Hanna

[Luis Rodríguez]

Hi Johanna, 

In some experiments such as robot_movement we store the webcam URLs as configuration variables of the experiment to avoid "hard-coding" them in the client. The advantage is that this way, once deployed, they are easier to change, and that we don't need to change the code of the experiment itself to change them; but the disadvantage is that, as you can see, the scheme is more complicated. You don't really need to do it that way, it is indeed easier to simply have a different server to provide the stream, and to just hard-code the URL in your HTML client.

Regarding your collaboration scheme (having one user "experiment" while the others look), I believe that it can be achieved relatively easily. Assuming that you are relying on Managed Experiments (which I believe you do):

An experiment's HTML client is loaded in the reserve screen, before the Reserve takes place. Many experiments simply hide the experiment's iframe until the Reserve succeeds, but you don't have to. So you can:

• Create a single experiment, non-concurrent (inheriting from Experiment.Experiment). 
  
• Create a single HTML client which initially displays the view for the "view-only" users. The experiment can be configured to show that view *before* the Reserve takes place, in the Reserve screen itself. Then, when a user (the one that wishes to control the experiment) clicks Reserve and reserves the experiment, the client will just receive the "onStart" callback and you can then display the "advanced" view.
  

An experiment that does something similar to this is: https://weblab.deusto.es/weblab/labs/Robot%20experiments/romie_blockly/

In that case, the experiment's client shows a blockly programming environment before Reserve so that many users can program at once but only one can control the robot at the same time. In your case, the concept would be mostly the same, except that they would see the webcam instead of the programming environment.

It is noteworthy that this scheme currently has some potential limitations (which may not affect you, though):

  - Before the Reserve takes place, the client can display anything, but the experiment server can't really be contacted. This typically means that you can easily display a webcam, but it also means that Not-reserved users cannot send commands to the server to, for instance, retrieve the state of sensors.

  - There are also some federation-related limitations.

Best regards!

  Luis

[Pablo Orduña]

Hi,

(Sorry for the format of the email, I'm writing a bus 0:-)   )

I've just read Luis' mail. I add more related info here.

There are three related topics described in more detail here (subsections "Load balancing", "Concurrency" and "Sharing resources"):

http://weblabdeusto.readthedocs.io/en/latest/remote_lab_deployment.html#step-3-registering-a-scheduling-system-for-the-experiment

Basically you can play with the scheduler to tune what you want. In the case you're describing, as far as I understand, you can have as you say two different experiments:

a) A regular one, with controlling access, and only one slot. So you would need its own queue and a single resource (unless you have multiple copies of the lab).

b) Another one, with multiple users accessing but no control access. So as to avoid that it interacts with the other lab, you should use a different queue. And you should "invent" different experiments only in the core server configuration file and in the lab configuration file.

So, you could have in the core server:

core_coordinator_laboratory_servers = {
'laboratory1:laboratory1@core_host' :
  {
'exp1|webcam|Dummy experiments' : 'webcam1@webcam_queue',
'exp2|webcam|Dummy experiments' : 'webcam2@webcam_queue',
'exp3|webcam|Dummy experiments' : 'webcam3@webcam_queue',
'exp4|webcam|Dummy experiments' : 'webcam4@webcam_queue',
'exp5|webcam|Dummy experiments' : 'webcam5@webcam_queue',
...
}, }

up to 60 or so. WebLab will think that there are those resources (as in load balancing) and will send them to the Laboratory Server.

Then, on the Laboratory Server you can do the same but pointing to a single experiment:

laboratory_assigned_experiments = {
'exp1:webcam@Dummy experiments' : {
'coord_address' : 'webcam:laboratory1@core_host',
'checkers' : () },
'exp2:webcam@Dummy experiments' : {
'coord_address' : 'webcam:laboratory1@core_host',
'checkers' : () },
'exp3:webcam@Dummy experiments' : {
'coord_address' : 'webcam:laboratory1@core_host',
'checkers' : () },
'exp4:webcam@Dummy experiments' : {
'coord_address' : 'webcam:laboratory1@core_host',
'checkers' : () },
'exp5:webcam@Dummy experiments' : {
'coord_address' : 'webcam:laboratory1@core_host',
'checkers' : () },
}

This experiment can be ConcurrentExperiment as you mention, but it can also be a regular experiment. The difference is that if it is a regular experiment, there is no way for you to differentiate among the different users. Suddenly it will be called the "start experiment" many times and when "send command" is sent, you don't know which user is calling it. But if you don't need it, then a regular experiment is fine.

And all this only makes sense if you need a feature of WebLab-Deusto. I mean, if it is only for the camera (and not for interacting with hardware in read-only mode), you're not interested in recording who enters in the camera, anyone can access and so on, it might be easier to put a static HTML file in the /pub directory and pass the link to the students.

Regarding your last paragraph, I didn't understand it. You meant like having an external site for the webcam so other people can access but also include it (e.g., in an iframe) in your experiment client? That would be perfectly fine.

Best,

[Johanna Roussel]

Ah I understand. Thank you both!

[Pablo Orduña]

Great! By the way, are you planning to upload somewhere the Flask + opencv example code? It would be great so as to know how to use USB cameras in Windows in this context!

[Johanna Roussel]

yes. I basicly focused on this website: http://blog.miguelgrinberg.com/post/video-streaming-with-flask
But I did the capturing with openCV (really simple). At the moment I'm testing with three different webcams on one computer. I can control two of them simultaneously. The third one sometimes makes problems when using with the others. I still have to figure out what's wrong here.
And I don't know if there are better ways to do the streaming..
By the way: did you consider using webRTC?

Can I find your webcam streaming server somewhere?

[Luis Rodríguez]

Hello Johanna,

We have uploaded the (quite simple) server that we are currently using to the "tools" folder:

https://github.com/weblabdeusto/weblabdeusto/tree/master/tools/cams

The "standard" method that we use to display the webcams is to simply, from the client, request snapshots continuously. In the client, from JavaScript, we replace the "src" so that the image changes.

The proxy is mostly to avoid overloading the webcams. The clients request images to the proxy, and the proxy checks whether it has recently obtained the image from the IP webcam. If it has, it returns the cached image. If it doesn't, it requests the image to the IP webcam. So, from the IP webcam's perspective, at any given time there is a single user (the proxy) requesting images.  

All of our experiments support the image refresh technique above (using the proxy), but, additionally, some of them, additionally support a "MJPEG" stream. We route these ones directly to the IP webcams, so the "proxied.py" file does not handle them. 

Best regards!

  Luis

[Johanna Roussel]

Thanks!

Using my python streaming server I recognized a mjpeg bug in firefox. The memory increases constantly (really fast) until firefox crashes after refreshing the website (minimum once) with the mjpeg stream.
It seems that this is not a bug of the server because it also happens if I try it with one of your mjpeg streams (like https://cams.weblab.deusto.es/webcam/fishtank1/video.mjpeg) (but slowly in this case)

With Chrome, everything is fine.

Best,
Hanna

[Luis Rodríguez]

Hi,

Yes. Native MJPEG support in browsers is, unfortunately, quite bad. Particularly, some issues that it seems to have:

• Bugs: The one you describe is a current example; but historically there have been lots of bugs in other browsers too (e.g., https://bugs.chromium.org/p/chromium/issues/detail?id=305215), they seem to take a long while to get fixed, and to not get too much attention.
• Latency: MJPEG streams typically send images at a fixed rate, so, typically, if the bandwidth is constrained, latency between the capture-time and the render-time will progressively increase. So, after a while, you can find that the image you are seeing is actually many seconds old. (This is particularly bad for some labs).
• It is particularly hard to detect errors or to detect out-of-sync issues with browser-native MJPEG, because, for them, JavaScript "load" and "error" events do not seem to be particularly useful (there is no "load" in each frame, and if the first frame succeeds, but later on the webcam fails, the "error" event does not seem to be raised; at least in some browsers). And there is no specific JavaScript API for them.

Currently, what we do is, basically, to make sure that every experiment supports image-refreshing based streaming. Though it might seem (and it is) quite a "vintage" approach, if the refresh rate is high enough the quality is generally acceptable, and it is very reliable (no problems recovering from errors; or adapting to changing bandwidth; or building up a delay).

Regards!

  Luis

[Johanna Roussel]

Hey!

Thank you Luis for the information! I'm totally new to the field and I thought webcam streaming is easier and better supported because I thought many people are doing it. First had the problems with WebRTC, which is not well supported by many browsers and now with mjpeg :)

I looked into your proxied.py which is really useful.

Let me summarize if I understand everything correctly:

Within the client html document the user can/should decide if he wants to access the livestream directly without using the proxied.py
e.g. <img id="cam_img" src="http://cams.weblab.deusto.es/webcam/fishtank1/video.mjpeg" >

or receive a cashed jpg via e.g.:
<img id="cam_img" src="http://cams.weblab.deusto.es/webcam/proxied.py/fishtank1>
from the proxy while "fishtank1" is a function defined in proxied.py

And the proxied.py is somehow published on an apache web server (I still have to figure out how to do this).

Then you have different streaming server probably on other machines relating to the experiments where the cams run on (by the way: are you using raspberry pis for that?). Are those streaming server special designed to work with the proxied.py?
I'm wondering if my mjpeg-Server would work with your proxied.py as well. Probably not because I think I can't access a single jpg via the url..

Best,
Hanna

[Luis Rodríguez]

Hi Johanna!

Yeah, WebRTC seems to have some portability issues. And to not be too oriented to centralised rather than P2P streaming. 

You are indeed mostly right. A couple remarks though:

- Client-side, the JPGs are automatically refreshed through JavaScript by changing the "src" (and in fact, appending a random number as an URL parameter to the URL, just to be sure no client-side caching occurs).  

- The "proxied.py" scheme that we use is somewhat simpler. We use mostly IP webcams, so the webcams themselves provide a URL through which you can access a snapshot. Thus, we don't really need additional streaming servers.  The "proxied.py" script accesses the IP webcam snapshots directly (though it only does so every X milliseconds at most), so we use a single server for all webcams (though, being IP webcams, the webcams are really servers themselves).

With USB webcams however, the scheme would need to be different. If you wanted to use a similar architecture, you could indeed add some "snapshooting" capabilities to your (raspberry-based?) streaming server, so that together (usb webcam + raspberry streaming server) would take the place of an IP webcam. Though there might be easier and/or cost-effective ways. For instance, depending on the raspberry model you have, the quality and FPS that you need, and the number of concurrent users that you want to support, using the streaming raspberry as a proxy / cache server for a single (or for a few) webcam(s) might work fine. In fact, depending on the same variables (quality, FPS, number of users, etc.), the server that hosts the Experiment Server *might* be enough to host both the Experiment and the USB webcam.

Regards!

  Luis

El viernes, 17 de junio de 2016, 10:07:31 (UTC+2), Johanna Roussel escribió

[Pablo Orduña]

Hi,

To make the client side clearer in addition to Luis' explanation, a simpler code of the client side using jQuery is this one:

   https://github.com/weblabdeusto/weblabdeusto/blob/master/server/src/weblab/core/static/jslabs/archimedes/js/widgets.js#L148

Feel free to copy it (Luis is the author).

One of the things we were considering to do would be some kind of library of widgets, so as to make the development process easier. In the past, we did that in GWT, but I'm not sure if it was useful at all for anyone outside Deusto. Now that we encourage "raw" JavaScript (or unmanaged labs), it could be especially useless: we don't want to reinvent the wheel, and for any type of widget (e.g., timers, buttons, whatever), there are probably much better and well maintained JavaScript libraries out there.

However, maybe it does make sense for very concrete widgets (such as the timer you see in VISIR or Archimedes; or for the camera). So it could end in something like adding a set of files <script src="../widgets/jquery/camera-0.1.js"></script> for those using jQuery or <script src="../widgets/angular/camera-0.1.js"></script> for those using angularjs; for those shared components. The alternative to that library would be what we're doing in this thread: just asking and sharing code snippets. Or maybe documenting them in the WebLab-Deusto docs.

Not sure what would be better, what do you think would be more useful?

Best,

Pablo Orduña ( http://morelab.deusto.es/p/pablo-orduna/ )

MORElab - Envisioning Future Internet (http://morelab.deusto.es)
WebLab-Deusto - (http://weblab.deusto.es)

[Johanna Roussel]

Hey,

I already analyzed your aquarium example last week to understand how the non-mjpeg streaming works. During debugging I found the CameraRefresher class in widgets.js.
At the moment I'm not using raspberry pi but I thought about using it in future maybe. Because of decreased network/concurrancy functionality my idea was to have a proxy with multi-user capability which is the only client accessing the camera (I think like your proxied.py). But you are right - using IP-cams would be much easier probably. But we are thinking about having a rack with multiple shelves and in each shelf an experimental setup equipped with a cam is situated (like 20 or so). I thought using raspberry pis would be the best way to handle that? And an IP-Cam + a raspberry pi is more expensive than raspberry pi + webcam (because all the 20 IP cams need to be connected somewhere..).
Having a library of widgets would be cool and useful if it's easy to realize. So - you are thinking of splitting e.g. CameraRefresher and TimerDisplayer into two files and providing more files for other widgets?

Best,
Hanna

[Johanna Roussel]

maybe IP-cams in combination with gige-switches would be better..

[Pablo Orduña]

Hi,

Just in case we in Deusto don't use raspberry pi + ip cams, but simply ip cams and 1 single server with the proxy.py for all ip cameras distributed phisically in the lab. It's convenient when you have multiple for having e.g., cams on the roof with WiFi and not dealing with so many cables, but I'm pretty sure you can reach that also with raspberries or USB depending on your physical constraints.

Regarding the library, we would have to discuss it further. The simplest approach is to provide simply a jquery-widgets-0.1.min.js file and a angularjs-widgets-0.2.min.js file, instead of one file per widget as I originally suggested. As the file gets bigger it might be interesting to split it, but maybe we can do it only if it makes sense in the future. As long as we keep the old versions of the file in -X.Y versions of the file it should be fine to maintain backwards compatibility (so we can always do a jquery-widgets-cam-0.9.min.js if really needed and keep the camera widget in the previous versions). Luis and me had discussed to put the library somewhere else too so that it can be as easy as adding an external server (e.g., http://cdn.wherever/weblabdeusto/jquery-widgets-0.1.min.js) to avoid dealing with internal paths in WebLab-Deusto), but it doesn't make sense for now and it would be easier to put it where the weblab.v1.js library is.

@Luis, can you think of any reason why we can't do that already?

To post to this group, send email to weblab...@googlegroups.com.
Visit this group at https://groups.google.com/group/weblabdeusto.

For more options, visit https://groups.google.com/d/optout.

[Luis Rodríguez]

Hi,

I think that we use IP cams mostly for convenience and for "historical" reasons. Raspberry + standard webcam + proxy/cache server should also work fine, I think. Maybe even better, at some respects (having your own raspberry as the cam's server probably gives you more control than the built-in servers of many webcams; and the raspberry *should* be powerful enough to handle both single-user streaming and the experiment itself).

Regarding the widgets library, yeah, I think we could indeed do that already.

Though doing it nicely would require some extra effort to tidy up the code of some widgets, ensure that the same JS / conventions are used, set up the grunt build file, the npm package, etc. But we could leave that for later.

[Johanna Roussel]

I will see when I know how the experiments do look like. If for each experiment a rasp pi is used anyway, a usb2.0 webcam would be fine. At present, however there are not many usb3.0 webcams available on the market. In that case a usb3.0-hub would work maybe, too.

Are you satisfied with your d-link ip cams (I couldn't find D-Link DCS-192L, is it VGA-resolution?)? Does the wireless work reliable? How many are you typically using in one lab? Customer reviews on ip cams are not convincing.

concerning the lib: sounds good!

[Luis Rodríguez]

The one we currently use the most is the DCS-932L (http://www.dlink.com/es/es/home-solutions/view/network-cameras/dcs-932l-day-night-cloud-camera) though some of our experiments rely on older models, and lately we have been buying different and/or better cameras. 

We are not really particularly satisfied. They seem to be rather expensive for what they offer. 

The advantages are essentially that:

  - We are used to them.

  - They provide (or used to provide) "reasonable" quality (640x480). (We have actually been buying better cams with HD quality lately). 

  - They support infrared vision.

  - They seem to be reasonably reliable. They don't typically break (some certainly have; but not very often - though we haven't really compared that against other brands or webcam types). 

The disadvantages are essentially that:

  - They are quite expensive; there might be cheaper alternatives with the same or better specs.

  - Their quality is not particularly high.

  - They only provide snapshots and MJPEG streams.

Wireless is indeed mostly reliable; though still, we try to rely on ethernet when we can; and use wireless only for cameras placed on-board on mobile robots, and the like, to avoid cluttering the wireless network. Once you have a certain number of cameras, the bandwidth they consume is significant. 

Most of our labs have 1-2 static webcams, connected through Ethernet. In a few other labs (such as https://weblab.deusto.es/weblab/labs/Robot%20experiments/romie_blockly/) we have sometimes an additional on-board camera to provide a "robot-level" view. 

Also, there are some exceptions, such as the VISIR experiment (which has no camera at all), or the archimedes experiment (https://weblab.deusto.es/weblab/labs/Aquatic%20experiments/archimedes/) which is made of several somewhat independent "boxes" and each has an IP ethernet-based camera, so it would sum 7 cameras. In fact, strictly speaking, each archimedes box also has an additional USB HD camera which only takes still images (so that would total 14 cameras).

[Johanna Roussel]

juhuuu my streaming server seems to work quite well!
I deployed your proxied.py on my apache and firefox doesn't seem to have a memory leak using the single image method in combination with the camera refresher.
Except for the edge browser at the moment (but I think this can be fixed (http://localhost:8000/weblab/static/js/jquery.min.map is missing))
Accessing with multiple clients also looks good.

I think I will upload the server soon. Still have to check the licence things.

[Luis Rodríguez]

That's great!

--
You received this message because you are subscribed to a topic in the Google Groups "weblabdeusto" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weblabdeusto/QqcHm7RtJpA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weblabdeusto...@googlegroups.com.

[Luis Rodríguez]

Hi Johanna!

Just so you know, we have started a widget library to place reusable laboratory widgets. 

https://github.com/weblabdeusto/jquery-lab-widgets

The goal would be to have two different libraries: one for jQuery-based widgets, and another for angular-based widgets.

It's still a work in progress: for now, we have only included the webcam widget (which you already have), so it will not be of much use to you. 

But, just so you are aware and so that you can provide feedback or suggestions if you wish.

Regards!

That's great!

To unsubscribe from this group and all its topics, send an email to weblabdeusto+unsubscribe@googlegroups.com.

[Johanna Roussel]

Ah - thanks! I will have a look on it later.

That's great!

To unsubscribe from this group and all its topics, send an email to weblabdeusto...@googlegroups.com.