Socket interface?

61 views
Skip to first unread message

Micah Wedemeyer

unread,
Dec 13, 2014, 2:45:56 PM12/13/14
to lightsh...@googlegroups.com
With people building web UIs, phone UIs, and other interfaces, I was wondering if it might be useful to have a control daemon that listens on a socket. Clients could control the music, lights, and such by sending messages to the socket. A socket interface would allow writing clients in other languages besides Python, and just generally make it easier to integrate with the underlying lightshow codebase.

Any thoughts on this?

Todd Giles

unread,
Dec 13, 2014, 6:55:10 PM12/13/14
to lightsh...@googlegroups.com

Totally, I've been having similar thoughts, just lack of time to implement ;)

Micah Wedemeyer

unread,
Dec 13, 2014, 11:26:24 PM12/13/14
to lightsh...@googlegroups.com
If I have some time, I'll poke around. It's been a long time since I've done any socket programming. I'm a web developer by trade, so it's strictly port 80 and 443 for me, but I'm curious to relearn some of the things I was supposed to learn in school.

Micah

Todd Giles

unread,
Dec 13, 2014, 11:36:24 PM12/13/14
to lightsh...@googlegroups.com
I'm sure there are quite a few ways we could go to implement this, but at a high level the idea of having the a lightshow daemon makes good sense moving forward and as you say will enable more individuals to integrate with it using their favorite method / programming language / etc... 


--
http://www.lightshowpi.com/
---
You received this message because you are subscribed to the Google Groups "LightshowPi Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lightshowpi-d...@googlegroups.com.
To post to this group, send email to lightsh...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lightshowpi-dev/b3a86a19-90d7-428b-894b-7217bde07638%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Micah Wedemeyer

unread,
Dec 15, 2014, 9:32:13 AM12/15/14
to lightsh...@googlegroups.com
I was looking around for advice on designing a remote control protocol and found this: http://json-rpc.org/

It's a way to create a remote API for something using JSON. The idea would be for a daemon to listen on a particular socket, waiting for JSON-RPC commands. We'd have to define the API methods available, but JSON-RPC saves us from having to come up with some kind of syntax, plus people on this project seem pretty comfortable with JSON already.

An example would be something like:
client sends: {method: "lightControl", params: ["on"], id: 12345 }
server responds: {result: "success", error: null, id: 12345}

There are several python libraries for JSON-RPC, plus libraries in other languages. So, if the server/daemon speaks JSON-RPC, people could create clients fairly easily.

Thoughts?

Paul

unread,
Dec 15, 2014, 9:35:57 AM12/15/14
to lightsh...@googlegroups.com
I think JSON would be a good way to go. XBMC (last I looked) was all JSON. They don't call it XBMC anymore, but that's beside the point. ;-)

Todd Giles

unread,
Dec 15, 2014, 10:30:06 AM12/15/14
to lightsh...@googlegroups.com
I'd like to suggest the use of protocol buffers:


I can be convinced to go the json route still - but would love your opinions on using a schema based definition of these RPCs such as protobufs.

-Todd

To unsubscribe from this group and stop receiving emails from it, send an email to lightshowpi-dev+unsubscribe@googlegroups.com.
To post to this group, send email to lightshowpi-dev@googlegroups.com.

--
http://www.lightshowpi.com/
---
You received this message because you are subscribed to the Google Groups "LightshowPi Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lightshowpi-dev+unsubscribe@googlegroups.com.
To post to this group, send email to lightshowpi-dev@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lightshowpi-dev/548EF1CB.2010204%40gmail.com.

chris usey

unread,
Dec 15, 2014, 10:53:22 AM12/15/14
to lightsh...@googlegroups.com
I don’t know much about protocol-buffers. From the little I do know, it seems to be a smaller footprint and promotes faster communication. I don’t think it will be as easy for future contributors to jump in and get going using protocol-buffers as it would be using JSON, but thats no deal breaker. I think either would work well.

- Chris

Todd Giles

unread,
Dec 15, 2014, 11:01:50 AM12/15/14
to lightsh...@googlegroups.com
One of the main things I like about protocol buffers is the future proofing it provides (i.e. the data format is easy to keep backwards compatible).  Here is an article that articulates some of these benefits I just came across:


Of course, I've been using them for the last 4 years almost exclusively when creating / defining RPC contracts so am very familiar with them so I obviously have some bias in that regard.  I agree that json is better understood by a wider swath of developers and that reason may be sufficient for us to lean towards using json.

To view this discussion on the web visit https://groups.google.com/d/msgid/lightshowpi-dev/1B2462F3-C69D-4BF6-B726-370D5EB27685%40gmail.com.

Micah Wedemeyer

unread,
Dec 15, 2014, 11:35:49 AM12/15/14
to lightsh...@googlegroups.com
I had never heard of protocol buffers. But, after reading the information you sent they sound very compelling. I'm always willing to try new things. So, I would vote for protocol buffers over JSON in this case. Especially for the backwards compatibility. I can imagine this protocol changing significantly at the start.

Micah
You received this message because you are subscribed to a topic in the Google Groups "LightshowPi Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lightshowpi-dev/oW1wLcnEz7Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lightshowpi-d...@googlegroups.com.

To post to this group, send email to lightsh...@googlegroups.com.

Paul

unread,
Dec 15, 2014, 11:47:30 AM12/15/14
to lightsh...@googlegroups.com
I have to add in that my 'programmer' career ended almost 15 years ago
and I've been a sys-admin ever since.
so, just to make sure I understand what is being discussed, an overview:

LSP-daemon running: web-interface, remote control, smartphone apps, etc
can all then send commands to the daemon to start the show, stop,
lights-on/off etc.
correct?

If that's the case, it would probably make sense then, to have the
daemon be _the_ controller and use the same protocol (json or protobuf)
internally.

the only experience I have with this sort of thing is with XBMC. That
one uses json. the XBMC remote app I have on my phone can do json to my
HTPC to tell XBMC start playing, stop, list available movies etc.
At the same time, XBMC uses the same thing "internally" so add-ins still
make json calls to XBMC for adding services, creating new streams etc.
(at least this is how it was when I was developing add-ins. It's now
Kodi.tv instead of XBMC, but as far as I know those internals/protocols
are the same)

This would work for network streaming too (like I've been working on)
Right now I just made up my own little protocol, but it could be wrapped
up with this mechanism too.

chris usey

unread,
Dec 15, 2014, 11:55:45 AM12/15/14
to lightsh...@googlegroups.com
I have to agree with Micah, Part of the reason I started with this was to learn electronics and python. So Im very open to learning protocol buffers ! +1 for protocol buffers.

- Chris

Micah Wedemeyer

unread,
Dec 15, 2014, 12:21:29 PM12/15/14
to lightsh...@googlegroups.com
Regarding the structure, my thinking was that the daemon/server would just control the lights and the music. Everything else would be considered a client. So, a web UI would talk to the daemon, as would an SMS interface.

The basic idea would be to make controlling the lights and music accessible via a socket interface, rather than needing to write python or run bash scripts. It would be similar to other socket-based services, like databases.

Does this match with others' visions on this?

Micah


To view this discussion on the web visit https://groups.google.com/d/msgid/lightshowpi-dev/EFAC1D02-BF09-4670-B935-EFF486822AF5%40gmail.com.

Todd Giles

unread,
Dec 15, 2014, 12:45:12 PM12/15/14
to lightsh...@googlegroups.com
Yes, that matches my vision on this front.  Perhaps you could draft up an initial service definition and we can iterate from there?

Looks like there are many open source protobuf RPC implementations:





To view this discussion on the web visit https://groups.google.com/d/msgid/lightshowpi-dev/CAGErOzFGMvJkeGLj2P7-cY-igLh6KASL4fb-sz26mFBfeFz-RQ%40mail.gmail.com.

Todd Giles

unread,
Dec 15, 2014, 12:47:58 PM12/15/14
to lightsh...@googlegroups.com
Sorry, that last one sent before I was done.


Looks like there are many open source protobuf RPC implementations:


But to simply define the RPC using protobufs the syntax can be found here:


It's pretty straightforward I hope ;)




For more options, visit https://groups.google.com/d/optout.

--
http://www.lightshowpi.com/
---
You received this message because you are subscribed to the Google Groups "LightshowPi Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lightshowpi-dev+unsubscribe@googlegroups.com.
To post to this group, send email to lightshowpi-dev@googlegroups.com.

--
http://www.lightshowpi.com/
---
You received this message because you are subscribed to a topic in the Google Groups "LightshowPi Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lightshowpi-dev/oW1wLcnEz7Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lightshowpi-dev+unsubscribe@googlegroups.com.

To post to this group, send email to lightshowpi-dev@googlegroups.com.

--
http://www.lightshowpi.com/
---
You received this message because you are subscribed to the Google Groups "LightshowPi Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lightshowpi-dev+unsubscribe@googlegroups.com.
To post to this group, send email to lightshowpi-dev@googlegroups.com.

--
http://www.lightshowpi.com/
---
You received this message because you are subscribed to a topic in the Google Groups "LightshowPi Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lightshowpi-dev/oW1wLcnEz7Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lightshowpi-dev+unsubscribe@googlegroups.com.

To post to this group, send email to lightshowpi-dev@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lightshowpi-dev/EFAC1D02-BF09-4670-B935-EFF486822AF5%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
http://www.lightshowpi.com/
---
You received this message because you are subscribed to the Google Groups "LightshowPi Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lightshowpi-dev+unsubscribe@googlegroups.com.
To post to this group, send email to lightshowpi-dev@googlegroups.com.

Tom Enos

unread,
Dec 16, 2014, 1:32:29 AM12/16/14
to lightsh...@googlegroups.com
Love the protocol buffers, wish I had known about them sooner,  and after reviewing some of the material it looks like it won't be that hard to implement.
I also like the idea of a deamon to control everything, I would still like a command line interface in there some place, but that is just me.

--
http://www.lightshowpi.com/
---
You received this message because you are subscribed to a topic in the Google Groups "LightshowPi Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lightshowpi-dev/oW1wLcnEz7Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lightshowpi-d...@googlegroups.com.

To post to this group, send email to lightsh...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
http://www.lightshowpi.com/
---
You received this message because you are subscribed to the Google Groups "LightshowPi Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lightshowpi-d...@googlegroups.com.
To post to this group, send email to lightsh...@googlegroups.com.

Todd Giles

unread,
Dec 16, 2014, 11:52:01 AM12/16/14
to lightsh...@googlegroups.com
I agree that we shouldn't sacrifice the command line interface either.  Let's agree upon the initial daemon API though before moving too far forward.  Whomever want's to through out the first proposal go for it!  I'll likely get around to thinking about it more deeply over Christmas break so if nothing is out before them I'll try to come up with an initial proposal in the form of a protobuf RPC service definition.

To view this discussion on the web visit https://groups.google.com/d/msgid/lightshowpi-dev/2b185867-77a0-42cb-89cd-ee796a39bb25%40googlegroups.com.

Todd Giles

unread,
Dec 16, 2014, 11:52:14 AM12/16/14
to lightsh...@googlegroups.com
s/throw/through/ ;)

Micah Wedemeyer

unread,
Dec 21, 2014, 10:30:39 PM12/21/14
to lightsh...@googlegroups.com
A few more thoughts on this subject:

Required Functionality

I've done a little more research and some brainstorming on the initial API. Here's a list of functions that I came up with. Note!!! I'm shooting for absolute barebones minimum functionality here. I'd prefer to start small and grow from there instead of trying to spec too much at the start.
  • lights control (on/off/fade/flash, support specific lights, cycle time, etc?)
  • play song
  • stop song
  • get status
  • change volume
Now, to make it actually useful, it probably also needs playlist support, so I have these as well:
  • play playlist
  • skip to next (or random) song in playlist
  • get playlist
I think those functions above, if implemented by a daemon, would expose a great deal of the underlying functionality. You'd still need to do a lot of setup (copy mp3s to the right place, create your playlist, edit your config files, etc.) but once up and running, you could easily control most aspects.

Socket Communication Layer

I did some reading and found that ZeroMQ is recommended for socket-layer communications. Since protocol buffers don't deal with the underlying transport layer, it's up to you to handle the actual message passing. ZeroMQ sits on top of TCP sockets and handles all the buffering and such, allowing you to write in messages (ie. a single request) as opposed to buffers and byte arrays and junk like that.

Further, someone has written a protobuf RPC stack on top of ZeroMQ with support for python: https://github.com/thesamet/rpcz

So, with RPCZ, we could write the .proto files and let RPCZ and ZeroMQ handle the transport/communication layer. The downside is that it seems like the dev environment setup is fairly involved.

Initial Stab at Definition

Finally, here's my initial stab at defining the messages and rpc for light control. I stole it pretty much directly from the hardware_controller main() function. I was hoping others could comment and critique on how these messages should be written.

message LightControlRequest {
  enum StateType {
    OFF = 0;
    ON = 1;
    FADE = 2;
    FLASH = 3;
    CLEANUP = 4;
  }

  optional StateType state = 1 [default = OFF];
  repeated int32 lights = 2;
  optional float sleep = 3 [default = 0.5];
  optional int32 flashes = 4 [deafult = 2];
}

message LightControlResponse {
  // Not really sure what the response should look like...status?
}

service LightControlService {
  rpc LightControl(LightControlRequest) returns (LightControlResponse);
}


Looking forward to feedback!
Micah

Tom Enos

unread,
Dec 23, 2014, 6:25:12 AM12/23/14
to lightsh...@googlegroups.com
Has anyone considered not reinventing the wheel?

This might also be an option to the socket interface.  One server that could work with many interfaces.

Todd Giles

unread,
Dec 23, 2014, 9:23:34 AM12/23/14
to lightsh...@googlegroups.com

Tom, took a quick look - how would you define a service using python-twisted on the server side but using other languages for building the clients? 

Micah Wedemeyer

unread,
Dec 23, 2014, 10:17:59 AM12/23/14
to lightsh...@googlegroups.com
Twisted definitely looks interesting. Building a good server is hard work, so using an existing one makes a lot of sense.

I thought twisted was only for web serving, but it looks like a perfect fit for a daemon like this.

Micah
--
http://www.lightshowpi.com/
---
You received this message because you are subscribed to a topic in the Google Groups "LightShow Pi Developers" group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/lightshowpi-dev/oW1wLcnEz7Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lightshowpi-d...@googlegroups.com.
To post to this group, send email to lightsh...@googlegroups.com.

Tom Enos

unread,
Dec 23, 2014, 2:57:11 PM12/23/14
to lightsh...@googlegroups.com
You could do it as a webapp, as one example.  I can do some more research and see if I can build a few examples.  

Tom Enos

unread,
Dec 23, 2014, 3:01:22 PM12/23/14
to lightsh...@googlegroups.com
I was looking for something that would run a function at set intervals, and came across it, then thought of this thread.  It could save a lot of time and work.  Twisted does a lot of stuff.  


On Tuesday, December 23, 2014 7:17:59 AM UTC-8, Micah Wedemeyer wrote:
Twisted definitely looks interesting. Building a good server is hard work, so using an existing one makes a lot of sense.

I thought twisted was only for web serving, but it looks like a perfect fit for a daemon like this.

Micah

On Tuesday, December 23, 2014, Tom Enos <tom....@overclocked.net> wrote:
Has anyone considered not reinventing the wheel?

This might also be an option to the socket interface.  One server that could work with many interfaces.

On Saturday, December 13, 2014 11:45:56 AM UTC-8, Micah Wedemeyer wrote:
With people building web UIs, phone UIs, and other interfaces, I was wondering if it might be useful to have a control daemon that listens on a socket. Clients could control the music, lights, and such by sending messages to the socket. A socket interface would allow writing clients in other languages besides Python, and just generally make it easier to integrate with the underlying lightshow codebase.

Any thoughts on this?

--
http://www.lightshowpi.com/
---
You received this message because you are subscribed to a topic in the Google Groups "LightShow Pi Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lightshowpi-dev/oW1wLcnEz7Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lightshowpi-dev+unsubscribe@googlegroups.com.
To post to this group, send email to lightshowpi-dev@googlegroups.com.

Micah Wedemeyer

unread,
Jan 11, 2015, 3:20:11 PM1/11/15
to lightsh...@googlegroups.com

I've been reading/thinking about this a lot more lately, and I have some more thoughts I'd like to run by the group.

Thrift

After looking over Protocol Buffers, Twisted, and various other communication methods, I found Thrift. It's very similar to Protocol Buffers, but goes the extra distance to provide a pre-made RPC layer. I experimented with creating a Thrift server to turn the lights on and off and had great success very quickly.

By using Thrift, we could create a server that listens for messages from clients written in any language that supports Thrift, and there are quite a few. So, I'm going to start working toward that end. I'll begin posting some Thrift definitions to get feedback.

Server / Daemon Structure

As part of daemonizing the code, I think there's an opportunity to refactor and separate some of the pieces a little. In particular, I'd like to re-envision the song playing as a pipes-and-filters approach. Instead of a single function that's in charge of doing everything (ie. play_song), it would be broken into smaller pieces that do a small bit of work and then put that into an output buffer that the next piece in the chain consumes. So, more like an assembly line where each worker does a small part that is then passed on to the next worker.

For example, here's a possible chain:

  • Reading the song/sound file (output is raw bytes of the mp3, or a byte stream)
  • Decoding the sound file (output is chunks of the decoded PCM audio)
  • Calculating the FFT/light channels for a chunk of audio (output is some kind of structure that contains the light channel settings as well as a chunk of the PCM audio data)
  • Playing the lights and audio

Each of those pieces would be its own little worker, and they would communicate via buffers that they read and write to.

A big plus of this is the ability to pull a piece out of that chain and replace it with something else that has the same inputs and outputs. For example, we could replace reading and decoding the mp3 file with another audio in source (like USB in or a Pulse Audio server) and leave the rest of the chain intact. Likewise, we could have alternate audio-to-light-channel analyzers that could be swapped in as well.

Finally, I would go so far as to make each worker a separate thread, and just ensure that the buffers used between them are thread safe. Even though the r-pi is a single core (right?) they will still work well in pseudo-parallel, and it will allow other work to be done when it's blocked on i/o, like when it's reading the audio stream or when it's actually playing the music and lights. Each worker would just continue consuming and filling its i/o buffers until one is empty or the other is full.

I'm going to start working on these items (Thrift and Daemon) on a fork. If anyone has feedback or thoughts, I'd love to hear it.


On Saturday, 13 December 2014 14:45:56 UTC-5, Micah Wedemeyer wrote:

Micah Wedemeyer

unread,
Jan 12, 2015, 12:15:59 AM1/12/15
to lightsh...@googlegroups.com
I've started with some refactoring of synchronized_lights toward the pipes-and-filters approach I mentioned earlier. Here's a commit that sort of gives an idea of where I'm heading: https://bitbucket.org/micahwedemeyer/lightshowpi/commits/de6579aa97969f52620a89f58a21bb2745a46afa?at=master

NOTE!!! This is very preliminary, it doesn't even work, but it's meant to show what I'm thinking.

Basically, I'd like to wrap all the audio sources in order to provide a consistent interface to them, similar to how decoded matches the wave interface. That way we could hopefully consolidate the audio_in and play_song methods.

In my example commit, I've added WavStream, DecodedStream, and PcmStream. All of these conform to a single interface, meaning they can be easily interchanged in the app, depending on what audio source you want to use. Plus, we could add PulseAudioServerStream or even MicrophoneStream if we wanted...

So, this would be the start of combining play_song and audio_in. There are some other differences there (like running until CTRL-C is pressed), but the basic idea is to make all input audio look homogenous and have a standard interface that the rest of the application expects.

I'm also working on breaking up the play_song method into smaller chunks. So far the result isn't much prettier, but I think I can continue cleaning it up.

Thoughts?
Reply all
Reply to author
Forward
0 new messages