WVR Arduino library use case for Star Trek model

[Robert Petersen]

I am interested in using WVR to control a spacecraft model I am building.  It is a 3' long Star Trek Enterprise 1701-D.  The model has around 100 WS2811 LEDs inside, and a small amplified speaker.

I want to control the Sounds and LEDs in the model using an embedded controller like the WVR running Arduino code, and operate the model features via an Android application running on a tablet.  The Android app currently uses WebSockets to control the embedded WVR.

The WVR can certainly handle the minimal polyphonic requirements of the model's audio.  I would need it to continuously loop an engine sound at a speed and volume sent over WebSockets.  It would then also need to play phaser and torpedo sound effects when told to do so over the WebSocket.

In order for this to work as I envision, I would need the following capabilities:

1)  WVR runs at the same time as the Arduino sketch I download (I understand how this works now from the previous thread)

2)  My Arduino sketch needs to control a string of WS2811 LEDs through one of the AVR output pins.  I see that the NeoPixel driver is included in WVR, but is it exposed to my Arduino sketch?  I would need to be able to tell the WVR code to not use some of the pins.

3)  It would be nice if I could also import and use FastLED in my Arduino code.  Theoretically this should not be a problem since it is just another Arduino library, unless WVR is using some of the same resources that FastLEDs needs.  In particular any DMA or I2S resources.

4)  My Arduino code needs to be able to trigger the WVR to play / pause / and set volume and speed of its downloaded audio files.  So the WVR Arduino Library would need to expose these as functions that I could call.

5)  My Arduino code needs to be able to either expose it's own additional WebSocket (to my Android tablet), or hook into the WVR WebSocket and see it's commands.  If the latter, the WVR would get WebSocket messages from my Android and need to pass them on to my Arduino code as an event.

6)  My Arduino code would need to be able to monitor a couple of input pins for some buttons that are on the base of my 1701-D model.  These buttons fire phasers and torpedoes (in addition to being able to be fired over WebSockets).  So I would need to tell the WVR to ignore a few more pins.

Does this all sound reasonable?

[WVR Audio]

Very cool :)

This mostly sounds reasonable!

WVR uses one i2s peripheral for its Audio, I know esp32 has 2 i2s channels, but I do not know if they can both be outputs.
An alternative to the i2s peripheral is using the RMT peripheral to drive WS2811.

To connect to an Android App via websockets, you would have to do some work, that is not setup ergonomically in WVR yet, but I am happy to help on this.

Setting volume of a looped sample is accomplished using standard MIDI commands, so that will not be hard.

Setting the "speed" of playback may be a challenge. WVR doesn't have the ability to pitch-shift. It can pitch interpolate, which provides discrete pitches, uploads them to separate memory slots, and makes them available as midi notes. That is a functionality of the web GUI. Depending on what effect you need it may be possible to achieve with some tricks.

Setting input pins to trigger sounds is a base functionality of WVR, so that can be accomplished without code.

Note that Web Midi could be an out-of-the-box replacement for websockets. If that interests you I can expand on that possibility, and there is a section on Web Midi in the documentation. This would be an out-of-the-box way to trigger sounds, control volume, and do any MIDI stuff remotely over WiFi.

Let me know how that all sounds, I will be away from the office for a few days, but available again on Sunday.

[WVR Audio]

Sorry Robert, I looked into the web midi code and my advice makes no sense! It's been a while since I worked on this code!

Websockets are the perfect approach for you, they are actually transformed directly into MIDI messages and parsed like any other MIDI source.

I will outline some parts of wvr_thames.ino that will come in handy for you to reference, but let me know if any of it is going over your head, some of the style I used for this code is not typical Arduino style programming.

line 285: wvr.setMidiHook() tells WVR to pass any incoming midi to a handler function, before acting on it. In this case the handler is midiHookKeyboard()

line 230: midiHookKeyboard gets a pointer to the midi data, and is allowed to modify it before returning.

line 206: the function sustain() checks what kind of a message it is, and may erase it if it is a NOTE_OFF. Using this approach you could check if the incoming midi is a particular note, and if so, turn on some LEDs, or whatever you like, and just return and leave the message untouched. If you prefer to have a separate message to control the lights, you could use a MIDI CC. midi_in.h defines the MIDI_CC vales that WVR currently recognizes, but you could use any other CC value, watch for it, and respond as you like.

All this assumes that you have a working knowledge of MIDI protocol. If you do not, it may take a minute to get your head around, it is a fairly old protocol, and a bit quirky, but there are lots of tutorials out there to help break it down. Accomplishing everything you want to do without using any MIDI is totally possible as well, there is wvr.play(uint8_t voice, uint8_t note, uint8_t velocity) and friends, that can do everything in a more direct fashion. It would not be hard to add some code to server.cpp, at line 85 the websocket data is present, and could be parsed and handled in any way that is convenient. There are the remnants of a JSON based rpc framework that I ended up abandoning there as well to inspire how it could look.

rgb.cpp exposes 2 functions rgb_init() and rgb_set_color(), but both are designed for one LED only. I have not experimented with more, but I'm sure the adafruit library has that included. If it is too slow, and if the i2s idea doesn't work, I could setup the RMT method, and expose that in the wvr class, I have all that code setup in another project and it works very well. Here is how that looks.

[WVR Audio]

FYI I just ran a fresh install of the whole toolchain, because I had not tested this in a very long time.

I installed the brand new Arduino IDE 2.1.0, and installed the newest ESP32-Arduino toolchain (version 2.0.8) and everything went smooth, and compiled first time.

phew :)

[Robert Petersen]

Thanks for all the support!  I will setup my system and get it all compiling over the weekend.

[WVR Audio]

great! no problemo.
I am going to setup a platformIO config for the project as well, in case you are using PIO.

I have been loving PIO lately, and it would make getting all the dependencies installed correctly a lot easier for new WVR users.

I should have that ready by monday.

[Robert Petersen]

I have the most recent versions of the Arduino environment, ESP32 Wrover Module, WVR libraries, and FastLED libraries all downloaded and compiling wvr_basic.ino without errors.  I did run into 2 issues:

1)  Your excellent WVR Arduino IDE setup instructions say to select the ESP32 board package version 2.1.  That option was not in the list of the Arduino IDE.  The highest ESP32 version was 2.08, so I installed that one.

2)  There were some Pin #define collisions between WVR and FastLED, so I had to add the following code before FastLED.h:

#undef T1

#undef D11

#undef D12

#undef D13

#include <FastLED.h>

I have ordered some of the WVR boards and will try running on the hardware as soon as it comes in.

I do not look forward to figuring out how to give some pins to WVR and some to FastLED.

[WVR Audio]

1) Right, 2.1 is a typo, sorry, it should read 2.0.1. However I am now testing 2.0.8 and it seems to work very well.

2) Although your solution may be ok, I am not sure why FastLED would need #defines for those pins, so I feel suspicious there could be a bigger issue. You can use wvr.resetPin(uint8_t pin) to remove all the settings and interrupts that WVR attaches to any pin, after running wvr.begin(), you can see this at work in the example file wvr_thames.ino, where I manually setup the hardware for 4 LEDs and 2 foot switches, at line 55.

I do have a board setup with a single WS2811, so I can run some tests and help debug if you run into issues with FastLED.

Really looking forward to seeing WVR run the enterprise, this will be amazing.

[Robert Petersen]

While waiting for parts to arrive I have browsed all the code and it all looks pretty straightforward to me.  I can't wait to get started once the hardware arrives!

1)  Do you still offer the WVR Makers Board?

2)   Using midiHookKeyboard seems like an excellent way for me to get my commands from the WVR webserver.  Very elegant way of allowing users to add their own behavior.  Thanks for pointing me in the right direction.

3)  I quickly browsed your wvr_ui code, and it seems like sending "Midi commands" to my code just boils down to opening a regular websocket to `ws://${WVR_IP}/ws` and sending JSON strings where the only thing I really change is 3 bytes:  Byte 1 = (channel & code), Byte 2 = Note, and Byte 3 = Velocity.

[WVR Audio]

1) I have about a dozen on hand, and would be very happy to sell them if you're interested. I can also publish the kicad files and fabrication files, they should be open source I just am a bit disorganized.

2) Thanks! void WVR::setMidiHook(void (*fn)(uint8_t *in)) is the one.

3) yes! It's not JSON, it's actually a Uint8Array. webMidi.js line 45 has a note about this, and some commented out logs that may help you test it out. The cross-over from JavaScript to C is a weirdly well-defined and elegant relationship, I have come to realize, so MIDI messages, JavaScript, web sockets, and C code can all be friends in a very efficient way!

[WVR Audio]

If you are comfortable with Git, I would suggest that you clone the WVR repo into your libraries folder, rather than using the release .zip files. That would allow me to push changes if you encounter any bugs in all this. If not, that's ok too, I can continue to publish .zip releases.

You may also want to get familiar with the protocol for MIDI CC messages, which differ a bit from note-on/note-off messages.

[Robert Petersen]

- I have cloned the repo

- I am looking into MIDI CC messages.  I am thinking of using the block of 102 - 119 of the undefined Midi CC list, unless you are already using some of these.

[WVR Audio]

Nope, WVR does not use those. Actually the complete list is here: https://github.com/marchingband/wvr#midi-control

[Robert Petersen]

Hi Andy,

I have some time to work on this project again.  Thanks for adding the local wifi feature to WVR.  It worked the first time I tried it!

Please let me know when you have updated the main repository with the local wifi feature so I can update my copy and start adding the code to trigger LED effects with Midi commands.

Thanks!

Bob

[WVR Audio]

Great!

Are you comfortable using git? it is in a branch called ap-station-mode

[WVR Audio]

https://github.com/marchingband/wvr/tree/ap-station-mode

[Robert Petersen]

I have successfully connected to the WVR via WebSockets.

Upon initial connection I get the following messages from the WVR:

ws_client 3

firmware v3.8.3

So the connection looks valid.

I am sending simple 3 byte messages with the following code:

Uint8List midiMessage = Uint8List(3);
midiMessage[0] = 0x90; // Note_on
midiMessage[0] += 0x00; // Midi Channel number (WVR currently set to OMNI)
midiMessage[1] = 43; // Note number Per WVR ui
midiMessage[2] = 20; // Velocity (Changing value seems to just change volume?)
webSocket.add(midiMessage);

The WVR is playing the notes I pick, including playing multiple notes at the same time.

I do not get any replies or messages back from the wvr.  I guess Midi over WebSockets does not have any high level handshaking, which is certainly fine.

Next I will start adding my own midi commands to be handled by the midiHookKeyboard we discussed earlier.

It is going really well!

[WVR Audio]

Great! So this looks like Dart code, are they coming from a Flutter mobile app?

This is awesome, I have only used React Native for this purpose in the past, great to know Flutter works well too!

[WVR Audio]

And yes, velocity is just the MIDI word for volume.
No handshake needed from user land, if there is one, it is handled in the underlying protocol drivers that build the connection.
There are examples in wvr_thames.ino on how to use the midiHook function to react to incoming midi events.

Good luck!