Game controllers?

[James Petry]

I watched Sam Aaron's latest Overtone video, very inspiring, many thanks!

In fact it inspired a bit of a mad thought.

Looking at the different kinds of controller devices in the video, I am wondering about all the buttons,

triggers and sticks on the USB XBox controller that is plugged into my home PC.

I'd like to get them connected somehow into Overtone.

Has anyone done anything like this?

If not, I'd like to have a go.

I mainly code in java and C# at work but we are starting to use clojure too. Haven't coded with Overtone yet at all.

A quick search shows there's a project called JInput for reading game controllers in java.

[Jeff Rose]

Hi James,
It would be great to have game controller support for manipulating synths and other stuff in Overtone. If you get something put together let us know, and feel free to ask here if you run into problems. I did experiment with JInput in the past to hookup with a playstation rock band drum kit, and it worked fine. Unfortunately I can't find that code right now, but I'll look around.

-Jeff

[Richie Cyngler]

SuperCollider's communication is built on OSC, I'm not sure how Overtone interacts with OSC but I'm guessing you should be able to get Wiimotes talking to SC via OSC using Overtone as the interface? (Someone please fill in the huge gaps in knowledge here).

--
Richie

www.glitchpop.com

[Jeff Rose]

Overtone speaks OSC to SuperCollider, but I think for game controllers we'll want direct communication from the underlying Java library to Overtone by registering a handler function that receives some kind of simple controller event map. That way we can use it to control anything, from sequencers to synthesizer controls or other abstract musical controls in clojure.

-Jeff

On Friday, February 24, 2012 at 9:34 PM, Richie Cyngler wrote:

> SuperCollider's communication is built on OSC, I'm not sure how Overtone interacts with OSC but I'm guessing you should be able to get Wiimotes talking to SC via OSC using Overtone as the interface? (Someone please fill in the huge gaps in knowledge here).
>

> On Fri, Feb 24, 2012 at 9:25 PM, Jeff Rose <ros...@gmail.com (mailto:ros...@gmail.com)> wrote:
> > Hi James,
> > It would be great to have game controller support for manipulating synths and other stuff in Overtone. If you get something put together let us know, and feel free to ask here if you run into problems. I did experiment with JInput in the past to hookup with a playstation rock band drum kit, and it worked fine. Unfortunately I can't find that code right now, but I'll look around.
> >
> > -Jeff
> >
> >
> > On Thursday, February 23, 2012 at 10:59 AM, James Petry wrote:
> >
> > > I watched Sam Aaron's latest Overtone video, very inspiring, many thanks!
> > >
> > > In fact it inspired a bit of a mad thought.
> > >
> > > Looking at the different kinds of controller devices in the video, I am wondering about all the buttons,
> > > triggers and sticks on the USB XBox controller that is plugged into my home PC.
> > >
> > > I'd like to get them connected somehow into Overtone.
> > >
> > > Has anyone done anything like this?
> > >
> > > If not, I'd like to have a go.
> > >
> > > I mainly code in java and C# at work but we are starting to use clojure too. Haven't coded with Overtone yet at all.
> > >
> > > A quick search shows there's a project called JInput for reading game controllers in java.
>
>
>
> --
> Richie
>

> www.glitchpop.com (http://www.glitchpop.com)

[James Petry]

As I'm not an idiomatic clojure programmer yet, and completely new to Overtone and OSC,

I'll build a throw-away bridge from my gamepad to OSC just to get working code.

It looks like I could write it in java for now using JInput and java-osc for the client, talking to osc-clj.

I've found some useful code to look at in Chapter 11 "Building a Game Pad Controller with JInput" of "Pro Java 6 3D Game Development",

which is available free on the author's website:

http://fivedots.coe.psu.ac.th/~ad/jg2/ch11/index.html

[James Petry]

On Saturday, February 25, 2012 8:37:29 AM UTC, James Petry wrote:

As I'm not an idiomatic clojure programmer yet, and completely new to Overtone and OSC,

I'll build a throw-away bridge from my gamepad to OSC just to get working code.

It looks like I could write it in java for now using JInput and java-osc for the client, talking to osc-clj.

Managed to get the java bridge working over the weekend. Most of the time was taken getting maven to work with 

the native libraries that are part of JInput.

Moving on to clojure, it was much easier getting Leiningen to do the right thing, because Sam Aaron had already

put JInput on clojars in the correct format. Many thanks!

I've got the basics of reading the controller in clojure going now. Still need to write the polling loop more elegantly and get the OSC bundles flowing. That's just a question of time though, no blockers at the moment.

[James Petry]

On Friday, February 24, 2012 10:28:22 PM UTC, Jeff Rose wrote:

Overtone speaks OSC to SuperCollider, but I think for game controllers we'll want direct communication from the underlying Java library to Overtone by registering a handler function that receives some kind of simple controller event map. That way we can use it to control anything, from sequencers to synthesizer controls or other abstract musical controls in clojure.

I've pursued the OSC route just for now as a learning exercise, as it's all new to me.

What I've learned so far:

The JInput library lets you fetch the whole state of the controller at once and then read the values of individual components.

On the XBox controller there is some chatter. For example, pressing a button often triggers the analog sticks too.

JInput does not detect the rumbler on the XBox controller :-(

[James Petry]

On Friday, February 24, 2012 9:34:05 PM UTC, Richie wrote:

SuperCollider's communication is built on OSC, I'm not sure how Overtone interacts with OSC but I'm guessing you should be able to get Wiimotes talking to SC via OSC using Overtone as the interface? (Someone please fill in the huge gaps in knowledge here).

 

I've done some searching, and it looks like java support for Wii remotes has been done but some of the code is now AWOL from the internet.

Wii remotes talk over Bluetooth and aren't supported by JInput by default.

I've got a Wii and a Bluetooth dongle so I could have a look at this some time.

It would certainly be interesting to find an abstraction over JInput devices and Bluetooth devices.

[Iain Wood]

Just a thought...

http://store.neurosky.com/products/mindwave-1

I doubt I can get one in this part of the world, but maybe next time I'm in the UK. Looks like there is already a pd/Max interface for it on github: 

https://github.com/qdot/np_mindset/

[James Petry]

Quick update, I've got the Satie example playing with the left and right shoulder buttons of an XBox controller using JInput and overtone.osc. I swapped in the (piano) for the sampled piano just to get some sounds for now.

[Tom Oinn]

Good stuff :)

We were talking a month or so ago [1] about a more general controller
abstraction, we came up with a list of hardware devices we'd like to
support and I suppose we should be adding game controllers to it -
everything else on the list is conventional (or not so conventional)
music production hardware. I should have a bit more time to actually
try to code some of the support up, will post more when I have
something working.

Tom

[1] https://groups.google.com/d/topic/overtone/o-ce252vRC0/discussion

[Richie Cyngler]

That EEG helmet looks mental! (arf arf), but I'd love to see one in action actually.

If you are on OSX http://www.osculator.net/ is great for routing OSC for things like Wiimotes and TouchOSC, etc. It's not FLOSS, but very reasonably priced and excellent and reliable software (unpaid endorsement).

cheers

--
Richie

www.glitchpop.com

[James Petry]

[1] https://groups.google.com/d/topic/overtone/o-ce252vRC0/discussion

Very useful doc. I should add to the "Types of physical device" section for game controllers to cover JInput notions like hats. The notion of buttons looks fairly similar already :-) 

[Tom Oinn]

Please do - I'd expect most game controller inputs to map to either
buttons, faders or endless encoders but you never know!

Tom

[James Petry]

Spent some time at the weekend reading overtone source, especially midi-clj, and cleaning up my code.

Here's my top-level server-side code using Satie now:

(defonce xbox (find-controller "XBOX"))

;; A ThrustMaster joystick has a plural name

(defonce stick (find-controller "USB Game Controllers"))

(defonce server (osc-server PORT))

(controller-handlers

  server

  xbox

  { BUTTON4 #(when (button-pressed? %) (play-next-lh 1))

    BUTTON5 #(when (button-pressed? %) (play-next-rh 1))

    BUTTON0 (fn [msg] (reset-pos))})

(controller-handlers

  server

  stick

  { BUTTON1 #(when (button-pressed? %) (play-next-lh 1))

    BUTTON2 #(when (button-pressed? %) (play-next-rh 1))

    BUTTON0 (fn [msg] (reset-pos))})

[James Petry]

On Friday, February 24, 2012 10:28:22 PM UTC, Jeff Rose wrote:

Overtone speaks OSC to SuperCollider, but I think for game controllers we'll want direct communication from the underlying Java library to Overtone by registering a handler function that receives some kind of simple controller event map. That way we can use it to control anything, from sequencers to synthesizer controls or other abstract musical controls in clojure.

Now I've got reasonably far with the OSC route, I'd like to look at this direct approach. Is there something already in Overtone that works this way, so I can read the code?

[Tom Oinn]

I don't know if it's the 'official overtone way', if there is such a
thing, but the examples I posted recently using my audiocube library
work for me and seem elegant enough. The API is at
https://github.com/tomoinn/overtone.device.audiocubes and an example
project using it to modulate synth properties at
https://github.com/tomoinn/audiocube-overtone-demo.

I don't think Jeff is referring to a specific overtone API (please
correct me if I'm wrong!) and more the general approach of having a
simple callback mechanism that can be used to glue your device
specific code into other bits of overtone.

The google doc I linked refers to a more general approach (and would
then *be* such an API) where an additional abstraction layer in the
form of a virtual console (in the 'recording studio' rather than
'terminal' sense) is interposed between physical hardware and consumer
code, but we don't have that yet.

Tom

[Jeff Rose]

The midi support works like this. Checkout
https://github.com/overtone/overtone/blob/master/examples/midi.clj to
see how a handler function is passed to midi-handle-events.

-Jeff

[Jeff Rose]

I think it would be ideal if the low level communication API worked
along the lines of the midi interface, where a handler function is
called with an event map. Then on top of this, we can build out some
more generalized abstractions for components like button grids, sets
of sliders / dials, keyboards, velocity sensitive drum pads for
percussion, etc. Seem reasonable?

-Jeff

[Tom Oinn]

On 6 March 2012 12:43, Jeff Rose <ros...@gmail.com> wrote:
> I think it would be ideal if the low level communication API worked
> along the lines of the midi interface, where a handler function is
> called with an event map.  Then on top of this, we can build out some
> more generalized abstractions for components like button grids, sets
> of sliders / dials, keyboards, velocity sensitive drum pads for
> percussion, etc.  Seem reasonable?

I think the API should be a bit smarter. MIDI has its own issues in
terms of resolution, but that's by the by and is largely fixed with
OSC. The more important point is that we want to track the state of
virtual controls, and both MIDI and OSC are event based protocols. I
think the API should be trapping these events and using them to both
call interested parties and update state of a software proxy to the
hardware and of an intermediate layer between the representation of
the actual device and the client code.

We need to handle and store state so that if I create e.g. a virtual
grid of 100x100 buttons which can be toggled, and want to surface that
onto a grid of 8x8 physical buttons + scroll keys (i.e. a launchpad or
small monome) there has to be somewhere to maintain the state of the
controls which aren't currently being made available so that when we
scroll across we can update the hardware appropriately. Client code
would bind to events from the 100x100 grid, and a mapping function
would manage the translation in both directions of events between the
client, through the virtual console, through the hardware's proxy and
to and from the actual device.

Given this someone could write code which required, say, a 4x5 button
grid, three rotary encoders and four faders, and depending on what
hardware you actually had on hand (degenerate case - mouse and screen
or keyboard) you could map one or many physical devices into this
virtual console. Those of us with home studios and lots of space get
to play with all the hands on controls, someone on a laptop still
accesses the control functionality through an on-screen or keyboard
mapped equivalent. It's also nicely modular in terms of effort, once
we have the virtual console protocols and types the work required to
add a new hardware device is a distinct package and can be done by
someone who actually owns one.

Tom

[Kristoffer Ström]

Have you looked at the supercollider HID support?

example here: http://whatsinthestew.blogspot.com/2006/09/week6-supercollider-hid.html

[Jeff Rose]

Yeah, I completely agree with you. I hacked a simple bit of state
into synths and instruments in order to modify their default parameter
values with GUI controls, but after reflecting on it I think we really
want to have this as a separate piece of functionality. The
synthesizer should be a simple specification of the directed graph of
ugens, and then on top of it we need machinery to store state and
functions that lookup this state when triggering the synth. Ideally
we could define groups of state, including named parameters with
documentation, initial values, range, stride (linear, exponential,
etc.), and probably other metadata. This would then let us map GUI or
physical controls onto the state, and we can have a library of
functions that help to do things like map from linear to logarithmic
within a certain range, or support scrolling subsets of controls like
you describe.

This is on top of the simple event layer though, rather than built
into it, because in many cases it is useful to have a raw interface to
easily hook into without any repercussions.

-Jeff

[James Petry]

On Tuesday, March 6, 2012 12:43:39 PM UTC, Jeff Rose wrote:

I think it would be ideal if the low level communication API worked
along the lines of the midi interface, where a handler function is
called with an event map.  Then on top of this, we can build out some
more generalized abstractions for components like button grids, sets
of sliders / dials, keyboards, velocity sensitive drum pads for
percussion, etc.  Seem reasonable?

I've made changes to my OSC code to try this out quickly.

[ The OSC server is replaced with an atom holding a map of message paths to handler functions. The client code is replaced with a dispatcher function which makes the controller values look enough like osc-clj message maps to invoke my existing example handler functions. ]

There is a difference from the midi code though, a push vs pull issue.

The javax.sound.midi code pushes midi messages to "listeners" (Receivers).

JInput-based code works by pulling, there is a .poll() method on each controller.

Currently I do the polling inside a loop-recur with a Thread/sleep. The loop is run in a future, just to put it into a background thread.

I'd like some guidance on how to improve this from more experienced clojurians please.

[Jeff Rose]

Yeah, I read that JInput used a polling interface because it maps to
the low level interfaces they connect with, and for many games that
operate in a run loop, it seems a sensible strategy. What you
describe, using a recursive loop with a short sleep call, should work
fine for Overtone though. Something like a 20-30ms sleep time should
result in controls that feel instantaneous.

-Jeff

[James Petry]

With the release of Overtone 0.7, I'll pick up the game controller stuff again and get it going with the global event stream.

[Sam Aaron]

Hey James,

On 27 Jun 2012, at 08:51, James Petry wrote:

> With the release of Overtone 0.7, I'll pick up the game controller stuff again and get it going with the global event stream.

That would be fantastic! Let me know if we can help in any way. I guess you'll be using jinput?

Sam

---
http://sam.aaron.name

[James Petry]

On Wednesday, June 27, 2012 8:59:43 AM UTC+1, Sam Aaron wrote:

> With the release of Overtone 0.7, I'll pick up the game controller stuff again and get it going with the global event stream.

That would be fantastic! Let me know if we can help in any way. I guess you'll be using jinput?

I'd like to use lein 2, and at one point I did the "unrolling" of jinput.jar in jinput-0.0.2.jar from clojars for my local copy. My environments have got a bit out of line, but I *think* the one on clojars is still jar-within-a-jar. Any chance of unrolling it and bumping the number please? Or let me know to clear my caches properly :-)

[Sam Aaron]

Oh, sure, I'll get onto it within the next day or so.

Sam

---
http://sam.aaron.name

[Sam Aaron]

I just threw up the latest Jinput nightly:

https://clojars.org/jinput

let me know if it works for you.

Sam

---
http://sam.aaron.name

On 27 Jun 2012, at 11:50, James Petry wrote:

[James Petry]

On Wednesday, June 27, 2012 12:32:16 PM UTC+1, Sam Aaron wrote:

I just threw up the latest Jinput nightly:

https://clojars.org/jinput

let me know if it works for you.

Yes, that's brilliant, many thanks, Sam. 

[Sam Aaron]

Awesome.

Also, this is something I put together quite a while ago now:

https://github.com/samaaron/space-navigator/blob/master/src/space_navigator.clj

it may be of some use - ignore if if not :-)

Sam

---
http://sam.aaron.name

[James Petry]

On Wednesday, June 27, 2012 8:38:45 PM UTC+1, Sam Aaron wrote:

Also, this is something I put together quite a while ago now:

https://github.com/samaaron/space-navigator/blob/master/src/space_navigator.clj

it may be of some use - ignore if if not :-)

 Thanks, I wish I'd found that before I started! :-)

I'm moving my code over to the latest overtone.libs.event, so it isn't working right now but

I've set it up on Github anyway with the rather original project name jinput-overtone :-)

[Sam Aaron]

On 27 Jun 2012, at 22:35, James Petry wrote:

> I've set it up on Github anyway with the rather original project name jinput-overtone :-)

Do you have a url for the project?

Sam

---
http://sam.aaron.name

[James Petry]

Do you have a url for the project?

Sure,

http://github.com/gavilancomun/jinput-overtone 

James

[James Petry]

I've added my percussion-toy.clj test code to the project, and uploaded to clojars

and github.

It is not a marvel of musical invention, but it lets me add more cowbell to Frere Jacques!

With the code working again on Windows with the latest lein, Overtone etc, my aim is to

get it working as-is on Ubuntu next. Then some serious simplifying / generalising and

documenting.

James

[Sam Aaron]

Awesome!

For those looking for a link it's here:

https://github.com/gavilancomun/jinput-overtone/blob/master/src/jinput_overtone/percussion_toy.clj

Keep up the excellent work James. We could do with a generalised library for working with game controllers for Overtone.

May I suggest you consider using the event stream? If you were to broadcast events via the event stream, then multiple clients could listen in and act on the events - similar to how MIDI instruments currently work.

Sam

---
http://sam.aaron.name