Composite device with Serial, KB, Mouse, and Joystick (eventually the Keyglove)

[Jeff Rowberg]

I've been working on the Keyglove project for a while now (details at
http://www.keyglove.net). I started with the Arduino Mega, then the
Teensy, and now I've designed my own board based on the AT90USB1287.
I've been able to make good progress with the LUFA library, which
seems to be frankly pretty awesome.

I'm trying to pack a lot of stuff into the MCU, and I can't figure out
how to make everything work at the same time. In short, I'm looking
for the following USB functionality:

- Virtual serial port
- Keyboard
- Mouse
- Joystick

My goal is to have a KB/mouse/joystick that works generically without
any drivers. The serial port can require drivers if necessary. I
calculate a total of 6 endpoints (2 serial, 2 kb, 1 mouse, 1
joystick), so I should be within the limitations of the AVR chip. I
was able to make the kb+mouse+joystick work together, but not after I
added the serial port. I looked through all of the demos, including
the DualVirtualSerial one. I scoured the support list here with no
luck. I'm implementing the low level interface, not the class library,
for what it's worth. It still comes up in the Device Manager as a
single "USB Composite Device" under the "Universal Serial Bus
controllers" section. I haven't been able to find, create, or install
any .inf files that will make it work. I'm running Windows 7 64-bit.

So, two questions:

1. Is it possible have a KB/mouse/joystick that works generically
without any drivers, plus a serial port that may require drivers, all
in the same device?
2. Can anyone help? :-)

All of my current code is available here:

https://github.com/jrowberg/keyglove/tree/master/keyglove_lufa

I can't help thinking I just have one little ID wrong in Descriptors.c
or something. Any help at all would be greatly appreciated.

Jeff

[Dean Camera]

Jeff,

Nice project!

Looking over your descriptors, I see an issue that will prevent
enumeration:

In Descriptors.h, you have this:

typedef struct
{
/// virtual serial
USB_Descriptor_Interface_Association_t CDC_IAD;
USB_Descriptor_Interface_t CDC_CCI_Interface;
USB_CDC_Descriptor_FunctionalHeader_t CDC_Functional_Header;
USB_CDC_Descriptor_FunctionalACM_t CDC_Functional_ACM;
USB_CDC_Descriptor_FunctionalUnion_t CDC_Functional_Union;
USB_Descriptor_Endpoint_t CDC_NotificationEndpoint;
USB_Descriptor_Interface_t CDC_DCI_Interface;
USB_Descriptor_Endpoint_t CDC_DataOutEndpoint;
USB_Descriptor_Endpoint_t CDC_DataInEndpoint;

// keyboard
USB_Descriptor_Configuration_Header_t Config;
USB_Descriptor_Interface_t HID1_KeyboardInterface;
USB_HID_Descriptor_HID_t HID1_KeyboardHID;

// ........... (snip)
}

This is an issue; when you enable the virtual serial descriptors,
suddenly your configuration descriptor turns to garbage (according to
the host). Why? The main configuration descriptor header:

USB_Descriptor_Configuration_Header_t Config;

Will be placed *after* the Virtual Serial CDC interface descriptors,
so the host will not be able to read the configuration, and thus
immediately reject it. Moving that entry to be the first in the struct
fixes the enumeration issue for me, allowing all the interfaces to
enumerate.


Cheers!
- Dean

On Jul 1, 1:58 pm, Jeff Rowberg <jeffrowb...@gmail.com> wrote:
> I've been working on the Keyglove project for a while now (details athttp://www.keyglove.net). I started with the Arduino Mega, then the

[Jeff Rowberg]

On Jul 1, 12:43 am, Dean Camera <abcminiu...@gmail.com> wrote:
> Jeff,
>
> Nice project!
>
> Looking over your descriptors, I see an issue that will prevent
> enumeration

Wow, that was fast! And exactly what I needed (and was hoping for). I
had to change the endpoint indexes in the HID descriptor request
function so that wIndex matches the respective HID EPNUM definitions
as well, and then everything enumerated into the right categories.
I've pushed the code changes to the githup repository.

I have two follow-up questions:

1. Although the serial port shows with no errors in Device Manager, I
can't get any data using Realterm to test. I have a test string being
sent every 250ms or so in Keyglove.c, but no data shows up. Am I
missing something?

2. Is there any way to specifically name each of the devices so that
in Device Manager they show up as "Keyglove Keyboard" and "Keyglove
Serial Port" instead of "Standard HID device" etc.? That would be
awesome.

Thanks for the great help so far. I'll be happy to throw together a
detailed demo with working serial, kb, mouse, and joystick to help
others put together complex multi-interface USB devices with LUFA--
once I get it to work. :-)

Jeff

[Jeff Rowberg]

I've just made another change to the code structure and pushed it to
github. One main difference is that it is no longer serial, keyboard,
mouse, and joystick, but is instead just serial + generic HID. I'm
hoping that I can use this approach and a sufficiently flexible
generic HID report structure to have the same functionality with fewer
endpoints, less complex code, and more room for functional expansion.
Can I use a single generic HID device to work as a kb, mouse,
joystick, and potentially other HID functionality with the same
driverless OS-independent support?

The second main difference is in the timing of the virtual serial
debug code. I was using the _delay_ms() function to help provide a
visual indicator (with an LED) of what is happening on the board. But
then it occurred to me that the various USB tasks rely on rapid
polling, and delaying for 250ms at a time probably wasn't a good idea.
So I eliminated the delays and instead have now setup an incrementing
counter variable using a mod-10000 test. Every 10000 cycles, it
triggers the debug string to be sent to the serial TX endpoint and
turns on the LED. Then 1000 cycles later, it turns off the LED. At an
8MHz clock speed and the code I have now, this works out to be a
visible blink roughly once per second.

However, I'm still having problems receiving any data in Realterm over
the virtual serial port. Everything enumerates correctly, and the port
shows up as COM20. I can open it in Realterm, and I don't see any
visible errors, but I receive absolutely nothing on from the device,
and I can't understand why. The relevant code is in CDC_Task(),
currently on Line 237 of Keyglove.c:

https://github.com/jrowberg/keyglove/blob/master/keyglove_lufa/Keyglove.c#L237

Am I doing something wrong here?

Something interesting to note is that the "blink" LED pattern
controlled by the CDC_Task() function doesn't start until I actually
open the port. I can plug the device in, and it shows up in Device
Manager immediately, but nothing happens with the LED until the port
is open. I believe this is what's supposed to happen, since it knows
not to send data until the LineEncodingData stuff has been set by the
host. But if I *close* the port on the host, the device keeps sending
data. Maybe this is also supposed to happen, but it kind of seems like
it shouldn't. Maybe the BaudRateBPS variable doesn't get cleared
automatically when the port is closed.

That's less important to me at this point than just getting some
serial debug data flowing from the device to the host though. Without
that, moving forward on the LUFA-flavored Keyglove code will be very
difficult due to a lack of simple debugging ability.

[Jeff Rowberg]

On Jul 5, 12:22 pm, Jeff Rowberg <jeffrowb...@gmail.com> wrote:
> However, I'm still having problems receiving any data in Realterm over
> the virtual serial port. Everything enumerates correctly, and the port
> shows up as COM20. I can open it in Realterm, and I don't see any
> visible errors, but I receive absolutely nothing on from the device,
> and I can't understand why.

Woo hoo, I figured it out! When I changed over to the Generic HID code
instead of the discrete kb/mouse/joystick code, I didn't correctly
update the HID endpoints, so that the HID in/out endpoint addresses
(*_EPNUM) were overlapping with the CDC notification and TX endpoint
addresses. I made everything unique and sequential, reprogrammed it,
and voila! Debug info flowing to the host exactly as it should.
Beautiful.

Now I need to figure out how to use that single generic HID report
structure to accomplish kb/mouse/joystick control, and I'd still like
to know how to change the enumerated device names shown in Device
Manager, if that's actually possible.

[Dean Camera]

Great to hear Jeff! I've now finally got some more time to dedicate to
LUFA, and of course LUFA support - but I'm glad you managed to get
sorted out in the meantime.

> Now I need to figure out how to use that single generic HID report
> structure to accomplish kb/mouse/joystick control

It's possible to do all three functions, but your throughput per
function will suffer; at a maximum you can send *one* HID report per
USB frame, which means up to 1000 frames per second. You can multiplex
several functions over the same HID endpoint using seperate HID
"Report IDs" (see Demos/Device/ClassDriver/KeyboardMouseMultiReport/
for an example) for each function, but then you need to choose which
one to send for each frame. Doing a round-robbin approach (keyboard,
then mouse, then joystick, then keyboard again) is the simplest
method, but then each interface only gets 1000/3 updates per second.

You can probably approach this a little more intelligently so that the
functions that require more reports in a given second get more update
opportunities, but for best performance give each one a seperate HID
interface and endpoint.

> and I'd still like to know how to change the enumerated device names
> shown in Device Manager, if that's actually possible.

This should be possible by creating new string descriptors describing
each interface, adding them to the CALLBACK_USB_GetDescriptor()
callback function's string table. Assign the unique string index
values you use there to each interface as the .InterfaceStrIndex
element, and the host should see the interface with a string
descriptor index, fetch that string, and use that as the interface's
description.

Cheers!
- Dean

[Jeff Rowberg]

On Jul 6, 1:32 am, Dean Camera <abcminiu...@gmail.com> wrote:
> Great to hear Jeff! I've now finally got some more time to dedicate to
> LUFA, and of course LUFA support - but I'm glad you managed to get
> sorted out in the meantime.

I've made more good progress, and in the process I created a "LUFA/
Demos/LowLevel"-friendly code structure which you are welcome to use,
as promised. It's a low-level implementation of a virtual serial port
combined with a generic HID and a multi-report implementation of a
keyboard, mouse, and joystick. It requires no hardware buttons and has
demo code to show all of the features working, including local echo on
the serial port and a ring buffer for TX and RX data.

https://github.com/jrowberg/keyglove/tree/master/keyglove_lufa/LUFADemo

Check it out, make sure it conforms to your code specifications, and
by all means include it with your demo code if you'd like. It should
be very helpful for other people who want to do something similar.

Thanks for a really great framework!

Jeff

[Dean Camera]

Jeff,

Great work! Unfortunately I'm trying to retain the copyrights to the
LUFA demos where I can now as a matter of policy (to keep my options
open regarding relicencing, etc. in the future) but I'd be more than
happy to add it to the LUFA Projects/ directory for you, where third
party demos and project sit.

A few tweaks/suggestions:

1) Try to use the C99 types (uint8_t, uint32_t, int32_t, etc.) where
possible. Not only does this make your code more portable/readable in
the long term, it will also reduce the code size usage and prevent
some subtle bugs. Those "long" counter variables you have will be
signed 32-bit integers on the AVR8 chips, which are quite inefficient.

2) I suggest you alter the report decision selection code so that the
report sent is selected based on which report was last send (i.e.
switch between then in the HID task each time a report is sent). This
will give even bandwidth between then, and will tighten the code a
little. That way your main routine just switches between sending a
serial string and sending a HID report.

3) Don't forget to add your name to the copyright header before I put
it in!


Cheers!
- Dean

[Jeff Rowberg]

Hi Dean!

I've been busy with the Keyglove code for a while making many changes
and improvements (which I've just pushed to the repository), which is
why it's taken me the better part of two weeks to reply.

>  1) Try to use the C99 types (uint8_t, uint32_t, int32_t, etc.) where
> possible. Not only does this make your code more portable/readable in
> the long term, it will also reduce the code size usage and prevent
> some subtle bugs. Those "long" counter variables you have will be
> signed 32-bit integers on the AVR8 chips, which are quite inefficient.

I've applied your first suggestion (about C99) types to both the demo
code and my whole project for consistency--thanks! I was until now
unfamiliar with the C99 type set, at least in practice. I'm sure I'd
seen them before, but I never paid attention and never learned them.

> Great work! Unfortunately I'm trying to retain the copyrights to the
> LUFA demos where I can now as a matter of policy (to keep my options
> open regarding relicencing, etc. in the future) but I'd be more than
> happy to add it to the LUFA Projects/ directory for you, where third
> party demos and project sit.
>

> ...

>
>  3) Don't forget to add your name to the copyright header before I put
> it in!

As for this, it's fine with me if you just keep the whole demo code
and all possible copyrights. I've made some slight modifications
(which I intended to do before I posted it anyway), and my name is now
entirely gone from that bit of code. It's just a demo, exactly like
the other demos, and honestly I'd rather have it be entirely yours so
you can modify/fix it however you like. Unless you really, really
don't want it, it's 100% yours. That's the least I can do to help such
a great project.

>  2) I suggest you alter the report decision selection code so that the
> report sent is selected based on which report was last send (i.e.
> switch between then in the HID task each time a report is sent). This
> will give even bandwidth between then, and will tighten the code a
> little. That way your main routine just switches between sending a
> serial string and sending a HID report.

This is an interesting concept which I don't think I fully understand
yet (especially in the much larger Keyglove project, let alone the
Demo code). One of the things I'm trying to do with the Keyglove code
is to make it as efficient as possible (duh). In light of this, is it
possible to let the USB use some kind of interrupt/event-driven
communication such that I don't have to call the CDC_Task() and
HID_Task() functions all the time when there isn't something specific
to send? I know the HID interface can receive keyboard LED reports
back from the host, and the CDC interface obviously receives serial
data from the host; neither of these events will come at predictable
times, so they have to be handled by either polling or interrupts on
the device side.

I see the HID functionality in what appears to be an interrupt/event-
driven bit of code inside the EVENT_USB_Device_ControlRequest
function. If this is called and processed automatically by the USB
library code as needed, would it therefore be safe to take HID_Task()
out of the main loop, since it's called specifically in other places
in the code right at the moment any given kb/mouse/joystick HID report
should be sent?

And, if so, is the same kind of functionality available for handling
CDC tasks?

Any pointers would be great. Actually, come to think of it, pointers
can be confusing sometimes. Maybe stick with a solution that doesn't
involve pointers. (kidding) And if there's some documentation or
discussion about this topic already, please feel free to direct me
there instead of explaining something again.

Thanks!

Jeff