Kinect for Windows
jeff kramer
-Jeff
Abdoulaye Camara
Sent from my iPhone
jeff kramer
drew.m...@gmail.com
First off, a call for help: if you have a K4W and a hardware USB
sniffer, I would appreciate USB logs of the device entering and using
near mode. If higher resolutions on the RGB camera are supported, I'd
love a log of that too. Feel free to ping me offlist if you don't
want thousands of people all downloading giant USB log files and using
all of your bandwidth.
Now: it turns out the K4W device is (from a protocol level)
practically identical. You have to set the K4W to use an alternate
USB configuration, and the productIds are different, but other than
that, all the rest of the code works as-is. The patch to get data
streaming, registration and all, was about 50 lines. It was somewhat
hackish, but for a proof of concept: very promising.
However, I have two significant concerns.
The first is that device enumeration is starting to get really
confusing. The K4Xbox has three subdevices: motor, camera, audio.
The K4W has only the camera and audio devices. There's no good
heuristic for telling which subdevices are grouped together besides
enumeration order (which libusb doesn't make any guarantees about).
As a bonus, the K4W has a serial number of 0000000000000000, so we can
probably no longer rely on freenect_open_device_by_camera_serial() to
distinguish between multiple K4W. This sucks from an design
standpoint, though it may not be a significant problem in practice
(anyone out there using multiple K4W with libfreenect? Color me
surprised).
The second is that since we no longer have a motor device at all, all
of the tilt-, motor-, and LED-related function calls make no sense on
the K4W. Even more problematic is that:
- our API expects the user to specify which subdevices should be opened
- the user can't (currently) tell programmatically before attempting
to open a device whether a device is a K4W or K4Xbox (this can
probably be fixed by adding to freenect_device_attributes)
- our API defaults to trying to open the Motor device and Camera device
- out API returns failure if it fails to open any of the requested
subdevices (like, say, the motor device, if you're trying to open a
K4W with the default subdevices).
The upshot of all this is that it is not currently possible to deal
with both the K4W and the K4Xbox without changing existing function
behavior or changing the enumeration API. My best idea so far is to
put a mask of the discovered subdevices in the
freenect_device_attributes struct, and punt picking the right
subdevices to the user, and then stub out the LED/motor/tilt functions
for the K4W. This will require users to start using
freenect_list_device_attributes() (or stop opening the motor device)
if they want to support the K4W.
I have no idea how to fix freenect_open_device_by_camera_serial()
given that the serial numbers on the K4W are utterly useless. Maybe
there's another call that gives a proper unique hardware identifier
for us to discover.
I would very much appreciate clever and not-so-clever ideas and design
discussion. :)
Best,
Drew
Jeff Kramer
Jeff, with a not-so-clever suggestion :)
Sent from my iPhone
Joshua Blake
First off, a call for help: if you have a K4W and a hardware USB
sniffer, I would appreciate USB logs of the device entering and using
near mode. If higher resolutions on the RGB camera are supported, I'd
love a log of that too. Feel free to ping me offlist if you don't
want thousands of people all downloading giant USB log files and using
all of your bandwidth.
However, I have two significant concerns.
The first is that device enumeration is starting to get really
confusing. The K4Xbox has three subdevices: motor, camera, audio.
The K4W has only the camera and audio devices. There's no good
heuristic for telling which subdevices are grouped together besides
enumeration order (which libusb doesn't make any guarantees about).
As a bonus, the K4W has a serial number of 0000000000000000, so we can
probably no longer rely on freenect_open_device_by_camera_serial() to
distinguish between multiple K4W. This sucks from an design
standpoint, though it may not be a significant problem in practice
(anyone out there using multiple K4W with libfreenect? Color me
surprised).
The second is that since we no longer have a motor device at all, all
of the tilt-, motor-, and LED-related function calls make no sense on
the K4W. Even more problematic is that:
- our API expects the user to specify which subdevices should be opened
- the user can't (currently) tell programmatically before attempting
to open a device whether a device is a K4W or K4Xbox (this can
probably be fixed by adding to freenect_device_attributes)
drew.m...@gmail.com
> On Tue, Feb 28, 2012 at 1:38 PM, drew.m...@gmail.com
> <drew.m...@gmail.com> wrote:
>>
>>
>> First off, a call for help: if you have a K4W and a hardware USB
>> sniffer, I would appreciate USB logs of the device entering and using
>> near mode. If higher resolutions on the RGB camera are supported, I'd
>> love a log of that too. Feel free to ping me offlist if you don't
>> want thousands of people all downloading giant USB log files and using
>> all of your bandwidth.
>
>
> The hardware USB sniffer was required for the Kinect for Xbox because we
> can't run software USB sniffers on the Xbox. For this, since there are
> existing Windows drivers for the K4W sensor, we might be able to use
> software USB sniffers.
>
> I have a K4W sensor but don't have time to sniff it at the moment.
Let me know if you find a software sniffer that works to log Kinect
traffic. From what I've seen, the Kinect won't appear in an
enumeration of USB devices on Windows if you have the SDK installed,
so all of the USB filter drivers that would log the traffic won't even
see the Kinect as a loggable device. While that was with the K4Xbox
beta SDK, and I haven't confirmed the same failure mode with the K4W
SDK, I'm betting they also made the K4W a separate device type.
>> However, I have two significant concerns.
>>
>> The first is that device enumeration is starting to get really
>> confusing. The K4Xbox has three subdevices: motor, camera, audio.
>> The K4W has only the camera and audio devices. There's no good
>> heuristic for telling which subdevices are grouped together besides
>> enumeration order (which libusb doesn't make any guarantees about).
>
> On Windows, K4W motor device shows up as a different device. I think you
> just need to figure out the different USB id to connect to. Also, there is a
> second audio device that shows up and exposes the Kinect audio array as a
> standard microphone array suitable for use with any audio app. That is most
> likely a software function installed with the SDK.
Hmm, at the USB level, the motor device doesn't appear to exist at
all; I only see a USB hub (045e:02c2), audio (045e:02be), and camera
(045e:02bf). The audio subdevice appears to use the same
firmware-loading scheme and protocol. I'm guessing the hardware is
the same; it works with both the UAC firmware (which will appear as
just a normal microphone array) and the Xbox360 firmware (which won't
work on Windows). I suppose they could have merged the motor device
into either the camera or audio device and added commands for it. If
this is the case, we've got even more complications - the motor device
can no longer be opened orthogonal to the device that it was merged
into, so our API still needs revision.
I suppose we could make something like usbmuxd for the Kinect, which
would let us preserve the API at the cost of running a system-level
daemon, but I'd prefer not to at this point.
>> As a bonus, the K4W has a serial number of 0000000000000000, so we can
>> probably no longer rely on freenect_open_device_by_camera_serial() to
>> distinguish between multiple K4W. This sucks from an design
>> standpoint, though it may not be a significant problem in practice
>> (anyone out there using multiple K4W with libfreenect? Color me
>> surprised).
>
> K4W has serial numbers and the K4W API lets you open a specific sensor by a
> UniqueKinectId string. It's most likely a case of we are asking the K4W
> sensor the wrong question (in the USB protocol.)
Guess that's exposed at a level higher than is typical for USB
devices; from lsusb -v -d 045e:02bf
iManufacturer 2 Microsoft
iProduct 1 Microsoft Kinect Camera
iSerial 3 0000000000000000
The takeaway is that a userspace driver can't know the serial number
of a device without opening the device and exclusively claiming an
interface on it. As a result, we can't know if a given device is the
right one to try without trying all of them. A kernel driver can
safely assume that it will always get an exclusive handle on the
hardware.
>> The second is that since we no longer have a motor device at all, all
>> of the tilt-, motor-, and LED-related function calls make no sense on
>> the K4W. Even more problematic is that:
>>
>> - our API expects the user to specify which subdevices should be opened
>> - the user can't (currently) tell programmatically before attempting
>> to open a device whether a device is a K4W or K4Xbox (this can
>> probably be fixed by adding to freenect_device_attributes)
>
>
> There is a new status API in the K4W SDK 1.0 that tells us useful things
> like whether the device is ready or error messages like insufficient usb
> bandwidth or device in use by another application. See
> http://msdn.microsoft.com/query/dev10.query?appId=Dev10IDEF1&l=EN-US&k=k(MICROSOFT.KINECT.KINECTSTATUS)%3bk(TargetFrameworkMoniker-%22.NETFRAMEWORK%2cVERSION%3dV4.0%22)%3bk(DevLang-CSHARP)&rd=true
>
> Some of these might be a runtime function but there's probably at least some
> that can be determined through USB commands or status messages.
Neat. At a glance, that might inspire a set of states for a
usbmuxd-alike daemon.
-Drew
Dwight Goins
some apps using Kinect for windows, and I'm the only one in the group
who does anything with freenect. So any plans here so that I can port
our code from windows sdk over to openNI and freenect?
On Mar 3, 5:56 pm, "drew.m.fis...@gmail.com" <drew.m.fis...@gmail.com>
wrote:
> On Sat, Mar 3, 2012 at 15:08, Joshua Blake <joshbl...@gmail.com> wrote:
Matthias Neutzner
device. For our use case it would be great to see K4W support in
libfreenect. While i'm not yet an experienced usb reverse engineer i'd
like to help getting K4W to work with libfreenect.
For the beginning i could provide information about the device. But
Maybe the wiki is a better place for this, than the list!? Briefly i
can acknowledge the technical details already given by Drew: differing
product id, no Motor device, Camera has no serial (000...), etc.
Beyond that i tried SnoopyPro as USB Software Sniffer in Windows 7 and
got some logs from the Windows driver's initalization sequence. I
didn't have the time to figure out what's the difference to the K4XBox
init sequence, but i supply the logs if someone is interested. I could
try to get some more USB logs if you tell me what could be relevant.
In the next step i'll try to replay the captured sequence to the
device and see what happens... Moreover i would appreciate any hints
and suggestions what to do in the next steps on the way to libfreenect
support.
Best regards
Matthias
On Mar 13, 10:44 pm, Dwight Goins <dngo...@gmail.com> wrote:
> Any more updates to this thread... I'm working with a team to develop
> On Mar 3, 5:56 pm, "drew.m.fis...@gmail.com" <drew.m.fis...@gmail.com>
> wrote:
>
>
>
>
>
>
>
> > On Sat, Mar 3, 2012 at 15:08, Joshua Blake <joshbl...@gmail.com> wrote:
> > > On Tue, Feb 28, 2012 at 1:38 PM, drew.m.fis...@gmail.com
> > > <drew.m.fis...@gmail.com> wrote:
>
> > >> First off, a call for help: if you have a K4W and a hardware USB
> > >> sniffer, I would appreciate USB logs of the device entering and using
> > >> near mode. If higher resolutions on the RGB camera are supported, I'd
> > >> love a log of that too. Feel free to ping me offlist if you don't
> > >> want thousands of people all downloading giant USB log files and using
> > >> all of your bandwidth.
>
>
> > > I have a K4W sensor but don't have time to sniff it at the moment.
>
> > Let me know if you find a software sniffer that works to logKinect
> > enumeration of USB devices onWindowsif you have the SDK installed,
> > SDK, I'm betting they also made the K4W a separate device type.
>
> > >> However, I have two significant concerns.
>
> > >> The first is that device enumeration is starting to get really
> > >> confusing. The K4Xbox has three subdevices: motor, camera, audio.
> > >> The K4W has only the camera and audio devices. There's no good
> > >> heuristic for telling which subdevices are grouped together besides
> > >> enumeration order (which libusb doesn't make any guarantees about).
>
> > > likely a software function installed with the SDK.
>
> > Hmm, at the USB level, the motor device doesn't appear to exist at
> > all; I only see a USB hub (045e:02c2), audio (045e:02be), and camera
> > (045e:02bf). The audio subdevice appears to use the same
> > firmware-loading scheme and protocol. I'm guessing the hardware is
> > the same; it works with both the UAC firmware (which will appear as
> > just a normal microphone array) and the Xbox360 firmware (which won't
> > this is the case, we've got even more complications - the motor device
> > can no longer be opened orthogonal to the device that it was merged
> > into, so our API still needs revision.
>