OSD v1 "mushy keys" in Arizona in x11vnc and with physical IR remote

[bmc]

I have the below code contributions for the OSDv1.

1) x11vnc as is, is unusable under Arizona on my OSD1.0. Under x11vnc
the
controls are effectively "mushy" - you hit a key, and then wait, and
the
OSD will act on that key once, twice, or a dozen times before settling
again and waiting for new input.

Tracing through the code shows that this is result of the Qt key
auto-repeat. There is race condition between the "send_ir" script
sending the keydown and keyup, and the Qt libraries reading them, and
things are sluggish enough that it is very random how many key repeats
it sneaks in before it sees the keyup.

Fix: I have added a "don't apply auto-repeat to the following key"
special key prefix. The x11vnc send_ir script now sends this via
/dev/neuros_ir before it sends the injected key. Corresponding
changes
in the code for libosdir-plugin.so now honor this special key.

Result: The x11vnc experience now feels crisp and precise.


2) Having done the above, I now found the physical IR remote
experience
frustrating by comparison. It occasionally triggers two button downs
for
one button pressed.

Knowledge from doing #1 showed this was a problem with the auto-
repeat.
The timing is too short/aggressive for this hardware running Arizona
firmware (400ms delay, 80ms repeat).

These defaults are hard-coded into the Qt library, and are not
(easily)
overridden. But it can be done.

Fix: Further changes to libosdir-plugin.so allows one to override the
delay and repeat values in the Qt keyboard handlers. I have it read
these values from a config file at /mnt/OSD/kbd_repeat_rate, leaving
the
Qt values if this file is not present. This is done at Qt startup,
and
another special key can be sent via /dev/neuros_ir to force reloading
of
these thresholds on demand, allowing one to tune them in real-time
without requiring a restart.

Result: Increasing the repeat delay to 750ms for initial repeat
gives a
solid experience for me - button presses on the remote navigating the
menus now feels precise. The slight extra delay for operations
depending on auto-repeat hasn't yet been noticed.

3) I made a small change to svn trunk to allow a build from a clean
checkout. Error is "can't cd to binaries/lib/pkgconfig/" from within
"Neuros-Cooler/external-libs/curl". Needed to add a dummy "pkgconfig"
and
"pkgconfig/curl.pc" into the repo before executing "./build-helper
all".

4?) Noticed, but not planning on fixing (probably harmless)

The current interaction of OSD IR remote with the Qt auto-repeat
results
in an event stream like:

key: 1 down not-a-repeat # initial down-up pair
key: 1 up not-a-repeat
<NOTE: double up here>
key: 1 up repeat # first up-down pair from autorepeat
key: 1 down repeat
key: 1 up repeat # 2nd up-down pair from autorepeat
key: 1 down repeat
<NOTE: missing key 1 up here>
key: 2 down not-a-repeat
...

The "right" way to do this is to send just a down during the initial
event, let Qt do its up/down pairs, and then send the final up during
the
IR keyup event. Not sure that it matters, I guess no existing OSD
applications track or care about up/down events as above - perhaps
handled
with a Qt event filter somewhere - if you filter to keep just Qt
keypressevent
keyup, I can see why it was left this way.

Please let me know what are the current practices for getting diffs
approved and committing back to the svn trunk.

Diffs for the above can be found at: http://oddren.com/files/osd/

Regards,

Brian

[Ugo Riboni]

> Please let me know what are the current practices for getting diffs
> approved and committing back to the svn trunk.
>
> Diffs for the above can be found at:  http://oddren.com/files/osd/

Thanks for the patches.

I have looked at them only briefly, but it seems to me that the
changes to the "osdir" keyboard driver aren't very kosher.
It's been a while since I've done any C++ to be honest, but it looks
to me what you're doing with QWSKeyboardHandler_Public and
QWSKbPrivate essentially breaks encapsulation, because you're
re-declaring a QT class just with a different name and exposing a
private member as public, then casting another class to it.
This works now because you know the current internals of that class.
But by definitions the internals are subject to change at any time, so
that may break unpredictably in the future and probably the compiler
won't even notice, you would get something ugly at runtime.
I may be wrong, but it just don't seem very kosher to me.

But the alternative is probably patching QT4 to expose the repeat rate
members as public (or even better add accessors).
Since we rebuild QT4 during the OSD1 firmware built, and we have
already in place a way to add patches to it automatically, it may be
cleaner if you do it that way.

For the rest, it seems ok from a quick look.
--
nero

[bmc]

> I may be wrong, but it just don't seem very kosher to me.

Agreed. It certainly is not kosher - this is the non-existent "strip
private-ness cast" which is of course forbidden by C++ and all all
that is right and holy in the world. :)

It's just one step above doing raw pointer arithmetic, to get at these
members hidden in a private pointer to an opaque internal class in Qt.

The trade-off is:

a) do something brittle like this, that allows the change to go out
encapsulated entirely inside the single plugin shared object.

b) rebuild the whole Qt library, and require people to replace Qt as
well as this plugin.

My choice of a), at least at this time, was strongly motivated by the
fact that
1) the current ./build-helper.sh doesn't rebuild the Qt library at all
- it's a frozen binary blob.
2) Adding accessors to an internal, private, opaque class the Qt
library goes to great lengths to hide from you is a little backwards
too.
3) Based on the lack of recent development and the frozen binary blob
for Qt - I thought it safe to consider the Qt release "carved in
stone".
4) I'm on an OSD 1.0, with the plugins living on the CF card, so it is
trivial for me to drop in a new plugin without going through a upk
firmware update.

I now see how to get your build system to forcibly rebuild qt-
embedded, and could put in more work to patch the Qt API to expose
this properly.

Let's see how the OSD 1.x development and user community responds
(hello out there? anyone?). If there's interest and support for
another full firmware release via UPK, I can do this "right".

Until then, the prebuilt plugin shared object is at the above URL too,
for casual hackers who want to drop it into their distribution.

Can you, or anyone, comment on what the system looks like for someone
with an OSD 1.1 (or whatever that is called) that is lacking a CF slot
and with bigger internal flash? Do they have the ability to replace a
plugin .so and/or the Qt library, or do they need a monolithic and
slow upk firmware upgrade? (i.e. is this stuff stored in a read-only
partition, or on the jffs2 r/w area?)

Are there just the two hardware releases (with and without CF slot?)
of the OSD 1.x, or are there other flavors in the wild?

I'm tempted to move as much as possible into a r/w area (jffs2 on the
flash, or USB/SD, or CF (for OSD 1.0s) ) to remove the need to often
(or ever) reflash via upk - that's slow, monolithic, and increases the
chance of bricking someone's OSD.

The trunk has lpkg in it; if we treat the OSD as a package-managed
platform, we want most everything mounted (or remount-able) as read-
write.

Sure, I can tftp/nfs boot my OSD, but that greatly limits who else can
benefit from this work.

- Brian

[Ugo Riboni]

> Let's see how the OSD 1.x development and user community responds
> (hello out there? anyone?).  If there's interest and support for
> another full firmware release via UPK, I can do this "right".
> Until then, the prebuilt plugin shared object is at the above URL too,
> for casual hackers who want to drop it into their distribution.

I agree with you on this one.

> Can you, or anyone, comment on what the system looks like for someone
> with an OSD 1.1 (or whatever that is called) that is lacking a CF slot
> and with bigger internal flash?  Do they have the ability to replace a
> plugin .so and/or the Qt library, or do they need a monolithic and
> slow upk firmware upgrade?  (i.e. is this stuff stored in a read-only
> partition, or on the jffs2 r/w area?)

If I recall correctly the extended area is just mounted in a rw
partition inside the internal flash, instead of the CF card.
The base fs is still on a readonly partition to be as similar as
possible to the previous system.

> Are there just the two hardware releases (with and without CF slot?)
> of the OSD 1.x, or are there other flavors in the wild?

I think they are the only ones, at least by Neuros.

> I'm tempted to move as much as possible into a r/w area (jffs2 on the
> flash, or USB/SD, or CF (for OSD 1.0s) ) to remove the need to often
> (or ever) reflash via upk - that's slow, monolithic, and increases the
> chance of bricking someone's OSD.
> The trunk has lpkg in it; if we treat the OSD as a package-managed
> platform, we want most everything mounted (or remount-able) as read-
> write.
> Sure, I can tftp/nfs boot my OSD, but that greatly limits who else can
> benefit from this work.

The problem with the package manager as of now is that it has no gui
and no repository for packages.
So the updates have to be manually initiated, and that has to be done
from the command line (telnet or serial).

That is great for you and me, not so much for any other end user that
doesn't know telnet from a baboon's arse ;)
Then again maybe all remaining OSD users right now are cool with
telnet, in which case no problem.
I like packages more, for the record, than flashing the entire thing
over and over.

In any case AFAIK there's no official release yet with lpkg inside
(not entirely sure though).
--
nero

[Fernando Cassia]

On Mon, May 11, 2009 at 2:24 PM, Ugo Riboni <neroc...@gmail.com> wrote:

> I like packages more, for the record, than flashing the entire thing
> over and over.

I am VERY annoyed with the OSD firmware new version-check aka updater because:

#1 it doesn't show the RELEASE DATE of teh firmware update available.
( downloaded dev release y-072 twice because I thought it were
different releases, several days apart, I has to start scribbling
release numbers in a piece of paper to keep track of what releases I
already tried).

#2 If you switch from Beta to dev and check for a new release IT
BEGINS DOWNLOADING without a confirmation if you indeed what to
install that. It should check and say "what I see in the dev branch is
y-072 dated 06/32/2049, install? OK/Cancel" not go straight into
downloading.

#3 Fix the bloody Youtube, please. It's unsusable as it is today. Two
out of three videos do not play anymore.

FC

[bmc]

I agree, FC, the firmware updater is unpalatable to me, just because
it is monolithic (not to mention other problems you cite above).

For my tinkering, I'd like to get things to a state where everything
is individually updatable.

I've read that youtube is a huge problem for most of the users - while
that's not something I use the OSD for, I looked into the problem and
it's a quick fix. Unlike the sticky keys fix above, it involves
replacing files that are on the read-only areas. You can manually run
the updated youtube + shared objects from another location, but no way
to tie that into the menu system (also read-only).

So at the least we're looking at one more upk update.

I'd like to make it the *LAST* upk update, and use a better update
process afterward.

I'm toying with one of the following options:

1) root entirely r/w jfffs2 on flash (but what about OSD 1.0 with
small flash?)
2) initramfs root, links to cramfs files (cramfs is faster) and links
to jffs2
3) initramfs, read-only root (cramfs), mini_fo to r/w overlay
filesystem

Or further tweaks of #3:

4) initramfs root from kernel, loopback mount of root filesystem
living on regular fat32 SD/USB/CF card (easy replacement of root if
"bricked")
5) Same as #4, but read-only root and overlay mini_fo mount of read-
write root filesystem.also on regular card
6) Same as #5, but additional mount of "user files" area on a 3rd r/w
filesystem on regular card

Playing around with the undersized flash in these days of multi-gig SD
and USB cards just seems counter-productive, especially since the two
releases of the OSD have different flash sizes (and mine has the small
flash). Just stick the kernel (and maybe a secondary backup kernel -
how to switch/fallback at u-boot time? u-boot can only see CF I
believe) in flash and offload everything else.

Nero: Can you comment on why the current root/jffs2 was chosen, and
any problems you'd anticipate with the above?

- Brian

[Ugo Riboni]

> So at the least we're looking at one more upk update.
>
> I'd like to make it the *LAST* upk update, and use a better update
> process afterward.
>
> I'm toying with one of the following options:

[[snip]]

> Playing around with the undersized flash in these days of multi-gig SD
> and USB cards just seems counter-productive, especially since the two
> releases of the OSD have different flash sizes (and mine has the small
> flash).  Just stick the kernel (and maybe a secondary backup kernel -
> how to switch/fallback at u-boot time? u-boot can only see CF I
> believe) in flash and offload everything else.
>
> Nero: Can you comment on why the current root/jffs2 was chosen, and
> any problems you'd anticipate with the above?

I can't actually say much about that decision. It was made by others
before I joined the project and personally I just took it as a given
and moved on with it. I can make some guesses.

The engineer who designed the OSD1 base system came from a strong
non-linux embedded background. He was used to monolithic firmwares and
monolithic update methods. He wasn't very familiar with package based
methods. So having a ro rootfs probably came natural since he assumed
it would be updated by flashing entire packages anyway.

Ease of support probably played a role too, since if we know the
version of firmware installed, we know what features it has, without
having them altered by the user.

There were also probably other reasons, but anyway, I wouldn't spend
too much time dwelling on those. What is done is done, and all that...

Regarding changing to a rw fs, i can tell you the software can cope
with that perfectly. During developement we use an NFS mounted rootfs,
and when that is rw things are just as peachy as when it's ro.

My favorite choice would probably be using some form of overlay fs (i
would've said aufs or unionfs, but mini_fo seems a much lighter
alternative that i didn't know about until now), using the CF or NAND
as writeable space (some considerations have to be made here on how
much space to allocate to this, and how). Using SDs for this of course
is also an option, or USB. but of course, that's only true if you
don't need it for recording or playback.

I'm not so sure how a true rw filesystem would behave on the NOR
memory. Something about speed of writing and especially limit on erase
cycles before it destroy itself bothers me, but right now I can't
elaborate on this much more. on NAND, it will probably be better to
have a regular rw filesystem.

Hope it helps,
--
nero

[bmc]

> My favorite choice would probably be using some form of overlay fs (i
would've said aufs or unionfs, but mini_fo seems a much lighter
alternative that i didn't know about until now),

mini_fo is very nice. I use it extensively with openwrt.

> Using SDs for this of course is also an option, or USB. but of course, that's only true if you
don't need it for recording or playback.

I was thinking about dedicating one of CF, SD or USB to always-there
storage for the improved OSD.

Since I'm on a 1.0, and am already doing that with CF, that's an easy
decision for me.

Don't know how that would fly for people without a CF slot. If they
had to dedicate one of SD or USB and leave plugged in while the OSD
was on, would that be a big deal?

Note that by doing this via loopback on a file in a regular fat sd/usb
stick, they could still be using it for movies, etc. Just that when
they wanted to unplug it and move it somewhere else they'd have to
shutdown the OSD.

I'm guessing the same sort of people adopting the OSD have plenty of
SD and USB sticks around, and could dedicate one to this while using
the other for their very large storage.

So the options would be:

a) Use one USB/SD/CF flash memory for movies and root - you need to
shutdown the OSD if you want to pull the storage.
or
b) Leave one USB/SD/CF flash memory plugged in all the time (like 1.0
and Arizona with the CF card)

Either way seems a lot better than what we have now, and it's easy to
engineer it to give people this choice.

> I'm not so sure how a true rw filesystem would behave on the NOR memory.

When are we talking NAND vs NOR? Since you've got a jffs2 on my OSD
1.x, I'm assuming it's NAND. Where is the NOR memory?

> Something about speed of writing and especially limit on erase cycles before it destroy itself bothers me,

Another reason to use the on-board flash just for the kernel, and let
the read/write cycles thrash your removable/replaceable storage...

[Gerry Boland]

These are interesting ideas that I've been pondering a bit recently too.

The readonly filesystem was great for me starting off with linux, since I couldn't break anything. But I see your point that it stifles fiddling, hacking and general messing around.

I had planned on experimenting with r/w filesystems on the OSD (though I've never heard of mini_fo, must look into that).

I was also planning on trying to integrate the OSD's firmware with the dd-wrt kamakaze build system. Then with some work, perhaps ipkg would be compatible, which would help in extending the OSD's abilities, and easing upgrades in future.

I think I'm right in saying that the OSD1 has all NOR flash (32MB), but the OSD 1.x (x>0) has NAND (256MB). However the OSD1 has a r/w partition, presumably on the NOR chip, so it must be do-able. How reliable jffs2 is on these NOR chips remains to be seen (I've helped 2 individuals in 3 years where the r/w partition corrupted). I've also heard of issues booting kernels off NOR chips with jffs2, I'm hoping that's an old problem now fixed.

It would be nice to have the option to boot from other media. I believe uboot can do it. Even so, it should be easy to point the kernel to any rootfs you want.

-G

2009/5/12 bmc <bmcarne...@oddren.com>

[bmc]

> the OSD1 has all NOR flash (32MB)

My 1.0 has the read-only cramfs root (mtd4) taking up 29M of that
flash.

Then mtd5 is 1.0 read-write jffs2 mounted at /mnt/OSD

So everytime a bookmark or a favorite is committed to jffs2 flash,
it's chewing up a NOR cycle...

Since I'm running Arizona, it also has a /mnt/tmpfs/mount_CF-card/data
directory holding a bunch of other stuff that is written more often.

Don't know what the OSD 1.x (x>0) layout looks like. Could someone
with a later rev send me a compressed "ls -lR /" from their device?

All the more reason to abandon the flash (NAND/NOR) for everything but
the kernel, and just use the card slots.

> The readonly filesystem was great for me starting off with linux, since I
> couldn't break anything. But I see your point that it stifles fiddling,

This is just as safe. If someone bricks their OSD with the rootfs on
the SD/CF/USB, they fix it by dropping clean rootfs file(s) back on
it. These are small enough we can even loopback mount them as regular
files on a vanilla FAT32 formatted card - unbricking is trivial even
if all you've got is windows.

Besides, if we go with overlays, they can just erase the file
containing the overlay partition and "start fresh".

Storing user settings outside of the basic rootfs + rootfs overlay
gives even more flexibility.

[Anthony DeRobertis]

Didn't the LINK have issues with rootfs on cheap flash drives? In
particular, with the rootfs getting randomly corrupted, especially if
power was pulled?

[bmc]

Was that with an ext2 rootfs or a jffs2 rootfs that experienced
corruption?

I'm thinking with a cramfs root and a overlay backed by jffs2 for the
rootfs, the reliability would be about the same for NAND/NOR vs a
cheap flash drive.

The ability to "un-brick" by just writing a new rootfs image file onto
a fat32 flash drive makes this less urgent, but it would be important
to make sure this doesn't happen more than once in a blue moon.

Can you provide more details on your problems with flash-based rootfs
on the LINK? Or point me to the relevant message threads?

On May 13, 7:14 am, Anthony DeRobertis <googlegro...@derobert.net>
wrote:

[Gerry Boland]

The Link is quite a different product. Its file system resides on a USB flash key, with ext2 partitions. Are people still experiencing corruptions now though? In the early days yes, but work has been done on fixing this.

This would be symptomatic of the errors involved:

http://forums.neurostechnology.com/index.php?topic=10001.0

http://forums.neurostechnology.com/index.php?topic=10140.0

In my email, the problem was with the /mnt/OSD partition being corrupted somehow. It is a jffs2 partition.

-G(greyback)

2009/5/13 bmc <bmcarne...@oddren.com>