OSK abstractions
billh
I started to go off on a tangent in the "Technical Approach" thread.
It seems to me it would be beneficial to try and converge on some of
the common abstractions that are useful when thinking about onscreen
keyboards and similar alternative-input-interfaces. (Is there a wiki
page where we could build this?)
At a high level, what about:
1) input (i.e. input devices and selection models)
(a) physical devices
(b) input stream type (continuous or intermittent, valuator or
binary-switch, number of
axes and number of discrete switch inputs)
(c) the concept of selection
(d) the concept of activation
[note: selection and activation are separate because selection
can be passive, i.e.
based on a scanning timer, whereas activation is likely to be
user-initiated]
2) logic
including concepts of
(a) currently active application(s)
(b) user 'focus' or context within application
(c) application-space selections and groups (i.e. menus, icon
lists, etc.)
(d) list of 'live' applications or interfaces which are not
currently active
[doubtless there are lots more abstractions of relevance that fall
within the "logic" category.]
3) presentation
(a) simulated physical keyboard
(b) selection from a list or menu
(c) presentation of 'actionable' items from current context
(d) a means of allowing the user to change his/her point of regard
Presentation of course feeds back into input via 1c and 1d. And
somewhere in there are the concepts of basic user interface tools such
as windows, the system pointer, workspaces, etc. since OSKs may wish to
offer the user an alternate way of interacting with those UI elements.
Perhaps the interfaces between 1 and 2 above, and between 2 and 3,
would be good places for "pluggability".
To bring it down to earth, in the "classic" case of a kiosk QWERTY OSK,
the input stage could be described as
1a:physical device: trackball or touchpad
b:input stream - 2D location (system pointer)
c:selection accomplished by placing the pointer within a presentation
onscreen "key"
d:activation by "click"
Logic (2) would probably reduce to "simulate corresponding key event
with XTestFakeKeyEvent(), possibly with some modifier-checks and
latching behavior, if XKB wasn't used for key latching.
Presentation would be a QWERTY look-alike drawn into an onscreen window
that didn't take focus, but was always on top.
Does this seem like a useful way of talking about the design
characteristics and goals?
Bill
davidb
Just a quick note that you may have missed a great resource Steve
pointed out to me:
http://ace-centre.hostinguk.com/assets/Product%20Downloads/SwitchScanningMaster_8_472.pdf
Among other things, it attempts to define a taxonomy.
cheers,
David
Steve Lee
the fundamental models. This will help provide flexible and workable
concepts to describe and discuss the requirements and drive the
implementation.
Can we boil down input into some sort of generic abstract 'super
device' of which actual devices implement a subset? That way any
combination of devices get handled the same way. That's probably
unrealistic.
We can come up with various scenareos (like the KIOSK OSK) and
detailed usecases for each described in the abstractions. At some
point we need to ground it in an implementation strategy.
I've created a wiki page to distill these ideas Bill
https://wiki.ubuntu.com/Accessibility/Projects/OSK-ng/Abstractions.
Steve
--
Steve Lee
www.oatsoft.org
www.fullmeasure.co.uk
Steve Lee
https://wiki.ubuntu.com/Accessibility/Projects/OSK-ng/Abstractions
My questions are around joysticks which can be digital(switch) or analogue (continous), My understanding is that older game joystick adapters (soundcard) were digital but now USB analogue is the commonest type. Digital joysticks just indicate direction, not value. Analogue give a value in 2 axis based on absolute position (could be relative motion). A digital joystick driver could factor in time to create a analogue position but that gives a very different operation to continuous.
Perhaps the distinction is irrelevant if rather than physical operation or interfaces we think about the logical interface exposed by the drivers/API. Thus either digital or analogue devices can control the system pointer like a mouse with suitable drivers. (Applications may have a choice how they accesses a physical digital joystick.)
Is a switch joystick a valuator with 9 values? Encoded 4 switches - centre, up down, left.....
I think this is not as useful as thinking of it as 4 switches.
Would a continuous (analogue) joystick be 1 or valuators 2 (1 per axis).
I think 1 as valuator does not specify degree.
Bill Haneman
> At the risk of being overly academic I'd like explore Bill's proposed
> valuator concept a little. A valuator is a device (physical or
> logical) that can take a range of values. Bill say's 'non-binary'.
>
> https://wiki.ubuntu.com/Accessibility/Projects/OSK-ng/Abstractions
>
> My questions are around joysticks which can be digital(switch) or
> analogue (continous), My understanding is that older game joystick
> adapters (soundcard) were digital but now USB analogue is the
> commonest type. Digital joysticks just indicate direction, not value.
> Analogue give a value in 2 axis based on absolute position (could be
> relative motion). A digital joystick driver could factor in time to
> create a analogue position but that gives a very different operation
> to continuous.
>
> Perhaps the distinction is irrelevant if rather than physical
> operation or interfaces we think about the logical interface exposed
> by the drivers/API. Thus either digital or analogue devices can
> control the system pointer like a mouse with suitable drivers.
> (Applications may have a choice how they accesses a physical digital
> joystick.)
>
> Is a switch joystick a valuator with 9 values? Encoded 4 switches -
> centre, up down, left.....
> I think this is not as useful as thinking of it as 4 switches.
appears as 4 switches (or 5, if there's a top-button on the joystick,
which there often is). Some joysticks have extra buttons as well, which
just appear as more switches.
This means of course that a "virtual valuator" can be constructed from a
switch plus some kind of auto-repeat/timer component (which is in
effect what commercial joystick drivers do).
Joysticks are useful for OSKs, by the way, for doing "directed
scanning", for instance if the buttons are arranged on a 2D grid. This
is one reason GOK renders its keyboards on a 2D grid rather than more
naturalistically; it makes it obvious which cell is "directly
above"/below the currently "highlighted" cell.
>
> Would a continuous (analogue) joystick be 1 or valuators 2 (1 per axis).
> I think 1 as valuator does not specify degree.
This sound to me like a single valuator with 2 axes, but of course you
could express the same thing as 2 1-axis valuators. I take it that you
mean by "continuous" joystick, not a truly analogue device (which USB
can't express anyway), but a discretized non-binary input, i.e. more
like a mouse...
Bill
Bill
>
> --
> Steve Lee
> www.oatsoft.org <http://www.oatsoft.org>
> www.fullmeasure.co.uk <http://www.fullmeasure.co.uk>
Steve Lee
In fact could I suggest 'continuous' as an alternative to valuator? It sounds a lot less techy.
I'm thinking we have
Device type (according to the values it can generate)
Switch: on | off
Continuous: n1 to n2
Event:
trigger - has no value (e.g hit)
discrete - has value from small set (e.g press release, left)
scalar value (e.g. slider)
delta - change in value
ordinal (e.g. 2 axis, pointer)
Logic can mutate these
discrete -> trigger by ignoring value.
scalar -> discrete by grouping ranges of values or detecting a change
delta -> scalar by using addition
discrete -> ordinal using timer or a counter with value ( e.g. +10,40)
Gestures are made up of sequenced events e.g. 3 presses, up then right. Possibly with events from multiple devices.
Actions are the result of gesture at time and app state or location e.g. active this cell, click this GUI button, open this link
I'm unsure whether switch is a trivial case of continuous - only 2 values. Are we forcing unnecessary distinctions at the device level which can be made at the event level?
Joysticks can be treated as either a continuous with several values or individual switches,
David Bolter
I also wonder if triggers can be individual values (e.g. switch-down).
I think we need to be careful about what layer we are talking about. We
need to define where an "event" sits in this context... because we can,
of course, impose additional layers (as GOK does or at least was
designed to do), where an user "action" can be defined as 1 or more events.
For example, an "action" could hypothetically be:
left-button-down-duration-100ms + headpointer-lower-left-screen-qaudrant
(this isn't a valid gok example but I'm just thinking aloud here).
We have to be careful with decisions around adding layers though... as
it does add complexity.
cheers,
D
> Joysticks can be treated as either a continuous with several values or
> individual switches,
>
> --
> Steve Lee
> On 11/10/06, *Bill Haneman* <Bill.H...@sun.com
> > www.oatsoft.org <http://www.oatsoft.org> < http://www.oatsoft.org>
> > www.fullmeasure.co.uk <http://www.fullmeasure.co.uk>
> <http://www.fullmeasure.co.uk>
>
>
Bill Haneman
> I prefer the term continuous for a range of values. Lets face it
> analogue is always quantised before it gets into the 'puter. But
> analogue seemed to be the term in use so I changed the text,
> forgetting the case you spotted.
>
> In fact could I suggest 'continuous' as an alternative to valuator? It
> sounds a lot less techy.
I am not keen on 'continuous', since it could mean continuous in
duration, as opposed to non-binary. "Valuator" is the term used in the
X world I believe.
>
> I'm thinking we have
>
> Device type (according to the values it can generate)
> Switch: on | off
> Continuous: n1 to n2
>
> Event:
> trigger - has no value (e.g hit)
> discrete - has value from small set (e.g press release, left)
> scalar value (e.g. slider)
> delta - change in value
> ordinal (e.g. 2 axis, pointer)
I agree with David, we need to be careful about where we put the event
abstractions, within our "layers". I think we should first just talk
about events in terms of "switch state change" and "valuator" (or
whatever - 'scalar input?') change.
The logic as to how to interpret the state changes above belongs, I
believe, in a separate state which involves both logic and
presentation. So I would like to distinguish between the basic input
device events (above - state-changes of both binary and scalar inputs),
and the way those events interact with the selection and navigation
models. Some of your concepts above seem useful to me in the latter
discussion (i.e. delta vs. absolute value).
It is certainly the case that single state-changes (i.e. press vs
release) need to be separately regarded in some cases - for instance in
the case of scanning techniques which advance the highlight as long as a
switch is pressed, and stop scanning when it is released, just to take
one example. It's also the case that one axis of automated (time-based)
scanning can be combined with one axis of valuator input - as I saw in
an interesting paper/demonstration of a system which was activated by
the expansion and contraction of a chest band for a person who had
control of his breathing but not of other voluntary muscles.
regards
Bill
>
> Logic can mutate these
>
> discrete -> trigger by ignoring value.
> scalar -> discrete by grouping ranges of values or detecting a change
> delta -> scalar by using addition
> discrete -> ordinal using timer or a counter with value ( e.g. +10,40)
>
> Gestures are made up of sequenced events e.g. 3 presses, up then
> right. Possibly with events from multiple devices.
>
> Actions are the result of gesture at time and app state or location
> e.g. active this cell, click this GUI button, open this link
>
> I'm unsure whether switch is a trivial case of continuous - only 2
> values. Are we forcing unnecessary distinctions at the device level
> which can be made at the event level?
>
> Joysticks can be treated as either a continuous with several values or
> individual switches,
>
> --
> Steve Lee
> On 11/10/06, *Bill Haneman* <Bill.H...@sun.com
> > www.oatsoft.org <http://www.oatsoft.org> < http://www.oatsoft.org>
> > www.fullmeasure.co.uk <http://www.fullmeasure.co.uk>
> <http://www.fullmeasure.co.uk>
>
>