Gestures like swipe

[Krukow]

I've just begun trying out Frank for a larger and quite complex iPhone
project. I'm really impressed with how well it is working out.

My current problem is that I need a gesture (swipe) to interact with a
view. UISpec doesn't seem to support it.

Do you think it would be somehow possible to combine the features of
UIAutomation for gui-manipulation, ie. have a step definition that is
implemented via UIAutomation (assuming I can programmatically launch
UIAutomation, read a script and run it).

Alá

When I "swipe" "left"

implemented by calling out to UIAutomation, bypassing Frank server for
that step?

Is it even making sense what I say? :)

- Karl

[Krukow]

On Jun 10, 10:35 am, Krukow <karl.kru...@gmail.com> wrote:
> My current problem is that I need a gesture (swipe) to interact with a
> view. UISpec doesn't seem to support it.

[snip]

I just found this thread

http://groups.google.com/group/frank-discuss/browse_frm/thread/4587423803db42e7

I'll take a look and see if I can implement swipe in uispec.

[Krukow]

On Jun 10, 10:48 am, Krukow <karl.kru...@gmail.com> wrote:
> On Jun 10, 10:35 am, Krukow <karl.kru...@gmail.com> wrote:> My current problem is that I need a gesture (swipe) to interact with a
> > view. UISpec doesn't seem to support it.
>
> [snip]
>
> I just found this thread
>

> http://groups.google.com/group/frank-discuss/browse_frm/thread/458742...

>
> I'll take a look and see if I can implement swipe in uispec.

I'm running into problems with this.

I was trying to extend UISpec to support swipe events. UISpec is based
on the article

http://cocoawithlove.com/2008/10/synthesizing-touch-event-on-iphone.html

My initial naive attempt was to simply mirror the touch method but
injecting some
UITouchPhaseMoved steps. But it didn't work. I was wondering about
the GSEventProxy.

Specifically the line
gsEventProxy->flags = ([touch phase] == UITouchPhaseEnded) ?
0x1010180 : 0x3010180;

It seems it only distinguishes between end and non-end. But I don't
know if this is the problem.

Note I'm running simulator 4.3.

In the EmployeeAdmin app if you right swipe on a table row it enters
"delete mode". This is what I'm trying to script. However nothing
happens

This was my naive code for a swipe

UITouch *touch = [[UITouch alloc] initInView:view];
UIEvent *eventDown = [[NSClassFromString(@"UITouchesEvent") alloc]
initWithTouch:touch];
NSSet *touches = [[NSMutableSet alloc] initWithObjects:&touch count:
1];

[touch.view touchesBegan:touches withEvent:eventDown];

BOOL done = NO;
CGPoint currentLoc = *((CGPoint*)[touch locationInWindow]);
while (!done) {
UIEvent *eventMove =
[[NSClassFromString(@"UITouchesEvent") alloc] initWithTouch:touch];

CGPoint oldLoc = currentLoc;
currentLoc = CGPointMake(currentLoc.x+2, currentLoc.y);
[touch setLocationInWindow:currentLoc];
[touch setPreviousLocationInWindow:oldLoc];
[touch setPhase:UITouchPhaseMoved];
[touch.view touchesMoved:touches withEvent:eventMove];
[eventMove release];
done = (currentLoc.x - view.center.x >50);
// [self wait:.01];
}


UIEvent *eventUp = [[NSClassFromString(@"UITouchesEvent") alloc]
initWithTouch:touch];
[touch setPhase:UITouchPhaseEnded];
[touch setLocationInWindow:currentLoc];

[touch.view touchesEnded:touches withEvent:eventDown];

[Krukow]

Code in more readable form

https://gist.github.com/1018757

[Steven Solomon]

As of yesterday we just extended UISpec to support tap gestures That
will give some insight into what needs to happen to have gestures
supported check out
https://github.com/moredip/UISpec to see the latest change to UISpec
and I believe Pete will have these changes updated to frank when he
has time.

[Pete Hodgson]

Thanks Steven for this great contribution. I really like the visual
touch indicator, it's a very nice touch (badoom!).

I just pulled these UISpec additions in to Frank's master branch. I
also added an example scenario to the EmployeeAdmin sample app to show
how one might use it.

One issue I noticed is that if you try and touch a specific x,y
coordinate of a subview then the visual indicator seems to be in the
wrong place. I'm wondering whether it's drawing at the absolute
position of the x,y coords from the origin of the entire window,
rather than the relative position from the origin of the view being
touched?

Also, regarding gesture support, folks might want to look at the iCuke
implementation. I'm pretty sure they have support for gestures by
sending a stream of touch events over the wire. I'm not a fan of that
low level approach to the API for Frank, but the implementation of
simulating the touches themselves could presumably be the same?

Cheers,
Pete

On Jun 10, 6:10 am, Steven Solomon <solomon.steve...@gmail.com> wrote:
> As of yesterday we just extended UISpec to support tap gestures That
> will give some insight into what needs to happen to have gestures

> supported check outhttps://github.com/moredip/UISpecto see the latest change to UISpec

[Steven Solomon]

yeah i was drawing based on the overall xy coordinate, however that
doesn't seem to feel so well thought out now

On Jun 10, 11:23 am, Pete Hodgson <phodg...@thoughtworks.com> wrote:
> Thanks Steven for this great contribution. I really like the visual
> touch indicator, it's a very nice touch (badoom!).
>
> I just pulled these UISpec additions in to Frank's master branch. I
> also added an example scenario to the EmployeeAdmin sample app to show
> how one might use it.
>
> One issue I noticed is that if you try and touch a specific x,y
> coordinate of a subview then the visual indicator seems to be in the
> wrong place. I'm wondering whether it's drawing at the absolute
> position of the x,y coords from the origin of the entire window,
> rather than the relative position from the origin of the view being
> touched?
>
> Also, regarding gesture support, folks might want to look at the iCuke
> implementation. I'm pretty sure they have support for gestures by
> sending a stream of touch events over the wire. I'm not a fan of that
> low level approach to the API for Frank, but the implementation of
> simulating the touches themselves could presumably be the same?
>
> Cheers,
> Pete
>
> On Jun 10, 6:10 am, Steven Solomon <solomon.steve...@gmail.com> wrote:
>
>
>
> > As of yesterday we just extended UISpec to support tap gestures That
> > will give some insight into what needs to happen to have gestures

> > supported check outhttps://github.com/moredip/UISpectosee the latest change to UISpec

[Krukow]

I've just begun pirring the icuke approch to swipe, following your
sugestion. Will report back Asap

On Jun 10, 5:23 pm, Pete Hodgson <phodg...@thoughtworks.com> wrote:
> Thanks Steven for this great contribution. I really like the visual
> touch indicator, it's a very nice touch (badoom!).
>
> I just pulled these UISpec additions in to Frank's master branch. I
> also added an example scenario to the EmployeeAdmin sample app to show
> how one might use it.
>
> One issue I noticed is that if you try and touch a specific x,y
> coordinate of a subview then the visual indicator seems to be in the
> wrong place. I'm wondering whether it's drawing at the absolute
> position of the x,y coords from the origin of the entire window,
> rather than the relative position from the origin of the view being
> touched?
>
> Also, regarding gesture support, folks might want to look at the iCuke
> implementation. I'm pretty sure they have support for gestures by
> sending a stream of touch events over the wire. I'm not a fan of that
> low level approach to the API for Frank, but the implementation of
> simulating the touches themselves could presumably be the same?
>
> Cheers,
> Pete
>
> On Jun 10, 6:10 am, Steven Solomon <solomon.steve...@gmail.com> wrote:
>
>
>
> > As of yesterday we just extended UISpec to support tap gestures That
> > will give some insight into what needs to happen to have gestures

> > supported check outhttps://github.com/moredip/UISpectosee the latest change to UISpec

[Krukow]

On Jun 10, 11:03 pm, Krukow <karl.kru...@gmail.com> wrote:
> I've just begun pirring the icuke approch to swipe, following your
> sugestion. Will report back Asap

Eh, pirring = porting with iPhone autocorrect and 1/2 bottle of red
wine.

Anyway,I've kind-of managed to port the iCuke technique for swipe, but
it is not yet working correctly. The iCuke approach to event synthesis
is somewhat different from UISpec. It is based on the Recorder
approach: iCuke sends a stream of events to the server which plays
them using

[[Recorder sharedRecorder] load: events];
[[Recorder sharedRecorder] playbackWithDelegate: self doneSelector:
@selector(finishResponse)];

As Pete suggested, I've instead created a "swipe" command which is
sent over the wire, and the events-stream is created and executed on
the server.

I've managed to synthesize up, down and move events. However when I
try to play them on the emploee-admin example to try and emulate a
touch-down-hold-0.2-swipe-right to enable "delete" on the cell, it
instead just activates the cell showing details. So it looks like the
move events aren't working as intended.

I'll work some more on it. Ideas on why the move-events aren't working
are welcome :)

- Karl

[Steven Solomon]

Hey Karl,

hows your implementation coming? mine is giving some headaches

[Krukow]

On Jun 13, 8:29 pm, Steven Solomon <solomon.steve...@gmail.com> wrote:
> Hey Karl,
>
> hows your implementation coming? mine is giving some headaches

Not too well, I'm afraid :( I'm in too deep, it seems. (Last hour I'm
looking into linking with the private framework: GraphicsServices, to
synthesize GSEvents/GSEventRecord but I can't figure out how to :( I'm
completely new to using private frameworks. )

My approach was to synthesize touch events in a form that is readable
by the Recorder.

http://networkpx.blogspot.com/2009/08/gsevent-recording-and-playback-in-30.html

which as far as I can tell is how iCuke works. I think that approach
is more robust if we can get it to work.

While I can touch down and up (which gives a different way of
implementing 'touch'), my move events dont seem to work for some
reason. Perhaps we can help each other out.

Let's discuss it here, if you are interested.

CAVEAT this is ugly and it doesn't work. I'm just trying to get
through with a proof of concept I'll make it general and clean it up
if I get it working.

https://gist.github.com/1023433

* the core part of swipe creates a gs_hand_info_t from a dictionary
representation of an event - this is taken verbatim from iCuke (but
doesn't work here for some reason)

* to get dictionary representations, swipe calls out to
swipeEventsForView which creates a touch "down" event, a series of
"move" events and a touch "up" event inside the selected view (lots of
hardcoded stuff there now should generalize).


/Karl

[Steven Solomon]

Hmm , did you take a look at some of the touch synthesis stuff done in
the UIQuery class, (Im not trying to claim i understand any of it;
because i don't) but maybe there is something in there that could
help.

My approach first was to wrap the tap event into an "InvisibleFinger"
class and from there i construct a line to trace the intermediate
points between a start and end point https://github.com/steven-solomon/UISpec/commits/frank_additions

On Jun 13, 3:02 pm, Krukow <karl.kru...@gmail.com> wrote:
> On Jun 13, 8:29 pm, Steven Solomon <solomon.steve...@gmail.com> wrote:
>
> > Hey Karl,
>
> > hows your implementation coming? mine is giving some headaches
>
> Not too well, I'm afraid :( I'm in too deep, it seems. (Last hour I'm
> looking into linking with the private framework: GraphicsServices, to
> synthesize GSEvents/GSEventRecord but I can't figure out how to :( I'm
> completely new to using private frameworks. )
>
> My approach was to synthesize touch events in a form that is readable
> by the Recorder.
>

> http://networkpx.blogspot.com/2009/08/gsevent-recording-and-playback-...

[Krukow]

On Jun 13, 9:12 pm, Steven Solomon <solomon.steve...@gmail.com> wrote:
> Hmm , did you take a look at some of the touch synthesis stuff done in
> the UIQuery class, (Im not trying to claim i understand any of it;
> because i don't) but maybe there is something in there that could
> help.
>
> My approach first was to wrap the tap event into an "InvisibleFinger"
> class and from there i construct a line to trace the intermediate
> points between a start and end pointhttps://github.com/steven-solomon/UISpec/commits/frank_additions

[snip]

Yes, originally my naive approach was something like this

https://gist.github.com/1018757

but it didn't work, so I started looking at how iCuke does it (since
it claims to support swipe - I haven't actually tested it with iCuke).

If I understand correctly what you are trying to do, I suspect you'll
run into the same problems as I had: once you construct the line
between start and end you need to synthesize "UITouchPhaseMoved"
events - this was where my problems started :) Touch up and down seem
to work, but it looks like my UITouchPhaseMoved events are ignored (so
I'm doing it wrong!)

Hope we figure it out.

/Karl

[Steven Solomon]

Well as you predicted =) im falling into the same trap. but i feel
like im just not quite hitting the definition of swipe (no the right
number of moved events maybe?)
whats your take?

[Krukow]

On Jun 14, 5:56 pm, Steven Solomon <solomon.steve...@gmail.com> wrote:
> Well as you predicted =) im falling into the same trap. but i feel
> like im just not quite hitting the definition of swipe (no the right
> number of moved events maybe?)
> whats your take?

As mentioned, I'm taking a different approach using this Recorder API.
I think this would lead to a more robust solution than manually
calling touches-methods on the views.

Using the Recorder API I've managed to record a series of events that
occur when I perform the swipe-right to enable the delete button on
the table cell. If I use these exact recorded events as my
implementation of swipe it works. So what I am saying is that it must
be possible to make it work using the Recorder API.

What I need to do now is to replace the loading of my recorded events
with the dynamic synthesis of events corresponding to a swipe.

I have to teach a course the next three days, but when I get back,
I'll try my best to make it work. Also, once I get back I can put
something up on Github in case you and other people are interested.

/Karl

[Krukow]

On Jun 14, 8:57 pm, Krukow <karl.kru...@gmail.com> wrote:
> What I need to do now is to replace the loading of my recorded events
> with the dynamic synthesis of events corresponding to a swipe.
>
> I have to teach a course the next three days, but when I get back,
> I'll try my best to make it work.  Also, once I get back I can put
> something up on Github in case you and other people are interested.
>
> /Karl

OK, I've managed to simulate a "swipe" on a table cell to enable the
showing of the "Delete" button. I've extended the UIQuery api with a
command that is simply "swipe" right now. It can be called using,
e.g.,

frankly_map( "tableViewCell index:1", "swipe" )

The swipe command is only at the "proof-of-concept" stage for now, but
I've gotten it to a place where I can at least run it in the
EmployeeAdmin example in the simulator.

Here is a short screen cast.

http://screencast.com/t/D6cgKv2DPz

During the next week I'll clean up the code and see if I can
generalize it. It still needs a lot of work.

/Karl

[Rodney Degracia]

Krukow,

Good work! I can't wait for your git pull request to come in. I think this is very useful!

Rodney Degracia

[Steve Solomon]

Awesome good work

On Jun 21, 2011, at 4:58 AM, Krukow wrote:

[Krukow]

On Jun 21, 10:58 am, Krukow <karl.kru...@gmail.com> wrote:
> During the next week I'll clean up the code and see if I can
> generalize it. It still needs a lot of work.
>
> /Karl

For those that want to *experiment* with it. I've created a pul
request

https://github.com/moredip/Frank/pull/23

Note this isn't exactly production code just yet :)

/Karl