Process of Automating Mobile Gestures
In this section, we will be learning about the process of Automating Mobile Gestures and the process to build up arbitrary gestures with multiple actuators.
You must understand that the Selenium WebDriver spec provides support for certain kinds of mobile interaction. Such that parameters it does not always match to the functionality of the underlying device automation. In which case, Appium implements the new TouchAction / MultiAction API defined in the newest version of the spec. Moreover, this is different from the earlier version of the TouchAction API in the original JSON Wire Protocol.
Also, these APIs allow you to build up arbitrary gestures with multiple actuators.
- W3C actions – W3C actions are implemented to the best of the limitations of the operating systems’ test frameworks. e.g. WDA cannot handle zero wait PR.
API doc and API docs of each client help to understand how to call them.
TouchAction – TouchAction objects contain a chain of events. In all the appium client libraries, touch objects are created and are given a chain of events.
The available events from the spec are: * press * release * moveTo * tap * wait * longPress * cancel * perform
Here’s an example of creating an action in pseudocode:
TouchAction().press(el0).moveTo(el1).release()
The above simulates a user pressing down on an element, sliding their finger to another position, and removing their finger from the screen.
Appium performs the events in sequence. You can add a wait event to control the timing of the gesture.
moveTo coordinates are now absolute to the current position. For example, dragging from 100,100 to 200,200 can be achieved by:
.press(100,100) // Start at 100,100
.moveTo(200,200) // Passing absolute values of 200,200 ending up at 200,200
The appium client libraries have different ways of implementing this, for example: you can pass in coordinates or an element to a moveTo event. Passing both coordinates and an element will treat the coordinates as relative to the element’s position, rather than absolute.
Calling the perform event sends the entire sequence of events to appium, and the touch gesture is run on your device.
MultiTouch – MultiTouch objects are collections of TouchActions. MultiTouch gestures only have two methods, add, and perform. add is used to add another TouchAction to this MultiTouch.
When perform is called, all the TouchActions which were added to the MultiTouch are sent to appium and performed as if they happened at the same time. Appium first performs the first event of all TouchActions together, then the second, etc.
Pseudocode example of tapping with two fingers:
action0 = TouchAction().tap(el)
action1 = TouchAction().tap(el)
MultiAction().add(action0).add(action1).perform()
Bugs and Workarounds – An unfortunate bug exists in the iOS 7.0 – 8.x Simulators where ScrollViews, CollectionViews, and TableViews don’t recognize gestures initiated by UIAutomation (which Appium uses under the hood for iOS). To work around this, we have provided access to a different function, scroll, which in many cases allows you to do what you wanted to do with one of these views, namely, scroll it!
Scrolling – To allow access to this special feature, we override the execute or executeScript methods in the driver, and prefix the command with mobile:. See examples below:
To scroll, pass direction in which you intend to scroll as parameter in Java –
JavascriptExecutor js = (JavascriptExecutor) driver;
HashMap<String, String> scrollObject = new HashMap<String, String>();
scrollObject.put(“direction”, “down”);
js.executeScript(“mobile: scroll”, scrollObject);
Sample to scroll using direction and element in Java –
JavascriptExecutor js = (JavascriptExecutor) driver;
HashMap<String, String> scrollObject = new HashMap<String, String>();
scrollObject.put(“direction”, “down”);
scrollObject.put(“element”, ((RemoteWebElement) element).getId());
js.executeScript(“mobile: scroll”, scrollObject);
Swiping – This is an XCUITest driver specific method that is similar to scrolling (for reference, see https://developer.apple.com/reference/xctest/xcuielement).
This method has the same API as Scrolling, just replace “mobile: scroll” with “mobile: swipe”
Automating Sliders in iOS using Java
// slider values can be string representations of numbers between 0 and 1
// e.g., “0.1” is 10%, “1.0” is 100%
WebElement slider = driver.findElement(By.xpath(“//window[1]/slider[1]”));
slider.sendKeys(“0.1”);
Android – The best way to interact with the slider on Android is with TouchActions.
