Advanced Interface¶

This chapter introduces more advanced automation interfaces. You can use these interfaces to perform a variety of detailed operations. The content in this chapter is extensive. If this is your first encounter, we recommend reading through each section patiently.

Getting Elements¶

You may have already gained some understanding of this in the Basics or earlier sections. You need to locate relevant elements via selectors to perform operations. You should also have seen where to obtain selector parameters. The following introduction will revolve around these elements. You can see information about the element “同意” (Agree) on the right side of this image.

For the above element, we generally obtain it via text. Using text works if there is no other element on the current screen that also has the text “同意”—this is the simplest method. Alternatively, you can use resourceId, but note that resourceId here does not represent a unique ID; it represents a resource ID, and a single screen may contain many elements with the same resource ID. Other fields like packageName, checkable, etc., are not commonly used, but if text, resourceId, description, etc., are unavailable, you can try these fields. We can locate this element in the following ways:

element = d(text="同意")
element = d(text="同意", resourceId="com.tencent.news:id/btm_first_agree")
element = d(resourceId="com.tencent.news:id/btm_first_agree")

Element Click¶

Call the following interface to perform a normal element click, which will achieve the effect of clicking “同意” (Agree) manually in the context.

element.click()

If you need to specify the click position on the element, you can provide the corner parameter when calling the click method. For example, Corner.COR_CENTER means clicking the center of the element, and you can also click its top-left corner or bottom-right corner (Corner.COR_BOTTOMRIGHT).

element.click_exists(corner=Corner.COR_TOPLEFT)

Perform a long click on this element, raising an exception if it does not exist. This interface also supports corner, but you cannot specify the long‑click duration.

element.long_click()

Click the element if it exists; if not, calling this method will not raise an exception. This interface also supports corner.

element.click_exists()

>>> element.click_exists()
True

Existence Check¶

In many situations, it is necessary to check the existence of an element before proceeding with further operations; otherwise, the subsequent flow may encounter exceptions or even perform wrong actions on an incorrect screen. You can use the following interface to determine existence.

element.exists()

Element Information¶

In some cases, you may want to retrieve part of an element’s information, such as its coordinates, region information, or the text, description, etc., displayed on the element. You can read element information with the following interface.

element.info()

For our test element above, the output information is as follows.

>>> info = element.info()
>>> print (info)
bounds { ... }
className: "android.widget.TextView"
clickable: true
enabled: true
focusable: true
packageName: "com.tencent.news"
resourceName: "com.tencent.news:id/btn_first_agree"
text: "\345\220\214\346\204\217"
visibleBounds { ... }

As you can see, this information is somewhat complex, in the default protobuf printing format. You can directly access the corresponding attributes to print their actual values. For example, to read the element’s text value, you can simply use the following.

>>> info = element.info()
>>> print (info.text)
同意

Of course, there is also element region/coordinate information, which you can likewise access. For instance, if you want to obtain the region information corresponding to this element, you can print the region as shown below, or save it as a variable for later operations.

>>> info = element.info()
>>> print (info.bounds)
top: 947
left: 338
bottom: 997
right: 743

The output or returned value is a region object (Bounds). You will find that this parameter is also used by some screenshot interfaces. You can pass this parameter to a screenshot interface to capture only that element. However, we have already provided a more convenient method for you.

You may also want to obtain the element’s width and height to calculate offsets, such as relative offsets of other elements. You can use:

>>> info = element.info()
>>> print (info.bounds.width, info.bounds.height)
484 138

Or obtain the center or corner points of the element, such as top-left, bottom-right, etc. The following interfaces typically return a Point object, from which you can retrieve the corresponding X and Y screen coordinates.

>>> info = element.info()
>>> print (info.bounds.center())
x: 792
y: 1908
>>> print (info.bounds.center().x)
792

The following call obtains the coordinates of a corner point. This example gets the coordinates of the top-left corner; additionally, you can obtain the coordinates of the other three corners: bottom-right, top-right, and bottom-left.

>>> info = element.info()
>>> print (info.bounds.corner("top-left"))
x: 550
y: 1839
>>> print (info.bounds.corner("top-left").x)
550

Element Traversal¶

You can also traverse all elements matched by a selector. Under normal circumstances, this selector in the current context may match only one element. If you want to test traversal, choose a selector that matches multiple elements. You can directly use a for loop or other methods on the selector to iterate.

for i in element: print (i.info())

Or if you know there are multiple matching elements and want a specific Nth match, you can obtain it as follows.

element_3rd = element.get(3)

Element Count¶

You usually will not use this interface directly. The following call returns the number of elements matched by your current selector.

>>> element.count()
1

Element Screenshot¶

We support element‑level screenshots; you can capture just the element’s image without taking a full‑screen screenshot and then cropping it.

element.screenshot(quality=60)

After capturing, you can directly use the getvalue() method to obtain the binary data of the screenshot, or pass it directly to PIL.Image.

>>> element.screenshot(quality=60).getvalue()
b'\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xe2\x02(ICC_PROFILE\x00\x01\x01\x00\x00\x02\x18\x00\x00\x00\x00\x02\x10\x00\x00mntrRGB XYZ \x00\x00...

Or, if you do not need further processing, you can save the screenshot directly to a local file.

>>> element.screenshot(quality=60).save("image.png")

Waiting for an Element¶

In some situations, you may need to determine whether the current page has finished loading. This can often be judged by checking whether related elements have appeared. The following example waits for the “同意” element to appear, with a maximum wait time of 10 seconds.

element.wait_for_exists(10*1000)

>>> element.wait_for_exists(10*1000)
True

Additionally, we support waiting for an element to disappear, i.e., waiting until the element is gone from the screen.

element.wait_until_gone(10*1000)

>>> element.wait_until_gone(10*1000)
False

Text Input¶

Text input requires special attention. You cannot input text onto a button because it is a button. Now we will reselect an input field element to introduce this; the basic information of this element is as follows.

For the above input field, we can call the following interface to enter the string “你好世界” (Hello World). You can also input English or other Unicode strings; simply use it as follows to input text into the box.

>>> element = d(text="搜索感兴趣的内容")
>>> element.set_text("你好世界")
True

If you want to retrieve the currently displayed text content of that input field, you can call it as follows.

>>> element = d(className="android.widget.EditText")
>>> element.get_text()
'你好世界'

You can also clear the currently input content. Usually, inputting text automatically clears the existing text, but you can also clear it manually.

>>> element = d(className="android.widget.EditText")
>>> element.clear_text_field( )
True

Normal Swiping¶

Use the following interface to perform swipe operations on the screen, such as scrolling up and down in a list. The following call performs an upward swipe; the larger the step value, the slower the swipe, making it suitable for swipes requiring higher precision.

d().swipe(direction=Direction.DIR_UP, step=32)

>>> element = d(resourceId="com.tencent.news:id/important_list_content")
>>> element.swipe(direction=Direction.DIR_UP, step=32)
True

Fling (Fast Swipe)¶

Flinging resembles a human’s fast swipe behavior. This operation will quickly swipe the screen and is suitable for simulating rapid browsing actions. The following example flings from top to bottom; the selector is empty in the example, but you still need to decide whether to provide a selector based on the actual situation.

d().fling_from_top_to_bottom()

Fling from bottom to top:

d().fling_from_bottom_to_top()

Fling from left to right:

d().fling_from_left_to_right()

Fling from right to left:

d().fling_from_right_to_left()

>>> element = d(resourceId="com.tencent.news:id/important_list_content")
>>> element.fling_from_bottom_to_top()
True

Element Dragging¶

Drag an element to the position of another element (e.g., dragging an app icon into a folder).

element.drag_to(Selector(text="购物"))  # Drag to the target element's location

Child and Sibling Queries¶

For repeated or featureless elements, you can first locate a parent container, then use child to get child elements and sibling to get sibling elements, thus narrowing down the scope.

form = d(resourceId="login_form")              # Locate the parent container
form.child().get(1)                            # Get the first child element under form
form.sibling(textContains="找回密码")           # Get a sibling element of form whose text contains "找回密码"

# Here is a slightly more complex query
# Obtain the first result matching resourceId=com.example.com:id/resource, select its child node with resourceId=com.example.com:id/abc, and within that child node, query for an element whose description contains "一天内", then output its information.
d(resourceId="com.example.com:id/resource").get(0).child(resourceId="com.example.com:id/abc").child(descriptionContains="一天内").info()

For the example layout information below, you can use the following query to precisely select that element.

d(resourceId="com.zhiliaoapp.musically:id/bxa").child().get(3).child().get(1).info()

Fling to Edge¶

Continuously fling in one direction until no further scrolling is possible; it may not always detect reaching the end, so you must specify max_swipes.

d().fling_from_top_to_bottom_to_end(max_swipes=32)   # Fling down to the edge
d().fling_from_bottom_to_top_to_end(max_swipes=32)   # Fling up
d().fling_from_left_to_right_to_end(max_swipes=32)   # Fling right
d().fling_from_right_to_left_to_end(max_swipes=32)   # Fling left

Scroll (Constant-speed Swipe)¶

Scroll with a fixed step size step, more mechanical than swipe, suitable for scenarios requiring steady stepping.

d().scroll_from_top_to_bottom(step=60)    # Down
d().scroll_from_bottom_to_top(step=60)    # Up
d().scroll_from_left_to_right(step=60)    # Right
d().scroll_from_right_to_left(step=60)    # Left

Scroll to Edge¶

Similar to flinging to the edge, but using constant‑speed scrolling instead of fling. Again you must specify max_swipes and step.

d().scroll_from_top_to_bottom_to_end(max_swipes=32, step=60)   # Scroll down to the edge
d().scroll_from_bottom_to_top_to_end(max_swipes=32, step=60)   # Scroll up
d().scroll_from_left_to_right_to_end(max_swipes=32, step=60)   # Scroll right
d().scroll_from_right_to_left_to_end(max_swipes=32, step=60)   # Scroll left