Advanced Interface Operations¶

This chapter introduces you to deeper automation interfaces. You can use these interfaces to complete various detailed operations. This chapter contains a lot of content, and if this is your first time, we recommend that you patiently read through each section.

Tip

When writing automation code, you can directly enter the command ‘lamda’ in the terminal on the right side of the remote desktop, and execute the following test code or perform element selection tests or click tests yourself. This can speed up your writing and verification process.

Getting Elements¶

You may already have some understanding of this from the basic knowledge or previous text. You need to find the relevant elements through selectors to perform operations. You should also have seen where to get selector parameters. Now some of our introductions below will revolve around this element. You can see information about the “Agree” element on the right side of this image.

Sample Element

Attention

The element you click directly on the left interface may not be the actual element, because it may overlap with other elements in size and position. Usually, we will display multiple overlapping elements in the right information bar, you can scroll up and down to see which one is really needed. You can also manually traverse all elements by pressing the TAB key on the left selection interface.

For the above element, we generally get it through text, using text when there is no other element on the current interface whose text is also “Agree”. This is the simplest method. Alternatively, you can also use resourceId, but you need to note that the resourceId here does not represent a unique ID, it represents a resource ID, and an interface may contain many elements with the same resource ID. Other fields such as packageName, checkable, etc. are generally not commonly used, but can be tried if text, resourceId, description, etc. are not available. We can get this element in the following ways.

element = d(text="同意")
element = d(text="同意", resourceId="com.tencent.news:id/btm_first_agree")
element = d(resourceId="com.tencent.news:id/btm_first_agree")

Element Click¶

Call the following interface to perform a normal element click operation. In the context, it will implement the effect of manually clicking agree.

element.click()

If you need to specify the position to click on the element, you can specify corner. Corner.COR_CENTER represents clicking the center point of the element. You can also click its top-left corner or bottom-right corner (COR_BOTTOMRIGHT).

element.click_exists(corner=Corner.COR_TOPLEFT)

Perform a long press operation on the element. If it doesn’t exist, an exception is thrown. This interface also supports corner and cannot specify the long press time.

element.long_click()

Click if the element exists. If the element does not exist, calling this interface will not raise an exception. This interface also supports corner.

element.click_exists()

>>> element.click_exists()
True

Element Existence¶

In many cases, you need to check the existence of elements before further operations, otherwise subsequent processes may result in exceptions or operating on the wrong things in the wrong interface. You may need to use the following interface to determine existence in some cases.

element.exists()

Element Information¶

In some cases, you may want to get part of the element’s information, such as the coordinates, region, or text or description information contained on the element. You can read element information through the following interface.

element.info()

In our test element above, the information output for this test element is:

>>> info = element.info()
>>> print (info)
bounds { ... }
className: "android.widget.TextView"
clickable: true
enabled: true
focusable: true
packageName: "com.tencent.news"
resourceName: "com.tencent.news:id/btn_first_agree"
text: "\345\220\214\346\204\217"
visibleBounds { ... }

Hint

You may find that some fields are missing in the printed information above, such as description, etc. This usually means that this field is an empty value or false. You can still access the field through attributes to get its value.

You can see that this information is a bit complex, which is the default print format of protobuf. You can directly access the corresponding attributes to print out their actual values. For example, if you want to read the value of the element’s text, you can directly use it like this.

>>> info = element.info()
>>> print (info.text)
同意

Of course, there is also some information related to element area coordinates, which you can also access. For example, if you want to get the area information corresponding to the element, you can use it like this to print area information, or you can obtain it as a variable for subsequent operations.

>>> info = element.info()
>>> print (info.bounds)

The output or returned value is area information (Bound), which you will find is a parameter that some screenshot interfaces will also use. Yes, you can submit this parameter to the screenshot interface to take a screenshot of this element alone, but we have already encapsulated it for you.

You may also want to get the width and height of the element to calculate some offsets, such as calculating the offset of other relative elements.

>>> info = element.info()
>>> print (info.bounds.width, info.bounds.height)
484 138

Or, get the center point of the element, or corner points such as the top-left corner or bottom-right corner. Of course, the following interfaces usually return Point information, and you can also get the corresponding X and Y axis device screen coordinates from the Point object.

>>> info = element.info()
>>> print (info.bounds.center())
x: 792
y: 1908
>>> print (info.bounds.center().x)
792

The following call is used to get the corner coordinates of the element. The example gets the coordinates of the top-left corner of the element. In addition to this, it also supports getting coordinates of bottom-right, top-right, bottom-left, etc., for a total of four corner points.

>>> info = element.info()
>>> print (info.bounds.corner("top-left"))
x: 550
y: 1839
>>> print (info.bounds.corner("top-left").x)
550

Element Traversal¶

You can also traverse all elements selected by the selector. Normally, in the context, this element may only have one, so if you are testing, please select other selectors for testing. You can traverse directly using a for loop or other methods on the selector.

for i in element: print (i)

Or, if you know that there are multiple matching elements and want to get the specified Nth matching element, you can use the following interface to get it.

element_3rd = element.get(3)

Element Count¶

Usually, you won’t directly use this interface. The following call can get the number of elements matched by your current selector.

>>> element.count()
1

Element Screenshot¶

We support element-level screenshots, which can take images of elements alone without needing to take a full-screen screenshot and then crop.

element.screenshot(quality=60)

You can directly use getvalue to get the binary data of the screenshot after taking it, or directly pass it to PIL Image.

>>> element.screenshot(quality=60).getvalue()
b'\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xe2\x02(ICC_PROFILE\x00\x01\x01\x00\x00\x02\x18\x00\x00\x00\x00\x02\x10\x00\x00mntrRGB XYZ \x00\x00...

Or, if you don’t need to process further, you can also choose to save the screenshot directly to a local file.

>>> element.screenshot(quality=60).save("image.png")

Waiting for Elements¶

In some cases, you may need to determine whether the current page has finished loading. Usually, you can determine whether the current page has finished loading by checking whether the relevant elements have been displayed. The following example will wait for the “Agree” element to appear, with a maximum waiting time of 10 seconds.

Hint

The waiting time here is in milliseconds, so 10 seconds needs *1000, 10 seconds = 10000 milliseconds.

element.wait_for_exists(10*1000)

>>> element.wait_for_exists(10*1000)
True

Of course, we not only support waiting for elements to appear, but also waiting for elements to disappear, that is, until the element disappears from the interface.

element.wait_until_gone(10*1000)

>>> element.wait_until_gone(10*1000)
False

Text Input¶

Text input is a relatively important area. We cannot input text on an agree button because it’s a button. Now let’s take an input box element as an example. The basic information for this element is shown below.

Text Input

Attention

There may be some things to note when getting input box elements. Please note that when getting input box elements, your input method must be in a popped-up state before finding the relevant elements, and it is recommended to look carefully, otherwise what you get may not be the real input box.

Hint

In the process of automation code, to have the input method in a popped-up state, you only need to write code to first click on the parent displayed input box.

For the input box above, we can call the following interface to input the string “Hello World” into the input box. Of course, it also supports you inputting English or other Unicode strings. You only need to use it like below to input text into the box.

>>> element = d(text="搜索感兴趣的内容")
>>> element.set_text("你好世界")
True

Or, if you suddenly want to get the currently displayed text content of this input box, then you can call it like this.

Attention

You can see that we have changed to a different selector here. Our initial selector was based on text, but after we input text, the content changed so that element no longer exists, so we use another selector. Using the appropriate selector is very important, but since we’ve already written it, let’s just go with it.

>>> element = d(className="android.widget.EditText")
>>> element.get_text()
'你好世界'

Or to clear the currently input content. Usually, inputting text will automatically clear the previous text, but you can also clear it manually.

Hint

Actually, continuously using the keypress interface to loop through pressing the BACKSPACE key can also achieve a similar effect.

>>> element = d(className="android.widget.EditText")
>>> element.clear_text_field( )
True

Note

In extreme cases, some places may not properly use this interface to input text. We are working on supporting this.

Normal Swipe¶

Use the following interface to perform swipe operations on the interface, such as swiping up and down to turn pages in a list. The following call implements swiping up, with step adjusted as needed. The larger the step, the slower the swipe speed, which is more suitable for swipes requiring high precision.

Attention

In lazy situations, this operation does not need to provide selector parameters, but if you encounter a situation where you cannot swipe, please set the selector condition to a suitable element, such as an element with scrollable or the first-level container of the list.

d().swipe(direction=Direction.DIR_UP, step=32)

>>> element = d(resourceId="com.tencent.news:id/important_list_content")
>>> element.swipe(direction=Direction.DIR_UP, step=32)
True

Direction Indicator	Description
Direction.DIR_UP	Swipe Up
Direction.DIR_LEFT	Swipe Left
Direction.DIR_DOWN	Swipe Down
Direction.DIR_RIGHT	Swipe Right

Fast Swipe¶

Fast swipe is similar to the behavior of a person quickly swiping the screen. This operation will quickly swipe the screen, suitable for simulating quick browsing operations. The following example swipes the screen from top to bottom. In the example, the selector is empty. You still need to decide whether to fill in the selector according to the situation.

d().fling_from_top_to_bottom()

Fast swipe from bottom to top

d().fling_from_bottom_to_top()

Fast swipe from left to right

d().fling_from_left_to_right()

Fast swipe from right to left

d().fling_from_right_to_left()

Attention

>>> element = d(resourceId="com.tencent.news:id/important_list_content")
>>> element.fling_from_bottom_to_top()
True

Updating…

Other Operations¶

# Drag this APP to categorize it into the "Shopping" folder (modify according to the actual situation)
element.drag_to(Selector(text="购物"))

#########
# Find sibling or child elements
#########
# Sometimes there will be some repeated elements or elements without obvious features, which are difficult to locate
# At this time, you can narrow down the search range by finding child/sibling elements
# Child elements, for example: a chat login box, the input box inside is a child element of the login box
# Sibling elements, for example: the username and password boxes in the chat input box are sibling elements (in normal cases)
form = d(resourceId="login_form")
form.child(index=1)
# This will get the element with index 0 under login_form
form.child(index=1).sibling()
# You can also find the forgot password button that is at the same level as login_form
# (Actually, it can already be determined by the string, so there's no need to do this, this is just a demonstration)
form.sibling(textContains="找回密码")
# They are elements themselves, you can do any element operation on them


# Others, keep swiping down/left/right/up until you reach the end
# Because it's not always possible to swipe to the bottom or detect that you've swiped to the bottom
# So the max_swipes parameter is required
d().fling_from_top_to_bottom_to_end(max_swipes=32)
d().fling_from_bottom_to_top_to_end(max_swipes=32)
d().fling_from_left_to_right_to_end(max_swipes=32)
d().fling_from_right_to_left_to_end(max_swipes=32)

#########
# scroll: More mechanical swiping
#########
step = 60
max_swipes = 32
# Swipe from top to bottom for step steps
d().scroll_from_top_to_bottom(step)
# Swipe from bottom to top for step steps
d().scroll_from_bottom_to_top(step)
# Swipe from left to right for step steps
d().scroll_from_left_to_right(step)
# Swipe from right to left for step steps
d().scroll_from_right_to_left(step)

# Others, keep swiping down/left/right/up until you reach the end
# Same as the fling description above
d().scroll_from_top_to_bottom_to_end(max_swipes, step)
d().scroll_from_bottom_to_top_to_end(max_swipes, step)
d().scroll_from_left_to_right_to_end(max_swipes, step)
d().scroll_from_right_to_left_to_end(max_swipes, step)

Interface Monitor

Image Matching Operations