Basic Knowledge¶

This chapter introduces basic knowledge related to Android automation. Please be sure to read this chapter; these topics will not be repeated later. Android automation differs greatly from conventional web automation, but they also share many similarities. In conventional web automation, you can simply use the F12 developer tools to inspect the web page layout, element IDs, and other information, and then use XPath to locate elements for operations such as clicking and waiting. The logic is similar in the Android environment: you can also select elements and perform operations such as clicking and checking through something called a selector, so there is no need to worry that it is difficult to get started.

Similarities and Differences Between Mobile and Web Automation¶

There are many similarities between mobile automation and web automation, and of course many differences as well. Take Selenium as an example: usually, three things are needed to control a web page with Selenium: first, a browser; second, WebDriver; third, Selenium. The same applies on mobile: the phone is equivalent to the browser, FIRERPA is equivalent to WebDriver, and FIRERPA's Python client library lamda is equivalent to Selenium. Their goals are also the same: to perform simulated user operations to implement testing, data collection, or automatic task execution. They are all script-driven and locate, click, screenshot, and check elements. From this perspective, the two are quite similar.

But they also differ. First, mobile automation requires a phone and a computer, whereas web automation can be done entirely on your own computer. Second, they do not use the same set of tools. Common web-side tools include Selenium, Puppeteer, and Playwright; on mobile, the usual tools are FIRERPA, AutoJS, Appium, uiautomator2, and so on.

For web automation, common locators are mainly XPath or CSS selectors based on HTML DOM structure, and element hierarchy is relatively intuitive. For mobile automation, common locators are selectors; of course, Android application interfaces also use XML layouts, so you can also use XPath selection through XML. Usually, web automation does not need to consider compatibility too much. In most cases, compatibility issues caused by device differences can be solved by pinning the browser version and launch resolution. But for Android, differences in brand, model, screen size, system version, and so on may affect the compatibility of automation code. However, do not worry too much; although there is an impact, it is limited.

Basic Automation Workflow¶

Usually, you need to research the approach first: should you perform only conventional automation, or also obtain application runtime data during automation? There are generally two data acquisition approaches: first, intercept HTTP/S communication through a man-in-the-middle; second, capture data through Hook. The man-in-the-middle method is relatively simple and suitable for regular use, but may not work for some apps. The Hook approach requires substantial reverse engineering knowledge, is difficult for beginners, and is suitable for edge cases.

Man-in-the-Middle Data Acquisition¶

The man-in-the-middle approach is relatively simple. You only need to find the relevant content in the documentation for Installing the Man-in-the-Middle Certificate and Setting the Proxy, and use it together with mitmproxy. If it is not clear, you can also refer to the official startmitm.py script, which already contains all the logic for you and can be copied or reused directly.

Hook Data Acquisition¶

The Hook approach requires at least beginner-level reverse engineering ability; if you have not learned this before, you can skip it for now. In general, the Hook approach involves writing Frida scripts, hooking relevant function calls, obtaining parameters or return values and submitting them, and then injecting the app, among other steps. You can learn simple demos and usage in the Using Frida to Report Data section.

Automation Code¶

Automation code is also indispensable, because you need automation to trigger the relevant logic. Writing automation code usually follows the following process. First, open the FIRERPA remote desktop, and you will see the interface shown below.

Now open the app you want to automate, then click the “little eye” icon in the upper-right corner of the remote desktop. You will see the interface shown below. At this point, select the element you want to operate on, and click to view its element information.

You can see the element information on the right, such as text and resourceId. Suppose we want to click this element, we can write the following code, which means “click the element whose text is ‘同意’.”

d(text="同意").click()

Alright, now you have learned the simplest way to write it. Next, write control logic such as if/else, together with interfaces such as exists, and you can implement a complete automation workflow. See, it is not that difficult.

Interface Layout Inspection¶

Under normal circumstances, writing automation code is inseparable from interface layout inspection, which is also the only way to obtain selector conditions. First, open the device's remote desktop in a browser. Then click the eye icon in the upper-right corner of the remote desktop to enter layout inspection. At this point, you can click the dashed boxes on the left screen to view the corresponding element information, and use its attributes as selector parameters. Clicking the eye icon again will close layout inspection. Please note that layout inspection does not refresh automatically as the page changes; it always stays at the screen layout at the moment you entered inspection. If you need to refresh, manually press the shortcut key Ctrl + R.

Interface Selector¶

The interface selector (Selector) is used to operate on Android elements. You can understand it as an XPath rule; although the mechanism is different, the purpose is similar. In FIRERPA, the selector class is Selector, and in most cases you do not need to use it directly. You have already seen its usage above. A complete Selector includes the following optional parameters.

In most cases, only resourceId, text, description, textContains, etc. are used as parameters. If an element has a normal resourceId, it should be preferred as the Selector, such as d(resourceId="com.xxx:id/mobile_signal"); otherwise, text can be used, such as d(text="点击进入"), or more vaguely d(textContains="点击"). description is similar to text, but description is used less often.

Screen Coordinate Definition¶

During automation, you will inevitably encounter situations where you need to operate through precise coordinates or regional coordinates. If you are not familiar with the concept of coordinates, here is an introduction to Android screen coordinates. Screens, like images, have a resolution. For Android screens, regardless of landscape, portrait, or auto-rotation, the upper-left corner is uniformly used as the origin (0,0), with the positive X-axis pointing to the right and the positive Y-axis pointing downward, forming a coordinate system, as shown in the figure.

From the figure above, the coordinate of the upper-left corner of the screen is (0,0), the upper-right corner is (1080,0), the lower-left corner is (0,1920), and the lower-right corner is (1080,1920). You can use this information to calculate the coordinates of any point on the screen.

Points on the Screen¶

In FIRERPA, some operations (such as clicking or taking screenshots) require you to provide region or coordinate information. For common coordinate points, we use the following definition, which represents the point with coordinates (100,100) on the screen.

Point(x=100, y=100)

Definition of a Region¶

A region refers to a rectangular area on the screen. Its definition is slightly complicated, so please read carefully. We use Bound to represent a region on the screen, and it requires four parameters: top, left, bottom, right. You may be a little confused; please be sure to understand the following carefully: top indicates the pixel distance from the top edge of the rectangle to the top edge of the screen; left indicates the distance from the left edge of the rectangle to the left edge of the screen; right indicates the distance from the right edge of the rectangle to the left edge of the screen; bottom indicates the distance from the bottom edge of the rectangle to the top edge of the screen. In short, all distances are X/Y-axis distances measured from the origin at the upper-left corner of the screen. The following figure helps with understanding. The phone screen is still 1080x1920 and is currently in portrait orientation.

Now suppose the screen is divided into four equal parts, and we need to obtain the definitions of the upper-left and lower-right regions shown in the figure. According to the rules, for Region 1: the distance from the top of the rectangle to the top of the screen is 0 pixels, from the left to the left edge of the screen is 0 pixels, from the bottom to the top of the screen is 960 pixels (1920÷2), and from the right to the left edge of the screen is 540 pixels (1080÷2). Therefore, its definition is

Bound(top=0, left=0, right=540, bottom=960)

Similarly, for Region 2: the distance from the top of the rectangle to the top of the screen is 960 pixels, from the left to the left edge of the screen is 540 pixels, from the right to the left edge of the screen is 1080 pixels, and from the bottom to the top of the screen is 1920 pixels. Its definition is

Bound(top=960, left=540, right=1080, bottom=1920)

Android Application Data¶

Each Android application has a dedicated data directory on the device. Usually, application data is stored under the /data directory. You can obtain an application's data directory by calling the d.application("com.example").info() interface. In most cases, you can also directly cd to /data/user/0/com.example.test to enter the user directory. In addition to /data, some apps also store multimedia and other files in the /sdcard/Android directory.

Viewing the SMS Database¶

Sometimes you may want to check where SMS messages received by the device are stored. This is very simple; by writing an extension, you can even read the content directly and obtain it in real time through an HTTP interface! We will introduce it according to the conventional Android approach. If your situation is different, please adapt accordingly. On Android, the package name of the SMS app is usually com.android.mms, so you can switch to the directory /data/user/0/com.android.mms. Through the following operations, you will see several databases in the databases directory, among which mmssms.db is what we are looking for.

 λ 10:12 /data/user/0/com.android.mms ➥ ls -la
total 82
drwx------    7 u0_a78   u0_a78        3452 Jan  2  2021 .
drwxrwx--x  381 system   system       53248 May  2 16:46 ..
drwxrws--x    3 u0_a78   u0_a78_c      3452 Jan  2  2021 cache
drwxrws--x    2 u0_a78   u0_a78_c      3452 Jan  2  2021 code_cache
drwxrwx--x    2 u0_a78   u0_a78        3452 Jan  2  2021 databases
drwxrwx--x    7 u0_a78   u0_a78       24576 Feb 26 13:43 files
drwxrwx--x    2 u0_a78   u0_a78        3452 May  4 10:12 shared_prefs
 λ 10:12 /data/user/0/com.android.mms ➥ ls -l databases/
total 504
-rw-rw----    1 u0_a78   u0_a78       24576 Jan  2  2021 dynamic_bubble
-rw-------    1 u0_a78   u0_a78           0 Jan  2  2021 dynamic_bubble-journal
-rw-rw----    1 u0_a78   u0_a78      491520 Feb 27 04:18 mmssms.db
-rw-------    1 u0_a78   u0_a78           0 Jan  2  2021 mmssms.db-journal
 λ 10:12 /data/user/0/com.android.mms ➥

Reading is very simple, because regular app databases on Android are SQLite. However, apps with higher security usually encrypt their databases. Nevertheless, FIRERPA is powerful: in addition to supporting regular SQLite, it also supports real-time reading of encrypted databases such as WeChat (SqlCipher) AES-256, WeCom AES-128, and Alibaba's SqlCrypto (AES-128) (provided that you find the key yourself). Below we simply demonstrate reading system SMS content with one command; you can also write an extension to read it.

sqlite3 databases/mmssms.db .dump

The output may be large, but you can quickly locate the table containing the required data, and then write SQL queries by yourself. This method applies to 98% of Android apps; the remaining 2% are encrypted databases.

Viewing Encrypted Databases¶

For encrypted databases, you need to find the database key or its generation method by yourself. The following briefly introduces how to read the databases of relevant apps, only explaining how to use PRAGMA to set the key. If you are not clear what this is, please first learn about SQLite.

WeChat (sqlcipher)

PRAGMA cipher = "sqlcipher";
PRAGMA legacy = 1;
PRAGMA key = "database-key";

WeCom (wxsqlite)

PRAGMA cipher = "aes128cbc";
PRAGMA hexkey = "database-key"

Alibaba (sqlcrypto)

PRAGMA cipher = "sqlcrypto";
PRAGMA key = "database-key"

Viewing Other Data¶

Of course, the application data directory contains not only databases, but also app parameters, configuration, cache, files, and so on, such as shared_prefs (XML files). We will not introduce too much here; you can explore by yourself.

Automation Assistance Measures¶

In automation work, not all apps are suitable for selector-based locating. Some interfaces (such as games) are rendered in real time and have no Android-level page layout. For such apps, you can only use OCR or image matching for recognition. Please refer to Text Recognition and Image Matching sections for specific interfaces and usage.