IX. Producing Highly Compatible Automated Test Scripts¶

Before reading this tutorial, please read the Airtest+Poco Quick Start Tutorial first.

1.Preface¶

This section of the tutorial contains the following content:

How to choose Airtest/Poco
Improve snapshot script compatibility
Improve script running speed
How to quickly verify script compatibility

2.How to choose Airtest/Poco¶

In the test script, we can use both image script (Airtest) and UI script (Poco) at the same time. Compared to image script, UI script is more precise, runs faster, and has lower iteration cost. When writing scripts, how should we use these two types of scripts? This is related to the type of application being tested:

Native app:
It is recommended to use Poco's UI script directly and Airtest's image script for necessary interface judgment.
Games:
For integrated poco-SDK, use Poco's UI script, and for the necessary interface, use image script.
For SDK that cannot be integrated, use image script.

3.How to improve snapshot script compatibility¶

1) Snapshot script common problem scenarios¶

During script testing, sometimes we may encounter the following situations:

① Resolution replacement recognition failed¶

The previously written snapshot script often encounters execution failure error when changing to a different resolution phone or changing the environment.

The error message is:

airtest.core.error.TargetNotFoundError: 'Picture Template(E:\\test\\pycli\\untitled7.air\\tpl1570692325989.png) not found in screen'

This is because image recognition has a certain level of error. In the case of devices with different resolutions or when the image changes to a certain extent, it may cause the image recognition effect to be lower than expected. We can try various methods to modify the snapshot to achieve a higher success rate in snapshot recognition.

② The operation passed but the position is wrongly identified¶

After running and checking the report, it was found that the recognized position is incorrect, but Airtest considers it correct. In this case, it is necessary to modify the script by adjusting the snapshot, threshold, and other similar methods.

③ Touch is in the right place but no touch effect¶

After running and checking the report, it shows that the correct position was identified and touched successfully, but in reality, it was not successfully touched.

This is because during the script writing process, if there are continuous touching operations, the screen content may continuously change, sometimes causing the situation where the script has clearly run to the touch operation, but it did not take effect. This is because the screen content switching speed is too fast. While the interface is not yet stable, airtest performs element recognition and operations, failing to touch the corresponding element successfully. Therefore, we usually recommend waiting for a suitable amount of time after some operation steps before proceeding to the next step, for example:

from airtest.core.api import *
start_app("test_package")
sleep(5)
touch([500, 500])
sleep(1)
touch([600, 0])

2) Image recognition algorithm¶

① The image recognition algorithm used in Airtest¶

By default, Airtest will try to use three algorithms: SURFMatching, TemplateMatching, and BRISKMatching for image recognition.

In Airtest version 1.2.0 and above, a new MultiScaleTemplateMatchingPre algorithm has been added:

Therefore, without specifying any additional algorithms, Airtest will use these image recognition algorithms in turn to perform recognition before timing out.

② How the program determines the success of image recognition¶

So how does the program determine the success or failure of image recognition? Let me introduce two essential terms: threshold and confidence. Their values range from 0 to 1. Each image recognition script has a threshold used for result filtering, with a default value of 0.7.

Confidence refers to the confidence level of the recognition results of the aforementioned image recognition algorithms:

We can understand it this way: within a certain timeout period, Airtest will use the above 4 algorithms in turn to perform image recognition in a certain order. If the result recognized by the algorithm has confidence greater than the threshold set by the image script, the image recognition is successful, and the recognition process ends. If the 4 algorithms have not found a result with confidence greater than the threshold within the timeout period, the program considers this image search to have failed.

③ How to view image recognition results on log¶

In general, if image recognition fails, the program will report an error airtest.core.error.TargetNotFoundError: 'Picture Template(D:\test\taikang_test.air\tpl1629971466235.png) not found in screen'. At this time, we can check the specific recognition results of this snapshot in the log, mainly looking at the threshold and the confidence of the results found by various algorithms, which can help us determine whether our threshold setting is too high, or the image is not captured well, or the confidence is too low.

3) Ways to improve snapshot script compatibility¶

① Avoid blending too much background in the snapshot¶

When taking snapshots, try to ensure that the captured image has high resolution and is clear and independent. For example, when capturing an image of a button, try not to include too many background patterns to avoid difficulties in successful recognition when the background changes. A clean snapshot like this one: is better than one with a lot of patterns in the background like this one: .

② Reduce text-only snapshot scripts¶

The algorithms used in image recognition are more suitable for identifying images of buttons (with borders) and icons. The success rate of recognizing only a few isolated words is very low. Please try to adjust the image's content to achieve better recognition results, and avoid capturing content with poor recognition effects. A snapshot with a border like this: would be better than a text-only snapshot: .

③Cleverly expand the snapshot range to add feature points¶

The recognition effect of pure text snapshots may not be good. In our eyes, perhaps the text of each icon is different. However, in the eyes of Airtest, they are too similar. We can try to modify the snapshot, use some other background styles, and modify it into a more recognizable image. For example, for an interface like this:

I need to touch the second [Install] button in the middle. If we take a snapshot directly,image we may not be able to touch it, or we may touch the wrong button. However, if we take a snapshot with more features, the returned result will be correct. Because Airtest defaults to touching the middle position of our snapshot, taking a snapshot in this way can effectively achieve the effect of touching the second [Install] button. In the following text, we can also touch the first or third [Install] button by setting the touch position target_pos.

④ Cannot rely entirely on the recorded snapshot script¶

Although we provide a convenient automatic recording function that can directly convert all current operations into code step by step, the images automatically captured in this situation are often not ideal and require manual adjustment. The following image is a snapshot taken using the automatic recording function, and it can be seen that the characteristics of the snapshot are not very clear. If the recorded snapshot script does not meet our needs very well, we need to try manually capturing snapshots with more obvious features.

⑤ Reasonably adjust the threshold¶

During each run of the image recognition script, three steps are involved: ① preliminary recognition results of the image; ② calculation of result confidence; ③ filtering of results through the threshold value in the script.

The threshold of the script plays the role of result filtering here. The threshold will affect the returned recognition results directly: If the threshold is set too high, it may filter out all low-confidence initial results, resulting in no effective image recognition results. If the threshold is set too low, it may return incorrect results (cases where there are no correct recognition results, because the threshold is too low, allowing incorrect results to pass the filtering). When improving the compatibility of image scripts, the most important part is to set a reasonable threshold. The default recognition threshold in the Airtest framework is 0.7, which can be adjusted appropriately.

Use the "image editor" to adjust the threshold of a single-line script

We generally use the "Image Editor" to modify the threshold. Double-click the image in the script box to open the "Image Editor". We can adjust the threshold value on the right side in real-time and touch the Snapshot + Recognition button to verify the recognition results in real-time. The confidence of the recognition result will be displayed directly on the status bar below, and the recognition result will only be returned when the confidence is greater than the threshold.
Note: It is recommended to set the threshold between [0.6, 0.9] to avoid mixing in incorrect results with a low threshold or frequently failing to find results with a high threshold.

Set the global threshold for the script

If we want to directly set the threshold for the image script of the entire test script, we can set the global threshold at the beginning of the script:

# The global threshold is in the range [0, 1]

from airtest.core.setting import Settings as ST
ST.THRESHOLD_STRICT = 0.7 # The default threshold for the assert_exists statement, generally higher than THRESHOLD
ST.THRESHOLD = 0.7 # Default threshold for other statements

⑥ Enable RGB color recognition¶

When recognizing images, Airtest will first turn images into grayscale images before the recognition. Therefore, if two buttons have the same shape and content but different colors, Airtest will consider them the same content. In the example below, if only the second red "[Delete] button is captured, Airtest will consider the other two gray-black [Delete] buttons to be the same.
By selecting the RGB option (double-click the image to open the image manager and select it) or adding rgb=True in the code, we can force the use of color images for recognition. This way, we can better recognize the red [Delete] button.

touch(Template(r"tpl1629971466235.png", rgb=True, record_pos=(-0.254, -0.238), resolution=(1022, 673)))

⑦ Set the touch position traget_pos to touch¶

When an image is recognized, Airtest will, by default, touch the center of that image. Sometimes, we hope to touch other positions after an image is recognized, which can be achieved by altering the attribute of target_pos. For example: We hope to touch the "Upgrade" button in the image above, not the other options. And only capturing the upgrade button cannot meet our needs. We can now consider expanding the snapshot area to the red dashed line box. After taking the snapshot, we can view this square image as a nine-square grid, where each dark red dot on the box represents a number. Set the value of target_pos to the position you want to touch. In this example, we can place the bottom of the snapshot area exactly on the "Upgrade" button, and then set target_pos = 8 to touch the button precisely. The value range of target_pos is 1 to 9, [1, 9], and must be an integer. The default value is 5 (the center of the image). Similarly, in the example of the install button mentioned above, the first and third buttons can also be recognized by setting the target_pos:

⑧ Customize statements to improve image script compatibility¶

We can improve compatibility through syntax for different device aspect ratios, different device resolutions, and multiple font types. This approach requires connecting devices with compatibility issues and including corresponding snapshots in the search list. The code script is as follows:

picList = [pic1,pic2,pic3]  # a list of objects for screenshot
for pic in picList:
     pos = exists(pic)
     if pos:
         touch(pos)
         break  # Execute touch when any image on the list is found

Note: If there is no break statement in the for loop, it will cause the logic to search through all the images (and perform a touch operation when found) instead of immediately returning upon finding the appropriate result.

⑨ Specify the resolution adjusting method for the game¶

When using devices with different resolutions for image recognition, it may lead to a low success rate. So Airtest provides a default resolution adjusting method (using the Cocos engine default scaling rules). The codes are here. The best way to improve the recognition accuracy of 2D games is to clearly specify the resolution adjusting method for your game. For example, you can directly write it at the beginning of the .air script file like this:

from airtest.core.api import *

def custom_resize_method(w, h, sch_resolution, src_resolution):
    return int(w), int(h)

# Replace the default RESIZE_METHOD
ST.RESIZE_METHOD = custom_resize_method

The above code specifies a custom scaling rule: return to the original value directly without considering the resolution, and all UI are not scaled (some games use this strategy). The RESIZE_METHOD here refers to the custom_resize_method we defined, are:

w, h # The width and height of the recorded UI image
sch_resolution # The screen resolution when recording
src_resolution # The screen resolution during playback

Output as:

The width and height of the UI image during playback To customize your RESIZE_METHOD, you only need to know the scaling rules of the game you are testing, and then implement it using code in custom_resize_method. This can greatly improve the success rate of image recognition on devices with different resolutions.

⑩ Expanded Reading¶

The comparison of the effectiveness of various image recognition algorithms used by Airtest can be found here.

4.Convenient operations in certain scenarios¶

In some scenarios, by using alternative operations (such as directly operating result, using button commands, or remembering element positions), it is possible to avoid compatibility debugging for image recognition, and even speed up script execution.

1) Operate result¶

① Result Operation Case 1: transition animation¶

If the transition animation is very slow, we can skip it, and then use result to operate the device. For example, in the game transition animation shown in the picture below:

# perform a simple click
touch([10, 10])

② Result Operation Case 2: App introduction page¶

After opening the Kaola app, 4 introduction pages needed to be swiped through before entering. If using the UI testing statements of airtest/poco, it may take a long time to run. However, by directly using fixed result positions to swipe, it can be done in just four swipes. It should be noted that if the continuous operation of the result script is too fast, the device may not be able to respond. Generally, a sleep(1.0) should be added after each statement to wait for the device's response.

If there are many such scenarios, they can be encapsulated into a universal function, which can be called when needed. This will save both code and time.

# get the device width & height
width, height = device().get_current_resolution()
# cal swipe (start/end) point coordinate
start_pt = (width * 0.9, height / 2)
end_pt = (width * 0.1, height / 2)
# swipe for 5 times:
for i in range(5):
    swipe(start_pt, end_pt)
    sleep(1)  # wait for the device's response

2) Use the phone key command¶

In many cases, the BACK button can be used flexibly. When you open a page and want to go back to the previous page, you can choose to press the UI button, but generally, keyevent("BACK") can also achieve the same purpose, which is simple, direct, and compatible.

3) Position Memory¶

There are also some special cases. In Hearthstone, the enemy's and our heroes' positions are fixed, and you need to use this position every time you attack the enemy hero. You can consider saving the position, and then call the corresponding position directly when you need to use it. It is important to note that the wait statement should be used to get the hero's position instead of exists, because here it is necessary to get the corresponding hero's position and need to wait for its appearance (not appearing directly will trigger an error), while the exists statement will not error if no hero appears. In the above picture, when you need to use followers to attack the opponent's hero at any time, you can simply swipe the follower's image directly to the position of the opponent's hero.

4) start_app ( package name ) replace touch (application icon)¶

start_app() supports Android and iOS devices. Compared to using snapshot scripts to launch an application, using this script will be more concise and have better compatibility.

# Open NetEase Cloud Music

start_app("com.netease.cloudmusic")

5) When switching the screen, use more wait or sleep¶

Many rookies are prone to making one mistake, which is accidentally performing many consecutive touch operations. In fact, after each touch operation, the application screen is also changing in real-time. If the next touch operation is executed while the screen is still loading, it will easily lead to recognizing the wrong position or recognition timeout. For example, when entering the NetEase Cloud Music app, after agreeing to the terms of service, there will be a very long startup animation. We can only proceed to the next step of touching "Experience Now" after the startup animation ends. Otherwise, this touch operation is likely to time out due to waiting for the startup animation process:

6) Properly using Poco script¶

If the projects tested can use the Poco framework, it is recommended that everyone flexibly mix Airtest and Poco scripts when writing automation scripts to help achieve better compatibility with the scripts: For example, in a playlist on NetEase Cloud Music, if you want to select the top 10 songs, you would need to write 10 snapshot scripts if you are using a snapshot script. However, if using the Poco framework, you only need a few lines of script to traverse the nodes to select the top 3 songs, as shown below:

And when the song name changes, the script snapshot also needs to be maintained accordingly; at this point, choosing a node that remains unchanged as the target of operation can obviously improve the compatibility of our script.

5.Use multiple devices to run compatibility tests on scripts¶

1) Multiple-device parallel operation on the official git¶

We have provided a demo in the git repository of AirtestProject, which allows for Android multi-device running and generates an aggregate report. You are welcome to use it.

2) Multi-device testing function of enterprise version IDE¶

Our paid products are aimed at enterprises. Among our private cloud solutions is a version of AirtestIDE, which includes a device batch-running function that can be used to verify script compatibility. By running the same test script on multiple devices with different resolutions simultaneously, it ensures script compatibility. Welcome to our private cloud official website to learn more about the detailed content of the enterprise version IDE.