Introduction to Automated Testing¶

1.How to help beginners with zero experience master automated testing?¶

How to help beginners with zero experience master automated testing?

This is what the Airtest team has been striving for over the past five years: continuously improving the toolchain to allow users to easily conduct automated testing with minimal difficulty. Whether you are a newcomer to the workplace or a testing engineer just starting to learn automation, we believe that this tutorial can help you clarify the context and master the learning path of automated testing. We also hope that this hand-on tutorial of Airtest can help you get started with automated testing. After finishing the tutorial, you are expected to:

Learn about mainstream frameworks and tools for automation.
Get started with Airtest and Poco.
Perform automated testing on Android devices.
Perform automated testing on iOS devices.
Perform automated testing for WeChat Mini Program and Mini Game.
Know how to write automation scripts with strong compatibility.
...

It should be noted that the testing and automation mentioned in this tutorial refer to software testing and software automation, and testing and automation in hardware and industry are not included in the scope of this tutorial.

Are you ready? Let's get started.

2.Let's briefly talk about testing¶

Software testing can be understood as a process used to verify whether software meets requirements and if there are any defects. Based on different dimensions, software testing can be subdivided into different types.

1) According to the testing methods, it can be divided into:¶

Black box testing: The most basic functional testing, it does not concern the internal code implementation, but only verifies the correctness of input and output.
White box testing: Based on logic-drive or code-based testing, it opens the internal implementation of the code to study the correctness of the interface or specific implementation in the source code.
Gray box testing: This is a third type of testing that falls between the two above.

2) According to the testing objectives, it can be divided into:¶

Functional testing: It tests each function of the product and module.
Performance testing: It tests the various performance indicators of the system. Stress testing: It tests the load capacity of the software or system and identifies any hidden risks.
Compatibility testing: It tests the compatibility between products and software/hardware, such as the compatibility of software on various Android models.
Security testing: Through various methods, it finds out security issues in software, such as information leakage, unauthorized use, malicious destruction, and so on.
Other specialized tests: such as weak network test, power consumption test, fluency test, etc.

3) According to the stages of software development, each stage can do:¶

Unit testing: Perform white box testing of independent modules in the program, with the purpose to verify the correctness of the fundamental components of software.
Integration testing: Perform combination tests on unit modules to verify whether the correctness of the interfaces between unit modules.
System testing: Conduct a complete test on the entire system to verify its correctness and compliance.
Regression testing: Validate the functional modules that may be affected by the software changes when they occur.
Acceptance testing: This is the final stage of testing and ensures the quality of the software before it is released or launched.

4) Other common testing concepts:¶

Smoke testing: It performs a simple test on the most basic functions of software to determine at low cost if the software is testable.

Smoke testing originates from hardware testing. When the circuit board is ready, it will first be powered on. If there is no smoke, the following tests will proceed. Otherwise, it indicates that the product does not meet the basic quality requirements and needs to be remade.

Exploratory testing: This type of testing relies more on the personal experience or expertise of the testers, in other words, the tester's subjective initiative.

The importance of exploratory testing can be referred to the field of game testing, where millions of players will play games in unexpected environments and with unexpected methods. Therefore, in addition to systematic testing methodologies and advanced testing tools, game testers must also have rich game experience and exploratory test thinking.

For all the tests mentioned, it is a simple and primitive way to perform them manually, but it creates massive repetitive work during the testing process. For example, regression testing is needed for every change; compatibility testing is needed to be performed on hundreds of mobile devices with the same use case. So we hope to use codes to help us do the testing, and this is where automated testing comes.

3.Introduction to mainstream automated testing frameworks¶

The automation of software testing can generally be divided into three levels.

Unit testing at the code level
Integration testing at the interface level
Testing at the UI level

1) Automation at the code level¶

Automating at the code level generally refers to unit testing of the code. Commonly used unit testing frameworks include Junit for Java, PyUnit for Python, and so on. As this is not the focus of this tutorial, we will not delve into it in detail here. If interested, you can learn more about it on your own.

2) Automation at the interface level¶

Automated testing at the interface level mainly tests the interfaces between systems and components, with the main goal of verifying data exchange and business processes. Interface testing can test functionality, as well as performance, stress, security, and more. As interfaces are much more stable than code units, the cost of maintaining automated scripts is lower and the benefits are greater, making it a cost-effective option. Commonly used testing tools include the following:

Jmeter: A Java-based testing tool developed by the Apache for interface testing, stress testing, and performance testing. Initially designed for web testing, it gradually expands to other fields and can be used to test static or dynamic resources.
LoadRunner: A performance testing and stress testing tool provided by HP. It can simulate thousands of concurrent user operations to test system performance, with detailed analysis of test results, and is thus a good choice for performance testing and stress testing.
Robot Framework: An open-source automated testing framework with excellent scalability. The framework is written in Python and provides cross-platform support.
Postman: A powerful and user-friendly API debugging tool, the preferred choice for API debugging.

3) Automation at the UI level¶

The automated testing framework at the UI level is much more complex. In terms of platform types, there are Windows, Linux, Android, iOS, Web, and the latest Mini-Programs, etc. The following will briefly discuss the principles, architecture, and cross-platform capabilities of mainstream UI-level automated testing frameworks.

① Appium¶

Appium is an open-source automated testing tool that supports IOS, Android, Windows, and Mac applications.

Cross-platform¶

Appium can run on desktops with OSX, Windows, and Linux.

Cross-language¶

Appium adopts the client-server design pattern and extends the WebDriver protocol, so that clients can be implemented in various programming languages such as Python, Java, JS/NodeJS, Ruby, OC, and C#.

Principle introduction¶

The core of Appium is a REST-compliance web server that accepts connections and commands from clients. With the unified interface design, it can implement test cases in multiple languages, allowing clients to use their preferred languages.

After receiving the testing instructions, the server will send them to the device. The device layer uses the native testing frameworks provided by the device manufacturers, such as XCUITest Driver and UIAutomation Driver for iOS, and UIAutomator and UIAutomator2 for Android.

Appium official website: http://appium.io/

Appium Github homepage: https://github.com/appium/appium

② Selenium¶

Selenium is an open-source web application automated testing tool that can run directly on multiple browsers, just like real user operations.

Cross-platform¶

Similarly, Selenium can run on desktops with OSX, Windows, and Linux.

Supported browsers¶

Firefox, Chrome, IE, Edge, Opera, Safari

Principle introduction¶

Selenium official website: http://seleniumhq.org/ Selenium Github homepage: https://github.com/SeleniumHQ/selenium

③ Airtest Project¶

Airtest Project is an automated testing framework developed and open-sourced by NetEase. Compared to other automated testing frameworks, it mainly has the following two advantages:

Significantly reduce the costs of writing and maintaining automation scripts.

The Airtest Project hopes to complete the script recording through a WYSIWYG approach, allowing even non-programming and script-illiterate testers to automatically record scripts through normal user operations such as touch and drag. This will significantly reduce the maintenance costs of automation for enterprises and projects.

Solve the pain points in game testing.

The Airtest Project hopes to become a truly cross-engine and cross-platform automated testing platform by supporting different game engines.

Architecture diagram¶

As the diagram shows, the main testing frameworks at the bottom are Airtest and Poco. The differences between them lies in:

Airtest: It is a python-based, cross-platform automated testing framework for UI, based on image recognition principles, suitable for games and apps.
Poco: It is an automated testing framework based on UI controls search. Its core advantage is that other than Android and iOS, it also supports games, as well as WeChat mini-programs, WeChat mini-games, and H5 applications.

The entire framework is equipped with a practical IDE. With AirtestIDE, it is easy to complete script recording, perform testing tasks, and generate testing reports. Please watch the demo video for the complete process:

In the following chapters, we will explain the automated testing technology based on Airtest and Poco for Android, IOS, and other platforms.

Airtest Project official website: http://airtest.netease.com/

Airtest Github homepage: https://github.com/AirtestProject/Airtest