NetEase Automated Testing Solution - Private Cloud

1.Private Cloud Solution Overview

1) Development History

The Airtest project (automated UI testing framework) was launched in 2014 and successfully served dozens of projects within NetEase. It was then invited to be jointly released with Google at the GDC conference in 2018. Since the project was open-sourced, it has been used by tens of thousands of developers and thousands of companies. But in the internal implementation process, we found that having only a bottom-level testing framework is insufficient to meet enterprises' needs for large-scale automated testing. For example, how to build a stable and efficient device cluster, how to scientifically manage and schedule devices remotely, how to manage testing resources and continuous integration, etc., are all issues that need to be addressed by a complete automated testing solution. Regarding the aforementioned requirements, we have developed our own mobile phone cabinet, device management system DeviceKeeper, Airlab cloud-based test lab, and other software and hardware. We have built a complete automated testing solution that serves Netease internally and many external enterprises, achieving outstanding results.

2) Program Introduction

The automated testing cluster solution launched by NetEase includes the construction of device clusters, device management, script writing, device scheduling, script multi-machine execution, report generation, continuous integration and other complete solutions. It is a software and hardware integration product that can help us quickly build a private mobile cloud testing platform within the enterprise.

Image Netease Automation Testing Solution - Private Cloud

2.Detailed Product Introduction

We will introduce our entire private cloud solution in detail from four aspects: cluster construction, cluster management, script writing, and platform building.

1) Cluster Construction

The traditional way of device management is manually managed by device administrators, and testers also need to frequently borrow and return devices offline to complete their complex testing work. In addition, as the number of internal testing devices in the enterprise gradually increases, device management will also be very time-consuming and labor-intensive, and the utilization rate of devices will also be very low. Imagine, how much time would it take for one device administrator to manually perform the routine on/off maintenance on 100 devices? So how can we efficiently manage testing equipment? How can we provide stable equipment resources for testing? Our solution is to build a real-device cluster. The real-device cluster can unify the management of equipment resources (pooling of equipment resources), which allows maintenance colleagues to manage and maintain these devices in bulk online, and enables testing colleagues to quickly and flexibly obtain designated or large numbers of devices for regression testing/compatibility testing. In order to build such a real device cluster, NetEase independently developed an industrial-grade device cluster cabinet:

Image Professional cabinet with the specification of 8*4

Image Cabinets in Netease's automated testing room

This cabinet's equipment carrying capacity and shelf height can be flexibly customized. There are different specifications of cabinets available, such as 84, 108, and 10*9. The equipment drawers on each shelf can be pulled out and work independently. The well-designed ventilation system of the cabinet and the built-in industrial-grade network solution can ensure the stability of the equipment and smooth network performance during continuous testing or bulk equipment tasks. In addition to the cabinet-style server rack (suitable for placement in a server room), we have also developed a portable cabinet:

Image Portable cabinet

Compared to the cabinet-style equipment rack, the portable cabinet has a smaller load capacity, up to 10 devices at most. However, it is small and flexible and can be taken and used anywhere, not limited to being placed in an equipment room. It is very suitable for some testing and demonstration scenarios, outdoor environment testing scenarios, etc. It is even a great choice to place it directly on a workstation for testing. In addition, in order to solve the problem of long-term online charging of cabinet equipment leading to battery expansion or shortened lifespan, we have also customized and upgraded the USB hub inside the cabinet, optimizing it into an intelligent power control version. Through programming, it is easy to control the power and data on/off of the equipment, as well as perform shallow charge and discharge operations on the devices at regular intervals, greatly extending the battery life of the equipment and reducing the maintenance and update costs of the hardware.

Image Cabinet - Intelligent Power Control Hub

2) Cluster Management

After the device cluster has been successfully built, the next issue we need to consider is cluster management: On the one hand, we need to be able to monitor and maintain our device cluster very conveniently to ensure the long-term stability of the equipment cluster; On the other hand, we also need to flexibly manage and allocate our device cluster, distinguish between public devices and project-specific devices, and ensure high utilization rates while also protecting the privacy of personal devices. To solve the difficulties mentioned above, we have developed a device management system called DeviceKeeper that is compatible" with the hardware of the server cabinet:

Image NetEase DeviceFarm Cluster Solution

DeviceKeeper allows device managers to view the online status of device clusters remotely and supports one-click batch maintenance and monitoring of our device clusters:

Image DeviceKeeper-Viewing the online status of the cluster

Image DeviceKeeper-Abnormality Monitoring

Of course, it also supports management and allocation of the entire cabinet or equipment on the cabinet, ensuring that personal devices can only be accessed and used by specific project teams or users (which is more in line with the needs and privacy of internal equipment usage in enterprises) :

Image DeviceKeeper - Cabinet Management

It's also worth mentioning that the well-known real device cloud is integrated with our DeviceKeeper management system, making it convenient for testers to perform some manual testing work remotely on the web:

Image DeviceKeeper - Cloud real device page

3) Script Writing

The purpose of automated testing is to discover product issues faster and better, and effectively improve product quality. Once we have solved the device resource issues for automated testing, we need to obtain these device resources to carry out tasks such as automatic script writing, script execution, and generating test reports to discover product issues. NetEase's Private Cloud Solution not only provides us with a bottom-level automated testing framework but also provides tools such as a script production editor and a test report editor for quickly producing scripts and test reports:

Image Automated testing - Framework and tool

When selecting an automated testing framework, we need to consider both the platform's universality and ease of use. Platform universality can ensure that the framework is compatible with popular platforms such as Android, iOS, mini-programs, H5, and various game engines. A low learning curve can enable enterprises to quickly adopt automated testing and implement automated testing solutions. The open-source Airtest and Poco frameworks were developed based on these pain points. With the dedicated framework editor AirtestIDE, even testers with zero coding experience can quickly produce "simple" automated testing scripts. (Community version of AirtestIDE website: https://airtest.netease.com/)

Image Automated testing pain points and solutions

Image Airtest - Framework principle and example

Image Poco - Framework principle and example

In addition, the enterprise version of the IDE in the private cloud solution has developed more advanced features based on the community version, such as batch running (small batch compatibility testing/regression testing), one-click generation of aggregate reports, high-definition mode connection to devices (good screen compatibility and smoother performance), etc., which are more in line with the enterprise-level requirements for automatic script editing.

Image Enterprise Edition IDE - Batch Run Tasks

Image Enterprise IDE - Aggregate Report Overview

In addition, the private cloud solution also provides a Klay unified installation program for script writing, which is an enhanced tool library. It encapsulates some commonly used functions, such as automatic clicking on popup windows, sliding according to direction, using swipe to unlock the screen, and enhances some of the Airtest API functions. This library reduces the difficulty of writing some scripts, and when testers encounter similar scenarios, they can directly call the provided API without additional encapsulation.

4) Platform Building

Although the enterprise version IDE supports completing small-scale compatibility testing/regression testing (with approximately ten devices), regression testing and device compatibility testing of a larger scale require an efficient automated testing case execution platform to complete. TheAirlab cloud-based test lab provided by NetEase's private cloud solution not only supports the scheduling and execution of basic batch test scripts, but also supports unified management of testing resources, report analysis of automated testing, and continuous integration.

Image Airlab-Platform Function Schematic

Using the Airlab cloud-based test lab, testers only need to prepare testing resources and submit task configurations to start testing tasks and conduct testing on many devices.

Image Airlab - Regression Testing Task Configuration

The whole testing process requires no manual intervention, and the complete run report of this test task will be provided automatically after the task is completed. The report includes report overview, performance report, error report, screen recording of run test use case, script log, etc., wh gression Testing Task Report

Image Airlab-regression test task report overview

For regression testing and compatibility testing of stable operation functions, it is only necessary to write a test script once and save it on the platform, which can be repeated automatically multiple times, maximizing testing efficiency and saving manpower costs for testing.

Another thing worth mentioning is that the Airlab automated testing platform supports batch device automatic packaging, patch downloading, and login requirements, which helps testing personnel reduce the time spent on packaging, launching applications, and logging in. The project team only needs to provide testing resources such as test packages, login use cases, login scripts, and test accounts to the Airlab automated testing platform, configure a compatibility testing task for testing logins on the platform, select the devices that need to be packaged, and the platform will automatically complete the testing application installation, patch downloading, and login operations, and notify the project team after the task is completed:

Image Airlab - Packing Task

3.Program Advantages

1) Flexibility of device cluster construction

We support real device cluster construction in various forms and scales, which can meet the different cluster construction needs of enterprises: Suppose you need a small-scale cluster that can be flexibly placed in offices and workstations. In that case, you can choose our portable cabinet, which can accommodate 10 devices, and supports Android and iOS, but only one platform of devices can be deployed on one portable cabinet.

You can choose our industrial-grade cabinets if you need a medium to large-scale cluster. Standard specifications include 48 (32 devices), 810 (80 devices), 9*10 (90 devices), etc. We also support customized cabinet specifications based on the actual equipment scale of the enterprise. In addition, industrial-grade cabinets also support custom layer heights, which can meet the needs of enterprises to install special equipment such as car machines, translation pens, special tablets, etc.

Of course, our industrial-grade cabinets also support Android and iOS devices and support a mixed deployment by row. For example, a 4*8 industrial-grade cabinet has a total of 4 rows, with 8 devices in each row. We can choose to deploy the first row as an iOS node and put up iOS devices, while the remaining 3 rows can be deployed as Android nodes and put up Android devices.

2）The professionalism of the cabinet

Our industrial-grade cabinets have gone through several generations of hardware trials and finally confirmed the current hardware solution, which can significantly ensure the cluster operation's stability.

And we are still constantly improving and optimizing our cabinets. For example, in 2021, we upgraded the regular USB hub to a smart-controlled USB hub, which significantly extends the battery life of devices when used with an intelligent power control system.

Also, the intelligent cabinet door lock solution we are currently testing and using can enhance the cabinet's security, making it convenient for colleagues to scan the code and open the door to borrow equipment offline.

3) The extensibility of the underlying testing framework

Our image recognition framework Airtest and control recognition framework Poco are encapsulated in pure Python. When used, they can be freely wrapped in secondary functions and scripts and flexibly organize test sets.

Additionally, our framework also supports the integration of your own testing services (such as game command services), allowing for freely calling commands in the script and completing more in-depth testing functions in conjunction with the testing framework's API.

4) Integrity of the toolchain

The NetEase private cloud solution provides detailed solutions from hardware to business layers, including cluster construction, cluster management, script writing, automated task execution, and report generation.

Enterprises can flexibly purchase our private cloud solution's hardware and software products according to their actual needs, including customized cabinets and software solutions selection.

5) Privacy of privatized deployment.

Nowadays, there are many companies on the market that can provide public equipment testing services. They charge based on the number of uses or duration of use, and can also meet the testing device needs of certain enterprises.

However, public device services have drawbacks: firstly, they cannot guarantee that the equipment models meet the testing needs of the enterprise; secondly, they cannot guarantee the privacy of the entire testing process. After all, we need to upload testing packages, scripts, and other resources to a public platform. Confidentiality is indeed a significant concern.

Therefore, nowadays, many companies are increasingly inclined towards the privatization deployment of device clusters. The private cloud solution provided by NetEase also involves deploying the rack and software platform within the enterprise, effectively ensuring the privacy of device usage and testing processes. At the same time, privatization deployment can also ensure that the enterprise is able to flexibly arrange its large-scale regression/compatibility testing tasks.

4.Program procurement and implementation process

Under normal circumstances, external companies need to go through the following process to purchase our private cloud solution:

Among them, the confirmation of enterprise demand is to understand each enterprise's unique private cloud deployment requirements/automation requirements. After a clear understanding of the enterprise's needs, our operation/business colleagues can provide suitable solutions for the enterprise and the pricing.

When the enterprise confirms that the plan can meet its own needs, we will come back to confirm the quotation and sign the contract.

After the contract is signed and the company pays the deposit, we will arrange for the hardware (such as racks and accessories) to be shipped to the company. We will also send engineers to deploy the hardware and software on-site to ensure the system is successfully deployed within the company and can be used normally.

At this time, the company can inspect our products and make the final payment after passing the inspection.

At this point, the plan has been fully purchased and implemented within the company. We will also provide a dedicated technical support group for the company, making it convenient for the company to receive prompt responses and solutions to any questions or other issues that may arise while using the plan.

5.Contact information

For enterprises interested in NetEase's Private Cloud Solution, you can find us through the following ways: Tel: 020-85105163 ext. 57744 WeChat: a19520913013 QQ: 1146924173 Official website: https://airlab.163.com/b2b