Skip to content

review of practices #30

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 13 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 9 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 31 additions & 25 deletions docs/content/practices/emulator_setup.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
# Emulator setup

Basically we have next choices:
Basically we have 2 choices:

* Manage devices automatically by `avd`
* Manage docker containers with emulators by `docker`
* Manage devices automatically via `avd`
* Manage docker containers with emulators with `docker`

Using docker image is the easiest way, however it's important to understand how docker creates device for you.
Using a docker image is the easiest way, however it's important to understand how docker creates emulators for you.

## Creating an emulator

Before starting to read this topic, make sure you've read
an [an official documentation](https://developer.android.com/studio/run/emulator-commandline)
Before starting to read this, make sure you've read
[the official documentation](https://developer.android.com/studio/run/emulator-commandline)

Firstly, you need to create an `ini` configuration:
Firstly, you need to create an `ini` configuration file for the emulator:

```ini
PlayStore.enabled=false
Expand Down Expand Up @@ -48,19 +48,19 @@ skin.name=320x480
disk.dataPartition.size=8G
```

Pay your attention that we disabled:
Pay attention to what we have disabled:

* Accelerometer
* Audio input/output
* Play Store
* Sensors:Accelerometer, Humidity, Pressure, Light
* Gyroscope

We don't really need them for our tests run. It also may improve our tests performance, because there are no background
operations related to that tasks.
We don't really need them for our test runs. It also may improve our tests performance, because there are no background
operations related to those tasks.

After that, you can run your emulator by `avd manager`, which is a part of android sdk manager. After your device
creation, you need change default generated ini file to custom one. You may see an example below:
After that, you can run your emulator via `avd manager`, which is part of the android sdk manager. After creating the emulator, you need to switch the default generated `ini` file to the custom one we defined previously.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I rephrased this. I believe this is what is intended after reading the code in the script

You can achieve that with a script like this one:

```bash
function define_android_sdk_environment_if_needed() {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I strongly recommend to have a "ready" bash file for this that we upload in this article (maybe extra subsection here to make it more prominent) and indicate how to execute -> ./bashFile.sh -args
The command can actually differ on windows though

Expand Down Expand Up @@ -109,22 +109,28 @@ define_path_environment_if_needed
create_and_patch_emulator
```

Pay your attention that you also need to wait until your emulator is fully booted.
Keep in mind that the emulator must fully boot before running any test. Otherwise the tests will fail because there is still no device ready
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, for snapshot testing I was using swarmer from Juno, which is deprecated but works pretty fine for starting several emulators and wait for all of them to be ready before running the tests.
Another new thing is Gradle Virtual Managed Devices...

Maybe worth mentioning later, after releasing the MVP?

on which they can run.

### Summary
1. create an `ini` configuration file for the emulator
2. run your emulator via `avd manager` and
3. switch the `ini` file generated in 2. with the one we create in 1.

## How to run an emulator in a Docker?

Running an emulator in a docker a way easier than manually, because it encapsulates all this logic. If you don't have an
experience with docker, you can check
[this guide](https://www.youtube.com/watch?v=zJ6WbK9zFpI) to check the basics.
Running an emulator in a docker is way easier than manually, because it encapsulates all this logic. If you don't have
experience with docker, check
[this guide](https://www.youtube.com/watch?v=zJ6WbK9zFpI) to get familiarized with the basics.

There are some popular already built docker images for you:
There are some popular docker images already built for you:

* [Official Google emulator](https://github.com/google/android-emulator-container-scripts)
* [Agoda emulator](https://github.com/agoda-com/docker-emulator-android)
* [Avito emulator](https://hub.docker.com/r/avitotech/android-emulator-29)

Talking about [Avito emulator](https://github.com/google/android-emulator-container-scripts), it also patches your
emulator with adb commands to prevent tests flakiness and to speed them up
Talking about the [Avito emulator](https://github.com/google/android-emulator-container-scripts), it also patches your
emulator with adb commands to prevent tests flakiness and to speed them up.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would be good to quickly mention how it does it.
From the description here, I take for granted that Avito emulator is the best out of the 3 here? If yes, should we stress the point in here?

What are the differences among them?

As a learner, I want to use the best for my needs


##### Run Avito emulator

Expand Down Expand Up @@ -154,18 +160,18 @@ docker rm $(docker ps -a -q)
## Conclusion

* Use docker emulators </br>
_You also will have an opportunity to run them with `Kubernetes`, to make it scalable in the future_
_You'll also have the opportunity to run them with `Kubernetes`, to make it scalable in the future_

* Start fresh emulators each test batch and kill them after all of your tests finished</br>
_Emulators tend to leak and may not work properly after some time_
* Start fresh emulators on each test batch and kill them after all of your tests finished</br>
_Emulators tend to freeze and may not work properly after idling for some time_

* Use the same emulator as on CI locally</br>
* Use the same emulator locally as on your CI </br>
_All devices are different. It can save you a lot of time with debugging and understanding why your test works locally
and fails on CI. It won't be possible to run Docker emulator on macOS or Windows, because
and fails on CI. It won't be possible to run Docker emulators on macOS or Windows, because
of [haxm#51](https://github.com/intel/haxm/issues/51#issuecomment-389731675). Use AVD to launch them on such
machines (script above may help you)_

!!! warning

To run an emulator on CI with a docker, make sure that nested virtualisation supported and KVM installed.
To run an emulator on CI with a docker, make sure that nested virtualisation is supported and KVM is installed.
You can check more details [here](https://developer.android.com/studio/run/emulator-acceleration#vm-linux)
38 changes: 19 additions & 19 deletions docs/content/practices/emulator_vs_real_device.md
Original file line number Diff line number Diff line change
@@ -1,37 +1,37 @@
# Emulator vs Real device

This question is a trade off and there is no right and wrong answers. We'll review pros/cons and basic emulator setup on
CI
Instrumented tests can run either on emulators or real devices. Which one should we used?
This question is a trade off and there is no right or wrong answer. We'll review the pros and cons of both approaches.

## Real device

Here is pros/cons
Here the pros/cons

➕ Real environment </br>
➕ Doesn't consume CI resources
➕ Doesn't consume significant RAM, CPU and memory of the CI.

➖ Breaks often </br>
➖ Requires special conditions </br>
➖ Breaks often: Battery swells, USB port failures, OS software failing... all this happens because real devices are not designed to be intensively used continuously. </br>
➖ Requires to place them in a room with special environment conditions </br>

A real device will help you to catch more bugs from the first perspective, however talking about scalability, if you
Although it seems that a real device is a better alternative because it helps you catch bugs on a full-fledged Android environment, it comes with its own issues. However, talking about scalability, if you
have a lot of devices, you need to locate them in a special room with no direct sunlight and with a climate-control.

However, it doesn't save from disk and battery deterioration, because they are always charging and performs I/O
operations. It may be a reason of your tests failing not because of them caught a real bug, but because of an issue with
a device.
But that doesn't prevent them from disk and battery deterioration, because they are always charging and performing I/O
operations. Therefore, if your tests fail, could be because of an issue with
a device and not because of a real bug in the app under test.

## Emulator

Here is pros/cons
Here the pros/cons

➕ Easy configurable </br>
➕ Can work faster than a real device </br>
➕ Can work faster than a real device</br>
_Keep in mind that it's achievable only if you applied a special configuration and have powerful build agents_</br>
Тot demanding in maintenance </br>
Not demanding in hardware maintenance e.g. battery, disk, USB ports, display...) </br>

➖ Not a real environment </br>
➖ Consumes CI resources </br>
➖ Not the real environment on which the app will end up running</br>
➖ Consumes significant resources of the CI like RAM, CPU and memory</br>
➖ Emulators might freeze if stay idle for a long time and need to be restarted.</br>

The most benefit that we may have is a fresh emulator instance each test bunch. Also, it's possible to create a special
configuration and disable features you don't need to have in tests which affects device stability. However, you need to
have powerful machine (and definitely not one, if you want to run your tests on pull requests)
The main benefit that we may have is a fresh emulator instance on each test run. Also, it's possible to create a special
configuration and disable features you don't need to have in tests, like sensor or audio input/output, which affect device stability. Nevertheless, you need
a powerful machine (and definitely not one, if you want to run your instrumented tests on very pull requests)
89 changes: 47 additions & 42 deletions docs/content/practices/state_clearing.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,23 @@
# State clearing
Whenever we execute UI tests, it is likely that we read/write some data locally.
These changes can affect the execution of the subsequent tests, for example:

This question appears as soon as you need to run more than 1 ui test.
* We run `Test1`, it performs some http requests, saves some data to files and databases.
* When `Test1` is finished, `Test2` will be launched.
* However, `Test1` left some data on the device which can be a reason of `Test2` failing.

## Problem
That's where *state clearing* comes to the rescue: clear the data before each test

We run `Test1`, it performs some http requests, saves some data to files and databases.
<br/>When `Test1` is finished, `Test2` will be launched.
<br/>However, `Test1` left some data on the device which can be a reason of `Test2` failing.
## Strategies for state clearing

Solution — clear the data before each test
There are a few strategies to deal with this:

## 1. Clearing within a process
1. Clearing within a process
2. Clearing package data

In this case, we don't kill our application process, and we have 2 options here:
### 1. Clearing within a process

The state clearing happens *without killing the application process*. We have 2 options here:

##### Use component from a real code base <br/>

Expand All @@ -28,16 +33,16 @@ application:
Databases, Files, Preferences and Runtime cache, and should be executed before each test.
!!! danger

This solution is a bottleneck and it's better to avoid it at all. If LogoutCleaner is broken, all of the tests will be failed.
This solution is a bottleneck and it's better to avoid it at all. If LogoutCleaner is broken, all of the tests will be failed.

<br/>

##### Clear internal storage <br/>

All cache in an android application is stored in the internal storage: `/data/data/packagename/`
All cache data (e.g.local databases, shared preferences and some files) in any android application is written in the internal storage: `/data/data/packagename/`
<br/>This storage is our application sandbox and can be accessed without any permission.

Basic idea is to avoid using components from a real code base. Instead of them, use some tests rules which do the job
In order to avoid any issues, the basic idea is to avoid using components from a real code base. Instead of them, use some tests rules which do the job
for us.

```kotlin
Expand All @@ -58,32 +63,33 @@ them [here](https://github.com/AdevintaSpain/Barista/tree/master/library/src/mai

!!! warning

This solution won't in 100% of cases:
This solution won't work in 100% of cases:

1. You may have runtime cache, which can also affect your tests
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not get this point with the runtime cache. What is stored in there and how it affects? Maybe an example would help

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Imagine if we store our data in databases and in runtime cache, like hashmap
Clearing of internal storage won't clear it. We have to clear it somehow, the only way -- use some production code, which is a bottleneck

2. Test or application process may crash and prevent the launch of next tests
2. The test or the application process may crash and prevent the launch of next tests

##### Conclusion<br/>

These are pros/cons for both solutions which don't kill the process:

Fast implementation<br/>
Easy implementation. Simply add the corresponding TestRules<br/>
➕ Fast execution in the same process<br/>
<br/>
➖ Don't give you any guarantee that your app will be cleared properly<br/>
➖ Application or Test process killing will break tests execution <br/>
➖ Can be a bottleneck<br/>
➖ Can be a bottleneck<br/>
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this needs some short hint of why


Use these solutions only as a temp workaround, because it won't work on perspective in huge projects
Use these solutions only as a temp workaround, because it won't work in long-term for huge projects

## 2. Clearing package data
### 2. Clearing package data

Our aim is to simulate the same behavior as when user presses the `clear data` button in application settings.
<br/>Application process will be cleared in that case, our application will be started in a cold start.
Our aim is to simulate the same behavior as when the user presses the `clear data` button in application settings.
<br/>Application process will be cleared in that case, our application will be initialized in a cold start.

##### Orchestrator

Basically, you can achieve an isolated state, if you execute your tests like this:
The Android Orchestrator aims to isolate the state of each test by running each of them in a separate process:
That can be achieved by executing your tests like this

```bash
adb shell am instrument -c TestClass#method1 -w com.package.name/junitRunnerClass
Expand All @@ -92,22 +98,17 @@ adb shell am instrument -c TestClass#method2 -w com.package.name/junitRunnerClas
adb pm clear
```

Each test should be executed in an isolated instrumented process and junit reports should be merged into a big one
report when all tests are finished.

That's the common idea of `Orchestrator`.
That's the idea behind of `Orchestrator`.
<br/>
It's just an `apk` which consist of
only [several classes](https://github.com/android/android-test/tree/master/runner/android_test_orchestrator/java/androidx/test/orchestrator)
and runs tests and clears data, as described above.
It's an `apk` which only consists of [several classes](https://github.com/android/android-test/tree/master/runner/android_test_orchestrator/java/androidx/test/orchestrator)
that run tests and clear data, as described above.

You should install an `orchestrator` along with `application.apk` and `instrumented.apk` on the device.
It is necessary to install an `orchestrator` along with the `application.apk` and the `instrumented.apk` on the device.

However, it's not the end.
But that's not all.
<br/>
Orchestrator should somehow execute adb commands. Under the hood, it
uses [special services.](https://github.com/android/android-test/tree/master/services)
It's just a shell client and should be installed to the device.
The Orchestrator also needs to execute adb commands. For that it uses [special services.](https://github.com/android/android-test/tree/master/services) under the hood.
It's just a shell client and should be installed on the device.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After reading this, I was doubting whether I have to install it on my own, or the Orchestrator does it for me (this one right?). This should be clarified


![alt text](../images/orchestrator.png "orchestrator and test-services")

Expand All @@ -118,37 +119,41 @@ It's just a shell client and should be installed to the device.
Despite the fact that it does the job, this solution looks overcomplicated:

1. We need to install +2 different apk to each emulator
2. We delegate this job to the device instead of host machine.
2. We delegate this job to the device instead of the host machine.
<br/>Devices are less reliable than host pc
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Less reliably in the sense of "they do not lose connection to adb server, issues with battery, internet, system dialogs," or sth like that?


##### Other solutions

It's also possible to clear package data by
using [3rd party test runners](https://android-ui-testing.github.io/Cookbook/practices/test_runners_review/), like
Marathon, Avito-Runner or Flank. Marathon and Avito-Runner clear package data without an orchestrator. They delegate
this logic to a host machine
this logic to a host machine.

##### Conclusion<br/>

These are pros/cons for an `orchestrator` and 3rd party test runners solution:

➕ Does the job for us in 100% <br/>
➕ Does 100% of the job for us<br/>
➕ Avoid test failing in cascade due to the application process being killed<br/>
<br/>
➖ Slow execution _(can take 10+ seconds and depends on apk size)_ <br/>
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

10+ was for ALL the tests together due to adb pm clear right? This I did not understand very well

Orchestrator — over-complicated <br/>
➖ Slow execution <br/>
Requires to install extra components — over-complicated: <br/>

Each `adb pm clear` takes some time and depends on apk size. Below you may see some gaps between the tests which
represent such a delay
The slow execution has 2 sources:
1. The time consumed in killing and restarting the process where each test runs, multiplied by the amount of tests.
2. Executing `adb pm clear` after each test takes some time and depends on apk size. Below you may see some gaps between the tests which represent such a delay.

![alt text](../images/package_clear.png "ADB package clearing takes some time")


## Final conclusion
!!! success

Only package clear can guarantee that your data will be celared properly.
Only package clearing can guarantee that the data will be cleared properly between test executions.
Marathon and Avito-Runner provide the easiest way to clear application data.

1. You can set them just by one flag in configuration
2. They don't use orchestrator under the hood
1. One simply needs to set a flag in their configuration
2. They don't use orchestrator under the hood, avoiding its caveats



Expand Down
Loading