End-to-end tests journey & Integration into a GitLab pipeline

Intro to end-to-end tests & GitLab integration

During our microservices journey from the past years, we have faced several issues when developing, maintaining, and integrating the end-to-end tests into our CI/CD pipeline. Let’s look at the most important topics that we encountered, and -where is the case- how we solved them:

• What are end-to-end tests?
• Why do we need to write end-to-end tests?
• When to write end-to-end tests?
• How much end-to-end test to write?
• GitLab basics
• How to: set up the runners and caching
• How to: set up the pipeline
• How to: isolate the tests
• How to: prepare the environment for running the tests
• How to: prepare the data for the tests
• How to: run the tests with Docker Compose
• How to: clean up after the tests
• How to: make the tests as fast as possible
• How to: maintain the test suite
• Conclusions

Prerequisites

In order to make the best use of this article, at least a basic understanding of Linux, Docker and Docker Compose is needed.

This article assumes that all microservices are packed as docker images and are published into a private or public docker repository.

The end to end test framework must support Junit-like runners that can be launched from Gradle, Maven, or command line.

Why write end-to-end tests?

The end-to-end tests are a safety net for refactoring and continuous enhancements. They increase the software quality by giving feedback sooner, before going into production.

Last but not least, end to end tests are a necessity because they increase the team confidence in each delivery.

When to write end-to-end tests?

Sometimes unit tests are hard to implement or to start with, especially for legacy applications where refactoring is needed in order to make the code more testable. In this case, it is safer to start with end-to-end tests.

Also, end-to-end testing can also be used as acceptance tests. If the UI mockups are clear, they can be written in parallel to the story implementation.

GitLab basics

GitLab is Berg Software’s choice for hosting the source code, and running the continuous integration and delivery pipelines. Their plans range from free to enterprise. You can create private or public repositories.

Some of GitLab’s nicest out of box features are:

• REST APIs
• CI/CD support
• protected branches
• code review support
• predefined variables
• custom runners support
• integration with different identity providers
• fine-grained permissions model
• scheduled jobs

You can define multiple organizations, each of them with their own projects and repositories.

Jobs can be triggered manually or automatically by other jobs or by user commits. Each job executes a pipeline with multiple stages. Each stage can have its own docker image that can be executed on the latest code extracted from the specified branch.

There are predefined environment variables accessible to the user inside the jobs, like branch name and commit message, which are very helpful during the build. There are also user-defined variables that can store sensitive values, and which can be protected. Only protected branches have access to protected variables.

How to: set up the runners and caching

Each job is executed by a certain GitLab runner. The runners are container images executed on the user-provided machines. They pick up jobs and execute them. You can define as many runners as you want. There are also shared runners provided free of charge, but you have to wait for other users’ builds.

We are not using shared runners. However, they can be found in GitLab in the project CI/CD settings section.

We are using a Docker Compose to register and initialize the runners one time:

version: '2'
services:
  register-runner-main-01:
    restart: 'no'
    image: gitlab/gitlab-runner:alpine
    volumes:
    - /opt/docker_conf/runner-main-01:/etc/gitlab-runner
    command:
    - register
    - --non-interactive
    - --locked=false
    - --name= Group - prj - MAIN - 01
    - --executor=docker
    - --output-limit=409600
    - --docker-image=alpine:latest
    - --docker-volumes=/var/run/docker.sock:/var/run/docker.sock
    - --docker-volumes=/opt/docker_data/runner-main-01-cache:/opt/docker_data/runner-main-01-cache
    - --tag-list=otcmain
    - --docker-privileged
    - --docker-cache-dir=/opt/docker_data/runner-main-01-cache
    - --cache-type=s3
    - --cache-s3-server-address=10.6.yyy.xx:9000
    - --cache-s3-access-key=minioxxxxx
    - --cache-s3-secret-key=yyyy
    - --cache-s3-bucket-name=runner-main
    - --cache-shared
    - --cache-s3-insecure
    environment:
    - CI_SERVER_URL=https://gitlab.com/
    - REGISTRATION_TOKEN=…

- --docker-image=alpine:latest

We have defined the volumes and mappings for the cache and for the configuration.

volumes:
    - /opt/docker_conf/runner-main-01:/etc/gitlab-runner
 
    - --docker-volumes=/opt/docker_data/runner-main-01-cache:/opt/docker_data/runner-main-01-cache
    - --docker-cache-dir=/opt/docker_data/runner-main-01-cache

    - --docker-cache-dir=/opt/docker_data/runner-main-01-cache
    - --cache-type=s3
    - --cache-s3-server-address=10.6.yyy.xx:9000
    - --cache-s3-access-key=minioxxxxx
    - --cache-s3-secret-key=yyyy
    - --cache-s3-bucket-name=runner-main
    - --cache-shared
    - --cache-s3-insecure

version: '2'
services:
  minio:
    image: minio/minio
    container_name: minio1
    restart: unless-stopped
    volumes:
    - /opt/docker_conf/minio-etc:/root/.minio
    - /opt/docker_data/minio-data:/data
    ports:
    - "9000:9000"
    environment:
    - MINIO_ACCESS_KEY=minioak
    - MINIO_SECRET_KEY=…
    command: server /data
networks:
  default:
    external:
      name: prj-bridge-network

- --tag-list=otcmain

version: '2'
services:
  runner-main-01:
    restart: unless-stopped
    image: gitlab/gitlab-runner:alpine
    container_name: runner-main-01
    volumes:
    - /opt/docker_conf/runner-main-01:/etc/gitlab-runner
    - /opt/docker_data/runner-main-01-cache:/opt/docker_data/runner-main-01-cache
    - /var/run/docker.sock:/var/run/docker.sock

In order to not block the main runners on the long execution of the end to end tests:

• we have to register and start in the same way as above;
• the trigger runners need to start the non-blocking execution of the pipeline from another project pipeline.

The startup script looks the same as above:

services:
  runner-guitrigger-01:
    restart: unless-stopped
    image: gitlab/gitlab-runner:alpine
    container_name: runner-guitrigger-01
    volumes:
    - /opt/docker_conf/runner-guitrigger-01:/etc/gitlab-runner
    - /opt/docker_data/runner-guitrigger-01-cache:/opt/docker_data/runner-guitrigger-01-cache
    - /var/run/docker.sock:/var/run/docker.sock

How to: set up the pipeline

Each project lives in its own git repository. In the project root folder, there is a file named .gitlab-ci.yml that configures the pipeline for each project. The file defines variables, cache settings, before and after scripts to be executed and the stages of the pipeline. The stages are basically the steps to be executed during the pipeline, and the conditions for the execution to be triggered. The last step of our pipeline is the end to end test stage.

You can define variables that can be used in the same file, or in all pipeline scripts:

variables:
GRADLE_OPTS: "-Dorg.gradle.daemon=false"
DOCKER_DRIVER: overlay2

stages:
  - opssetupchmodfix
  - test

To speed up the jobs execution time, we need to cache the dependencies of each build so they are not downloaded every time. The cache has a key that is picked from the predefined git environment variable CI_COMMIT_REF_SLUG, that contains the git tag or branch name for which the project is built:

<pre class="text" style="font-family:monospace;">cache:
 key: ${CI_COMMIT_REF_SLUG}
 paths:
 - .gradle/wrapper
 - .gradle/caches</pre>

We can define a script to be executed before starting the execution of the stages:

before_script:
  - export GRADLE_USER_HOME=`pwd`/.gradle

guitest:
  image: registry1.projekt.de/prj-dind-chrome
  stage: test   
  script:
    - ./ci/bin/guitest.sh "$EXECUTE_TEST_FOR_ENVIRONMENT" "$SRC_TRGR_BRANCH"
  tags:
    - guirunner

The image registry1.projekt.de/prj-dind-chrome is a custom docker image based on selenium/standalone-chrome:latest enhanced with bash, curl, openssl, git, x11 server, jdk and other tools, in order to be able to execute the end to end test in a browser environment.

Excerpt from the Dockerfile used to build the image:

RUN sudo apt-get -y install bash curl git openssl openssh-client openjdk-8-jdk libxpm4 libxrender1 libgtk2.0-0 libnss3 libgconf-2-4 xvfb gtk2-engines-pixbuf xfonts-cyrillic xfonts-100dpi xfonts-75dpi xfonts-base xfonts-scalable x11-xserver-utils x11-xkb-utils

script:
     ./ci/bin/guitest.sh "$EXECUTE_TEST_FOR_ENVIRONMENT" "$SRC_TRGR_BRANCH"

The process that triggers the execution can be another pipeline that calls the command in a stage

trigger -a ${GITLABTRIGGER_APITOKEN} -p ${ GITLABTRIGGER_GUITESTTOKEN} -t ${branch} ${GITLABTRIGGER_GUITESTID} 
-e SRC_TRGR_BRANCH=${CI_COMMIT_REF_NAME} 
-e EXECUTE_TEST_FOR_ENVIRONMENT=${GUITEST_ENVIRONMENT^^} 
-e EXECUTE_TEST_FOR_PIPELINE=${CI_PIPELINE_URL}

The artifacts section labeled ‘artifacts:’ defines what files should be made available for download after the job finishes, what is the source of those files, and for how long they should be preserved.

artifacts:
    when: always
    paths:
      - target/*
      - /opt/selenium/config.json
    expire_in: 1 week

only:
    - branch_dev1
    - branch_dev2
    - branch_preview
    - master
    - /^story-/
  except:
    variables:
      - $CI_COMMIT_MESSAGE =~ /^opssetup-*/

The first approach used the commands:

Xvfb -ac :99 -screen 0 1280x1024x16 &
export DISPLAY=:99

./gradlew -i clean runAParallelSuite aggregate -P webdriver.base.url=$TARGET_BASE_URL

This approach doesn’t scale very well. We got frequent memory issues so we decided to start the tests on a dedicated machine using the commands:

openssl enc -aes-XXalgo -d -pass env:GRP_SSL_ENCSECRET -in ci/secrets-enc/id_rsa-usr-dockerexec.dat >| ci/secrets/id_rsa-usr-dockerexec

ssh -i ci/secrets/id_rsa-usr-dockerex usr-dockerex@10.6.xx.xx  -o StrictHostKeyChecking=no "/opt/docker_exec/launch-tests.sh ${CI_JOB_ID} ${TARGET_LABEL} ${TARGET_BRANCH} ${TARGET_ENV}"

Another stage called ‘opssetupchmodfix’ is executed only when the provided commit message contains the text ‘opssetup-chmod-fix’, and only on the runner labeled with ‘otcprj’. The stage changes the permissions of the executable bash files checked out from git into the current project workspace folder and pushes the changes back to git. The default permissions are denying the execution.

opssetup-chmod-fix:
  stage: opssetupchmodfix
  image: registry1.projekt.de/prj-dind-base
  script:
    - chmod oga+x ./ci/bin/*.sh
    - ./ci/bin/fixexecuteflag.sh
  tags:
    - otcprj
  only:
    variables:
      - $CI_COMMIT_MESSAGE =~ /^opssetup-chmod-fix*/

How to: isolate the tests

You might think that isolating the tests is a bad thing because you don’t get feedback from the real system, data, and users. But actually, most of the time, the purpose of the end to end tests is to verify that the existing business-critical functionality still works as expected after performing any changes. Verifying the system consistency or performance must not be done using the end to end tests.

How to: prepare the environment for running the tests

We have created a cloud machine for running the end to end tests using terraform scripts. If multiple machines are needed, they can be created dynamically. The machine is configured using Ansible. The user ‘usr-dockerex’ is created and allowed (A) to connect via its ssh key, and (B) to execute the end to end tests launch script. The script launch-tests.sh is copied during the machine setup.

We need to be able to run in parallel multiple end-to-end tests for different branches and environments. Therefore, we are using the job id to prepend all services started by one run. We are using Docker Compose to start the containers.

To each container name we append the job ID like in the following:

container_name: ui-client${JOB_ID}

image: "registry1.projekt.de/ui-client:${ CLIENT_IMAGE_TAG}"

The microservices that are not changed have to be started as well for the end to end test. For this, we need to specify a target environment where to get the latest images for that environment. Each environment is bound to a protected branch.

Let’s suppose we have 3 microservices and only 2 are changed. Each one resides in its own git repository. The changes for the same story/feature are performed in a branch with the same name, let’s say feature1. The first environment where the feature is deployed should be the dev environment. So, the unchanged microservice3 has to use the latest image built for the dev environment. The used images are in this case the following:

•	image: "registry1.projekt.de/microservice1:feature1" 
•	image: "registry1.projekt.de/ microservice2:feature1" 
•	image: "registry1.projekt.de/ microservice3:dev1_latest"

function check_branch_existence(){
     local result=$1
    local project=$2
    local branch=$3
    local exists=$(curl -I -so /dev/null -w "%{http_code}" "https://gitlab.com/api/v4/projects/${project}/repository/branches/${branch}?private_token=${GRP_GITUSER_ACCESSTOKEN}")
    eval $result="'$exists'"
}
 

We have to check the existence of a docker image tag to launch the right container or fall back to a default image which is actually the dev image. We use the following function:

function check_image_tag_existence(){
    local result=$1
    local project=$2
    local image_tag=$3
    local repo=$4
    echo "docker inspect --type=image ${repo}/${project}:${image_tag}"
    local exists=$(docker inspect --type=image "${repo}/${project}:${image_tag}" --format "{{.Id}}" > /dev/null 2>&1 && echo $? || echo $?) 
    echo "Exists image? ${exists}"
    if [ $exists -eq 1 ]; then
	echo "try to pull image"
	local pullresult=$(docker pull "${repo}/${project}:${image_tag}";echo $?)
	echo "pull result ${pullresult}"
	echo "tried to pull image ${repo}/${project}:${image_tag}"
    fi
    exists=$(docker inspect --type=image "${repo}/${project}:${image_tag}" --format "{{.Id}}" > /dev/null 2>&1 && echo $? || echo $?) 
    echo "after second inspect $exists"
    eval $result="'$exists'"
}

We are iterating through all our GitLab microservices projects to compute the right environment variables with the container images and tags.

projects=("1233xxx" "1224xxxx"… )
project_names=(..)
project_vars=(..)
for i in ${!projects[@]};
	do
checkImageTag ${projects[$i]} ${project_names[$i]} ${project_vars[$i]} $imageTag "dockerregistry1.server" "dev1-latest"
	done
 

function checkImageTag(){
	    local curProjId=$1
		local curProjName=$2
		local curProjVar=$3
		local curProjTag=$4
		local curRegistry=$5
		local envBranch=$6
		check_image_tag_existence imageexistresult $curProjName $curProjTag $curRegistry
		if [ $imageexistresult -eq 0 ]; then 
			eval "export $(echo ${curProjVar}_IMAGE_TAG | tr [a-z] [A-Z])=$imageTag"
		else
			#fallback to the default tag
			check_image_tag_existence imageexistresult $curProjName $envBranch $curRegistry
 
			if [ $imageexistresult -eq 0 ]; then 
				eval "export $(echo ${curProjVar}_IMAGE_TAG | tr [a-z] [A-Z])=dev1-latest"
			else
				echo "image ${curProjTag} for project ${curProjName} does not exist!"
				exit 1
			fi
		fi
}

So far, we have computed (A) the container names to be unique for different jobs, and (B) the right docker image tags based on the branch name and target environment. We still miss the proper cleanup before launching the new containers for the tests.

The execution of the end to end tests are taking less than one hour so we can stop all containers started more than one hour ago, that are still running. The containers have the prefix ad. We filter those containers and count them. If there are such containers, we use docker stop giving the list of filtered containers IDs.

if [ `docker ps --filter "status=running" --filter "name=ad" | grep 'hour.* ago' | awk '{print $1}' | wc -l ` -gt 0 ] ; then
docker stop $(docker ps --filter "status=running" --filter "name=ad" | grep 'hour.* ago' | awk '{print $1}')
fi

if [ `docker ps --filter "status=exited" | grep 'hour.* ago' | awk '{print $1}' | wc -l ` -gt 0 ] ; then
docker rm $(docker ps --filter "status=exited" | grep 'hour.* ago' | awk '{print $1}')
fi

if [ `docker inspect --format='{{.Name}} ' $(docker ps -aq --filter "status=running" --filter "name=admicroservice1*" )  | cut -d r -f 3 | wc -l` -lt 5 ] ; then
 echo 'allowing not more than 5 running gui tests'
else
 echo 'there are running in parallel already 5 gui test, please retry later '
 exit 1
fi

export NETWORK_BRIDGE=nwkbguitest${JOB_ID}
[ ! "$(docker network ls | grep $NETWORK_BRIDGE)" ] && docker network create -d bridge $NETWORK_BRIDGE || echo "Network present!"
 

vardockers=`docker ps -a --filter "name=*${JOB_ID}"| wc -l`
if [ $vardockers -eq 1 ]
then
 echo "no containers"
else
        echo "stop and remove all containers"
        docker stop $(docker ps -a -q --filter "name=*${JOB_ID}")
        docker rm $(docker ps -a -q --filter "name=*${JOB_ID}")
fi

docker-compose -f /opt/docker_compose/docker-compose-apps.yml up -d

docker inspect --format='{{.Name}} {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $(docker ps -q)
docker logs -f endtoend-tests${JOB_ID}

result=`docker inspect $(docker ps -aq --filter "name=endtoend-tests${JOB_ID}") --format='{{.State.ExitCode}}'`
docker-compose -f /opt/docker_compose/docker-compose-apps.yml down
docker network rm $NETWORK_BRIDGE
docker network prune -f
docker image prune -f
docker volume prune -f
exit $result

How to: prepare the data for the tests

We use the Spock framework for system testing of our rest API. Spock is a testing and a specification platform for Java and Groovy applications that can be easily integrated with IDEs and CI pipelines via the Junit runner. Spock allows us to write specifications that describe expected features from a system under test. The tests can produce documentation for a wider audience than developers, by using the labeled blocks “given”, “when” and “then” (like in the following):

given: "an admin user"
// ...code
when: "his company settings are changed"
// ...code
then: "the user is notified"
//...code

Specific users are used for testing, in order to isolate (as much as possible) the interferences with the existing users active on the system.

At the beginning of the execution, the test data is prepared. In the end, the cleanup is performed.

The Docker Compose file from above contains the MySQL service that executes an SQL script on startup, to create the empty databases.

db-mysql:
        container_name: prj-mysql${JOB_ID}
        image: mysql:5.8
        volumes:
            - "./setmode.cnf:/etc/mysql/conf.d/setmode.cnf"
        ports:
            - "3306"
        entrypoint:
             sh -c "
               echo 'CREATE DATABASE IF NOT EXISTS at_projects  /*!40100 DEFAULT CHARACTER SET utf8 */; CREATE DATABASE IF NOT EXISTS at_server; …' > /docker-entrypoint-initdb.d/init.sql;
               /usr/local/bin/docker-entrypoint.sh --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
             "

How to: run the tests with docker compose

All microservices and required runtime dependencies are declared in the Docker Compose file. The container names are unique, while the image tags are computed from the launching script and provided as environment variables.

All services for one job are started in the same network including the database server.

networks:
  default:
    external:
      name: ${NETWORK_BRIDGE}

A typical back end service looks like this:

services: 
 prj-server:
    image: "registry1/prj-server:${PRJ_SERVER_IMAGE_TAG}"
    container_name: prj-server${JOB_ID}
    environment:
      - "SPRING_PROFILES_ACTIVE=otcdev,swagger"
      - "SQL_SERVER_IP=db-mysql${ JOB_ID}"
…
    ports:
      - 8080
    restart: always
    mem_limit: 2500m
    depends_on:
        - db-mysql

ui-client:
    image: "registry1 /ui-client:${UI_CLIENT_IMAGE_TAG}"
    container_name: ui-client${JOB_ID}
    environment:
      - "BACKEND_SERVER_URL=http://prj-server${ JOB_ID}:8080"
      - "BACKEND_CHECKLIST_URL=http://prj-checklists${JOB_ID}:8084"
    ports:
      - 8085
    restart: always
    command: /bin/bash -c "envsubst < /opt/enviroment-endpoints.template > /opt/enviroment-endpoints.json && cp /opt/enviroment-endpoints.json /usr/share/nginx/html/assets/enviroment-endpoints.json && exec nginx -g 'daemon off;'"

Running the test is actually performed in a special container started from a chrome image. The image with all dependencies set is prepared to run the end-to-end test.

services:
 endtoend-tests:
    image: registry1.projekt-adis.de/adis-dind-chrome
    container_name: endtoend-tests${JOB_ID}
 

volumes:
      - /opt/docker_data/guitest:/repo/tests

entrypoint: >					           
        /bin/sh -c "

git config --global user.name \"${GRP_GITUSER_NAME}\";	
git config --global user.email \"${GRP_GITUSER_EMAIL}\";
 

sudo mkdir repo;
sudo chmod a+rwx repo;
cd repo;
sudo mkdir tests/${JOB_ID};
sudo chmod a+rwx tests/${JOB_ID};
sudo rm -rf adis-systemtest;

git -C . clone --branch ${PRJ_SYSTEMTEST_BRANCH} https://${GRP_GITUSER_LOGON}:${GRP_GITUSER_ACCESSTOKEN}@gitlab.com/company/projects/cust/prj/prj-systemtest.git;
        chmod oga+x ./prj-systemtest/ci/bin/systemtest.sh;
        cd prj-systemtest;

./ci/bin/systemtest.sh ISOLATED http://prj-server${JOB_ID}:8080 http://prj-projects${ JOB_ID}:8081 http://prj-checklists${ JOB_ID }:8084;

cp -R build/test-results/test /repo/tests/${ JOB_ID}/test-results-system;
cat /repo/tests/${ JOB_ID}/test-results-system/TEST-specs.TestSuite.xml;
cat /repo/tests/${JOB_ID}/test-results-system/*.xml >> /repo/tests/${ JOB_ID}/test-results-system/systestresult.log;
[ `cat /repo/tests/${JOB_ID}/test-results-system/systestresult.log | grep '\failures=.[^0].' | wc -l` -eq 0 ] && echo 'system test success' || exit 1;

cd ..;
sudo rm -rf prj-guitest;
git -C . clone --branch ${PRJ_GUITEST_BRANCH} https://${GRP_GITUSER_LOGON}:${GRP_GITUSER_ACCESSTOKEN}@gitlab.com/company/projects/client/prj/prj-guitest.git;
chmod oga+x ./prj-guitest/ci/bin/*.sh;
cd prj-guitest;
 

./ci/bin/guitest.sh ISOLATED http://prj-client${ JOB_ID} http://prj-admin-client${ JOB_ID};

cp -R target/site /repo/tests/${ JOB_ID}/test-site;
        cp -R build/test-results/runAParallelSuite /repo/tests/${ JOB_ID }/test-results;
        cp -R build/reports/tests/runAParallelSuite /repo/tests/${ JOB_ID }/test-reports;
        cat /repo/tests/${ JOB_ID }/test-site/serenity/results.csv;
        [ `tail -n +2 /repo/tests/${ JOB_ID }/test-site/serenity/results.csv | grep -v SUCCESS | wc -l` -eq 0 ] && echo 'gui test successful' || exit 1;
        "
    mem_limit: 7500m

How to: clean up after the tests

After running the test suites, we execute the cleanup step. The cleanup can fail because of various reasons, such as abrupt program termination due to the lack of resources, corrupted data or network failure, bad cleanup procedure, or redeployment in progress. One way to deal with the situation is to also do the cleanup at the beginning of the test. If the cleanup does not succeed, then fail the test. Sometimes you cannot delete any created resources if the delete API is not available or the cleanup is too expensive.

Our solution is to always use a clean database for each end-to-end test execution and to populate it with the system tests. In order to do that, we needed an isolated environment.

How to: make the tests as fast as possible

The end-to-end tests are executed in parallel, to minimize execution time. In order to do that, the tests are split into independent test suites. This means that the data consumed or produced by one suite does not alter the data of other suites running in parallel. Pay attention when certain users’ actions impact all other users’ running sessions. For instance: the user language is changed, the user preference to not display the menu is changed, the privileges are changed.

As seen above, there is a Gradle task that extends the standard test task to which we provide the application URLs where to point the browser:

For the standard test task (i.e. the one where the browser should point to, based on the application’s URLs), there is a Gradle task that extends it.

task runAParallelSuite(type: Test) {
    systemProperty "webdriver.base.url", findProperty("webdriver.base.url")
    maxParallelForks = 3
    forkEvery = 1
    include '**/**TestSuite.class'
    testLogging.showStandardStreams = true
}
 

Other factors to influence the performance are the logging, and the capturing of screenshots. We have reduced the logging to an acceptable level, and have set up the screenshots to only be captured in case of failure. The serenity.properties file, located in the root of the end to end project, contains the configurations:

serenity.take.screenshots=FOR_FAILURES
serenity.logging=NORMAL

chrome.switches=--headless --disable-extensions --disable-gpu --disable-dev-shm-usage --no-sandbox

Inside the end-to-end tests xpaths based on ids have to be used for maximum performance.

How to: maintain the test suite

User journeys are written for business-important scenarios. Here, the user performs different actions, to achieve a certain goal. The tasks are performed in the user interface by interacting with it. The test asks questions or makes assumptions about the result of the visible task.

Group the code into reusable tasks or actions performed by the user:

public abstract class BaseLoginTask implements Task
public class OpenMyProfilePage implements Task
public class EnterValueIntoField implements Task
 

public class ConfirmationMessageAppears implements Question<String>
public class ValueInColumnOfTable implements Question<List<String>>
 

public static Target ADD_TASK = Target.the("AddTask button").located(By.id("btn-add-tasks"))

Target.the("The second selected interaction").locatedBy("//div/span/app-link-or-text/a");

WaitUntil.the(ChecklistTemplates.ADD_CHECKLIST_TEMPLATE_BUTTON,             WebElementStateMatchers.isEnabled()) 
.forNoMoreThan(TimeConstants.SECONDS_10). seconds().performAs(actor);
 

public static final int SECONDS_1 = 1 * factor;
    public static final int SECONDS_3 = 3 * factor;
    public static final int SECONDS_10 = 10 * factor;    
    public static final int LOADER = 5 * factor; 
    public static final int LONG_LIST = 10 * factor;
 

#How long webdriver waits for elements to appear by default, in milliseconds.
webdriver.wait.for.timeout=5000
#How long webdriver waits by default when you use a fluent waiting method, in milliseconds.
webdriver.timeouts.implicitlywait=5000
#How long should the driver wait for elements not immediately visible, in milliseconds.
serenity.timeout=5000
 

WaitUntil.the(MainUIUtilities.LOADER, WebElementStateMatchers.isVisible());
WaitUntil.the(MainUIUtilities.LOADER, WebElementStateMatchers.isNotPresent()).
forNoMoreThan(TimeConstants.SECONDS_10).seconds().performAs(actor);
 

Conclusions

• Private git runners have to be registered for job execution
• Minio can be used as a pipeline cache provider
• The end to end tests must have their own GitLab project set up.
• The execution can be triggered from any project from a stage defined in gitlab-ci.yml file, using the pipeline-trigger container image.
• The tests should be executed on a dedicated/provisioned environment, isolated from external influence for better stability.
• GitLab API can be leveraged for complex situations
• The tests should be broken into independent test suites that must run in parallel, headless, with screenshots on failure only, for achieving the best performance
• User journeys must be defined to test only the business-critical happy paths.
• The backend integration tests can be used to populate the data for the end to end tests.
• For maintainability, ids must be used everywhere in UI in addition to the code review and code reuse practices.
• Wait for condition, with a timeout, must be used instead of fixed wait time.
• A cleanup of the test data should be performed on every run

＿

How do you approach end-to-end testing / pipeline integration? Got any experience that you can share? Let us know!