Random-generated tests using a simulated environment

Customer expectations are growing, for want of better digital experiences. However, even the slightest code change could take four to six weeks, which inevitably delays the overall production. How does one tackle the situation when everyone demands quality without compromising the delivery speed, and how does one stay ahead of the competition?

Both speed and quality are equally important, therefore automation in testing and continuous integration is no longer an option but a necessity. ^[1]

For large automation projects, this might not be enough. Interactions with hardware, and even between software modules, add an extremely high level of complexity and a large range of possible fail modes that cannot be predicted. Therefore, a simulated random test environment might be the solution to overcome these problems.

1. Overview of frequently used QA measures
   • 1.1. Automated testing
   • 1.2. Simulation
     • 1.2.1. Usage
     • 1.2.2. Objectives of simulation
     • 1.2.3. Simulation Activities
   • 1.3. Continuous Integration
   • 1.4. Are simulations and automated testing enough?
2. Introducing randomness to the test environment
   • 2.1. The basic solution
   • 2.2. Pros
   • 2.3. Cons
3. Intermediary conclusion
4. Using Azure to support simulation of random-generated tests
   • 4.1. Create a virtual machine as a base
   • 4.2. Create your image library
   • 4.3. Add a virtual machine image to your image library
   • 4.4. Create a virtual machine from an image in the gallery
   • 4.5. Run the virtual machine and start the simulation
   • 4.6. Use scripts to optimize image creation
   • 4.7. Conclusion

1. Overview of frequently used QA measures

1.1. Automated testing

Automated testing relies on executing multiple predefined scenarios each time a part of the base code has been changed, and always getting the same results. This means that when changing the code base all components are working as expected.

Automated testing involves the following components ^[2]:

• The use of scripts to drive tests. The tests are prepared in parallel with the development, and it is considered that all possible scenarios are thought of in advance.
• Hardware to support the scripts. The scripts then either run automatically (at a certain interval or triggered by an event such as a commit in the source control) or manually (usually after completing a feature). In any case, they need to run on separate machines than those used for development, so that the developer can continue working.
• The use of other scripts to record results, or at least a summary of what tests ran and how they performed (succeeded or failed).
• A theory or philosophy of testing. This requires thinking in advance on all possible interactions of all modules.
• The ability to detect faults. Consider what should or should not happen.

So even though automated testing is suitable in some situations (e.g. testing all regularly used scenarios in optimal conditions or close to them), it might not be enough for projects with a large number of interacting modules. On one side, it would prove very difficult to predict all possible scenarios and all interactions at specific times; on the other side, the investment in time and hardware to run all these possible scenarios would be significant.

1.2. Simulation

Simulation relies on modeling the hardware in software, in order to determine how would the software behave once it is installed on the hardware.

1.2.1. Usage ^[2]

• Facilitate the generation of requirements
• Uncover unknown design interactions and details
• Reduce development cost by having fewer actual parts

1.2.2. Objectives of simulation ^[2]

• A quick evaluation of various design concepts, without material investment
• Demonstration of system integration (peer reviews and customer feedback)
• Refinement of design concepts by predicting effects of performance parameters on system behavior
• Verification that the simulated object performs correctly across a wide range of nominal and fault scenarios
• Identification of key variables that impact the system, and realization of the system implications, particularly with respect to potential interactions
• Reliability consequences
• Theoretical simulation results as a reference for the practical verification

1.2.3. Simulation Activities ^[2]

In order to begin developing a simulation, it is important to go through the following process flow:

• Identify the simulation goals for our specific project
• Prepare for simulation by:
  • Identifying parameters
  • Modeling parameters
• Run the prototype simulation
• Perform the test (physical test to compare the actual performance to simulation results)
• Gather data
  • Compare test results with model parameters
  • Identify new parameters needed
• Re-run simulation if needed, and gather data again
• Analyze data
• Update models
  • Determine if additional parameters are necessary
  • Review the range of parameter values as a sanity check
• Design updates

1.3. Continuous Integration

Continuous Integration (CI) is a development practice where developers integrate code into a shared repository frequently, preferably several times a day. Each integration can then be verified by an automated build and automated tests. While automated testing is not strictly part of CI, it is typically implied.

One key benefit of integrating on regular basis is that you can detect errors quickly and locate them more easily. As each introduced change is typically small, pinpointing the specific change that introduced a defect can be done quickly.

In recent years CI has become a best practice for software development and is guided by a set of key principles. Among them: revision control, build automation and automated testing. ^[3]

Continuous integration ensures that your software is built and tested regularly. It can help you to demonstrate that your software does what it claims to do, and that it does so correctly. It also helps you to rapidly release bug-fixes and more functional versions of your software. Continuous integration can also be used to automate experiments that run using software. ^[4]

1.4. Are simulation and automated testing enough?

By combining simulation and automated testing, and running these scenarios regularly, we can consider that a certain quality has been reached. But is this enough, considering the current competitive market? For certain products it is, but for large systems it might not be. Due to the complexity of these large systems, many scenarios may be overlooked and only get discovered too late. when the costs of fixing them are much higher, and can also affect the company’s image.

2. Introducing randomness to the test environment

To enhance the quality and stability of large systems, we propose adding a random input to the already existing techniques.

Random Testing is a black-box test design technique, where test cases are always selected randomly through an algorithm (aptly named “pseudo-random generation”) which is used to match an operational profile. This technique of Random Testing is mostly used for nonfunctional testing attributes, such as performance and reliability points.

For small systems, Random Testing doesn’t make sense since it gives rather (oh well…) random results. But for larger systems, if the pseudo-randomness is done properly, it could bring great benefits.

In short, you would be running your system over a rather long time, and each module would randomly generate events at random periods of time. If this is done from the early stages of the product development, it will greatly improve quality and reduce costs since it can find bugs without putting too much effort into defining all possible scenarios.

An interesting view on Automated Test Generation using Random Testing is presented here:

“So, when you’re testing your program systematically, the problem that you tend to have as a tester is you think like a good user, you tend to write test inputs that correspond to common cases in the program. But when we’re looking for things like security bugs, what we really want to do is to act like bad users, like malicious hackers or other people that are trying to break the system.”^[5]

You might be also looking for race conditions, or 1 in a million cases when your app fails. Adaptive input generation (knowing the range and scope of the inputs that one chooses) can bring a great improvement in random input generation.

2.1. The basic solution

An overall solution would have the following steps (but it can be tailored to each product’s needs):

• Create a simulated environment of the hardware and integrate it with the software.
• Define a set of rules for generating system events and inputs. Here we can have two categories:
  • Normal working conditions: inputs that are regularly received by the module from outside (normal user interaction, correct sensor values, etc.)
  • Error conditions: inputs that occur when something goes wrong (hardware faults, misuse of the equipment, etc.)
• Enhance the simulation by feeding the events and inputs in a controlled random manner. Consider various situations that can happen and generate your inputs within those margins.
• Create automated tests based on this simulated environment, both with a predefined event sequence and random event injection. Here you can define some scenarios depending on what is needed (enhanced quality, stability, performance, code integrity, etc.)
• Use short running simulations/tests to validate code updates. (Include building the solution → BnT / “build and test”)
• Use complex scenarios on a regular basis, to test the overall product. (Combination of predefined order of events with random generated events.)

2.2. Pros

• Finding non-common, or relatively rare error cases such as race conditions faster than regular automated testing
• It does not have any bias. Since the inputs are pseudo-random, the tests are not focused on whether people know how the system works, meaning it can break it easier.
• Use of less hardware and less time spent on hardware can greatly reduce costs
• QA engineers can spend time analyzing faults, rather than running infinite tests

2.3. Cons

• Takes resources to design and implement
• Needs to be updated together with the product
• This technique will create a problem for continuous integration if different inputs are randomly selected on each test

3. Intermediary conclusion

By adding a non-biased random input to automated tests, you may improve the quality of large systems with fewer costs. The value added by this pseudo-random testing comes from identifying rare problems earlier, and from the high rate of performance issue detection.

4. Using Azure to support simulation of random generated tests

Maintaining a simulation environment up to date with the latest changes across the organization may prove difficult if everybody regenerates their own scenario. Luckily, working on the cloud can mitigate this risk.

Let’s have a look at how we can use Azure to set up a simulation environment and spread it out throughout the organization.

4.1. Create a virtual machine as a base

First, you need to create a virtual machine where you would run your simulation. This can be done easily using the Azure portal. When creating the VM you need to consider the resources needed for running the simulation and find the best performance/cost ratio. More on creating virtual machines can be found here.

4.2. Create your image library

The next step is to create a shared image gallery where you would put your VM definitions. (For more info check here.)

The image library will keep images of all virtual machines that have a preconfigured testing environment. In this way, anyone in the organization can have access to the testing environment without having to configure it each time something changes. They can just deploy a virtual machine and directly run the simulation. This ensures that everyone involved in the process has the same base settings and also that the environment is always up to date.

4.3. Add a virtual machine image to your image library

After the gallery has been created you can add VM definitions to the gallery. Here you will define VM specifications for the VM that would run different types of simulations. Note that you need to have an existing virtual machine from which you can create the image.

Types of simulations to consider (but not exclusively):

• Latest version of the code
• Release versions
• Customer configurations
• A simple configuration (only what is needed to get the core functions working)
• A complex configuration (for checking how the system performs while heavily loaded)

For each of the types mentioned above, check if introducing random generated input can add value to verification.

In this step you might want to consider the resources needed for type of simulation. You can set the guidelines under the publishing tab. This gives the people using the image an idea about what system they should set up for the simulation.

One option is to add an image of the formerly created virtual machine to your library. Go to the Virtual machine configuration in the Azure portal and select Capture.

4.4. Create a virtual machine from an image in the gallery

From the image created earlier, we can create new Virtual Machines as follows:

Select the image from the image library

4.5. Run the virtual machine and start the simulation

Now that you have the virtual machine created, you must start it and connect to it in order to run your simulation.

First hit the start button to have the VM prepared and started, and then use the connect button to connect to it.

4.6. Use scripts to optimize image creation

Now that we have seen how images can be used to run simulations, we can further optimize the process by using Azure PowerShell scripts to do all the steps above automatically.

This is explained very well in the following tutorials from Microsoft:

• Tutorial: Create and Manage Windows VMs with Azure PowerShell
• Tutorial: Create Windows VM images with Azure PowerShell

4.7. Conclusion

In order to keep costs down while increasing quality, Azure can be a very useful tool for maintaining a good environment for simulation tests, especially if you need to cover a great deal of scenarios.

• It can provide almost 100% uptime due to redundancy configuration.
• You pay only for the time the machines are used.
• You don’t need to worry about hardware upgrades or malfunctioning hardware.
• Everyone in your organization can have access to the exact test scenario that is needed
• You can run any number of tests that are needed in parallel without overloading your system.

References

[1] https://www.kovair.com/blog/how-to-build-a-successful-continuous-testing-environment/

[2] https://www.embedded.com/what-you-need-to-know-about-automated-testing-and-simulation/

[3] https://www.cloudbees.com/continuous-delivery/continuous-integration

[4] https://software.ac.uk/using-continuous-integration-build-and-test-your-software

[5] https://www.coursera.org/lecture/automated-analysis/automated-test-generation-using-random-testing-aB9YS