Svetlana Grinevich. Artificial Intelligence QA - Testing Automation with AI

Introduction

Introduction

Hello! My name is Svetlana Grinevich, and I am a tech manager at Globant. I oversee various QA automation projects and work closely with cross-functional teams. I have a deep passion for technology and am currently diving into machine learning frameworks and Internet of Things (IoT) platforms. Today, I want to discuss testing automation using artificial intelligence (AI).

The Impact of AI

When people hear about AI, they often think it is a domain exclusive to big tech giants and doesn't impact their daily lives. However, AI is something we encounter daily, from unlocking our phones with face ID to using digital voice assistants.

AI in Testing

Both large and small software companies are constantly seeking ways to reduce costs and improve testing reliability. Under this pressure, testers are increasingly turning to AI and machine learning to enhance their testing processes.

Enhancing the Software Testing Process with AI

This presentation will cover the benefits of integrating AI into the software testing process. We will explore how AI can help teams test faster and more effectively. I will demonstrate a concept where we will use machine learning to speed up testing by adopting a big data approach. We will create tests using natural language and eliminate the need for extensive test scripts.

Building AI Models for Testing

In this article, I will guide you through building two AI models that enhance software testing:

1. Detection Model: This model will use convolutional neural networks (CNNs) to identify UI elements.
2. Action Model: This model will use deep reinforcement learning and natural language processing (NLP) to execute actions on detected elements.

Building the Detection Model

The detection model will identify elements on a page. To build this model, you need the following dependencies installed:

• Python
• TensorFlow: A powerful open-source library for machine learning.
• ImageAI library, Pillow, and Python struct: These libraries simplify the code for object detection.

Here is a common flow for building machine learning models:

• Create a dataset.
• Train the dataset.
• Create a model.
• Use the model for data analysis and prediction.

Creating the Dataset

Collect images of web pages showcasing various web elements. A minimum of 500 images is recommended. Automate the collection process by using developer tools in your browser to capture pages and save them.

Annotating the Data

Using tools like LabelImg, annotate the objects in your images. This process involves marking the locations of elements like buttons and saving the annotations in XML files.

Training the Model

Split your dataset into training (80%) and testing (20%) sets, then train the model using the ImageAI library. This training process can be time-consuming, taking up to 72 hours for 20 experiments.

After training, evaluate the models to identify the most accurate one. Use Mean Average Precision (MAP) to gauge accuracy.

Example Scenario

For example, if you have images from Netflix or Amazon, the model should accurately detect buttons within those images.

Building the Action Model

The action model will utilize NLP to perform specific actions like clicking a button or navigating to a URL.

Preparing the Dataset

Create a collection of BDD (Behavior-Driven Development) steps from various sources. Classify these steps by labeling similar actions under one category (e.g., all navigation steps as "navigate_to_url").

Training the Model

Train the NLP model using the spacy library and other Python dependencies. After training, evaluate the model to ensure it can accurately predict actions based on natural language input.

Applying the Models

Once you have both models ready, you can integrate them into your automation framework. The framework will transform natural language test scenarios into actionable test scripts, which will be executed by the models.

Conclusion

Using AI in testing can eliminate the need for extensive manual test scripting and provide more reliable, faster results. Hopefully, this process will help you build a powerful AI-based testing automation platform.

Keywords

• AI in Testing
• Machine Learning Frameworks
• QA Automation
• Convolutional Neural Networks
• Natural Language Processing
• TensorFlow
• ImageAI Library
• BDD Steps

FAQ

Q1: How does AI help in software testing? AI helps reduce costs and improves testing reliability. It can detect UI elements and perform actions using machine learning models, eliminating the need for extensive manual test scripts.

Q2: What technologies are used in building the detection model? The detection model uses Python, TensorFlow, ImageAI library, Pillow, and Python struct for training CNNs that identify UI elements.

Q3: How are datasets prepared for the detection model? Datasets are prepared by collecting images of web pages, annotating the objects using tools like LabelImg, and then splitting the dataset into training and testing sets.

Q4: What is the role of the action model in testing? The action model uses NLP to transform natural language test scenarios into actionable steps, like clicking a button or navigating to a URL.

Q5: How are BDD steps used in the action model? BDD steps are classified and used to train the NLP model to recognize and perform specific actions based on natural language input.