Is ChatGPT coming to take automation engineers’ jobs? The short answer to this question is not so easy or straightforward.

The long answer involves understanding exactly what Generative AI Tools and ChatGPT can and can’t do within the circumstances of the software testing and test automation. In so many other domains, ChatGPT is able to do wonderful things that feel like a threat to engineering careers. But there are so many other things it can’t do, or at least can’t do well. Automation engineers interested in using AI tools, to do their duty better rather than being replaced by the AI tools, need to know this difference. 

AI and Test Automation, Pre-ChatGPT

The idea of using AI tools in software testing is not new. For many years, various software testing and deployment domains have provided AI-powered “bots” (Robo on Google Firebase is one of the general examples) that can navigate across applications and decide what to test, and then execute testing. 

These agents can help automation test engineers explain test cases and execute them. But they have clear flaws, such as a difficulty in working past authentication walls within the application, and have a tendency to get blocked in infinite loops while performing testing.

For this reason, automation test engineers have historically thought of AI as a tool that can reduce some of the toil and boredom work required to carry out basic tests, but it’s not anything approaching a replacement for human testing experts. 

How Good Is ChatGPT at Software Test Automation?

This tool represents solutions that give greater flexibility compared to old AI tools. These tools are easy to use by using natural-language questions, making them more comfortable than many other conventional solutions.

Imagine that you want to test the UI of a browser-based web application. We can use this tool to create an automation script with a prompt such as the following: 

Create a Selenium test script that opens the kobiton.com website, selects the “Platform” menu option and then clicks on “Real Device Testing.” 

ChatGPT’s response:

from selenium import webdriver 

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# Set up the Chrome webdriver and navigate to kobiton.com
driver = webdriver.Chrome()
driver.get(“https://kobiton.com/”)

# Wait until the “Platform” menu option is clickable and then click on it
platform_menu = WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.XPATH, “//a[text()=’Platform’]”))
)
platform_menu.click()

# Wait for the option to be clickable
real_device_testing = WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.XPATH, “//a[text()=’Real Device Testing’]”))
)
real_device_testing.click()

# Close the browser
driver.quit()

The above script is close to testing what the user asked for, but there are some mistakes in the above script. The biggest mistake is that the “platform” is not a clickable element on kobiton.com website (you have to hover the mouse over it to make the “Real Device Testing”). Here ChatGPT failed to identify this fact because it parsed kobiton.com in a simple way. 

An automation tester can identify this issue easily and can change the Selenium script as desired. So, in this scenario, generative AI tools are capable of doing maximum work that’s required to solve a test-automation challenge; an automation engineer with domain-specific knowledge would need to do the rest. 

ChatGPT and Mobile Testing

Now let’s take an example of testing the UI of a mobile application that runs on a specific device. In this instance,things get difficult for generative AI tools.

As the ChatGPT tool doesn’t have access to a fleet of mobile devices,there’s no way to prompt ChatGPT with the following request- “Load my app on a OnePlus 9 Pro and test the login page.” This is a task where automation testers need to handle mostly manually, with the help of software-testing platforms that give access to mobile devices.

But there are many ways in which generative AI tools canassist engineers in this case. It can review the automation scripts written by automation test engineers, and suggest additional test cases they may have overlooked, such as testing for scenarios where users leave password blank, rather than only testing for the scenario when users enter the wrong password.

Here, ChatGPT is capable of doing only a small part of  work. The maximum work has to be done by human test-automation engineers, who have domain-specific expertise and access to testing infrastructure. 

Conclusion 

Generative AI is growing quickly, and it’s not impossible to imagine it becoming capable of working on the past limitations. Eventually someone might find a method to give AI tools access to mobile devices, to assist it to write automation scripts for mobile apps and not just for web applications. 

But even then, it’s difficult to imagine a world where tools such as ChatGPT become capable of handling all aspects of test automation in an end-to-end manner. There will always be gaps and oversights that humans will have to address. Testers who want to glow in an AI-centric world must use generative AI to automate the simple and tedious aspects of test automation, while focusing their energies on performing the things that AI can’t.