An LLM a Day Keeps Manual Testing OK: Using Large Language Models to Generate and Maintain Manual Tests

Manual tests, tests which are written in natural language, are essential in contexts where automated testing is infeasible or insufficient. However, they often suffer from incompleteness, ambiguities, outdated steps, and test smells, which can increase maintenance costs and reduce reliability. In this talk, we will explore applications on how Large Language Models (LLMs) can assist in the generation and maintenance of manual tests. We will discuss and show techniques to create tests from requirements and from screenshots of system interfaces, as well as strategies to update tests when requirements or user interfaces change. We will also focus on how LLMs can help with the task of removing test smells from manual tests. The talk will focus on real-practice manual tests and also will discuss open challenges and research opportunities.

Márcio Ribeiro is an Associate Professor in the Computing Institute at Federal University of Alagoas. He holds a Doctoral degree in Computer Science from the Federal University of Pernambuco (2012). He was the recipient of the ACM SIGPLAN John Vlissides Award (2010). In addition, his PhD thesis has been awarded as the Best in Computer Science of Brazil in 2012. He has experience in Software Engineering and Artificial Intelligence. Ribeiro was the President of the Special Interest Group on Software Engineering (CEES) of the Brazilian Computer Society (SBC) in 2021-2022. His main research interests are Configurable Systems, Empirical Software Engineering, Software Testing, and Refactoring. Ribeiro has published papers in top-ranked conferences and journals such as ICSE, ESEC/FSE, ECOOP, PLDI, AOSD/Modularity, ICSME, SPLC, GPCE, ICPC, ISSRE, TSE, IST, and JSS. In 2014, Ribeiro was the General Chair of the most important academic event on Software of Brazil, CBSoft 2014 (https://cbsoft.sbc.org.br/2014/). He designed and created the "Aglomerações" app, an app to register and track agglomerations. The app pioneered the idea in which the own user registers agglomerations during the COVID-19 pandemic in Brazil. He is the coordinator of the Research Center on Engineering and Systems (EASY) at the Federal University of Alagoas (https://easy.ufal.br). The center has executed several R&D&I projects. Some of the projects are used in all Brazilian states and has brought savings of millions of reais to brazilian public departments. The center also executes a telehealth project in the context of the brazilian state Alagoas, in partnership with the Ministry of Health of Brazil.