Starting at the end of 2018, I joined the AI4Test team at T-Systems Multimedia Solutions as a working student. As the name indicates, the project aims at supporting test automation tools by providing AI tools such as vision-based GUI element detection, the intelligent exploration of web pages and automatically generating test scripts.
For that, a neural network was initially trained on synthetically generated training data, which was later substituted by crawled and DOM-analyzed web page data. During my diploma thesis, I significantly improved the recognition performance by implementing a combination of weakly supervised pre-training based on web page crawls and then utilizing transfer learning to fine-tune the model using manually annotated high-quality data. This also included the conception of labeling principles, implementation of the labeling pipeline as well as instructing the labeling team and providing feedback.
The time at T-Systems MMS was also important for my understanding of workflows of a larger company outside the university environment. This includes for example internal events or presentations and production-oriented workflows using technologies such as docker and managing a server environment.
As a summary, the following tasks were part of my work at AI4Test:
- Refactoring and modularization of an existing code base
- Configuration/Administration of a GPU server used in production
- Analysis of intelligent crawling methods of web applications for automated extraction of test cases
- Conception of data sets and annotation pipelines as well as instructing a labeling team
- Implementation and evaluation of an object detection pipeline using weak supervision and transfer learning to recognize GUI elements on web pages and other applications (Diploma Thesis)
