Mobile Test Automation on Transformation and SaaS Device Farm Integration Service

Yıldız Tech is the digital innovation-driven R&D center under Yıldız Holding, designated as an On-Site R&D Center. It focuses on technology and innovation, especially in e-commerce, providing solutions for Yıldız Holding and external companies. Its goal is to help various industries adopt new technologies, achieve their objectives with digital and analytical solutions, and capitalize on emerging business opportunities.

İstegelsin, and Şok, two of the most popular brands of Yıldız Tech, were the main focus of our Mobile Test Automation Transformation and SaaS Device Farm Integration Service.

Türkiye's first fully online supermarket, İstegelsin, develops technologies to offer mobile means that will eliminate the difficulties of grocery shopping and offer new-generation solutions to modern life. İstegelsin is a brand of Yıldız Holding, which also includes well-known brands such as Godiva and Pladis.

Architecture

Initially, there were 4 test automation projects that were planned to be created and developed for İstegelsin. These projects were Web, Android, iOS, and API test projects. A roadmap covering a period of 6 months was planned to be executed on these projects.

A test automation project can be prepared with different programming languages and various frameworks. There were two main factors for us to decide which toolset we were going to use:

• Toolset was to cover everything İstegelsin needed and to offer a tailor-made infrastructure for them.
• The internal test team of Yıldız Tech, including people who had never written automation tests before, could adapt and contribute to the project swiftly.

Considering these main factors, we prepared POCs with different language and framework combinations. In the end, we decided on the following software language and framework combinations for each project:

Platform Development

Once the architecture was defined, Kloia's platform development team worked alongside TOGG to implement the Trumore platform. They utilized industry best practices and followed agile development methodologies to ensure the timely delivery of high-quality software components with AWS best practices.

Methodology

It was a fresh start, and we needed a fresh approach to test automation project management. We decided to practice Scrum methodology with 2 weeks long sprints for the projects mentioned above. Created a Jira board to monitor and log sprint tasks. We set regular meetings including daily meetings, groomings, sprint reviews, and retrospectives.

We have created a list of smoke test cases to be automated for Web UI, Mobile, and API automation projects by closely working with the internal team of Yıldız Tech. There were two main reasons for starting with the smoke test cases:

1. Smoke test cases allow us to cover core business functionality which is essential.

2. Since the smoke tests are related to various points of the application, after the development of the smoke tests we get:

• A strengthened project structure
• A set of reusable functions which makes developing further test cases easier
• Integration points for data and reporting
• Coding conventions adopted by the team

We determined the reviewers of test automation projects, formed project development teams, and assigned projects to manual testing and test automation teams.

Development

We created test automation project architectures for Web, Android, iOS, and API projects. We defined project standards. These standards included coding conventions such as test case writing with Cucumber Framework, defining variables, and Cucumber tag naming.

We supported the team in setting up the project installations on different operating systems. To share our know-how, we scheduled onboarding sessions. These sessions included topics such as "Web element inspection", "Git usage", "Ruby language", "Capybara Framework", "Karate Framework", and "Appium". These sessions were especially important for our teammates with limited or no hands-on coding experience to adapt and contribute to the project swiftly.

All these conventions, installations, and usage information related to these projects were documented on Confluence.

We followed the Behaviour Driven Development (BDD) approach which enables us to reduce test steps to everyday English on these projects. Thus, making the test cases understandable even by the people outside of the developing circle.

We also built our test architectures for them to work as a distributed system, running tests in parallel. So we can run different tests in different instances of various browsers or devices.  

With further analysis of the newly-developing projects, a new requirement emerged. To reduce repetitive code and to provide data for the client projects WEB, Android, and iOS, we created another project, which was a private Ruby gem (a library) covering more than 150 test data needs, that can be called from client projects. In the scope of this project, every data of the client projects required such as user info, customer cart, or payment info, were generated in the run time dynamically. 

CI/CD Integration

When the test automation coverage exceeded the 30% threshold, we established Jenkins pipeline architecture for WEB and API projects. We designed the pipelines as able to run Smoke, Regression, Production test cases, or specific features.

Projects that are built in test environments are automatically triggered and test results are automatically transmitted to the Slack channel created for report tracking.

Smoke cases are run automatically every morning. After deployments to the production environment, the production pipeline is triggered and the cases in the production environment are run regularly on a daily basis, just like Smoke cases.

After the pipeline integration, we informed the whole organization about what we have achieved during that 3-month period. We added the entire project team to the test run groups, so we dynamically shared the test run reports with all stakeholders. Since every team in the organization was able to see these results, it became much easier to track down broken features and amend them.

Mobile Device Farm and Pipelines

We carried out two different operations for mobile projects’ pipeline integrations simultaneously.

• Managing Apps Dynamically: We integrated the application build tool Yıldız Tech used with Bitrise to manage apps dynamically. With Bitrise APIs, the app can be downloaded and installed automatically on a device with a specific app version or with the last successful build version.

• Purchasing a Device Farm Service: We prepared numerous POCs with different Device Farm services. Ultimately, our customer decided to use the digital.ai Continuous Testing platform in light of the POCs we presented. With the device farm, we integrated mobile test automation projects into the CI/CD pipeline.

Load Tests

In the beginning, there were only 5 automation projects. But in the meantime, we also included internal projects serving Cepte Şok product in the Yıldız Holding.

After making progress in UI and API projects, we developed load test scripts using Karate - Gatling and integrated these scripts into the pipeline. We dynamically generated the data we needed in load tests and built an architecture where the whole project team could run load tests.

As a result of a one-year work, what we have developed throughout the company are:

• 3 API Test Automation Projects
• 3 Android Test Automation Projects
• 2 WEB Test Automation Projects
• 2 Data Management Projects
• 1 iOS Test Automation Project

In each project, we carried test case automation coverage up to an average of 80%. We reached over 1200 test case coverage. We integrated more than 30 pipeline scripts with respect to the test type.

Our custom-made data-providing gem satisfies more than 150 test data needs.
With our parallel test architecture, the test runs that have 2-3 days of run time just take up a maximum of 2 hours to be completed.

300+ builds per month through scheduled and triggered test runs.