In today’s modern era of building software, all well managed software development teams should understand the value of testing. New applications should be built with near perfect unit test coverage for automated testing. Additionally, browser based functional testing should be fully automated using open source tools like Selenium in order to capture known patterns of human interaction.
The Challenges of incorporating Functional Tests in CI/CD
Functional use cases tend to evolve over time and change on a regular basis as a software suite’s use cases grow and evolve. As a result many companies struggle with how to run an ever evolving set of selenium tests as part of an automated test and deployment pipeline.
In particular, in today’s world of SaaS software, many multi-tenant SaaS applications have entire suites of Selenium tests that need to be run before a new tenant is provisioned. These tests should be run as part of the automated pipeline of the actual tenant provisioning.
The reality, however, is that most SaaS companies are not doing this. The result shows up as constant inconsistencies during tenant or private cloud provisioning.
So then, what’s the solution?
Even though Selenium is an effective and popular way to automate browser based tests, organizations need a way to automate the inclusion of properly versioned Selenium test buckets into deployment pipelines.
ProcessMaker is an ideal solution for automating a deployment and provisioning pipeline of selenium tests directly from their repositories in Github. Let’s look at how this can be done.
Selenium test suites in Github
Imagine for a moment a company that has a bucket of automated Selenium tests which are stored as Python files in GitHub. These python files need to be executed each time before a new workspace tenant or private cloud install gets provisioned. If any of the test cases fail, we want to make sure that the provisioning does not go forward.
Additionally, we recognize that the test cases will be constantly changing and evolving. In this case, we have multiple teams that want to be able to contribute tests to a versioned repo, and we want to be sure that our selenium tests are always run using the latest stable and approved version of tests.
In this case, the testing will be done with Selenium. Of course, Selenium is an extremely popular open-source tool that is used by developers throughout the world. With Selenium, you can automate every task in your browser. In addition, it supports Chrome, Firefox, Safari, IE, and Mozilla.
In ProcessMaker we have created a deployment workflow which pulls the appropriate Selenium test harness from github, runs it, checks for issues, and if everything passes, then calls a Lambda function in order to deploy a new AWS server with the verified code.
ProcessMaker gives us a visual representation of the workflow, an ability to track performance, time, and errors. Here is what the workflow looks like for one company currently using ProcessMaker in this way:
Flexibility is key to test management. ProcessMaker makes it easy to regroup and refactor test cases so that you can improve maintainability while decreasing complexity and duplication. With ProcessMaker’s easy-to-use tools, you can reduce testing execution time and even run several tests in parallel.
Using ProcessMaker, Selenium tests are pulled from the code repository and executed through a series of script tasks in ProcessMaker which funnels the test through a Selenium Docker image and then stores the results of the test in the Request.
Part of the beauty of ProcessMaker is that it manages data as a very lightweight, unstructured, and easy to understand Json object. During the life of a request (a process instance), this Json can receive data from many different sources. In the case of the Selenium tests, the results that are returned from the Selenium tests are simply added into the Json object. Then, depending on how our workflow is supposed to work, we can use external rules or gateways to evaluate the results and change what happens in the workflow.
To illustrate, ProcessMaker can be configured to execute follow-up actions based on the test results returning results == “success”. At the gateway, we evaluate the variable of results. If the results were successful, then we have configured the workflow to call a new script tast which is the lambda function we use to spin up a new tenant (workspace). If the results show “failure” then we would not go forward with spinning up the tenant. In that case, we may want to post details of the failure into a Slack channel or via an email using connectors in ProcessMaker for either Slack or Email.
Using ProcessMaker as a test harness in this way, we can test any custom application before we implement it on a server for a customer. We get flexibility in complex workflows and the ability to scale and fire off tests on demand.
Process execution via the ProcessMaker API
Because Process execution can occur through API calls, the usage of ProcessMaker can form part of your CI/CD pipelines. If an environment is deployed or upgraded, the CI/CD pipeline can execute a REST API call against ProcessMaker to begin executing a test suite based on the API call request payload.
Using a CI/CD process you get faster issue closures because you have faster bug identification and fixes. Also, test planning and execution is maintained consistently. Technical skills are not required. Deployment is easier and faster with almost no downtime. Higher quality applications can be created because you have more time to focus on usability, security, exploratory, and performance testing. Continuous feedback improves overall quality, as well.
Sub process execution
A simple process can be designed to execute one single selenium test and return the result. A test suite can be a parent process which loops through execution of a defined test suite and performs more complex workflow decisions based on the results.
At its core, the ProcessMaker workflow engine is highly customizable. You can track testing with complete transparency. Some of the benefits include the following:
- Better test efficiency
- Decreased maintenance costs
- Optimized test coverage
- Little manual intervention needed
- Code reusability
- Test automation expertise is not necessary
Containerized selenium based testing with script tasks
Script Tasks run in Docker containers for the lifecycle of that script. We can leverage Selenium’s docker image in order to execute a test with sandboxed isolation. We can also customize the docker image to inherit from the selenium base image and then provide scaffolding to load the test and execute it which returns the results in a proper JSON format that ProcessMaker expects.
Why does ProcessMaker like JSON? Well, for starters, JSON is more readable and compact than XML, making it easier to work with complex systems. Since JSON uses less data than XML, software parsing is much faster. JSON also makes it easier to time map to domain objects. JSON’s map-data structure is also easy to understand. Additionally, JSON ensures the matching of objects and codes.
Scaling
Execution of test suites are done on-demand and can be configured as a single environment deployment or large scale (multiple environment upgrades). You can easily scale out test suite execution by utilizing ProcessMaker’s autoscaling infrastructure.
Storing test suites as code
The intent is to store the definition of a test suite as a JSON file in a GitHub code repository. The JSON file will define the tests to execute with each test being stored in the code repository as well. This utilization of git code repositories to store test suites allows us to leverage branching and other git characteristics to control and manage test suites. This also allows the use of executing said tests, without the use of ProcessMaker, locally for development and debugging purposes.
So why do we use GitHub? Honestly, because GitHub’s wiki and issue tracker gives us the feedback and in-depth documentation we require. Further, you can track revisions across multiple versions. Moreover, in terms of version control, we love Git’s branching capabilities. Through the branching features, you can work in an isolated environment for every change you need to make to your codebase. Since GitHub uses a distributed version control system, every person working on the same project can have their entire history saved in their own localized repository. Further, if two contributors make changes to the same file, GitHub will intelligently combine the modifications. It’s also quite easy to revert back to a specific version. Overall, it enables a faster release cycle and GitHub works seamlessly with CI/CD environments.
Conclusion
When selecting an automated testing tool, you should look for one that is agile and offers support for a large range of languages and applications. This will enable your team to contribute to your testing cycles without the need for technical skill sets.
