Is there a way to automate these reports? - asked the business analyst.
I had no idea what to do, but as someone who relies heavily on analytical thinking, I did not hesitate to answer that there was a way.
My name is Emil Bibinovksi, QA Senior Manager & Vice President at VSG Bulgaria. We are a team of experienced professionals, always ready to provide the best software solutions for our clients and solve even the most complex problems. And the project we have been working on for many years for one of our clients is software for risk analysis when borrowing, or according to our colleagues from the USA - Loan Origination System. If we stick to the modern approach to expression, I will say that the product is space. Extremely serious and complex - with many layers of business logic.
When I first started at VSG, I was the new, young, ambitious QA who was ready to automate everything. But how do you automate something that doesn't even have a single test case? This was the start of my six-year adventure. We spent time measuring quality and distributing work among colleagues.
We began by documenting everything we could find as a specification, dividing the system into domains, and gradually amassing a collection of praiseworthy test cases. It also served as the foundation for the regression and smoking sessions that we began.
We built a framework over Selenium, changed the working pace to one at even faster speeds and started catching up. And just when it got calm that we had described everything, automated it, we had to remake it because of a redesign. Here I would like to express special thanks to VladimirKisyov. Vladi, who did not show a drop of fatigue throughout this extremely complicated process!
At one of our biggest releases, we had significant changes at all levels of the project - a major redesign, a transition from .NET framework 4.6 to .NET 6. We rewrote the front-end part of the project, which combined different technologies by switching to React with Typescript.
The regression passed and we fixed everything we found. It was time for a release, which also went smoothly, and we had a quiet month. Have you ever been very tense because the project is suspiciously calm? People who are in the industry would understand, I am sure :)
Just then, I received an invitation to speak with a lady from the business analysis department, and the dubious calm ended:
"Emil, we have a serious and negative response from several clients that the reports they generate have significant discrepancies with the truth!"
In the banking system, it all comes down to reporting. As a banking software company, we support about 1,600 diverse types of reports.
How is a report generated? The bank employee goes to the report page, selects the report, and presses Print. What happens in the background?
To verify that a report works, we need to read and analyze about 20 forms with over 30 fields, have detailed information about all reserved procedures, back-end transformations and front-end logic involved in its generation. This is quite often about 3-4 pages pdf or xls.
Throughout the conversation, I hoped she would not ask me the question I did not want to hear until she did - is there a way to automate these reports?
Of course, there is, how else will we develop if we do not challenge ourselves?
I went through many options in my mind - each one more difficult than the other - and finally saw quite a simple solution: What does the manual QA do while testing the report? He checks your input, applies logic, and verifies output. That is, it! If we guarantee that the input data has not changed, there are no changes in the business logic, then the report should not have changed either. We guarantee that it works as expected.
How to guarantee the incoming information, given that it includes 8 forms, 30 fields, some of which are dropdowns, radio buttons, multi-selects and everything that can come to mind as web elements? We were considering the Selenium option but repeatedly the answer was the same. Take information from the database and compare it somehow? No.
As is often the case in life outside of work, the answer was right in front of us. We needed an extra "pair of eyes," not our own, to see that nothing had changed and that everything was as it should be. We went to Visual Testing after a lot of headaches.
This is a new method for "capturing" so-called Snapshots, which the system uses as a basis for all future comparisons and performs the next pixel comparison check. That is, someone captures the entire input for us every day and informs us that it has not changed. Then he films the completed report and tells us that it looks the same as before! Here is our "second set of eyes" that notices that with the same input data and without changing the business logic, we get only one output in the form of a report! Voila!
There are still things that surprise me and here I noticed that the comparison found a difference from 1px. Of course, in the settings of the framework there is an option for tolerance. We made the most of the advantages of the system by including all complex grids and tables. We removed from the test case management system the test cases with which we verify columns in tables and grids. For the first time, we had the option to test a design without using an abstraction for web elements one that uses UI testing frameworks.
In the process of working on visual testing, we covered a large area of uncreated tests with an entirely new layer, which improved the project significantly. A moment of gratitude, but this time to Denislav Dimov - he took to heart the construction of a visual testing framework, which we use based on playwright js. Today, we already have over two hours of visual comparison running, and in the whole adventure we managed to attract three more junior QAs.
Everything in the world of automation is very dynamic; we get carried away because we have access to so many resources; we pile technology on top of technology until we find ourselves in an unnecessarily complicated world simply because we can make it so.
Passionate about innovation and the integration of processes, we sometimes forget that with simpler actions and the right tool, we can often achieve much more.
