We may define performance testing as a testing measure used for evaluating the speed, responsiveness and stability of a computer, network, software program or device under a workload. The objective of any organization is to run performance tests so as to identify performance-related bottlenecks.
Roles and Responsibilities of a Performance Tester
- Identifying SLOs(service level objectives) / SLI (service level indicators)/Golden signals metrics
- Performance Test Requirements Analysis (NFR) and Planning
- Script Development (using LoadRunner / Jmeter/ Neoload)
- Test Execution and Server monitoring
- Results Analysis and Reporting
- Creating service virtualization
Let’s listen to Performance Testing Expert – Gourab Mitra. He has over 12 years of experience the in software industry and currently working as Lead-Performance Engineer in American Eagle Outfitters Inc. Previously he has worked with Cognizant as a Senior Associate-Performance Testing & Engineering. He has rich experience in Performance / Reliability Testing. Also he has worked extensively on Application Performance Monitoring (Appium) tools (like Dynatrace / AppDynamics / NewRelic) developing alerts and dashboards using ELK / SPLUNK/ CloudWatch(Monitoring as Code) and doing root cause analysis with profiling /debugging tools like JVisual VM/Heap Hero/JMC.
1. Why did you choose this field and how would you describe your journey so far?
Answer: I have been in performance engineering for 12 years now. Unlike many performance engineers who mostly have graduated from different traits of software engineering to perform testing engineering. I have begun my career in performing engineering, I have been in this domain and doing this for this long. So mostly, things which attracted me towards this domain like when I started performing testing and performance engineering, it was a niche area in software engineering. So basically it is a domain where you need knowledge in diverse areas of software development and life cycle as well as the entire length and breadth of software engineering. It’s more about understanding the functionality of the application, underlying steps details, behavioural management, and infrastructure.
So that’s what like attracted me because you get to know soo much in the software engineering field by being in this domain. This is what led me on this journey. The journey so far has been pretty good and rewarding. There are few technical traits which the domain needs and is not possessed by many and also like you won’t find many people working in this domain.
That is why it is a relevant and kind of skill event today. It was unskilled when I started, it is still today, a lot of things had changed though. A lot of requirements be it a performance engineer have changed but still it is considered the same.
2. What is performance testing and how is it relevant in the new age of software development paradigms?
Answer: Performance testing by bookish language is basically an art of accurately understanding and benchmarking the responsiveness, stability, and speed of a digital system which is under the desired workload. It’s kind of like used to analyze various success factors like response times, different client-side and service-side stats and also potential areas which might come when the system is under a kind of workload. So with these performance results in hand, we can identify different bottlenecks like different bugs and defects and can try to optimize our application to eliminate these kinds if problems.
So performance testing engineering helps us in solving problems which are non-functional in nature, which is basically related to the stability and reliability of the system. It mainly tries to benchmark a system. For example, an application which gives out a temperature of a certain location is a weather app. When a user is trying to access the application, it gives the correct result and in a fair bit of amount of time, like a second, it gives the result. But as my application becomes popular and more users like coming to my application to check the weather, humidity and other features which I offered, I see the application slowing down over time.
Under the huge amount of user load, it is now servicing and slowly slows down. It’s slowing down and on the verge of being almost inaccessible as the number of users grows. The users would definitely be coming to the systems if the weather app takes such a long time to tell what is the temperature today. Those are the problems which might happen. So performance engineering tries to tackle these problems as to why is it taking so much time to open and how can we resolve it. So all these things are taken into consideration and comprise a performance and reliability engineering base.
3. How would you select a test case for performance testing?
Answer: In selecting a test case for performance testing, there are a few metrics which can be used to identify a candidate for performance testing. What we generally see is what is the throughput of that application test or application program interface or API. So we consider the business impact and criticality of the app, Even if it’s considered low traffic app. Also, other things are the routs or the workflows depending on the system it touches. During its execution how it is interacting with a database, and what queries it is doing. Based on these factors we take into consideration whether it could be a performance testing candidate or not.
In some cases, I have seen that we have done performance testing and we have tuned applications like done for performance engineering for that even though it will be used once per week. But it is such a resource-intensive app or process that the criticality of the process generates a certain report which goes to very critical stakeholders. The performance of that app becomes very critical for that business domain. If seen from an organizational perspective, how it is impacting them, in terms of their revenue or their time.
There are 2 branches of thinking. I am currently in retail which relates to revenue like how to increase the throughput of the system and how more users can come in and do their transactions easily. That is one branch of thinking and enabling the customers. But again there is another part of it, where in our internal system like where our VPs, upper cader management users for them also printing out a report and getting certain status before the meeting or a certain point of a day becomes difficult, They also look for data while taking decisions.
4. What are the most common tools for performance testing—open source or proprietary?
Answer: When I started performance testing and engineering, I started working with one tool called load runner. At that time it was market interactive then it got accord by HP and now with Microfocal. We are still using it. It’s a licensed tool so it has a kind of test of time and it’s still there and one of the most used performance testing tools. When I worked with other performance testing tools which are open source like Jmeter. And other license products called NeoLoad. It was by Neotest initially but now it’s with Tricentis.
There are also a host of other performance testing tools like Gatling, k6 which we have worked but there are a few performance testing tools like Jmeter which have been there for a long period of time. Their companies are also like BlazeMeter which even markets their product as a licensed product. But their product and platform is built on Jmeter. So kind of like a win-win situation.
If somebody learns Jmeter, they will understand and find their way into the BlazeMeter world and navigate their platforms. These different tools work on very common basic guidelines & principles. So the guideline principles for all the tools are very similar except for their packaging. But if you learn one tool, it would be easier to transition to other tools. Since the thumb rule of doing the work in all tools is almost same. So if you know the basic principle of doing it, it’s pretty easy.
5. Which test management tool is suitable for performance testing?
Answer: There are so many good platforms available which are equally good. Performance testing from engineering point of view is kind of loosely coupled. There are so many tools I worked on like, Jira, Xray, Bugzilla and many more. Everything works pretty well with our domain. Most of the leading test management tools which we care about like the ones I mentioned, work pretty well. As you know, most organizations are removing tools that were agile. In the last 4-5 years, the companies I had worked for mostly preferred Jira. In the market, Jira holds a good market share.
6. How to perform performance testing of SOAP and REST Web Services?
Answer: So there are a plethora of ways to adjust performance testing and engineering and create solutions for SOAP & REST web services. But the most important thing that come to the point is what kind of a tool I am going to use at the end of the day. So in simpler terms, we need to have a fair bit of idea of what kind of tool we will be hitting as part of our testing. We need to have a pair of request responses to validate it and depending on the tool wrap the request in a specific format. For example, load runner needs to the request to be in a c-format if you are using it, although it is not required for other tools.
But again like, it depends on what kind of tool we use, definitely for a beginner who has the knowledge of the matter of experience how to deal with it, like changing the format. And also like we have to create some appropriate checkpoints and generate a response based on it. Just to have a checkpoint, my request is actually doing the intermediate thing or not, having a checkpoint for response code whether it responds through or not. So those kinds of things.
7. Which tool is suitable for performance testing of mobile applications?
Answer: Mostly using load runner for mobile performance testing. And earlier we used HTTP protocol for mobile performance testing. Basically, we extract mobile request responses by using a tool called Fidler. And create an expansive script in automated like in load runner. It can be done easily if you have the request and responses captured through fidler or any other network like charles proxy etc. We can import those request responses into our script into the load runner editor and from there we can easily do debugging and create interactive scripts.
There are also so many ways in which we can do it like if we want to use an open source tool. We don’t have or want to procure licenses to run these kinds of tests. We can create hard files and use Jmeter, as it is an open-source tool. There are ways to do it instead of paying for soo much license cost to these companies. There will be pros and cons and certain limitations of using open source. But definitely ways out to do it.
8. How does performance testing integrate with DevOps?
Answer: It really gels well with DevOps because as you know it has been around for many years. A term called early performance testing where we tried to get into performance testing and engineering in the early stages of the software development life cycle. It is very important to integrate these performance testing jobs as-built jobs which can get triggered after deploying after getting a certain built. That too at a pre-determined time to evaluate stability and performance of that specific built. So we can put checks in place to pass/fail a build accordingly. And mark the build as not as performance-ready even though it has passed quality assurance.
In that way, we can integrate performance testing jobs, and in-built pipelines which could be done through any tools Jenkins or Bamboo. Like I personally worked in these 2 of the tools. But it has become important to have these kinds of jobs. It’s very important to have that kind of job in place to give an indicator/marker. It also helps you in understanding in an incremental kind of way instead of waiting for so much time when the full product is available.
Testing those smaller bills, weekly bills or daily bills and marking them as pass or fail depending on the market face. So it’s very important to integrate performance testing as part of the life cycle and as part of your built plan. Any kind of built, early integration helps not only in your developing environment but in a smaller way. It gives an idea to a developer and DevOps engineer that how that piece of code or that built might actually perform and get promoted to the next environment.
So basically, just to reduce the amount of time, when we go into production. So that we can test from the reliability point of view, if we can test it way ahead of the development of life cycle.
9. Cloud computing has become the default IT platform. How is performance testing performed on software running on AWS, Google Cloud, or Azure?
Answer: Cloud performance testing determines like how will the application function in a cloud-hosted environment. Be it anything, AWS cloud or google cloud. So you can mitigate performance and bottlenecks of how your application is best suited to handle peak workloads. In terms of like saying, with economies at scale, we have so much computing power at hand but still fail to optimize those things. For example, in the current organization, we are trying hard to optimize the kind of infrastructure we need. In order to do that, you need to have a fair bit of idea of how to do it.
If your application is skilling up to your current infrastructure, and if not what is the best possible configuration to do it? Along with that what is the best possibility to have it in place to best utilize your resources? If it’s underutilized how can you eliminate the not utilized resources. Basically depending on how you tune up your environment, what kind of computing power you need and what kind of setting from a memory kind of perspective can we do. What kind of tuning you can do to pods, and what kind of memory management can be done.
So that it is optimized to the best of it’s potential. Also the resources it uses also get optimized so that my application does not utilize what it needs. So it’s very important to optimize the application first and then to optimize your platform. For example, there was a certain application which was related to reprocessing some orders. So there were batch processes involved when it fails. So it was found that our application was not properly tuned to reprocess those orders. Whenever those orders used to fail, they used to come back to our queue because of our memory management, the algorithm was not appropriate for it. So we had to tune the application first, then tune the pods which was to be handled.
We fine tuned the application then the platform which will be hosting the application. So in doing that there are a lot of other techniques we adopted to do the reliability part of the cloud. Performance testing and engineering are fast domains. Earlier reliability was not considered a part of this domain, but now it’s getting crucial day by day. Because we are using cloud solutions, very important for us to know how these cloud solutions work or reliable they are in terms of handling our peak workload.
10. What is your opinion on the next five-year outlook for the performance testing domain?
Answer: Outlook is very promising and bright and performance testing not only involves performance testing but also engineering as well. And it is just not to give out a black-and-white report like this meets the SLA and something does not meet the SLA. But also to work from an engineering site and give recommendations and work with diverse teams from application development to platform engineering, DevOps and reliability. It’s not limited to understanding reliability, to report and say this is not working. But to know, why is it not working and to have the acumen to take a role of a consultant than a tester.
To give out recommendations and have that acumen to understand how to improve the application performance by understanding the internal workings of the software. It’s not just about testing the application, but also to develop in this domain, we need to have acumen of an engineer, who basically has the capability to give us results and suggest changes. Be it code review, building utility or capability which helps in like betterment of software.
11. What is your advice for freshers who wish to make a career in this field?
Answer: My first and foremost advice will be to pick any language, be it most preferably open source, understand the language how multithreading works, how memory management works, and how the garbage collection algorithm works in that language. It is of utmost importance how these specific features of that language works. Because in the actual industry, these are the certain things which we try to understand when we look into the performance part of it. So if a certain intern or fresher has a fair bit of idea of how these things work and where we need multi-grading and where we don’t. What kind of garbage collection algorithm works with what kind of test cases.
Start with this. Also like, beginning with picking a performance testing tool would be a good starting point. Along with that, the thumb rule of performance testing is kind of similar if you master one tool, it’ll be easier to transition to other tools. Just master one language, one open-source performance testing tool be it Jmeter or Gatling, and your journey should be pretty smooth.
