UAT Test Scripts – ITSM/ITAM/ESM Blog

Is 2 tests per UAT script enough?

Recently a customer asked me if 2 tests per User Acceptance Test (UAT) Script are enough.

This is a good question, UAT testing is key to successful Ivanti HEAT implementations and rollouts of major features/functionality.

How many UAT Tests should be performed per Test Case?

As many as it takes for you and your team to feel comfortable that you can perform your day-to-day job function.

Recommendation and Best Practice is a minimum of 5 tests per UAT Test Script. Anything less is a risk that the client need to evaluate. Some detailed, complex, and high impact UAT Test Scripts may require at least a dozen of tests for the various scenarios that you have identified. It is better to test more than less, that’s for sure. Perform as many tests as it takes for you and your team to feel comfortable that you can perform your day-to-day job function.

Experience dictates that if UAT (validation of business requirements) isn’t thorough then the likelihood of post-implementation issues, gaps, and problems is much much higher, and can take more time and effort to address than during UAT.

Intangibles like morale and perception of the new Ivanti HEAT Service Manager system/features can also be negatively impacted.

Also note that you are not bug hunting, so you don’t need to test every single feature and menu item, however you are testing that you can perform your daily job function, so test all the features related to the use case (business process) that apply.

It’s not unlike ordering a fleet of vehicles simply by kicking tires and doing a lap around the parking lot versus more extensive test-drives tailored to the company wide business needs. Sure, everything “might” appear fine, and “might” go fine upon delivery. But those are huge risks to take for a high dollar, high impact, high exposure item that ultimately could lead to work slow down or stoppage.

No matter how detailed your UAT Test Scripts, you need to have a good range of test case scenarios (sample input data) for every test script. Sample input data should be real data, from your existing system, whether it’s a database, excel sheet, or paper forms. For example tickets from your old HEAT Classic system, Paper Forms from a manual system, if you’re going digital, sample rows from an Excel sheet if that’s what you’re replacing. Bottom line, use real data that represents the typical type of data to expect and the UAT Team members can relate to. Tip: Ask yourself if the data could be used for training purposes. If it can, then include it as an artifact for training documentation. If it can’t, then ask yourself if the sample data is a good representation of a typical day in the life of the end user.

The time and effort required is usually negligible. Minutes per test.

A great amount of time and effort has gone into the Ivanti HEAT project, and UAT is a critical milestone in the Software Development Life Cycle (SDLC). So make it count and be thorough on your validation of business requirements, UAT that is.

Trust the Process

As you may have noticed, the last few Ivanti HEAT ITSM Blog posts had a common theme. UAT , Test Scripts, Pitfalls of UAT.

User Acceptance Testing (UAT) is without a doubt the most critical stage of the SDLC. This is where “the rubber hits the road”, and the customer and subject matter experts (SMEs) validate the business requirements.

This is also where typically some customers and SMEs tend to panic, get confused, worry, or simply get overwhelmed by the project at hand. Rightfully so, there is usually a lot at stake and the customer wants to make sure their business requirements are met, as does the consultant.

On one side, the customer and SMEs look up to the consultant for guidance drawn from years of experience, on the other hand, customers and SMEs also tend to think they know better.

So who is right?

Over 2 decades with HEAT Consulting, Business Analysis, and Project Management have taught me and reinforced, is that the customer isn’t always right and neither is the consultant. Looking at who is right is wrong! Instead of building a law case and looking to disprove one another, everyone needs to TRUST THE PROCESS.

If you have made it to UAT so far. Best Practice is to always review the steps taken (process) to-date.

Scope of Work was Signed Off by Customer
Business Requirements Elicited & Reviewed by Customer and Ivanti HEAT Consultant
Prototype Reviewed & Signed Off by Customer
System Testing performed by Ivanti HEAT Consultant
UAT Test Script Module implemented by Ivanti HEAT Consultant
UAT Test Script Examples provided by Ivanti HEAT Consultant
UAT Test Scripts created by Customer and SMEs
UAT Test Scripts imported into UAT Module by Ivanti HEAT Consultant
UAT Kicked Off
UAT Training Provided by Ivanti HEAT Consultant

…and the steps (milestones) ahead:

UAT Testing by Customer
UAT Test Scripts review by Ivanti HEAT Consultant
UAT Remediation (to address failed tests and gaps) by Ivanti HEAT Consultant
UAT Sign-Off
Go-Live (Rollout) Prep by Customer/SMEs, and Ivanti HEAT Consultant
Go-Live (Rollout) by Customer/SMEs, and lead by Ivanti HEAT Consultant
Go-Live (Rollout) Support by Ivanti HEAT Consultant
Post Go-Live (Rollout) Remediation by Ivanti HEAT Consultant
Minor Enhancements (if needed and within scope)
Major Enhancements (if needed and within scope)

In Summary, using the analogy of purchasing a car, the buyer (customer) has done their research and made a list of must-haves, and nice-to-haves (business requirements), identified the car type, model, and budget (scope) with the help of the dealership (Ivanti HEAT Consultant), and now it’s time to take the car (system) for a road test (UAT). This is where “rubber hits the pavement”.

UAT. This is where some customers tend to experience a little anxiety and suddenly question their very own requirements and the very system they helped build, see Common UAT Pitfalls for more details, while other customers shine and jump into the drivers seat and master the road test (UAT).

Acquiring a new car (System/Major Functionality Changes/Upgrades) can be overwhelming for many reasons, covered in the Common UAT Pitfalls Ivanti HEAT ITSM Blog Post. But like anything else in life, you need to do a self check and trust the process.

You’ve made it this far, sure the new car can be overwhelming, but once you hop into the driver seat (take responsibility) and go on that road-test (UAT), familiarize yourself with the new vehicle, adjust the mirrors, seat, study the controls (learning curve) and see how the vehicle handles (learning curve), you soon start parallel parking, emergency breaking tests, city driving, highway driving, and so on.

The road-test (UAT) is the time to stop kicking tires and take action. Trust the process and go for that test-drive. The goal is to report back any issues (UAT Test Scripts), collaborate with the family, (UAT Meetings with UAT Lead to review UAT Test Script Results), and ask questions (UAT Test Script Module has comments for the consultant), and not only CONTINUE with the road test (UAT Test Scripts), but go on several road tests (many tests per UAT Test Script), and before you know it, you master the new car (system).

The last thing you want to do is stop the road-test every time you are unsure (learning curve), have a question (Learning Curve / UAT Meeting), or feel something is not right (UAT Test Script Result). You keep driving and report back your findings (UAT Test Script results) and collaborate with the family (UAT Team) to see if your findings are issues or just a learning curve (new controls, new way of doing things), or terminology (gas tank versus fuel tank), or it’s a major change, like driving on the right side of the road to the left side (major system upgrade), which requires more test drives. Or maybe you’ve never driven a car before and you are upgrading from a motorbike, bicycle, or walking on foot (legacy systems, Excel, Word, handwritten notes). Assumption is that you’ve taken your driving test and have a valid license of course (experience in the business, and have used Windows and web-apps before). Nowadays web applications are intuitive, as is learning to drive a different type of car (sports car, SUV, mini-van, luxury car). The days of long thick training manuals and days of classroom training are in the past although training videos can be provided as needed to fill any major training gaps. In other words, a learner’s license and some time with an experienced driver are just as efficient as a 2 week intensive course. At this stage of the implementation all SMEs are typically well versed with handling a car (Windows, Web Applications).

The learning curve will vary and you must TRUST THE PROCESS. You’ve made it this far, and should feel confident you have the support of the internal teams and external Ivanti HEAT Consultant, however ultimately you need to do the test-drive. The Ivanti HEAT Consultant has already done his. Now is the time to roll-up-your-sleeves-get-to-work-test-test-test with the goal of testing and providing concise precise results back to identify any gaps and more importantly, CELEBRATE THE WINS, mark test results as passed. So focus on the EASY TESTS FIRST, you know the parallel parking, driving in the parking lot, city driving on a slow day, before you go on a multi-lane highway or off-road in the Rocky Mountains on one of the toughest terrains.

You will get there, just keep testing, and know that the combination of your planning of must-haves (business requirements), intimate knowledge of your business processes, and the extensive experience and guidance by the Ivanti HEAT Consultant will get you there!

Who owns UAT Test Scripts?

Sometimes confusion arises over who owns the User Acceptance Testing (UAT) Test Scripts.

The UAT Test Scripts are owned by the Customer, this is a best practice as the intended use for the UAT Test Scripts is to validate business requirements, which should be aligned to the customer’s operating procedures. The consultant is external and does not own the business process, operating procedures, or business requirements.

The HEAT Consultant owns the Solution Design Documentation and System Test Scripts.

Keep in mind that the UAT Test Scripts are not training guides to the system, but rather steps to validate the business requirements by the owning team. It is up to the owners of the scripts, to maintain the scripts and word the steps in a way that is meaningful to their users.

Are UAT Test Scripts considered Training Material? No. UAT Test Scripts could form a basis for Training Material, however Software has come a long way and the old ways of creating extensive training materials with step by step instructions and days of classroom training are in the past. Web applications nowadays are intuitive and require very little training. Albeit there is always a learning curve for any new application. If more detailed training materials are needed, over and above the training videos and workshop recordings provided by the HEAT Consultant, then a best practice is to have the customer’s training lead or knowledge manager to work with the consultant to tailor customer specific training materials.

Going back to the Analogy used in my blog post “Common UAT Pitfalls“, that of test-driving a car. You as the customer are ultimately responsible for determining the best vehicle make, type, and test driving (UAT Testing) it with your set of criteria. You never want to rely on the, potentially biased, dealership (Consultancy) to tell you what to test-drive. The dealership may give you some tips as to what road to use, just as the HEAT Consultant provides you with some sample UAT Test Scripts, but ultimately the customer is the decision maker and responsible for making the purchases and test-driving before you sign on the dotted line. So be sure to take the system for more than just 1 test-drive and test thoroughly with all conditions and criteria you require.

What are ITSM, ITIL, and Ivanti?

ITSM began with the Information Technology Infrastructure Library (ITIL), which provides best practice guidance and an extensive set of processes and policies for improving IT service delivery and customer satisfaction.