While obvious to the most seasoned Quality Assurance Analysts and HEAT ISM Consultants, the gathering of UAT Sample Data can be confusing to some and is often poorly documented and executed.
The worst thing you can do is test milli-vanilli and only make up input data without using real data. Using made-up sample data can be confusing to UAT Testers, difficult to follow, and lead to false test results.
UAT is all about validating business requirements and confirming that you can perform your day to day job functions. So why would you want to use some made up data? You don’t. You want to use real live actual data to verify that the new system or feature behaves exactly like your old system, whether HEAT Classic, another software system, Excel, manual forms, legacy application, and so on. In my 25+ years of Ivanti HEAT ISM Implementations I have not come across an implementation that didn’t have some sort of tracking tool or method, spreadsheet, or other artifact that can be used as sample data. Even handwritten logs could make up sample data, and I don’t remember the last time I have seen one of those.
Sample input data should be real data, from your existing system, whether it’s a database, excel sheet, paper forms, or another system or artifact. For example tickets from your old HEAT Classic system, Paper Forms from a manual system (Invoices, Logs, etc), if you’re going digital, or sample rows from an Excel sheet if that’s what you’re replacing. These are just some examples.
Tip: Create a “complete workflow” artifacts of examples from start to finish in the process. One of my clients recently added procurement workflows to the Ivanti HEAT System and created a PDF of sample data for the relevant process, from inception of the procurement request, approval, quote, shipment, receipt, asset tagging, stock administration, invoice, and payment. The artifact was through and included sample data from every step. Some digital, others print outs, and even handwritten notes on the invoices. All of which were important sample data for User Acceptance Testing. It was a one time effort by the Subject Matter Expert (SME) and was invaluable throughout UAT, Training, and overall for discussion purposes. Intangibles were positive user morale, adoption, and a clear understanding of UAT Testers as to what data to use and how the new system would benefit them relating back to how they were doing things before and how that was going to change in the new way of doing things. In other words, the sample data artifacts were valuable Training Knowledge and used for review and discussion purposes, and during ongoing UAT Status Calls and meetings.
Bottom line, use real data that represents the typical type of data to expect and the UAT Team members can relate to. Tip: Ask yourself if the data could be used for training purposes. If it can, then include it as an artifact for training documentation. If it can’t, then ask yourself if the sample data is a good representation of a typical day in the life of the end user.
Furthermore, keep in mind that when there are changes, upgrades, or issues, you will want to have sample UAT Test Data to fall back to.
This is a good question, UAT testing is key to successful Ivanti HEAT implementations and rollouts of major features/functionality.
How many UAT Tests should be performed per Test Case?
As many as it takes for you and your team to feel comfortable that you can perform your day-to-day job function.
Recommendation and Best Practice is a minimum of 5 tests per UAT Test Script. Anything less is a risk that the client need to evaluate. Some detailed, complex, and high impact UAT Test Scripts may require at least a dozen of tests for the various scenarios that you have identified. It is better to test more than less, that’s for sure. Perform as many tests as it takes for you and your team to feel comfortable that you can perform your day-to-day job function.
Experience dictates that if UAT (validation of business requirements) isn’t thorough then the likelihood of post-implementation issues, gaps, and problems is much much higher, and can take more time and effort to address than during UAT.
Intangibles like morale and perception of the new Ivanti HEAT Service Manager system/features can also be negatively impacted.
Also note that you are not bug hunting, so you don’t need to test every single feature and menu item, however you are testing that you can perform your daily job function, so test all the features related to the use case (business process) that apply.
It’s not unlike ordering a fleet of vehicles simply by kicking tires and doing a lap around the parking lot versus more extensive test-drives tailored to the company wide business needs. Sure, everything “might” appear fine, and “might” go fine upon delivery. But those are huge risks to take for a high dollar, high impact, high exposure item that ultimately could lead to work slow down or stoppage.
No matter how detailed your UAT Test Scripts, you need to have a good range of test case scenarios (sample input data) for every test script. Sample input data should be real data, from your existing system, whether it’s a database, excel sheet, or paper forms. For example tickets from your old HEAT Classic system, Paper Forms from a manual system, if you’re going digital, sample rows from an Excel sheet if that’s what you’re replacing. Bottom line, use real data that represents the typical type of data to expect and the UAT Team members can relate to. Tip: Ask yourself if the data could be used for training purposes. If it can, then include it as an artifact for training documentation. If it can’t, then ask yourself if the sample data is a good representation of a typical day in the life of the end user.
The time and effort required is usually negligible. Minutes per test.
User Acceptance Testing (UAT) is without a doubt the most critical stage of the SDLC. This is where “the rubber hits the road”, and the customer and subject matter experts (SMEs) validate the business requirements.
This is also where typically some customers and SMEs tend to panic, get confused, worry, or simply get overwhelmed by the project at hand. Rightfully so, there is usually a lot at stake and the customer wants to make sure their business requirements are met, as does the consultant.
On one side, the customer and SMEs look up to the consultant for guidance drawn from years of experience, on the other hand, customers and SMEs also tend to think they know better.
So who is right?
Over 2 decades with HEAT Consulting, Business Analysis, and Project Management have taught me and reinforced, is that the customer isn’t always right and neither is the consultant. Looking at who is right is wrong! Instead of building a law case and looking to disprove one another, everyone needs to TRUST THE PROCESS.
If you have made it to UAT so far. Best Practice is to always review the steps taken (process) to-date.
UAT Remediation (to address failed tests and gaps) by Ivanti HEAT Consultant
UAT Sign-Off
Go-Live (Rollout) Prep by Customer/SMEs, and Ivanti HEAT Consultant
Go-Live (Rollout) by Customer/SMEs, and lead by Ivanti HEAT Consultant
Go-Live (Rollout) Support by Ivanti HEAT Consultant
Post Go-Live (Rollout) Remediation by Ivanti HEAT Consultant
Minor Enhancements (if needed and within scope)
Major Enhancements (if needed and within scope)
In Summary, using the analogy of purchasing a car, the buyer (customer) has done their research and made a list of must-haves, and nice-to-haves (business requirements), identified the car type, model, and budget (scope) with the help of the dealership (Ivanti HEAT Consultant), and now it’s time to take the car (system) for a road test (UAT). This is where “rubber hits the pavement”.
UAT. This is where some customers tend to experience a little anxiety and suddenly question their very own requirements and the very system they helped build, see Common UAT Pitfalls for more details, while other customers shine and jump into the drivers seat and master the road test (UAT).
Acquiring a new car (System/Major Functionality Changes/Upgrades) can be overwhelming for many reasons, covered in the Common UAT Pitfalls Ivanti HEAT ITSM Blog Post. But like anything else in life, you need to do a self check and trust the process.
You’ve made it this far, sure the new car can be overwhelming, but once you hop into the driver seat (take responsibility) and go on that road-test (UAT), familiarize yourself with the new vehicle, adjust the mirrors, seat, study the controls (learning curve) and see how the vehicle handles (learning curve), you soon start parallel parking, emergency breaking tests, city driving, highway driving, and so on.
The road-test (UAT) is the time to stop kicking tires and take action. Trust the process and go for that test-drive. The goal is to report back any issues (UAT Test Scripts), collaborate with the family, (UAT Meetings with UAT Lead to review UAT Test Script Results), and ask questions (UAT Test Script Module has comments for the consultant), and not only CONTINUE with the road test (UAT Test Scripts), but go on several road tests (many tests per UAT Test Script), and before you know it, you master the new car (system).
The last thing you want to do is stop the road-test every time you are unsure (learning curve), have a question (Learning Curve / UAT Meeting), or feel something is not right (UAT Test Script Result). You keep driving and report back your findings (UAT Test Script results) and collaborate with the family (UAT Team) to see if your findings are issues or just a learning curve (new controls, new way of doing things), or terminology (gas tank versus fuel tank), or it’s a major change, like driving on the right side of the road to the left side (major system upgrade), which requires more test drives. Or maybe you’ve never driven a car before and you are upgrading from a motorbike, bicycle, or walking on foot (legacy systems, Excel, Word, handwritten notes). Assumption is that you’ve taken your driving test and have a valid license of course (experience in the business, and have used Windows and web-apps before). Nowadays web applications are intuitive, as is learning to drive a different type of car (sports car, SUV, mini-van, luxury car). The days of long thick training manuals and days of classroom training are in the past although training videos can be provided as needed to fill any major training gaps. In other words, a learner’s license and some time with an experienced driver are just as efficient as a 2 week intensive course. At this stage of the implementation all SMEs are typically well versed with handling a car (Windows, Web Applications).
The learning curve will vary and you must TRUST THE PROCESS. You’ve made it this far, and should feel confident you have the support of the internal teams and external Ivanti HEAT Consultant, however ultimately you need to do the test-drive. The Ivanti HEAT Consultant has already done his. Now is the time to roll-up-your-sleeves-get-to-work-test-test-test with the goal of testing and providing concise precise results back to identify any gaps and more importantly, CELEBRATE THE WINS, mark test results as passed. So focus on the EASY TESTS FIRST, you know the parallel parking, driving in the parking lot, city driving on a slow day, before you go on a multi-lane highway or off-road in the Rocky Mountains on one of the toughest terrains.
You will get there, just keep testing, and know that the combination of your planning of must-haves (business requirements), intimate knowledge of your business processes, and the extensive experience and guidance by the Ivanti HEAT Consultant will get you there!
The UAT Test Scripts are owned by the Customer, this is a best practice as the intended use for the UAT Test Scripts is to validate business requirements, which should be aligned to the customer’s operating procedures. The consultant is external and does not own the business process, operating procedures, or business requirements.
The HEAT Consultant owns the Solution DesignDocumentation and System Test Scripts.
Keep in mind that the UAT Test Scripts are not training guides to the system, but rather steps to validate the business requirements by the owning team. It is up to the owners of the scripts, to maintain the scripts and word the steps in a way that is meaningful to their users.
Are UAT Test Scripts considered Training Material? No. UAT Test Scripts could form a basis for Training Material, however Software has come a long way and the old ways of creating extensive training materials with step by step instructions and days of classroom training are in the past. Web applications nowadays are intuitive and require very little training. Albeit there is always a learning curve for any new application. If more detailed training materials are needed, over and above the training videos and workshop recordings provided by the HEAT Consultant, then a best practice is to have the customer’s training lead or knowledge manager to work with the consultant to tailor customer specific training materials.
Going back to the Analogy used in my blog post “Common UAT Pitfalls“, that of test-driving a car. You as the customer are ultimately responsible for determining the best vehicle make, type, and test driving (UAT Testing) it with your set of criteria. You never want to rely on the, potentially biased, dealership (Consultancy) to tell you what to test-drive. The dealership may give you some tips as to what road to use, just as the HEAT Consultant provides you with some sample UAT Test Scripts, but ultimately the customer is the decision maker and responsible for making the purchases and test-driving before you sign on the dotted line. So be sure to take the system for more than just 1 test-drive and test thoroughly with all conditions and criteria you require.
User Acceptance Testing (UAT) is where some Ivanti HEAT ISM Projects tends to fall apart. Today I will cover the most common UAT Pitfalls, recommendations, and best practices to put your Ivanti HEAT Project back on track!
Thank you to Test Lodge for allowing us to use their Image on Slide 3
Perspective
First and foremost you want to do a review of where the HEAT Implementation Project is at. The typical implementation milestones such as Scope of Work, Requirements Gathering, Solution Design, Prototype, and Business Requirements Sign-Off and/or Prototype Sign-Off have been met. That alone should give the customer confidence. And if it doesn’t, then now is a good time to re-iterate all the successes and milestones met.
Drawing from and relating to previous implementations at the same customer site also gives perspective, for example, if several UAT Test Teams have completed their UAT Test Scripts within 5 days and one team is struggling and taking weeks on in, then that is worth sharing to the struggling team and having other UAT Teams share their experiences and successes.
Try to use analogies wherever possible to explain and clarify where things are at. I like using the analogy of purchasing a car. The customer has requirements, such as family of 4, active life style, living in a 4 season climate, with a medium size budget. The UAT Test Scripts are the test drive. The family is the UAT Team. The consultant wears hats of the sales person and mechanic. Ultimately the family needs to go for a test drive , mom or dad are the ultimate decision makers and lead (UAT Lead) the family, perhaps with their oldest child (the UAT Lead), and all aspects of the requirements are discussed with the sales person and perhaps the mechanic (consultant). The sales person (Consultant) gives some advice on what route to take (sample UAT Test Scripts), and helpful advice as to how the car handles and what features the car has. Ultimately though the Customer needs to test drive the vehicle. Get in, familiarize yourself with the control, adjust the mirrors, seat, and so on. Not unlike familiarizing yourself with the new HEAT System and/or features. And then you take the vehicle for a spin, on the highway, on a city road, on rough terrain, in a parking lot to practice emergency braking, parallel parking, check the trunk, storage space, and all aspects of the day-to-day use of the vehicle (the UAT Test Scripts) and then comes back to the Salesperson with questions. It is very seldom that after carefully studying the market and available vehicles that the customer will come back saying the car is undriveable. So why is it that some User Acceptance Testing sessions go haywire? Keep Reading!
Not having UAT Test Scripts This by far is the biggest issue that can de-rail HEAT Projects. Validation of Business Requirements is essential, as is a proper UAT Test Plan. UAT Test Scripts are a must have to achieve Implementation Success! Who owns the UAT Test Scripts? The customer. The consultant owns the System Test Scripts and Solution Design. Business Processes, Standard Operating Procedures, Business Requirements, and validation of business requirements (UAT) are owned by the customer. Hence that’s why the consultant does not own the test scripts. Simply put, if you are purchasing a car, you typically have a checklist of requirements that you came up with, not the sales person. You don’t walk unprepared into a car dealership and then sign-off after reading the pamphlet. Do your due diligence!
Not following UAT Test Scripts Having UAT Test Scripts is a step in the right direction, however execution is key. UAT scripts must be followed to ensure you’re validating business requirements, and the UAT Lead must work closely with the UAT Test team to ensure team members understand what is expected and asking them why they aren’t following a proven process? When you walk into the car dealership, you don’t just start up the car and say ok I’ll take it. You go for a test drive.
Not following the UAT Support Structure As with any process or procedure, it is important to understand the support structure. The UAT Team Members must know who to turn to when there are questions or issues. The UAT Lead is responsible for coordinating UAT Test Scripts with the Team and ensuring Team Members know what is expected. In the event of questions, the UAT Lead should channel that question to the correct individual, if the UAT lead is unable to answer. Note that T.ogether E.veryone A.chieves M.ore, so be sure to have regular UAT Team Meetings where the UAT Team Members can review their finding s and ask for support (HEAT How-To, Procedural, etc) amongst the TEAM and with the UAT Lead and getting direction from the Decision Maker.
When you purchase a car as a family, then everyone gets a chance to participate in the test-drive, unless of course the kids are toddlers, but even then you need to include them in the test-drive, and strap them into the baby-seat to see if it is up to your requirements. So when family members identify an issue, it goes to the parent. Ultimately it’s the parents that sign off, make and own the decision.
No Collaboration Some UAT Testers tend to work in silos, creating a list of all the problems the find whereas the best practice is to encourage collaboration, as per the above support structure, and emphasize solutions, not problems.
If your team insists of creating a separate list of problems in outdated tools such as Word, Excel, Notepad, or handwritten, then it’s time to re-emphasize the importance of collaboration and discussing the issues at the next UAT Team Meeting rather than letting the frustrating build up.
When the family members decide to raise problems at every turn of the test-drive, then it’s time to have a family meeting and address the concerns. Is it really a problem with the vehicle? Is there something else going on? Is there some confusion around an aspect of the vehicle?
No Internal Meetings One of the best practices for UAT is to ensure you have regular standup UAT meetings, daily at first, until you have momentum and are able to move to weekly. These meetings are essential for collaboration and ensuring everyone is working together to find solutions, not problems. Not unlike family meetings, the parents need to be on the same page and the family deserves to know what’s going on and have some input on the decision and test drive results.
Learning Curve Any new software product or major feature has a learning curve. HEAT is no different. Moving from Excel to HEAT ISM, even moving from HEAT Classic to HEAT ISM has a learning curve, as do major functionality enhancements to HEAT. Without the support structure, collaboration, internal meetings, that that learning curve can be detrimental. It is important that any issues, no matter how big or small or channeled to the UAT Lead and/or Training Lead. If needed, the UAT/Training Lead should make a list of questions for the HEAT Consultant so that areas requiring assistance are addressed. Tip: This is where the a19 UAT Test Scripts Module comes in, by entering Comments for the consultant.
When you’re test-driving a vehicle, there is a leaning curve too, so what do you do, first you do a visual inspection around the car, then inside inspection, check the side mirrors, rear view mirror, familiarize yourself with the control, adjust the seat, and then get familiar with how the car handles. Just like with any new HEAT System or new Ivanti Service/Asset Manager features. It’s quite intuitive. Of course unless you’ve never driven a car or never even used a PC or Google. Then the learning curve will be harder and you need some basic training.
Terminology When implementing a new system, there often is new terminology or a change in terminology, be it technical or business related. You’ve made it this far, so any new terminology has without a doubt been covered in the many workshops, design sessions, prototype reviews, and at sign-off.
Any questions about terminology should be logged by the UAT Lead, reviewed with the Decision Maker, and if needed, reviewed with the HEAT ISM Consultant. Often it’s just simple changes in terminology, like gas tank or fuel tank. Trunk or boot of the car. If you haven’t heard one term before well then you don’t raise that as a flag and stop driving the car and say it’s unusable. You just ask for clarification and keep on plugging away. It’s not the end of the world, but if you’re confused then raise your hand and talk to the UAT Lead.
Resistance to Change Change is inevitable. The only constant is change. Yet it’s human instinct to resist change. Change needs to be enforced by the Decision Maker and supported by the UAT Lead. Sometimes the best answer is to re-emphasize “the new way of doing things” and gradually over the time, users will adapt. Like when you purchase an SUV because it fits all the business requirements but many family members really had their heart set on a Sports Car or family members were used to the old gas guzzling family car that had lots of good memories, was no longer economical and the family had outgrown.
Not following or understanding business process/procedure The key to remember is that the Business drives Technology, and not the other way around. Focus needs to be on the business requirements and operating procedures. Be sure to seek guidance from the Decision Maker if any business operating procedures are unclear. When you’re test driving a car, then you test various aspects of your day-to-day use of the car. Highway driving, city driving, off-road, parking, etc. Everyone on the UAT Test Team (participants) should be well acquainted with their role. New drivers will need more help than others.
Not understanding the UAT Test Script Some UAT Testers might find UAT Test Scripts hard to follow. Remember these scripts are built by the decision maker and/or UAT Lead, so be sure to raise any questions and make note of any improvements to the UAT Test Scripts. UAT Test Scripts aren’t written in stone and can be updated as needed. Some UAT Test Scripts need no explanation at all, while others may require explanation of a standard operating procedure, steps to take, and input date. If you take a car for a test drive and milli-vanilli decide to do emergency braking on the highway, well that’s going to shock and confuse everyone. So make sure you are clear on what and why you’re testing.
Not having UAT Test Data While you may get away with using milli-vanilli data (arbitrary data), the best practice is to always use real data and examples from your existing system, whether that system is a sophisticated computer system, excel, or hand written artifacts. You will want to have real life data available that accurately reflects the type of input the tester will be using in day-to-day operations. Simple put, if you’re replacing your Helpdesk Ticket System, use ticket data. If you’re replacing sales order, use sample sales orders. Similar to regression testing, put your old system on the left hand side and the new system on the right hand side and then go through the motions in your old system and replicate in the new system. When you’re test-driving a car, your test data is your current way of doing things, plus some artifacts such as the oversized golf clubs or family bikes or skis that need you need to make sure fit.
Focusing on nice-to-haves versus MUST-HAVES The key intention of UAT is to validate business requirements (must-have-requirements), not to minimize the number of clicks, or nit-pick UI Design. There is always room for UI Improvement (nice-to-have requests) and that time is after UAT is complete and business requirements have been signed off. This tends to be where some UAT Testers get stuck, focusing on the window dressing instead of the architecture and foundation. When you purchase a car, the accessories are just that. Sure the super expensive surround sound is nice, but is it needed? Are the fuzzy dice, bumper stickers, upgraded paint job, and fancy wheel covers really imperative when conducting road test?
Training Material Keep in mind that the UAT Test Scripts are not training guides to the system, but rather steps to validate the business requirements by the owning team. It is up to the owners of the scripts, to maintain the scripts and word the steps in a way that is meaningful to their users.
Software has come a long way and the old ways of creating extensive training materials with step by step instructions and days of classroom training are in the past. Web applications nowadays are intuitive and require very little training. Albeit there is always a learning curve for any new application.
If more detailed training materials are needed, over and above the training videos provided by the HEAT Consultant, and workshop recordings of all HEAT Project Sessions to date, then a best practice is to have the customer’s training lead or knowledge manager to work with the HEAT Admin to clarify and if needed, work with the HEAT consultant to tailor customer specific training videos.
However, that shouldn’t stop UAT testing. As always keep moving forward and test what you can, collaborate with your team, and ask for help from the UAT Lead, who can always get help with HEAT specific functionality question from the HEAT Admin.
More often then not, it’s just a matter of getting used to the new way of doing things, reading between the lines, and trust that the most important interface, the chair to keyboard interface, will figure it out.
It’s not unlike getting used to a new card that you’re test driving, it will handle differently then what you’re used to, the gas tank may be called fuel tank and placed on the opposite side of what you’re used to, but doesn’t stop you from the road test. You continue as needed and trust that you will figure out the changes. Owner manuals are time consuming to read, hardly every read, and just collect dust in the glove compartment. A few questions, some help from others, from the experienced drivers, and voila you are mastering the road test in no time.
Waiting until the final hour to ask for help Some implementations tend to go very smooth while others tend to stagnate with the UAT Team either not testing, focusing on nice-to-haves versus validating must-have requirements, and any combination of the above pitfalls, and then ultimately raising a red flag out of the blue with a long list of issues, that could have been easily addressed as per the above recommendations and best practices.
You don’t wait until the day of the paper signing to mention that rattling noise or performance issue during the test-drive, you mention it right here and then. But it can happen, and if it does, then you go for another test drive and move forward!
Gaps Although technically not a pitfall the UAT Team might stop testing all together or raise flags when gaps are identified. Ironically Gap Analysis is the intention of UAT, to validate the Business Requirements and identify any show stoppers (gaps). When a gap is identified, it should be raised to the UAT Lead to verify, and if confirmed, reviewed with the decision maker, to prioritize. Showstopper versus implement-later versus nice-to-have. For example, that ski-rack that you need, and isn’t available. Is that a show stopper? If it’s the middle of summer and you can wait until December? Not having 4 wheel drive for country roads in winter conditions for a work truck could be a major gap. And you know what we do with gaps right, we address them, and fill them. Such as ask for the 4 wheel drive model. However make sure you adequately prioritize gaps and just focus on needs, not wants.
In summary, the best practice for Ivanti HEAT ISM UAT is to ensure that UAT Support Structure is followed, UAT Test Scripts are actively used and maintained, real test data is used from your existing system, and the team looks for solutions, not problems, collaborates regularly, and focuses on must-have versus nice to haves, and moving forward with the new way of doing things, realizing there is a learning curve, and the decision makers have signed-off after many workshops, prototypes, and discussions to move ahead and need you to validate that the day-to-day requirements have been met.
It’s not unlike test-driving a new car you want to purchase. You take it for a spin, and take it through the motions of your day-to-day activities. Parking, driving on the highway, check the storage space, handling, ask your significant others opinion, collaborate on your findings, and then focus on must-haves. The nice-to-haves like the fuzzy dice on the rear-view mirror aren’t important. Those you can get to later, after you did some emergency braking, speed tests, and the likes and find the new car is up to snuff. Sure, you need to get used to how it drives, there is a learning curve to any new car. And if the ski rack you want isn’t available now, in mid-summer, well no biggie, not the end of the world. Also rest assured that the decision makers have done their research, kicked the tires, and had many discussions with the dealership to make sure it’s the right fit. And of course the dealership is committed to address any reasonable concerns or issues.
Regression testing refers to testing of existing functionality to ensure that recent changes do not adversely affect existing features.
For example, when implementing Asset Management, the existing Incident functionality is tested to test use cases such as creating an incident, creating a task, updating incidents, closing tasks, incidents, etc
In addition to UAT Test Scripts, you will want to do a comparison of existing workspaces, existing functionality, and existing features, to ensure existing functionality has not been impacted.
Ivanti Best Practice: If you want to run Ivanti Service Manager UAT and PROD tenants side by side, you need to use a different browsers (not just different tab), as otherwise the browser cache may become corrupted or cause issues. So for example you can use Google Chrome for UAT and Internet Explorer for PROD.
The purpose of User Acceptance Testing (UAT) is to test use cases, that is specific day-to-day operations in the life of an end-user and their required tasks to perform their job function
UAT Testing differs from System Testing in that System Testing is focused on software functionality. UAT Testing is focused on job function, that is day to day operations can be performed. Therefor access to Ivanti Service Manager (powered by HEAT) is not needed to build UAT Test cases, in fact it is discouraged to use HEAT as your focus should be on building use cases about day to day operations, operating procedures, such as Logging an Incident, searching for customers, assigning 2nd level support, and so on.
Regression testing refers to testing of existing functionality to ensure that recent changes do not adversely affect existing features. For example, when implementing Asset Management, the existing Incident functionality is tested to test use cases such as creating an incident, creating a task, updating incidents, closing tasks, incidents, etc. In addition to UAT Test Scripts, you will want to do a comparison of existing workspaces, existing functionality, and existing features, to ensure existing functionality has not been impacted.
One of the biggest issues with writingUAT test cases, is knowing where to start and insisting on building your test script around the software. The fact is, you do not need access to Ivanti Service Manager (ISM, powered by HEAT IT Service Management) and it is actually encouraged not to use ISM at all when devising UAT Test Scripts.
Writing UAT Test Scripts for ISM HEAT is not unlike writing a letter to a long lost friend. At first you don’t know where to start, the first few words are tough to get out. But as you start writing, and you focus on the subject matter, the words turn into sentences, paragraphs, and pages.
Same goes for UAT Test Cases. Start simple, and remember, it’s about defining test cases for day-to-day job functions, not system testing of software, features, or functionality.
You should focus on:
Test Script Case Subject | What are we validating?
Test Script Case Details | Optional if you need to expand on the subject at hand
Subject Matter Experts | Who is validating?
Input Steps | What are the steps, think SOP, that testers need to take to carry out the test
Input Data | Sample test data and supporting artifacts
Expected Results | What is the expected output?
For example, if you were to write use cases for a kitchen, you would make a list of all the use cases (criteria if you will), such as Making Breakfast, Making Lunch, Coffee, Washing Dishes, etc. You don’t need to have a kitchen, and you don’t need to cook an omelette to document a use case for breakfast. All you need to do is visualize the steps, intended input, and expected results. High level, bullet form. Anything more becomes training documentation.
In this Use Case example, steps would be prepare ingredients, follow recipe, cook, season to taste, and eat. Input would be your ingredients, output is an omelette. The case steps may reference a combination of training materials (how to crack eggs, mix ingredients, recipe steps if you will) and standard operating procedures, aka SOPs (using the stove, properly food handling, storage, etc). Training materials and SOPs are important references, however separate from UAT Test Script Cases.
As you can see, UAT Test Cases should be simple, high level, and do not require ISM HEAT access, and shouldn’t be confused with Training Materials or SOPs. You should consult Subject Matter Experts (SMEs), the end users, to ensure you have covered all the possible scenarios. In this example, this would be the chef. In the real world this could be the Service Desk Manager, Service Desk Analyst Lead, Storage Techs for Asset Scanning and Storage Management, Procurement Lead for Accounting, Purchase Order related cases, and so on.
Last but not least, your focus should be on Test Case Scenarios, not features, functionality, or software. That’s System testing. We’re not testing the recipe or stove’s every button and dial. We are constructing high level use cases and scenarios, of day-to-day operations.
If you’re looking for sample ISM UAT Test Cases and Scripts from the a19 Consulting – Ivanti Best Practice System then be sure to contact us! We’ve been providing HEAT (now Ivanti Service Manager) Professional Services since 1996 and can help!
Blackout, is a point in time in the development process when a system becomes unavailable (off-line) by design. The term comes from Electricity “blackouts”when there is a complete electric shut down.
A best practice is to take the UAT system off-line when UAT is undergoing maintenance or configuration in preparation for End User Acceptance testing. This is to prevent any unauthorized access by end users that might want to start testing the system before it’s ready, which is never a good idea. After all, you don’t call the building inspector until you’re up to code, right? Leaving UAT available during configuration can be disastrous. Don’t make this rookie ISM mistake, you will thank me later!
When you are gathering Ivanti Service Manager (ISM) and Ivanti Asset Manager (IAM) requirements to build your Solution Design Documentation and Scope of Work (SOW) there is one very important best practice. Set Implementation Priorities!
setting priorities for your ISM project
Scoping is an art of finding the right balance between effort, benefits, and long term gains. Quick Wins should be your 1st Priority (P1) followed by long term gains which are Priority 2 (P2) and have the highest benefits.
For example, on a recent Asset Management implementation the core project team determined that the quickest wins and highest benefits would be asset scanning and a foundation for procurement and assets (hardware and software) that we could built upon on future phases.
ITSM Priorities by Benefit and Effort
That lead us to this high level scope:
P1 – Asset Scanning / Procurement Foundation
P2 – Software Asset Management (SAM)
P3 – Hardware Asset Management Enhancements
What this meant was that we would build asset scanning capabilities not only for assets but also packing slips, replace the existing purchase order tracking excel sheets, and vendor inventory tracking sheets from Softchoice and various software vendors, with an Ivanti Service Manager product catalog for hardware and software, utilized the purchase order workspace for sales orders, enabled contracts and entitlements for software inventory, and built a softchoice integration to automatically update invoice and shipping information and functionality to reconcile against packing slips and purchase orders.
These were just our P1’s and P2’s! With further integration to discovery tools, enhancements to assets, and workflow automation for P3. Of course keep in mind that upon project completion some of the other requirements from the parking lot will become P1’s and P2’s and the cycle begings.
Along the way we identified morequick wins that improved overall product usability and project tracking:
ISM Administrator Level 2 Role – for limited admin access with dedicated top level tabs to manage assigned validation lists and some configurations without full administration console access
a19 Consulting Tracker– a custom built project tracking tool by a19 Consulting with backlog, artifacts, parking lot, status updates, project dashboard, heat release project and package tracking, to-do list, and many more features
a19 Consulting Change Control with Announcements, Alerts, Notifications, and approval process for User Acceptance Testing (UAT)
a19 Consulting UAT Test Scripts Workspace for ISM Administrators and Consultants to develop test scripts, run test scripts, log defect, pass/fail tests, request enhancements and identify show stoppers, must-haves, and nice-to-haves
ITSM/ITAM/ESM Blog 
You must be logged in to post a comment.