TechNow Ep 65 | ATF Custom UI and Parameterized Tests

[Chuck Tomasi]

In this episode, we'll cover what Automated Test Framework (ATF) is and why your organization needs to be using it. We'll also spotlight the latest Madrid features around testing custom UI pages and using parameterized tests to reduce the number of tests you need to build. Come learn how to accelerate application delivery cycle and reduce upgrade testing time using ATF.

Quick tip: Using ** in a reference field (Chuck Tomasi)

Originally aired: July 30, 2019 8:00 AM PT

(apologies for the audio quality on this one)

Featured Experts

Shrity Verma is a Sr. Product Manager at ServiceNow managing various products in App Development and Deployment area. She has over 17 years of experience in Data Warehousing and Cloud Computing as a Database Software Architect and Product Manager. Shrity has broad expertise in several areas which includes Application Development, Big data & Analytics, Database management and Internals. Shrity holds a computer science and engineering degree from JNTU College of Engineering and a MBA from UCLA.

Kreg Steppe is a Training and Certification Program Manager within ServiceNow developing and supporting cloud training infrastructure. He specializes in developing integration solutions, automating repeatable processes and Cloud Management in ITOM. Kreg's prior experience includes operating his own ISP, developing web applications in PHP, network integration, managing network support, Application Development on cloud based networks, DNS and email server maintenance. He is a Linux enthusiast and enjoys Photography.

Stacey Bailey is a Training and Certification Program Manager for ServiceNow with over 20 years of IT experience in various roles. Since 2009 she has been working with the ServiceNow platform customer, partner, and employee. She has earned numerous awards in her career including: ITSM MVP 2017, InfoWorld Top 100 Award, DCI Portal Excellence Award, Financial Times Award for Excellence in E-Learning. Stacey proudly refers to herself a s a "gadget girl". In her spare time she likes trying new gadgets, learning new forms of art, playing in the makerspace, collecting new / unique skills, listening to audio-books, trying new life experiences, and hosting parties to build communities.

Chuck Tomasi is a Sr. TPMM for ServiceNow. He is a computer science major with over 35 years of IT experience. As a former ServiceNow customer, Chuck won the first Innovation of the Year Award at Knowledge 10. Since joining ServiceNow in 2010 as a Technical Consultant, he has done many large scale ITSM implementations and custom applications, acted as an adjunct instructor for Education Services, created and lead the Technical Best Practices program, makes appearances on Live Coding Happy Hour, created dozens of fit for purpose custom applications, and co-hosts the ServiceNow series "TechNow".

---

LIST ALL EPISODES

Questions and Answers

What was that about ** in reference fields? -- Michael White
It will give you a quick search in a reference field without having to open the search dialog. -- Kreg Steppe

do you still support Jasmine assertions? -- David Dietrich
Yes you can write Jasmine assertions in a "Run Server Side Script" step -- Matt Coleman

Will there be integration between ATF and Test management module? Especially to use Test Plan functionality to sign-off test executions. -- satya Tota
this is under consideration with the test management team -- Joel Fischer

It is not that - her accent is very heavy - sorry - I'm going to drop off cause I cant hardly understand a word she is saying. 😞 -- Jon Myers
Thank you for the feedback. We'll consider this when selecting other guest speakers. -- Chuck Tomasi

Is a certain role needed to be able to see the Page Inspector function? -- Kenya Greene
Should be visible to admins and atf test designers. -- Stacey Bailey

I was trying to test adding a KB block to a KB article, but the UI was not supported by Page Inspector. How can we know what is supported and what is not by the page inspector? -- Sushant Sood
please submit an incident so that we can review your case -- Joel Fischer

Can you update Automated tests? -- Karen Guerra
Yes, of course! That's essential for when you create tests for your business functionality as your business evolves. For "quick start" tests, you can create a copy of those tests and tailor them for your particular implementation. It's best practice to make sure you keep your ATF tests up to date as your instance evolves. -- Stacey Bailey

Can image be clicked or handled thorugh Custom UI? -- Sovan Banerjee
Yes you can use a 'Click Component (Custom UI)' step to click an image -- Matt Coleman

When will the ability to invoke buttons in Related Lists tabs? -- David Pax
It's on the roadmap. No specific release planned yet. -- Matt Coleman

Are we able to create tests for the CSM portal using ATF? -- David Dietrich
Try using the Automated Test Framework > Manual Page Inspector to open the CSM portal page and interact with it. In Madrid it appears the CSM Portal framebusts the test runner frame so it will not be possible to run tests against it. Submit an incident requesting that the CSM team address this. -- Joel Fischer

Isn't that what * does? I guess I'm misunderstanding the difference -- Michael White
* doesn't show any records unless you suffix it (e.g. *ga). Just by itself, nothing is displayed. ** shows a list of records in the "preview" dropdown. -- Chuck Tomasi

Will ATF support UI Pages that pop-up on a request in the future? -- Mark MacConnell
In Madrid and up, the Custom UI step category can test GlideModals opened from a form, but not new browser pop-up windows -- Joel Fischer

I asked the mobile testing question to the former CEO of Sky Giraffe (can't remember his name) and he told me if used as designed ATF is not needed for mobile because they own the inner workings and they make sure it is working as designed. Not sure how ATF would be used, but maybe there is a way. Glad they are considering. -- Jay Freise
Thank you for your comment -- Stacey Bailey

Will you ever be able to impersonate user in an ATF, test runner currently logs you out of current session? -- Stan Haag
please create an incident if an ATF test is logging you out of your instance during a test. Make sure your user has roles and is active -- Joel Fischer

Thank you Stacey! -- John Gubatayao
Thank you, John! -- Stacey Bailey

Are the test still valid to export from Developer.servicenow.com to our instance? -- Steve Crocker
you'll find varying degrees of success exporting and importing data between two unrelated instances. Tests are metadata, and adhere to the instance's configuration and database. If you import a test and notice that data is missing, you can still update the test with data available to your instance -- Joel Fischer

Would you please send a link to the recording -- Deepti Gupta
You can find this listed as episode 65 at http://bit.ly/servicenow-technow -- Chuck Tomasi

is there a documenttaion on what is not testable through ATF? -- Chetan Gandhi
Yes, on docs.servicenow.com there is a (Madrid and up) page "Custom UI test steps" that lists testable and untestable ui components -- Joel Fischer

Can we pass variable from one test case to other within a test suit? ideally within a test case INC number can be passed from one step to other but its not allowing to relate all test cases within a suit. Or is my understanding incorrect -- Pavan Amarala
Tests cannot share data. All data created or modifiied during a test is rolled back to its original state at the end of each test. We are aware of the interest to support calling tests within other tests, but please submit an enhancement request to have your scenario reviewed -- Joel Fischer

In Custom UI - If the clickable component have same details and name, How can we know which component needs to be clicked ? -- Manoj Soundararajan
The easiest way to find your component is to add an 'id' attribute to the HTML element. Then you can search for that ID when building the step -- Matt Coleman

If you create a record at the beginning of a test do all of the tests in that suite use that record? -- Steven Bogart
Data is rolled back after each test execution, so a record created by one test in a suite cannot be used by the other tests in the suite -- Matt Coleman

Have you experienced or seen customers ATF do NOT roll back properly? -- Kimberley Sanders
No. I have not heard of that. If you are having issues with that behavior, you may want to reach out to customer support. -- Chuck Tomasi

Currently the test results go into a log - are there plans to be able to display via a dashboard -- Jerry Groesser
Per Shrity, you can use the standard reporting and dashboarding functionality to create a dashboard. -- Stacey Bailey

HI, for the regular incident, problem forms with multiple UI action etc, which testing approach is preferred? [Should we use this UI testing approach?] -- Balaji Jaganathan
You can use the Form category of steps to interact with standard platform forms. There are some cases like the Change table's form interceptor that will require a Custom UI step to click the interceptor links. -- Joel Fischer

How does the option for Custom UI shows up in the Add Test Step popup? -- Sankalp Joshi
There is a new 'Custom UI' category in the Add Test Step modal that contains all the Custom UI steps -- Matt Coleman

I missed this due to my connection, but how is page inspector different to steering by the DOM? If you have good stable IDs for objects, upgrades shouldn't affect a POM approach. -- Alistair Driver
The page inspector shows how the ATF will interact with a page during test execution. It enables a developer to confirm their page is testable outside of writing an ATF test. Having IDs on your testable elements is recommended -- Joel Fischer

would this work on a modal page like the risk survey? -- Chris Bui
Yes you can use Custom UI steps to interact with modals -- Matt Coleman

Is there a possibility the parameters will be able to read directly from existing tables (i.e., won't have to create a choice list)? -- Susan Williams
yes parameters are dictionary records, and can be defined as references to tables -- Joel Fischer

Is the training content only available to partners or regular users as well? -- Sheikh Ayub
Anyone -- Kreg Steppe

how can we search the articles? -- Sovan Banerjee
Try the Form category > Open an existing Record step to the article's definition form, then Form category > Click UI Action "View Article" -- Joel Fischer

Will all the components identified by page inspector be available to custom UI testing? -- Aditya Reddy Avula
Yes -- Matt Coleman

i think for the last list value defined the value and position have been entered incorrectly -- Umesh krishnan Perumbala
Thank you! She ended up correcting it. -- Joel Fischer

Are these examples for this Maually adding step by step available somewhere? -- Kimberley Sanders
If you're referring to adding steps from the Custom UI step category, check out docs.servicenow,com (Madrid and up), page "Create a custom UI test". For just adding any steps to a test there is an eLearning Series "Automated Test Framework (ATF) Fundamentals (ID: ES-SN-ATFF-WBT, Version: 1)". Create an incident if you're unable to find it -- Joel Fischer

Agree I see in the screen share of the Shirty showing but I am asking does it shows up for current SN with London or something more needs to be added in the application? I don't see it in London instance, thanks. -- Sankalp Joshi
Parameterized tests and custom UIs came out in Madrid (Q1 2019) -- Chuck Tomasi

Can the shared parameters be shared/referenced by other test suites? -- Thach Nguyen
yes Shared Parameter Definitions are available to all parameterized tests -- Joel Fischer

How to move ATF scripts from one servicenow instance to another instance -- Nisha Auti
using update sets -- Shrity Verma

can you pull your choices from an existing table -- Raymond Andrews
try creating a parameter definition of type reference -- Joel Fischer

what is the capacity for interacting with UI pages and Client side alerts and/or confirm? -- Mark Johnson
You can use Custom UI steps to interact with UI pages. Browser-level alerts & confirms (ex. window.alert('hi'), window.confirm('hi')) are automatically captured and recorded in the Test Log table, which is a related list on a Test Result form -- Matt Coleman

will the ability to turn off rollback be available in future? -- David Pax
we currently do not recommend disabling rollback because it affects the ability to repeat execution of tests. A way to disable rollback for specific scenarios is being reviewed, but please log an enhancement request with your use case -- Joel Fischer

5-Planning -- David Pax
Thank you! -- Stacey Bailey

Is there a plan to expand the "template" concept so that multiple template tests could be used to create tests, like building blocks. This way standard repeatable steps could be created once and then combined as needed to create new tests along with new steps. -- Matt Spangler
We are looking into supporting this, but there is no specific release planned yet. -- Matt Coleman

Can we test UI actions under related list using custom UI? -- Kushal Dangare
related list support is on the roadmap but not available to date -- Joel Fischer

Will ATF have variables that can be used across Test Sets? for example, I create a variable called "my_color" and set it to "blue" then in the test set, I can set the Test Run data Parameter to the variable. Allows me to use the same value across multiple test set without having to enter it in every test set. -- James Kailukaitis
Parameter Definitions can be shared between tests, but Test Run Data Sets are only available to the test on which it's defined. -- Joel Fischer

If records are created from ATF, can we easily track what records are created so they can be removed post testing? Or removed automatically at end of testing...thinking about production upgrade testing... -- Ashley Myers
They should be removed automatically. However, there are some limitations regarding cleanup. See details here: https://docs.servicenow.com/search?q=Tables excluded from rollback after running an automated test &labels=2 -- Stacey Bailey

Very NICE feature !!! -- Jerry Groesser
Thank you! -- Chuck Tomasi

is there a way to save the records produced during ATF? -- Ikram Rehman
this is currently not available, please request an enhancement and provide your business use case -- Joel Fischer

Custom UI Testing - Will this work for the "Risk Assessment" on a change request? -- Timothy Onyskin
Yes -- Shrity Verma

How come Shrity is so awesome? -- Jay Freise
She really is! I've been a fan-girl of hers for quite a while!! 😉 -- Stacey Bailey

Can ATF be used to test custom UI forms that have checkboxes? -- Kenya Greene
Yes you can set a checkbox using a 'Set Component Values (Custom UI)' step -- Matt Coleman

can we test custom service portal widgets using this fesrure ? -- Anand Lall
Yes you can use Custom UI steps to test your custom service portal widgets -- Matt Coleman

Will ATF include ability to test Mobile in future releases? -- James Kailukaitis
It's on the roadmap. No specific release planned yet. -- Chuck Tomasi

When personnel changes at our company I may need to change the Impersonate step and approvers with the new users names. This is in our service portal ATF test. I plan on making a table with catalog item, workflow catalog step, assignment group and assigned user. Then in the test get the assigned user from the table given the catalog item and catalog step. To help propagate the table what would the script look like where I can get the assignment group from a published workflow for a catalog item? -- Edward Parker
You may benefit from the parameterized tests that Shrity will show in this video. -- Chuck Tomasi

Can you use ATF Inspector to test Service Portal pages? -- Kourtney Kirshman
You can inspect a portal page using the ATF > Manual Page Inspector module -- Joel Fischer

Will they ever make ATF's usable with frame busters, when making an ATF for Major Incident workbench it doesn't work. -- Stan Haag
For now framebusting pages are not supported within the ATF test runner due to the number of ways a page can interact with parent frames. You can submit an incident to the Major Incident workbench team to request that their page does not framebust when rendered in the ATF test runner's test frame. -- Joel Fischer

Will ATF be able to test order guides that are not currently using service portal? -- Jerome Turner
If it's not currently supported, submit an incident to the Service Catalog team to request this enhancement -- Joel Fischer

Will ATF be able to handle UI pages? -- Johnnie Ortiz
Yes you can use these Custom UI steps to test UI pages -- Matt Coleman

will you cover ... one test script that uses multiple selections ... for example a script that might read its input from a file vs cloning a script to go -- Jerry Groesser
For Parameterized testing, you can "Import" a formatted excel spreadsheet to a test's Test Run Data Set tab. See the Import button there for instructions and an interface to generate the spreadsheet template -- Joel Fischer

thank you for the response. -- Kenya Greene
of course! -- Stacey Bailey

I saw "parallel testing" in one of the slides...could you talk about that? -- Neil Moore
The parallel testing feature is available in New York. Shrity covered this about 55 minutes in. -- Chuck Tomasi

or can they be saved off, so cloning over an instance will not erace them -- Howard Richter
Howard, I would encourage you to deploy these through your instance stack so they exist on your production environment, even though you would not run them on prod. That way, they will be included on any clones back down to non-prods. -- Stacey Bailey

Any tips/suggestions on how to pick the correct object. Ex. On incident when looking to hit save button, it shows 6 save buttons. Not sure on what to pick. -- Bhaskar TELKALA
The best way to find the right button to click is to search by sys_id. ex. If you right click the Save button on an incident form you can hit 'Edit UI Action' and then right click the header to copy the sys_id of that UI action. Then when you are building your Click a UI step, you can filter down to that sys_id -- Matt Coleman

For parameterized testing, would a single parameter with a list of fields that need to be mandatory work on a form ? -- Edward Parker
you'll need to populate each field with a value out of the test run data set -- Joel Fischer

It was a great session, but would need to revisit, since it went by pretty fast. So a link to the video recording will help. Thanks -- Deepti Gupta
Definitely. We will post that. Also check out these posts: * ATF course in developer portal (for new york instances): * https://developer.servicenow.com/app.do#!/trainlist/app_store_learnv2_atf_newyork_automated_test_framework?v=newyork * NY - parameterized tests in docs * https://docs.servicenow.com/bundle/newyork-application-development/page/administer/auto-test-framework/concept/parameterized-tests.html -- Stacey Bailey

Can tests be moved between instances (like from asandbox to dev) -- Howard Richter
Absolutely! They go nicely into scoped applications as well as update sets. -- Stacey Bailey

What would be the best way to test domain configurations? -- Kelly Campbell
The ATF can run and validate scenarios in a domain separated configuration, but it does not support altering the configuration -- Joel Fischer

is what we are seeing Madrid or New York? -- James Kailukaitis
Probably Madrid, but this tip is for any version. It has been around for a while. -- Kreg Steppe

More detail to my previous question: A UI page in a pop out window, such as a Risk assessment on Change Request -- Johnnie Ortiz
Yup.. you can test that -- Shrity Verma

How do you navigate to one of the tabs on the form? -- Wade Mitchell
Interacting with related lists is not yet supported in ATF, but it's on the roadmap. -- Matt Coleman

Do you create an update set to capture these to move to Production from test -- Steven Bogart
Yes, capture tests just as you would with other development work and promote through your instance stack. Even though you wouldn't run your ATF tests in PROD, it's handy to have them there for when you clone back to refresh your stack. -- Stacey Bailey

Is parameters the way that ATF will now be able to test/verify dropdown lists? -- Matt Spangler
Custom UI steps can identify the contents of a choice list at runtime for testing on custom ui pages (ui pages, portal pages, widgets, GlideModals), but scripted choice lists on forms and service catalog item variables are not supported in any step category. If you use scripted choice lists, we recommend changing to a dictionary/variable with defined choices for maintainability, testability, and access to the choice list security model -- Joel Fischer

How would this work when testing domain separated applications? -- Kelly Campbell
If you start your test with an Impersonation step for a user in a specific domain, the UI will render as the person in that domain and the test should execute as normal allowing you to test the custom UI as rendered for that domain / user. -- Stacey Bailey

Is there a way to copy the existing choice list from the table/field that you are testing? -- Eric Riemer
Copying the contents of a choice list for reuse is not currently supported -- Joel Fischer

Can we pass values from one test case to another using ATF? -- Meghna Shaw
You can only pass values from steps within a test. Data is rolled back after each test execution, so it cannot be passed between tests -- Matt Coleman

That screenshot is really bad resolution -- Karen Guerra
We appreciate the feedback and will focus on good quality screenshots in future presentations. Thank you! -- Chuck Tomasi

and what if the user is in multiple domains? -- Kelly Campbell
A user can only exist in one domain. If you're asking about users who have visibility to see multiple domains, I don't think there is support yet to select a domain as a test step. However, if you add some more details, we can look into what you're trying to do a bit more. -- Stacey Bailey

So you mean ATS designer role can see page inspector? -- Karen Guerra
yes with the atf_test_designer or atf_test_admin roles -- Joel Fischer

Can we perform parallel test execution through ATF in Madrid? -- Anand Lall
In Madrid only 1 test at a time can be running on the instance -- Matt Coleman

Is ServiceNow Mobile App testing also available with ATF? -- Meghna Shaw
It's on the roadmap. no specific release target yet. -- Chuck Tomasi

for the parameter test, can we test choice field only? Is it possible to test against reference fields? -- Jian Zhong
Yes it is possible to create parameters that are reference values -- Matt Coleman

Our Department has not yet released servicenow to all users but, it will within the year. Where can I get a copy of this for reference when we are at the point to use servicenow for automation? -- Christina Chaw
You can find this information on our community. This is episode 65. You can also find a complete index at http://bit.ly/servicenow-technow --

can ATF test on form with complex workflow? -- Chinh Vo
Yes. You are doing functional testing. The workflow will do what the workflow is going to do at the specific times when you trigger it. -- Chuck Tomasi

Is there a way to test for an expected failure (so if the test fails it is actually a success)? For example, a catalog client script used for form field validation may restrict what can be entered into a text field and you might want to verify that it is working by intentionally entering data that should fail. -- Ed Davis
Many steps have an 'Assert type' field that allow you to assert that a user is not allowed to do something -- Matt Coleman

Can the parameterized testing be used for UI policy verifications? -- Balaji Jaganathan
Custom UI step category has a component state validation step that can check if the ui component is enabled/disabled or editable/read-only depending on the type -- Joel Fischer

copied test feature is good for OOB tests, but seems to be annoying for new tests that are copied -- Vyjayanth Uttanoor
please create an incident so we can look into the issue -- Joel Fischer

Can we write manually custom UI field descriptions instead of learning each time? -- satya Tota
You can change how a field appears in a Custom UI step by adding detail to the actual page's elements - this requires the page developer to alter the page. For example specify or change that element's attributes: aria-label, title, alt, name, id, and the text between element tags. The Custom UI step Retrieve Component feature identifies the structure of a page and its testable UI components in real time during test execution. It's able to record data that may only appear to specific users and configurations for the defined test scenario, and enables the Add Test Step interface to adjust to changing scenarios without requiring the user to manually define HTML that would otherwise be necessary to represent interacting with those fields. -- Joel Fischer

yes my user has visibility into multiple domains when im testing im trying to set the domain first to ensure that im testing the correct form/ visibility -- Kelly Campbell
ATF is able to test a domain separated configuration, but it's not recommended to alter domain configuration during a test -- Joel Fischer

How to test Integrations using ATF? -- Meghna Shaw
There are test steps to test outbound REST integrations. To test inbound integrations, though, typically, you would set your test to trigger the integration condition and then test the results. When testing the results, on steps like Record Query, there is a timeout setting that can give the integration time to run. -- Stacey Bailey

Not sure if mentioned yet but what are the new ATF abilities in New York? -- David Lindberg
parallel testing, steps: upload attachment to form, upload attachment to record, Record Query supports Enforce Security, and Create and Impersonate a user with roles/groups -- Joel Fischer

hope there is a way to control when the warning message is displayed as in new test created vs OOB test copied -- Vyjayanth Uttanoor
Copy Test only shows a modal warning about data requirements for base system Quick Start Tests, not tests created by users -- Joel Fischer

Is there a plan to have the parameterized value inherit the choice list values when they exist? -- Johnnie Ortiz
try creating a parameter definition of type reference -- Joel Fischer

can we disable auto rollback from an atf test case ? -- Anand Lall
No you cannot disable rollback -- Matt Coleman

Prioty has 5 levels of value, I think maybe it should be 4, not 3? -- Johnnie Ortiz
Thank you!! -- Chuck Tomasi

sorry I joined late what version of ServiceNow we are using for the demo? -- Gino Sferra
Madrid is being used for this demo. -- Chuck Tomasi

blacklisted page can be tested using ATF? -- Reshma shirur
pages that are blacklisted from Custom UI step category are currently not supported, but there are some scenarios under review for future releases. Please submit an enhancement request with your business use case to make sure we've looked at your scenario! -- Joel Fischer

Will Email in be added to the testability? -- Todd Volpe
support for simplified email testing is under consideration, but please request an enhancement with your use case so that we can review it -- Joel Fischer