TechNow Ep 82 | Instance Data Replication

[Chuck Tomasi]

Join us for this webinar to find out how fast and easy it is to configure a ServiceNow-to-ServiceNow® data integration with Instance Data Replication (IDR). Every day more customers like you are increasing the number of instances they use. Additionally, they are exchanging data with other ServiceNow customers. Why not supercharge your efficiency while making things easier, too?

Originally aired: December 15, 2020 8:00 AM PT

Featured Experts

Kreg Steppe is a Staff Programmer Lead within ServiceNow Training and Certification developing and supporting cloud training infrastructure. He specializes in developing integration solutions, automating repeatable processes and Cloud Management in ITOM. Kreg's prior experience includes operating his own ISP, developing web applications in PHP, network integration, managing network support, Application Development on cloud based networks, DNS and email server maintenance. He is a Linux enthusiast and enjoys Photography.

Chuck Tomasi is a Sr. Developer Advocate for ServiceNow. He is a computer science major with over 35 years of IT experience. As a former ServiceNow customer, Chuck won the first Innovation of the Year Award at Knowledge 10. Since joining ServiceNow in 2010 as a Technical Consultant, he has done many large scale ITSM implementations and custom applications, acted as an adjunct instructor for Education Services, created and lead the Technical Best Practices program, makes appearances on Live Coding Happy Hour, created dozens of fit for purpose custom applications, and co-hosts the ServiceNow series "TechNow".

Jeremy Duncan is a Platform Architect on the Workflow Design Studio team at ServiceNow. Jeremy is an Enterprise Architect with over 16 years of IT and Service Management experience. His experience and certifications span across retail and manufacturing with focuses on hardware, software, and information security specialties. He has done many implementations in both the Enterprise Federal and Commercial spaces since 2011 and brings a breadth of platform knowledge that aids him in sound architectural decisions. He achieved his B.B.A. at MTSU and majored in Information Systems. He spends his free time fighting crime as a volunteer certified police officer in North Nashville since 2012 and enjoys family time, camping, with his wife and kids!

---

LIST ALL EPISODES

Questions and Answers

can an MSP use it for its customers?

-- Chris Bui

Yes, but I'd be curious to know the use case. Normally the MSP has the customer data already.

-- Chuck Tomasi

Is this IDR not for an on-premise customer? If so, why?

-- Akira Muranaka

It's not available for on-premise. I don't have the details, but it has something to do with the Kafka server and key management.

-- Chuck Tomasi

What happens to two task records (for example, incident) with same number in bidirectional sync?

-- Akshaya Shetty

It uses the sys_id as the unique identifier.

-- Chuck Tomasi

Thank you, I can see that, however "number" INC123435 could exist in both instances with different SYS_IDS but also have unique actual incident needs.

-- Art Anderson

That's true. Like any other integration, IDR doesn't free you from the philosophical things you need to consider. :)

-- Chuck Tomasi

What roles do we require for approval and can we gave this authority to anyone

-- Imran Rasheed

You need either idr_admin or admin roles to administer IDR. Technically, yes, you can assign these roles to anyone. Given that this involves two systems, and potentially massive amounts of data, consider this role (like many others) very carefully.

-- Chuck Tomasi

Can IDR be used to replicate Group, Roles, Users records?

-- Avelino Pinto

Yes.

-- Chuck Tomasi

What about PA Dashboards???

-- Imran Rasheed

I wouldn't as dashboards are part of the sys_metadata construct (captured by update sets, app repo, etc.)

-- Chuck Tomasi

Are you going to show how to make a catalog item?

-- Agnès de Briey

Sorry, that's beyond the scope of this webinar topic on IDR.

-- Chuck Tomasi

Sorry if I missed this. Can you use IDR between 2 Production instances (separate companies) but own by one.

-- Ken Rempe

Yes you can. Any two ServiceNow instances can use IDR.

-- Chuck Tomasi

how does it keep track of order? sys_updated_on?

-- Mason Wong

It does it in the background. It sends the latest modifications last. I don't know the specific details.

-- Chuck Tomasi

Use case for domainsepraion:I have a Legal MSP (domain separated) who uses their central instance to manage incidents for multiple clients in a first line capacity, but these are sync'd to multiple other SN instances for 2nd and 3rd line support within the clients instances.

-- Karl Hardiman-Douglas

Yes, it could potentially be used for that Karl.

-- Chuck Tomasi

Is this enabled by a plug-in? The only "replication" I see in Filter Nav is "Replication Properties" and it talks about master/slave for data replication. Same ? Different?

-- Jon Epstein

You need to reach out to customer support on this one. Even as 'maint', I couldn't activate it as it requires some set up with keys and the replication server (kind of like a store.)

-- Chuck Tomasi

Are there any limits for the amount of exchanging records as they are in IntegrationHub? Or is this 'inclusive' because it's a paid plugin?

-- Felix Großer

There's a practical limit of 3 million records. Yes, it uses your IH transactions to the best of our knowledge. Please check with your account team with any pricing questions.

-- Chuck Tomasi

Will we be able to know how much time will it take to sent the data. As in prod there we will be dealing with bulk data.

-- Imran Rasheed

It's 'near real-time', so if you import a lot of data to prod, it will take some time to replicate to the consumer(s). If it's just a record here or there changing from time to time, you won't notice it.

-- Chuck Tomasi

what happens with the associated tasks

-- Latha Suvarna

They're not brought over unless you add that table to your replication set.

-- Chuck Tomasi

If I'm the producer (Company A) and my customer(Company B) is the consumer. Do both companies need to purchase this product?

-- Brenda list

I believe so, but please check with your account team.

-- Chuck Tomasi

are there any limitations with the attachments or attachment size?

-- Manjeet Singh

I haven't heard, but that doesn't mean much. 🙂 Please reach out to customer support.

-- Chuck Tomasi

I am struggling to understand the use case for IDR if it isn't meant to replace cloning (especially to get configuration/code changes to consumer) and if a more formal e-bonding solution is coming in a future release. Can you expand a bit on the slide #1 speaking to specific use cases IDR was meant to handle?

-- Julie Leung

The primary use case is to share data between ServiceNow instances quickly and easily. eBonding traditionally involves non-SN systems and potentially other data transormations.

-- Chuck Tomasi

Are you going to show how to work on the change management application?

-- Agnès de Briey

Sorry, that's beyond the scope of this webinar topic on IDR.

-- Chuck Tomasi

Does IDR replace a back-up? What are ultimate benefits of IDR?

-- Akira Muranaka

No, it does not replace backups. Benefits are listed on that Top Takeaways slide.

-- Chuck Tomasi

I am sorry, can you repeat about on-premise customer for IDR? I missed it.

-- Akira Muranaka

IDR is currently not supported.

-- Chuck Tomasi

What is Discrete for?

-- Akira Muranaka

It allows you to create a main producer set with a "sub filter" of sorts. E.g. send incidents, but only where company=Amazon.

-- Chuck Tomasi

Are the incident records on the Consumer view-only, or they can be modified/acted upon from the Consumer instance?

-- Cris Diaconu

No, they are live on both instances (ACLs dependent of course.) This is not a read-only replica feature.

-- Chuck Tomasi

What if the incident table are configured differently on both instances? Like Producer has 4 incident states and Consumer has 5 incident states?

-- Anish Reghu

It will send over the value, but obviously, if the sys_choice entry is not available, it won't be able to display (and show up in the list blue). In the case of a reference field, you'll just get a blank entry (because we don't like to display sys_ids)

-- Chuck Tomasi

can you synchronise fields with different names on producer and the consumer?

-- Mohammed Talha

Yes. It's in the transform section.

-- Chuck Tomasi

What will happen if the consumer already has existing incident or problem numbers that the producer is sending? (But different sys_ids)

-- Julie Leung

You'll end up with duplicates

-- Chuck Tomasi

Am I right that the vendor is paying for IDR Plugin in a Customer/Vendor model? Like they bought support from the vendor and both companies are using ServiceNow.

-- Felix Großer

I don't recall. Please check any licensing questions with your account team.

-- Chuck Tomasi

What happens to fields like - Opened, Resolved values?. They are timestamps. Sure that they are not getting imported? Or do they?

-- Anish Reghu

The values (not display values) of the date and date/time fields comes over and is stored on the consumer. If your instances are in two different time zones, it will be addressed/displayed as expected.

-- Chuck Tomasi

Can you use scripts to transform data?

-- Jeffrey Kucaj

Yes

-- Chuck Tomasi

Is SN considering ever allowing data replication to other systems, say a SQL database?

-- Dave Pacheco

I haven't heard of such plans. You may want to suggest it to the Idea Portal.

-- Chuck Tomasi

Are there any tables that are restricted from use in the producer?

-- Joe Leser

There are. I don't have that at this time. It's documented in the IDR docs page.

-- Chuck Tomasi

I'd think one would need to setup any related/referenced tables as well, e.g. sys_user table for caller etc

-- teunis schouten

Yes. Related records also need to be replcated.

-- Chuck Tomasi

Does IDR work with replicating to Developer instances?

-- Jeffrey Kucaj

Yes. I assume you mean sub-prod, and not PDI.

-- Chuck Tomasi

what about business rules and client scripts that may be associated with incidents? We have to replicate those separately?

-- Cris Diaconu

Please do not put things like BRs in IDR. That would be moving data in the wrong direction. Hope I explained that fairly well about 40min in.

-- Chuck Tomasi

isn't export xml and import is faster for this?

-- Latha Suvarna

I haven't done a timed trial. Remember, import/export XML is once, this keeps data updated in near-real time.

-- Chuck Tomasi

IDR is not a back-up use. Is it correct? What are the benefits of IDR?

-- Akira Muranaka

No, it is not a replacement for backups. The benefits of IDR were mentioned several times in the webinar, notably at the beginning and end.

-- Chuck Tomasi

how do you manage duplicate numbers?

-- David Yano

You can do that with the adapter field on the numbers.

-- Chuck Tomasi

Is ebonding for TSM Telecommunication Service Management build on top on IDR.

-- Ganesh Punalkar

Not that I'm aware. We are planning on a configurable eBonding solution in Q. #safeharbor

-- Chuck Tomasi

Does the replication use standard table APIs or another method?

-- Dave Pacheco

It uses another method to keep it fast and secure.

-- Chuck Tomasi

Is there a cost associated with IDR?

-- Chris Bui

Yes. This is a paid for product with its own SKU

-- Chuck Tomasi

I would imagine the answer yes... If I have more than 3M rows, I can do this in stages to commit all 4M rows?

-- Art Anderson

If you have that much data, I'd reach out to your account team to talk to the product team if you have requirements that go beyond the limits we suggest.

-- Chuck Tomasi

Would/ could this be an eBonding replacement?

-- dan cavanaugh

Theoretically yes, but we have an optimal solution coming specifically for configurable eBonding.

-- Chuck Tomasi

It's pretty much like Sys ID is mandatory for import Data.

-- Anish Reghu

Correct.

-- Chuck Tomasi

Paid option or included with the monster subscription charge 🙂 ?

-- Jon Epstein

Separate product.

-- Chuck Tomasi

We saw the Producer request and Consumer approve. That's like a push. Does it work the other way round? for pull as well?

-- Anish Reghu

Each consumer needs to make a request to the producer. The push/pull concept doesn't really apply here (at least in my mind.)

-- Chuck Tomasi

is it easy to map fields (eg if systems have different categories)

-- David Yano

Yes, You can do that with the Adapter.

-- Chuck Tomasi

Is it possible to configure problem data in producer table to be replicated to problem_task data in consumer table using IDR?

-- Kshitij Bhatnagar

Yes

-- Chuck Tomasi

Why is the "Producer URL" not Mandatory ?

-- Christopher Crefcoeur

Good point. It should be.

-- Chuck Tomasi

How do you know if IDR is enabled in your instance?

-- Avelino Pinto

You will have an application menu called "Instance Data Replication"

-- Chuck Tomasi

back to business rules, do they get triggered on the consumer instance?

-- Mason Wong

The BRs on the consumer will run as they normally would. The incident, for example, doesn't change behavior when it goes from the producer to the consumer. It's just incident data.

-- Chuck Tomasi

Can we make each instance (producer and consumer) to use their own record numbering to keep the local numbering sequence?

-- Ahmed Hussein

By default, they use their own numbering sequence. Remember, numbering is assigned when the record is created. Since IDR is functionally the same as exporting data from one system to another, you'll get the numbers from the other system. You may want to look at using an adapter to change the value in the number field if you need, say, PRBs in one instance and DEFs in another.

-- Chuck Tomasi

When we enable Transform, does it provide options for Mapping Assist?

-- Anish Reghu

Not the same UI we use for import sets, but there is a field matching UI.

-- Chuck Tomasi

Does it work perfectly with On-premise implementations

-- Sameer Thakare

On prem is currently not supported.

-- Chuck Tomasi

How does this deal with collisions of ID's between the instances?

-- Art Anderson

Sys_ids are the unique identifier.

-- Chuck Tomasi

how the systems are talking without authorization?

-- Rajat Gupta

I hope we cleared that up in the discussion (around 28 min past)

-- Chuck Tomasi

How do you I enable attachments?

-- Rakesh Makam

Replicate data in the sys_attachment and sys_attachment_doc table.

-- Chuck Tomasi

In case of bi-directional IDR, does the consumer need to approve as well (as it is also a producer)?

-- Neha Natu

For bi-directional, the consumer is also a producer. Hopefully your question was answered in the demo.

-- Chuck Tomasi

In case of Requested Items if I replicate - "sc_req_item" table will it automatically send the variables as well which are in items ? Or i need to send all the related sc_item_mtom tables as well ?

-- Rohit Bhadoria

I suspect they will not come over. I haven't tested this case specifically.

-- Chuck Tomasi

If we just want to once in a while update sub-prod with some tables on a schedule, is this a solution for that?

-- Murali Kaza

That sounds like a scheduled export. Take a look at https://docs.servicenow.com/bundle/geneva-servicenow-platform/page/administer/export_sets/concept/c_ScheduledExportSets.html

-- Chuck Tomasi

does IDR manage encrypted fields ?

-- Simon Trudeau

No

-- Chuck Tomasi

just joined, does IDR also helps in real time syncing of data between instances like similar to ebonding?

-- Rajat Gupta

Essentially, yes. It's what we call "near real-time"

-- Chuck Tomasi

IDR is not supported for on-premise customers. Why?

-- Akira Muranaka

It has to do with the Kafka server, key management, and some other things that I'm not totally privvy to. If you like, we can reach out to a product manager and broker an introduction if you would like to understand further.

-- Chuck Tomasi

Is it OOTB that edge encrypted fields will not be allowed to be selected?

-- Anish Reghu

Yes

-- Chuck Tomasi

How is the referenced field handled? do we need set IDR for each table? is the IDR order matters?

-- Lon Hu

Yes, you need to send any related records that may be involved or they will display as blank entries. Order generally doesn't matter.

-- Chuck Tomasi

How does choices gets managed, if consumer does not has the choices which are send by Producer ?

-- Rohit Bhadoria

The values will be transferred in the record, similar to any other values (dates, integers, etc) but will show up as blue in a list to indicate there's no corresponding choice in sys_choice.

-- Chuck Tomasi

Why isn't IDR part of Integration HUB already?

-- Stefan Ojeblom

Someone above my pay grade makes decisions like that. :)

-- Chuck Tomasi

How to map choice lists items, such as categories and subcategories in case if they don't match between producer and consumer tables?

-- Ahmed Hussein

You would do that the same way you would without IDR. An update set, an app repo, etc. to move config changes like choice lists between instances.

-- Chuck Tomasi

Is IDR supported for different Release Versions like Customer is still on Orlando, Vendor is already on Paris?

-- Felix Großer

You can replicate between different versions, but there may be some limitations. Talk to customer support for more details.

-- Chuck Tomasi

Does IDR have back-out option?

-- Akshaya Shetty

No.

-- Chuck Tomasi

In UNI directional IDR, if you enable transform at the consumer, is there an option/transform script to add a prefix or suffix in the short description or any field so that we can differentiate the incidents

-- Neha Natu

Yes, you should be able to do this with an adapter

-- Chuck Tomasi

Are sys_ tables eligible with IDR, e.g. sys_auto_script?

-- Stefan Ojeblom

Some are, some not. Check the docs for full details.

-- Chuck Tomasi

That pretty much simplified the way SNOW instances will talk in the future.

-- Anish Reghu

Yes

-- Chuck Tomasi

What if the consumer number has a a record with the same incident number?

-- Pavan Kumar

You will end up with duplicate numbers.

-- Chuck Tomasi

Is this practical limit of 3mio records/transactions per year or one time? Is it only for 'real' records like Incidents or also for each update on the records? Also, for each connection or for all connections (e.g. the Instance is connected to multiple Instances)?

-- Felix Großer

Three million records is total (for all time.) For the second part, it would be for each connection. If you are considering IDR for millions of records, reach out to customer support.

-- Chuck Tomasi

Is the configuration of IDR being kept in update set? I want to config that on dev first and validate on test before I enable it on production.

-- Christopher Crefcoeur

It doesn't appear to be and I can understand why given the security considerations behind the scenes. Given that it only takes a few minutes to set up each consumer, I don't consider this a major liability.

-- Chuck Tomasi

can we update replication set to bi-directional ?

-- Nitesh Kumar

Yes

-- Chuck Tomasi

IDR implementation is first step for TSM?

-- Ganesh Punalkar

I don't understand the question, but since it's not the first step to anything, I'll say No.

-- Chuck Tomasi

Is it possible to test IDR using two temporary PDIs or are there any restrictions?

-- Felix Großer

IDR is not currently available on PDIs. Wish it was. If you feel strongly about this and have a use case, contact the developer program people and ask. Things can change. We may also have instances available for a lab/workshop at Knowledge 21. Stay tuned.

-- Chuck Tomasi