<img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=266211600481844&amp;ev=PageView&amp;noscript=1">

Business Intelligence

How Forward-Thinking Businesses Deal with Data Integration

by Dennis Junk
on October 30, 2015

Data Integration SystemOne of the big changes among cutting-edge businesses in recent years is that they’ve begun thinking of data integration as its own area of focus, separate from the application development projects or B2B pipelines that call for single connections. The old default approach to integration has been to address it at the end of projects, treating it as a one-off task. You finish developing an app, for instance, and then you custom-code the connectors to any other apps it needs to share information with. Or you set up a channel for communication with a partner company, maybe using EDI documents formatted to the standards of your industry. If the standards change, or if you bring on a new partner, you simply build another connector. 

These types of ad hoc integrations usually work fine in the short-term, so tasking your IT team with coding custom integrations can seem like the most cost-effective strategy. The problem is that over time, as your business grows, your network evolves, your apps get upgraded, and the regulatory standards for forms undergo major changes, your collection of single integration fixes quickly becomes a prohibitively complicated tangle—or as Aptera’s Integration Specialist Richard Spice calls it, “a spaghetti mess.” You end up being terrified to update or change anything because you have no way of knowing what it’s going to break.

Integration System as a Central Hub

Rather than coding a specific connection between any two applications or setting up a dedicated pipeline for each EDI, many businesses are using technologies like SAP, Cloverleaf, or BizTalk to set up centralized integration platforms. With these systems in place, you no longer have to sift through endless streams of legacy code every time you want to bring a new app online. You basically plug the app into the integration system and it will connect to the necessary information. All the data that passes through this central hub essentially gets labeled and tracked, so any application that needs a particular piece of information can pull it out.

Spice explains, “It’s important to understand why there’s an advantage in these integration models—and most of it is the reusability of logic. If you’ve got the same type of data that needs to be shared with six or seven different systems, you don’t want to have write that extract six or seven different times. You want to be able to reuse it when you bring in new pieces of software. And you also want to make sure you can guarantee these messages are delivered. If you’ve got a system down, what does that mean? Does that mean you have to run your extract again? Does it mean when the system comes back on you can just pick right back up where you left off? It’s important to model based on that.”                                              

Appointing a Team and Developing a Plan

As part of treating data integration as its own area of concern, these businesses develop comprehensive strategies that include budgeting, long-term planning, ongoing maintenance, and dedicated teams. Aptera’s BI Practice Leader Aaron Crouch explains, “An integration project is bigger than the sources it’s integrating—if you’re thinking I just want to connect A to B, you’re really not thinking forward enough.” What this means is that instead of thinking about integration as a phase for a single project, these businesses make a plan for integrating systems across their organization. “The integration system itself doesn’t belong to the finance system,” Crouch points out. “It doesn’t belong to the CRM system or the department that oversees that system. It’s really its own system unto itself.” 

As its own system, integration needs its own team. And one of the most commonly overlooked roles on such a team is also one of the most important. For every integration project—and for integration as its own ongoing focus area—you need someone who has clout to help get buy-in form decision-makers. Whoever fills this role of flagbearer or champion will be the one who’s ultimately accountable, for both the success any given project and for the system as a whole.

The team will ideally also include a Business Analyst, someone who understands the context and purpose of each project. A representative of each department or vertical the project will impact should also be involved, along with a Project Manager and, of course, an integration architect. Each of these roles may not necessarily be filled by a different individual, but they should give you a sense of the various aspects involved in ensuring the projects are successful in the long run. 

Making an Upfront Commitment

The main reason that treating integration as individual add-ons to development projects is no longer viable for most businesses is the steady and rapid rate of change. Focusing exclusively on the present project, with little to no thought about what will happen six months or two years down the line, is perfectly natural. But it can also be costly. As Crouch points out, “You’ve got multiple source systems and each of them is changing on its own; if you don’t take that into account, things can get out of hand very quickly.” If you don’t plan ahead, in other words, you could end up with one of those spaghetti messes.

What does this sort of upfront commitment to an integration system entail? First, it’s making that comprehensive plan, and then it’s setting up the team. But beyond these two steps it’s also a mindset. The temptation is always to forget about your integrations until the day they stop working for some reason. But integration isn’t a set-it-and-forget-it type of project. You need a general plan, and you need someone focusing on the system’s day-to-day operations, things like server health and scalability, error handling, version upgrades, hot fixes, and so on.

A good example of the set-it-and-forget-it mindset is neglecting to do sufficient testing. For the businesses still doing integration as one of the final stages of a software project, it’s pretty common to rely on auto-generated tests. But, with systems as complex and interdependent as the ones modern companies tend to use, you really can’t get by without testing material for all the possible inputs and outputs.

This is where a larger upfront investment can really save you money in the long-term. Crouch explains, “The cost of spinning up a QA or test environment—a nonproduction environment to play with—is typically going to be a lot less than the amount of money you’re going to spend fixing and re-fixing issues that come up in production.” And this logic applies to all the other elements of your business’s integration strategy as well.

Popular posts like this:

How Businesses Are Using BizTalk and Other Integration Technologies

Exchanging Healthcare Claims Digitally with Data Integration Tools

When Should You Start Thinking about an Integration Strategy?

Why the Market for Data Integration Tools Is Exploding Right Now

Guide to New Power BI

Digital Marketing Assessment
Dennis Junk
I'm Aptera's Content Strategist. I've been writing about tech and marketing for 5 years and have certifications from HubSpot and The Content Marketing Institute. A big science and literature geek, I taught college rhetoric and composition while I was still busy going to school for way too long, earning bachelor's degrees in anthropology and psychology, along with a master's in British and American literature. Look me up on LinkedIn.
You may also like:
A Results-Driven Digital Agency www.apterainc.com