'Digital transformation' is one of those Harvard Business Review terms so overplayed in the tech lexicon that its very definition is something of an enigma. Over the last 10 years, we’ve seen digital transformation hinge on everything from IT systems to new business models, from 'mobile first' customer experiences to virtual reality. Today, we are back to taking an IT-centric view of digital transformation as companies recognize that, like Dorothy in The Wizard of Oz, the keys to innovation lie in their metaphorical back yard: the vast stores of data already distributed across the enterprise.
Why manual labor just doesn’t cut it
Today, many businesses try to digitally transform by becoming data-driven. They are bringing in new cloud applications that both analyze and create enterprise data. Unfortunately, the way they’re connecting these cloud applications is almost like the pharaohs building the pyramids –– with lots and lots of manual labor.
It’s understandable; our industry has traditionally relied on manual labor to get things done. Whether it’s throwing armies of developers at a problem, or hiring floors of consultants to drive a project to fruition, IT initiatives have always depended on people as much as technology.
AI aids in digital transformation
But there is a better way. Artificial intelligence (AI) can automate repetitive development tasks, dramatically changing the economics (time and cost) of cloud, analytics, and digital transformation initiatives. Data lies at the heart of digital transformation, and AI-driven self-service will soon be able to trump code-centric data integration tools and APIs.
Just as AI can free humans from the burden of driving a car, it can also free up technology teams from the burden of manually managing data pipelines. With line-of-business users empowered to manage their own data pipelines, IT organizations will be able to focus on higher-value design and deployment needs.
'Self-driving' integration powered by AI
As an industry, we’re starting to see a real difference with new 'self-driving' technology that applies artificial intelligence to enterprise integration. In this application of AI, advanced algorithms learn from millions of metadata elements and billions of data flows that occur through an enterprise integration cloud. In this way, the AI models use that learning to improve the speed and quality of integrations across data, applications, and things.
An easy way to think of AI and data integration is as 'self-service to the nth degree,' the latest iteration of the ongoing self-service IT evolution. Walt Mossberg describes this in his final column, 'The Disappearing Computer.' AI will drive significant improvement in enterprise data integration over the next two to three years, and today is being used to power a recommendation engine that uses machine intelligence to give business users and analysts the optimal next steps in building data pipelines. And the truly great thing about the self-learning models at the heart of AI-driven data integration? Increased usage will only further improvements.
Innovation and architecture matter
Success with autonomous integration requires two ingredients: a born-in-the-cloud platform and a metadata architecture:
- Being cloud-native is an essential enabler of the data science necessary to find patterns and features that can be used to train machine learning models. You simply can’t do this kind of stuff with on-premises legacy technology rooted in the 1990s.
- A modern and robust metadata architecture is required to learn from data flows, integration paths, and patterns across the enterprise integration platform. The most robust enterprise integration platforms execute upwards of a million pipelines every hour, and the right AI analytics models can self-learn from each one.
For line-of-business leaders, AI-driven data integration shortens the learning curve for ad hoc and citizen integrators. For IT managers, it helps to automate highly repetitive development tasks.
Autonomous integration is the future
I’m not a fan of using outdated modes of transportation when a self-driving car can get you there instead. Similarly, I believe the future will be autonomous integration that blends the best of machine and human intelligence. The end game: fueled by AI-driven data integration, enterprises will be far better equipped to harness their immense data stores to achieve digital transformation.