To further strengthen our commitment to providing industry-leading data technology coverage, VentureBeat is pleased to welcome Andrew Brust and Tony Baer as regular contributors. Check out their articles in the Data Pipeline.
Data quality, part of data intelligence, is a topic that worries many business leaders – 82% cite data quality as a barrier to their business. With many data quality solutions with different approaches available in the market, how do you choose?
Alation’s CEO and co-founder Satyen Sangani said today’s announcement of his Alation Open Data Quality Initiative (ODQI) for the modern data stack is designed to provide customers with freedom of choice and flexibility in selecting the best data quality and data observation providers to meet the needs of their modern, data-driven organizations.
Alation’s Open Data Quality Framework (ODQF) opens Alation Data Catalog to any data quality vendor in the data management ecosystem and modern data stack. Initially, data quality and data observation vendors have joined such as Acceldata, Anomalo, Bigeye, Experian, FirstEigen, Lightup, and Soda, as well as industry partners such as Capgemini and Fivetran.
Some of them were already partners with Alation, while others are new and drawn to the idea of having a standard to merge around. The company hopes that ODQF will become the de facto standard.
From data catalogs to data intelligence
Sangani, with a background in economics and experience in financial analysis and product management at Oracle, co-founded Alation in 2012. However, the company remained in stealth until 2015, working with a handful of clients to define what the product and what the company was. . really reachable and for whom.
Sangani’s experience has also informed Alation’s approach. He said that selling large-scale packages to big companies to help them analyze their data meant that the companies didn’t really understand the data themselves:
“Two years, hundreds of millions of dollars would be spent … and often a lot of that time was spent locating which systems have the correct data, how the data was used, what the data meant,” Sangani said. “Often there were multiple copies of the data and conflicting records. And the people who understand the systems and the data models were often outside the company.”
The realization was that data modelling, schemas and the like were more of a knowledge management problem than a technical problem. Sangani says he believes it encompasses aspects of human psychology as well as a didactic aspect, in terms of empowering and teaching people how to use quantitative reasoning and thinking.
Over time, Alation’s trajectory has been associated with a number of terms and categories. The most prominent of these were metadata management, data governance, and data catalog. Today, however, Sangani says these three are all coming together in a broader market space: what was originally IDC identified as data intelligence†
A few years after Alation launched in 2015, the company attempted to create the data catalog category, which Sangani says was new to many. After that, other players from metadata management and data governance also started working together on building a data catalog.
At the same time, the timeline from 2012 to today also includes developments on the technology side, such as the democratization of big data through the Hadoop ecosystem, as well as the adoption of regulations such as HIPAA and GDPR. All of these responded to the need to create inventories aimed at facilitating people’s use of data, which Alation sees as a competitive differentiator.
Alation as a platform for data quality
For Alation, the data catalog is the platform for the broader category of data intelligence. Sangani says that data intelligence has many components: master data management, privacy data management, reference data management, data transformation, data quality, data observation, and more. Alation’s strategy is not to “own one box of each of these things,” as Sangani put it.
“The real problem in this space isn’t whether you have the ability to tag data. The biggest problem is engagement and adoption. Most people don’t use data properly. Most people have no idea what data is out there. Most people don’t care about the data. Most of the data is insufficiently documented,” Sangani said.
“The idea of the data catalog is really about involving people in the datasets. But if that’s our strategy, to focus on engagement and adoption, that means there are some things we’re not doing strategically,” he said. “What we’re not doing is building a data quality solution. What we are not doing is building a data observation solution or a master data management solution.”
Alation considered expanding its offerings in the data quality market, but declined. It is a rapidly changing, densely populated market and solutions approaches can vary widely. Sangani said Alation doesn’t have huge competitive differentiation beyond the information in its data catalog. Sangani added that parts of Alation can make a data quality platform and that is what the Open Data Quality Initiative aims to achieve.
Whether standards live or die, however, is really determined by customer acceptance, Sangani said. This initiative is a follow-up to Alation’s Open Connector framework, which allows third parties to build metadata connectors for any data system.
Sanitary as a basis for value-added applications
Sangani said Alation will continue to build open integrations and frameworks over time, because in the world of data management there needs to be a consistent way to share metadata. In a sense, Sangani added, what Alation has built is now plumbing, and the ODQF is an example of more plumbing.
While plumbing is essential, the company has already started moving the pile to provide value-added features. For example, by using natural language processing (NLP) to perform name entity recognition for recommendations or allowing people to write English sentences and convert them into SQL to perform interactive interrogation of searchable datasets.
Sangani cited technologies such as knowledge graphing, AI, and machine learning as ingredients to build a more intelligent data intelligence layer.
“I’m probably more excited about what we can do in the next five years than what we’ve been doing in the last five years, because this all lays the groundwork for some really cool applications that we’re going to see in the near term,” he said. .
