Data Mesh Architecture Using HDInsight Trino

[Guest nijelsf]

What is Data Mesh?

 

A data mesh architecture is a decentralized data storing approach that enables domain teams to use the storage technologies of their own choice. The primary component of a Data mesh architecture is a processing engine which is capable of performing cross-domain data analysis.

 

 

 

Typical data mech processing engine should be capable of processing the data resides in various sources. Few example data storage options can be:

 

• Relational data in SQL Server.
  
• Delta files in Azure Data Lake Storage Gen2
  
• Images and Videos in ADLS Gen2
  
• PDF reports/logs/Text stored in Blob.
  
• Streaming data in Kafka/Event-hub
  

 

 

 

Data Mesh Using HDInsight Trino

 

In recent years, many organizations successfully use modern architectural and analytical patterns that combine data warehousing technologies and more recent big data technologies.

 

Microsoft is committed to provide best possible options to customers and chosen Trino for one of the options for Data Mesh architecture.

 

Trino is a highly parallel and scalable query engine. It supports federated Query over wide range of data sources.

 

Azure HDInsight provides AKS based Trino managed service which can be used for developing the Data mesh architecture. Its PaaS form of service supporting key features like AutoScale, Security management, Library Management and In-Place Upgrade. HDInsight Trino supports various storage formats like ORC, Delta and etc.

 

 

 

Typical Data mesh architecture using HDInsight Trino looks as below.

 

 

 

 

 

 

 

 

 

 

 

 

This is reference architecture. Trino can support much larger variety of sources and visualization tools.

 

Feel free to try out the boundless capabilities with Trino.

 

Continue reading...