Announcing public preview of Apache Flink with Azure HDInsight on AKS

  • Thread starter Thread starter sairamyeturi
  • Start date Start date
S

sairamyeturi

We're thrilled to introduce the public preview of Apache Flink with HDInsight on AKS.



Streaming data is now pervasive in a business context and with the ability to process data streams on the fly, enterprises will be able to proactively respond to the timely insights and innovate at scale. Stream processing helps in computing on data directly as it is produced or received.



Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. It is one of the top projects of the Apache Software Foundation, it has emerged as the gold standard for stream processing. With high performance, rich feature set, and robust developer community; Flink makes it one of the most popular choices for low-latency, high throughput stateful stream processing.



Flink enables enterprises to harness the value of data immediately, making it a valuable tool for time-sensitive applications and scenarios requiring up-to-the-minute insights. Stream processing systems excel at handling high-velocity, unbounded data streams, such as click streams, log streams, live sensor data, social media feeds, event streams, transactional data, and IoT device data.



Learn more about some key usecases flink has to offer here



HDInsight on AKS offers Flink in a Platform as a service offering, some of the key benefits include:



  • Easy to use:
    • HDInsight on AKS Flink simplifies the complexities associated with stream processing.
    • With its user-friendly interface and intuitive design, enterprises can easily develop, deploy, and manage stream processing applications without the need for extensive expertise in distributed systems or complex programming.
    • The platform provides a visual interface for developing and monitoring streaming workflows, making it easy to use by all engineers, developers and architects.
  • Auto Scale:
    • Apache Flink with HDInsight on AKS platform allows enterprises to scale their processing clusters based on scheduled demands, ensuring high throughput and low latency even as raw data volumes grow significantly.
    • This scalability feature is particularly valuable in scenarios where real-time processing of massive data streams is required and is predictable.
  • Fault Tolerance and High Availability
    • HDInsight on AKS Flink incorporates robust fault tolerance mechanisms, which are crucial for maintaining data integrity and ensuring uninterrupted processing.
    • It leverages the fault tolerance features inherent in Apache Flink, including checkpointing and stateful recovery, to handle failures and system disruptions effectively.
    • By providing high availability and seamless fault recovery, the platform virtually eliminates data loss and guarantees reliable stream processing in the face of unexpected events.
  • Advanced Stream Processing Capabilities
    • HDInsight on AKS Flink offers advanced features and capabilities that enhance stream processing workflows.
    • It provides support for job management using native Azure interface, along with the rich APIs which help customers manage their jobs and stay focused on developing the business logic and less worry on the infrastructure.
    • Rich extensions to support Flink SQL capabilities, help customers perform SQL on Streams, with support for varied table formats on catalogs like Delta, Iceberg, Hive
    • Perform Change Data Capture on popular databases like PostgreSQL using Flink
  • Integration and Ecosystem Compatibility
    • HDInsight on AKS Flink seamlessly integrates with various data sources, messaging systems, and storage systems commonly used in the data ecosystem.
    • It provides connectors and integrations with popular technologies like Apache Kafka® (Azure Event Hubs and HDInsight Kafka) and many more, enabling Enterprises to leverage their existing data warehouse infrastructure.
    • With the ADLSGen2 having an ability to function as a file sink, streaming into ADLSGen2 is now more powerful and customers can leverage ADLS Gen2 for real time analytics needs.
    • With integration to Azure Data Explorer (ADX or Kusto), customers can always perform time series analysis on the processed stream events, enriching the real time event streams.
    • This compatibility allows for easy integration and interoperability, reducing the complexities of data ingestion and enabling smooth data flow into the stream processing pipelines.
  • Monitoring and Observability
    • HDInsight on AKS Flink emphasizes comprehensive monitoring and observability capabilities, critical for ensuring the health and performance of stream processing applications.
    • It offers real-time monitoring dashboards and metrics, enabling Enterprises to gain insights into the state and behavior of their streaming workflows.
    • With integrated Prometheus and Grafanna - logging, alerting, and visualization features. Enterprises can now have ability to proactively monitor, troubleshoot, and address issues immediately.
  • Enterprise-Grade Security and Compliance
    • HDInsight on AKS Platform prioritizes security and compliance, making it suitable for enterprise-level deployments.
    • The platform provides robust security measures, including authentication, authorization, and encryption, to protect sensitive data and ensure compliance with regulatory requirements.
    • Enterprises can confidently deploy stream processing applications on the platform, knowing that their data is secure and their operations align with industry standards.

Apache Flink natively supports Kubernetes, and HDInsight on AKS runs Flink using the Flink Kubernetes operator.



Some of the key use-cases of Apache Flink spoken about in last few Flink conferences are:


Flink is used for unbounded, real-time stream processing and SQL on Streams in HDInsight on AKS. Stay tuned to our roadmap and release updates.



We are super excited to get you started, lets get to how?




Continue reading...
 
Back
Top