Running your first Apache Flink job with Azure HDInsight on AKS

[sairamyeturi]

Could you already complete creating your first Apache Flink® cluster and submit your streaming job on it with HDInsight on AKS?



Well, if you are yet to do that - Let me help you get started.

• Make sure you’ve got your subscription and tenant ready for the lift off!

• Tenant registration (For HDInsight on AKS app) steps & Feature registration (for first time AKS users) - read more in detail here (for the initial steps)
• Complete your pre-requisites, using the ready to use ARM templates
  • We will need MSI and ADLSGen2 for your first cluster creation, we will not use the Hive Metastore for this exercise.

• Create your cluster pool

• Let's get started to setup your Flink Cluster

• You can opt-in to use hive catalog during creation, and provide the database details.
• You can configure your TMs based on the SKUs you select on the configuration page during creation. Add SSH nodes to use the Web based secure shell and Flink CLI access.

• Opt in for Autoscale based on any predictive schedules that suits your event stream requirements.
• Integrate with Prometheus, Log analytics (provided you have enabled this on the cluster pool creation step, as shown above on cluster pool creation phase).
• Tag your resources.
• Review & Validate your creation, and kickoff creation!

Voila! The cluster is up and running in next few minutes.

• Next you can go to Flink jobs page to run your first Flink job.
  
  
  
• Download the sample Flink sleep job (sample job jar) and upload it in your primary storage account which is used during Flink cluster creation.
• Create a new job from the Flink jobs section by giving the details of the jar location.

• Hurrah!
  • Congratulations, on your first Flink job
  • You can click the job name to view the JSON view of the job submitted anytime.
  • Alternatively, you can always use Flink dashboard in the Flink web UI and check the job statistics

We are super excited to get you started, lets get to how?

• Get started today - Microsoft Azure
• Read our documentation - Azure HDInsight on AKS (Preview) - Azure HDInsight on AKS
• Join our HDInsight community page, share an idea on what you would like to see next? - Sign Up | LinkedIn
• Have a question on how to migrate or want to discuss a usecase - Microsoft Forms

Apache, Apache Flink, Flink and the Flink logo are trademarks of the Apache Software Foundation.

Apache® and Apache Flink® are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

No endorsement by The Apache Software Foundation is implied by the use of these marks.

Continue reading...