Azure Synapse Analytics March Update 2023

[Guest ryanmajidi]

Azure Synapse Analytics March Update 2023

 

Welcome to the Azure Synapse Analytics March 2023 update! This month, we have the General Availability of Multi-Column Distribution and Prive Endpoint support for Cosmos DB to Azure Data Explorer Synapse Link. We also have additional updates in SQL, Apache Spark for Synapse, and Synapse Data Explorer! Read about all of these new features below!

 

 

 

Don’t forget to check out our

on the Azure Synapse Analytics YouTube channel!

 

 

 

 

 

 

Table of contents

 

• SQL
  
  • Multi-Column Distribution (Generally Available)
    
  • Greatest and Least TSQL Functions
    

  [*] Apache Spark for Synapse

  • Library management new ability: in-line installation
    

  [*] Synapse Data Explorer

  • ADX Dashboards (Generally Available)
    
  • Private Endpoint support for Cosmos DB to Azure Data Explorer Synapse Link
    
  • Kusto Dashboards Plotly visuals support
    
  • Amazon S3 support in Kusto Web Explorer (KWE)
    
  • View cluster history in Kusto Data Explorer (KWE)
    

SQL

 

Multi-Column Distribution (Generally Available)

 

 

We are excited to announce that Multi-Column Distribution (MCD) for Azure Synapse Dedicated SQL pools is now Generally Available! MCD is highly desirable for easing migrations, promotes faster query performance, and reduces data skew.

 

 

 

 

 

 

You can choose to distribute data on multiple columns to balance the data distribution in your tables and reduce data movement during query execution. Multi-Column distribution will allow you to choose up to eight columns for distribution.

 

 

 

To enable MCD, change the database's compatibility level to 50 with this command:

 

 

 

 

 

ALTER DATABASE SCOPED CONFIGURATION SET DW_COMPATIBILITY_LEVEL = 50;

 

 

 

 

 

Multi-Column Distribution is supported by the following commands:

 

1. CREATE MATERIALIZED VIEW
   
2. CREATE TABLE
   
3. CREATE TABLE AS SELECT
   

 

 

 

To learn more about setting the database compatibility level, read Alter Database Scoped Configuration.

 

To learn more about MCD, read Multi-Column Distribution for Dedicated SQL pools is now GA!

 

 

 

Greatest and Least TSQL Functions

 

 

GREATEST and LEAST functions are now available in Azure Synapse Analytics Dedicated SQL pools! GREATEST and LEAST are scalar-valued functions and return the maximum and minimum value, respectively, of a list of one or more expressions.

 

 

 

These new T-SQL functions will increase your productivity and enhance your experience with Azure Synapse Analytics. Providing the GREATEST developer experience in Azure is the LEAST we can do.

 

 

 

To learn more about these functions, read GREATEST (Transact-SQL) and LEAST (Transact-SQL).

 

 

 

Apache Spark for Synapse

 

Library management new ability: in-line installation

 

 

%pip and %conda are now available in Apache Spark for Synapse! %pip and %conda are commands that can be used on Notebooks to install Python packages.

 

 

 

The availability of these commands will increase your productivity and ensure the agility of managing packages on Apache Spark for Synapse during the interactive run of Notebook.

 

 

 

'%pip install' is one of the powerful commands that enables you to install new libraries from a public repository like PyPI, install your custom libraries from storage, or install the full list of libraries from an environment specification file to your Notebook Spark session. You can use '%pip show' to investigate the installed library version when you want to retrieve this information during development.

 

 

 

To learn more about the full abilities of these magic commands, read Manage session-scoped Python packages through %pip and %conda commands.

 

 

 

Synapse Data Explorer

 

 

 

ADX Dashboards (Generally Available)

 

 

Azure Data Explorer Dashboards are now Generally Available!

 

 

 

Azure Data Explorer is a fast and highly scalable data exploration service for log and telemetry data. Using Azure Data Explorer web UI, you can explore your data from end-to-end, starting with data ingestion, running queries, and ultimately building dashboards.

 

 

 

Each ADX dashboard is a collection of tiles, optionally organized in pages, where each tile has an underlying query and a visual representation. Using the web UI, you can natively export Kusto Query Language (KQL) queries to a dashboard as visuals and later modify their underlying queries and visual formatting as needed. In addition, to ease data exploration, this fully integrated Azure Data Explorer dashboard experience provides improved query and visualization performance.

 

 

 

ADX Dashboards, which are part of the Azure Data Explorer web UI, have a user-friendly interface, allowing you to quickly explore and analyze data without the need for extensive technical knowledge. They offer a range of customization options and are designed to handle big data.

 

 

 

 

 

 

To learn more about ADX Dashboards, read General availability: ADX Dashboards

 

 

 

Private Endpoint support for Cosmos DB to Azure Data Explorer Synapse Link

 

 

2 months ago, we announced the Public Preview of Cosmos DB to Azure Data Explore Synapse Link. This new data connection enables you to ingest a Cosmos DB container in real time.

 

 

 

We now support Cosmos DB accounts behind a Private Endpoint or Service Endpoint.  To connect to such an account, you simply add a  Managed Private Endpoint of type DocumentDB (the original name of Cosmos DB) from the Networking pane:

 

 

 

 

 

 

And fill out the resource details:

 

 

 

 

 

 

To learn more about Private Endpoint support for Cosmos DB, watch

and read Create a managed private endpoint for Azure Data Explorer.

 

 

 

Kusto Dashboards Plotly visuals support

 

 

We recently added support for Plotly graphing library. You can render your data as a Plotly visual and use its power, diversity, and advanced properties.

 

 

 

 

 

 

For more details on how to leverage Plotly visuals with or without Python, read Plotly visualizations in Azure Data Explorer.

 

 

 

Amazon S3 support in Kusto Web Explorer (KWE)

 

 

You can now ingest data from Amazon S3 seamlessly via the Ingestion Hub in Kusto Web Explorer (KWE).

 

 

 

The Ingestion flow can be completed with following selections:

 

1. Source type: Amazon S3
   
2. Link to Source: Enter the connection string of a bucket / object. 
   

 

 

 

To learn more about Amazon S3 support, read Storage connection strings.

 

 

 

View cluster history in Kusto Data Explorer (KWE)

 

 

It is now easier to track the history of queries and commands run on a Kusto cluster using .show queries and .show commands-and-queries. These commands allow you to view admin commands and queries that have reached a final state.

 

 

 

To access and run the set of KQL statements that will let you view the cluster history, simply right click the cluster name in the Connection pane, and select ‘View history’. A new query tab will be opened with the relevant statements.

 

 

Continue reading...