G
gia_mondragon
We're excited to announce the general availability of integrated vectorization with Azure OpenAI embeddings in Azure AI Search. This marks an important milestone in our ongoing mission to streamline and expedite data preparation and index creation for Retrieval-Augmented Generation (RAG) and traditional applications.
Integrated vectorization simplifies RAG pipelines
Integrated vectorization simplifies RAG pipelinesWhy is vectorization important?
Vectorization is the process of transforming data into embeddings (vector representations) in order to perform vector search. Vector search aids in identifying similarities and differences in data, enabling businesses to deliver more accurate and relevant search results. Getting your data prepared for vectorization and indexed also involves various steps, including cracking, enrichment and chunking. The way you perform each of these steps offers opportunities to make your retrieval system more efficient and effective. Take a look at the blog post Outperforming vector search with hybrid retrieval and ranking capabilities that showcases the configurations that would work better depending on the scenario.
What is integrated vectorization?
Integrated vectorization, a feature of Azure AI Search, streamlines indexing pipelines and RAG workflows from source file to index query. It incorporates data chunking and text/image vector conversions into one flow, enabling vector search across your proprietary data with minimal friction.
Integration vectorization simplifies the steps required to prepare and process your data for vector retrieval. As part of the indexing pipeline, it handles the splitting of original documents into chunks, automatically creates embeddings with its Azure OpenAI integration, and maps the newly vectorized chunks to an Azure AI Search index. It also enables the automated vectorization of user queries sent to the AI Search index.
This index can be used as your retrieval system wherever you are building your RAG application, including Azure AI Studio and Azure OpenAI Studio.
Integration vectorization simplifies the steps required to prepare and process your data for vector retrieval. As part of the indexing pipeline, it handles the splitting of original documents into chunks, automatically creates embeddings with its Azure OpenAI integration, and maps the newly vectorized chunks to an Azure AI Search index. It also enables the automated vectorization of user queries sent to the AI Search index.
This index can be used as your retrieval system wherever you are building your RAG application, including Azure AI Studio and Azure OpenAI Studio.
What functionality is now generally available?
The following functionalities within integrated vectorization are generally available as part of REST API version 2024-07-01:
- Azure OpenAI embedding skill and vectorizer: These features allow for automatic vectorization of text data during data ingestion and query time.
- Index Projections: This feature enables mapping of one source document associated with multiple chunks, enhancing the relevance of search results.
- Split skill functionality for chunking with overlap: This functionality divides your data into smaller, manageable chunks for independent processing.
- Custom Vectorizer functionality: This allows for connection to other embedding endpoints apart from Azure OpenAI.
- Shared Private Link for Azure OpenAI accounts: This feature, which is part of the latest AI Search management API version 2023-11-01, provides secure and private connectivity from a virtual network to linked Azure services.
- Customer Managed Keys for indexes with vectorizers: This feature allows for additional security and control over your data through the use of your own keys. When you configure CMK in your AI Search index, your vectorizers operations at query time are also encrypted with your own keys.
How can you get started with integrated vectorization from the Azure portal?
The Import and vectorize data wizard in the Azure portal simplifies the creation of integrated vectorization components, including document chunking, automatic Azure OpenAI embedding creation, index definition and data mapping. This wizard now supports Azure Data Lake Storage Gen2, in addition to Azure Blob Storage and OneLake (in preview), facilitating data ingestion from diverse data sources. Coming soon, the wizard will also support source document additional metadata mapping to chunks and the Azure portal will provide debug sessions functionality for skillsets configured with index projections.
ADLS Gen2 Support in "Import and vectorize data" wizardAzure AI Search also allows you to personalize your indexing pipeline through code and take advantage of integrated vectorization using any of its directly supported data sources. For example, here’s a blog post of how to achieve this for Azure SQL Server data with integrated vectorization: Vector Search with Azure SQL Database.
What's still in public preview?
We also have support for image (multimodal) embeddings and Azure AI Studio model catalog embeddings which remain in public preview. For more information about this functionality visit Azure AI Search now supports AI Vision multimodal and AI Studio embedding models - Microsoft Community Hub.
Customers and benefits
Streamlined RAG pipelines allow your organization to scale and accelerate app development. Integrated vectorization’s managed embedding processing enables organizations to offer turnkey RAG systems for new projects, so teams can quickly build a GenAI application specific to their datasets and need, without having to build a custom deployment each time.
Customer: SGS & Co
For over 70 years, SGS & CO has been at the forefront of design, graphic services, and graphic manufacturing. Our specialized teams at Marks and SGS collaborate with clients worldwide to ensure a consistent and seamless brand experience.
“A key priority has been to equip our global teams with efficient tools that streamline their workflows, starting with our sourcing and research processes. We recognized the need for a system that allows for searchable assets without depending solely on order administration input, which can be inconsistent or deviate from actual data. This discrepancy posed a challenge for our AI modules.”
“SGS AI Visual Search is a GenAI application built on Azure for our global production teams to more effectively find sourcing and research information pertinent to their project. The most significant advantage offered by SGS AI Visual Search is utilizing RAG, with Azure AI Search as the retrieval system, to accurately locate and retrieve relevant assets for project planning and production.”
“Thanks to RAG's Azure AI Search's vector search capabilities, we can surpass the limitations of exact and fuzzy matching with contextual retrieval. This allows our employees to access information swiftly and effectively, enhancing service delivery to both our internal teams and global clients.”
“Additionally, the integrated vectorization feature in AI Search has greatly streamlined our data processing workflows. It automates batching and chunking, making it faster and easier to index data without requiring separate compute instances. Azure’s seamless handling of vectorization during live searches saves development time and reduces deployment costs. This capability enables us to efficiently create and manage indexes for multiple clients without extensive pipeline management. Moreover, integrating this feature with other RAG applications, such as chatbots and data retrieval systems, further enhances our ability to deliver comprehensive solutions across various platforms.”
Laura Portelli, Product Manager, SGS & Co
“A key priority has been to equip our global teams with efficient tools that streamline their workflows, starting with our sourcing and research processes. We recognized the need for a system that allows for searchable assets without depending solely on order administration input, which can be inconsistent or deviate from actual data. This discrepancy posed a challenge for our AI modules.”
“SGS AI Visual Search is a GenAI application built on Azure for our global production teams to more effectively find sourcing and research information pertinent to their project. The most significant advantage offered by SGS AI Visual Search is utilizing RAG, with Azure AI Search as the retrieval system, to accurately locate and retrieve relevant assets for project planning and production.”
“Thanks to RAG's Azure AI Search's vector search capabilities, we can surpass the limitations of exact and fuzzy matching with contextual retrieval. This allows our employees to access information swiftly and effectively, enhancing service delivery to both our internal teams and global clients.”
“Additionally, the integrated vectorization feature in AI Search has greatly streamlined our data processing workflows. It automates batching and chunking, making it faster and easier to index data without requiring separate compute instances. Azure’s seamless handling of vectorization during live searches saves development time and reduces deployment costs. This capability enables us to efficiently create and manage indexes for multiple clients without extensive pipeline management. Moreover, integrating this feature with other RAG applications, such as chatbots and data retrieval systems, further enhances our ability to deliver comprehensive solutions across various platforms.”
Laura Portelli, Product Manager, SGS & Co
Customer: Denzibank
Intertech is the software house of Denzibank, Turkey’s 5th largest private bank. They built one centralized RAG system using Azure AI Search and integrated vectorization, to support multiple GenAI applications and minimize data processing and management.
“At Intertech, we were in search of a solution to disseminate and more efficiently utilize information from our current documentation, solutions offered in our ticket system, and company procedures. This solution needed to act as a central vectorization and search solution for our various, different GenAI applications being built. Thanks to Azure AI Search’s integrated vectorization, we had access to the latest models offered by OpenAI, including embedding-3-large, and our job became much easier, allowing us to develop various applications very quickly and effortlessly.”
Salih Eligüzel, Head of DevOps and MLOps, Intertech
“At Intertech, we were in search of a solution to disseminate and more efficiently utilize information from our current documentation, solutions offered in our ticket system, and company procedures. This solution needed to act as a central vectorization and search solution for our various, different GenAI applications being built. Thanks to Azure AI Search’s integrated vectorization, we had access to the latest models offered by OpenAI, including embedding-3-large, and our job became much easier, allowing us to develop various applications very quickly and effortlessly.”
Salih Eligüzel, Head of DevOps and MLOps, Intertech
FAQ
What’s integrated vectorization pricing?
As part of your AI Search service pricing you have an allowed included limit of built-in indexers. Split skill (data chunking), native data parsing and index projections, which are necessary for integrated vectorization are offered at no extra cost. Azure OpenAI embedding calls are billed to your Azure OpenAI service according to its pricing model.
What customizations are available with integrated vectorization?
Azure portal supports the most common scenarios via the “Import and vectorize data” wizard. However, if your business needs extend beyond these common scenarios and require further customization, Azure AI Search you can customize your indexing pipeline through code and use the integrated vectorization functionality using any of its directly supported data sources.
Customization options include enabling features available through other skills in the AI Enrichment suite. For instance, you can make use of custom code through Custom WebApi skill to implement other chunking strategies, utilize AI Document Intelligence for chunking, parsing, and preserving table structure, and call upon any of the available built-in skills for data transformation, among others. Skillset configuration serves to enhance functionality to better suit your business needs.
For a more comprehensive understanding, we encourage you to explore our AI Search vector GitHub repository, which houses sample codes, and our Azure AI Search Power Skills repository, containing examples of custom skills. For example, this custom skill code is used to call an external embedding endpoint (aside from Azure OpenAI) and can be called the custom indexing pipeline and vectorizer at the query time.
Some scenarios that are a good fit for integrated vectorization
Integrated Vectorization is particularly beneficial when preparing data with AI enrichment before chunking and vectorizing it. Azure AI Search provides AI enrichment capabilities for OCR and other data transformation before placing it in the index for convenience.
Integrated vectorization is ideal for RAG solutions that require quick deployment without constant developer intervention. Once identified, needed patterns can be made available for teams to use for their convenient RAG and constant deployments. Examples of this would be projects, per-use-case scenarios with specific documents, etc.
In essence, if you aim to expedite your time to market for RAG scenarios with low/no-code for retriever creation, integrated vectorization offers a promising option.
More news
Azure AI Search is also launching binary quantization, along with other vector relevance features, to General Availability today! Dive into the details of these new additions in our Binary Quantization GA announcement blog post.
What’s next?
Stay tuned for more updates on the latest features of Azure AI Search and their role in simplifying integration for RAG applications!
Getting started with Azure AI Search
As part of your AI Search service pricing you have an allowed included limit of built-in indexers. Split skill (data chunking), native data parsing and index projections, which are necessary for integrated vectorization are offered at no extra cost. Azure OpenAI embedding calls are billed to your Azure OpenAI service according to its pricing model.
What customizations are available with integrated vectorization?
Azure portal supports the most common scenarios via the “Import and vectorize data” wizard. However, if your business needs extend beyond these common scenarios and require further customization, Azure AI Search you can customize your indexing pipeline through code and use the integrated vectorization functionality using any of its directly supported data sources.
Customization options include enabling features available through other skills in the AI Enrichment suite. For instance, you can make use of custom code through Custom WebApi skill to implement other chunking strategies, utilize AI Document Intelligence for chunking, parsing, and preserving table structure, and call upon any of the available built-in skills for data transformation, among others. Skillset configuration serves to enhance functionality to better suit your business needs.
For a more comprehensive understanding, we encourage you to explore our AI Search vector GitHub repository, which houses sample codes, and our Azure AI Search Power Skills repository, containing examples of custom skills. For example, this custom skill code is used to call an external embedding endpoint (aside from Azure OpenAI) and can be called the custom indexing pipeline and vectorizer at the query time.
Some scenarios that are a good fit for integrated vectorization
Integrated Vectorization is particularly beneficial when preparing data with AI enrichment before chunking and vectorizing it. Azure AI Search provides AI enrichment capabilities for OCR and other data transformation before placing it in the index for convenience.
Integrated vectorization is ideal for RAG solutions that require quick deployment without constant developer intervention. Once identified, needed patterns can be made available for teams to use for their convenient RAG and constant deployments. Examples of this would be projects, per-use-case scenarios with specific documents, etc.
In essence, if you aim to expedite your time to market for RAG scenarios with low/no-code for retriever creation, integrated vectorization offers a promising option.
More news
Azure AI Search is also launching binary quantization, along with other vector relevance features, to General Availability today! Dive into the details of these new additions in our Binary Quantization GA announcement blog post.
What’s next?
Stay tuned for more updates on the latest features of Azure AI Search and their role in simplifying integration for RAG applications!
Getting started with Azure AI Search
- Learn more about Azure AI Search and about all the latest features.
- Start creating a search service in the Azure Portal, Azure CLI, the Management REST API, ARM template, or a Bicep file.
- Learn about Retrieval Augmented Generation in Azure AI Search.
- Explore our preview client libraries in Python, .NET, Java, and JavaScript, offering diverse integration methods to cater to varying user needs.
- Explore how to create end-to-end RAG applications with Azure AI Studio.
Continue reading...