Microsoft’s Azure OpenAI Service: Efficient, Intuitive, and Individualized

by Katie Bates

by Katie Bates

August 08 2023 Blog

A hush fell over the crowd as the software developer began to build service chatbots in real time.  Will the test code work as expected? The Wi-Fi seems to be too slow to run the simulation! “Unplug it and plug it back in,” a jokester types into the session’s Q&A chat.  Dom Divakaruni, the Head of Product for Azure OpenAI, and Pablo Castro, a top engineer and manager of Azure Cognitive Search, were eager to show the world the software they had kept well under wraps until this moment.

AI, LLMs, and other terms you need to know

First, a little background.  Artificial intelligence (AI) is a field of computer science that seeks to create intelligent machines that can replicate or exceed human intelligence.  Though work in this field dates back to the mid-1950s, the past few years have seen a frenzy of development unmatched by anything in recent history; ChatGPT alone reached 100 million users in a mere two months, greatly outpacing the adoption of any other modern technology.

Modern AI projects have generally relied on large language models (LLMs), which are massive data sets of natural language that can include billions of parameters.  LLMs are an essential component in, for example, predicting the next word in a sentence and comparing inputs to known words, both processes you may not be aware that Microsoft uses to check spelling and grammar in their Word application.  LLMs compare new text to text that they already have and analyze the similarities.

This all sounds great, so what’s Microsoft announcing?

AI has historically used LLMs to make 1:1 matches of user-input queries to existing data.  To get the specific result you need, you have to input a very specific prompt.  This poses a problem when you’re not sure of the exact language used in the original text.  Newer applications like ChatGPT and GPT-4 make use of OpenAI, a world-wide collaborative research laboratory that develops LLMs and publishes them in Microsoft Azure.  More than 4,500 Microsoft customers are already using OpenAI services.  This means that Microsoft is in a unique position to compile all of the machine learning of these services and use that information to not only reliably return search results using LLMs, but to generate new content in the apps, suites, and programs we already use.  Microsoft’s Azure OpenAI promises to be your co-pilot in completing complex language tasks including summarizing information, creating documents, and generating code, while keeping true to their reputation for prioritizing enterprise security, privacy, and compliance.

Introducing: Azure OpenAI Service on your data

No longer will the LLM be limited to information already input by services like ChatGPT and GPT-4.  For the first time, Microsoft will allow users to easily add their own data to the learning model to deliver results relevant to their particular business needs.  These LLMs will come pre-configured, and all manual additions and adjustments to that model reside solely in the user’s tenant.  That is, the data added to the learning model stays securely in your tenant and is not added to any third-party model, keeping your data secure.

In their six months of user testing, existing Microsoft clients who were given a preview of the product reported back with great success.  Three common use cases emerged from that testing:

1. AI was easily embedded into their existing products like Search to help better discern what the user was looking for.
2. AI assisted in generating blog posts, emails, documents, and even generated tutorials for novices to learn about new topics using Q&A style learning.
3. AI helped experts in their fields to be more efficient in their jobs by offloading tasks. A healthcare provider was able to automate health record generation; a data processing company was able to enhance their data sets and speed up their experimentation to build better data models faster than ever before.

How does it work?

In the live demonstration during the announcement event, Microsoft engineer Pablo Castro walked the crowd through two scenarios related to a chat session. First, he presented a scenario in which an employee wants to know whether their employer-provided insurance policy covers annual eye exams. The chatbot queried the standard LLM and, because the LLM doesn’t have access to the employee’s particular insurance policy, returned a null result, “I don’t have access to the specific information you’re looking for.” When Castro added the insurance policies to the data set and ran the same query, the chatbot now replied, “Yes, eye exams are covered¹ subject to certain limitations and exclusions²” and provided two citations from source texts from the data set. By uploading information specific to the example company via Blob storage, uploading files, Azure Cognitive Search, or Cosmos DB, the AI chatbot was able to return information specific to the employer and specific to the employee’s needs. This chatbot with access to the company-specific data set was then easily published as an Azure App Services web app, making it accessible for all employees of the company.

In his second example, Castro walked the crowd through a complex process known as vector-based retrieval that makes it easier than ever to find the results you’re looking for.  Microsoft’s Azure OpenAI service translates both the data set and the question prompt into vector data, derives a similarity rating between the two vectors, and returns results with highly similar semantic meanings.

That is, not only will a new AI model return examples of the exact phrase that was searched (e.g., search the data for the phrase maroon car), but also examples that carry a similar meaning (in this case, also include examples that reference a “red vehicle”).  The infinite complexity of human speech makes this quite a dauting task, indeed.

In his live demonstration, Castro input the unwieldy prompt, “Does my plan cover things that help me better perceive mechanical waves reaching the sides of my head?”  As with the first example, the result was inconclusive because those exact words were not found in the employee insurance plan.  BUT after adding Azure Cognitive Search with vector support to the query box, the chatbot was quick to say, “Yes, hearing aids are included in your insurance plan.”  Castro also pointed out that the Azure Cognitive Search function is also highly customizable with filters to make sure that the correct data finds its way to the user.

I’m sold, when do I get it?

Azure’s OpenAI Service is gradually rolling out to users with limited availability for most products by this summer!