azure-samples

Open Source

miyagi

# Project Miyagi - Envisioning sample for [Copilot stack](https://learn.microsoft.com/en-us/semantic-kernel/overview/#semantic-kernel-is-at-the-center-of-the-copilot-stack) > “Start with the customer experience and work backwards for the technology” - Steve Jobs > > "Change is the only constant" - Ancient wisdom <p align="center"><img src="assets/images/1.png" width=20% height=20% /></p> Project Miyagi showcases Microsoft's Copilot Stack in an [envisioning workshop](https://github.com/Azure-Samples/intelligent-app-workshop) aimed at designing, developing, and deploying enterprise-grade intelligent apps. By exploring both generative and traditional ML [use cases](https://iappwksp.com/wksp/05-use-cases/), Miyagi offers an experiential approach to developing AI-infused product experiences that enhance productivity and enable hyper-personalization. Additionally, the workshop introduces traditional software engineers to emerging design patterns in prompt engineering, such as chain-of-thought and retrieval-augmentation, as well as to techniques like vectorization for long-term memory, fine-tuning of OSS models, agent-like orchestration, and plugins or tools for augmenting and grounding LLMs. > **Note** > *Work in Progress*. Meanwhile, signup at [intelligentapp.dev](https://intelligentapp.dev) for updates and checkout our related repo that showcases Generative AI capabilities for cloud-native, event-driven microservices: [Azure/reddog-solutions](https://github.com/Azure/reddog-solutions#readme). > > :tv: For a preview, catch the [recording on Cosmos DB Live TV](https://www.youtube.com/watch?v=V8dlEvXdGEM&t=144s) > The project includes examples of usage for [Semantic Kernel](https://learn.microsoft.com/en-us/semantic-kernel/overview/#semantic-kernel-is-at-the-center-of-the-copilot-stack), [Promptflow](https://promptflow.azurewebsites.net/overview-what-is-prompt-flow.html), [LlamaIndex](https://github.com/jerryjliu/llama_index), [LangChain](https://github.com/hwchase17/langchain#readme), vector stores ([Azure AI Search](https://github.com/Azure/cognitive-search-vector-pr), [CosmosDB Postgres pgvector](https://learn.microsoft.com/en-us/azure/cosmos-db/postgresql/howto-use-pgvector), and generative image utilities such as [DreamFusion](https://huggingface.co/thegovind/reddogpillmodel512) and [ControlNet](https://github.com/lllyasviel/ControlNet). Additionally, it features fine-tuned foundation Models from AzureML such as Llama2 and Phi-2. Utilize this project to gain insights as you modernize and transform your applications with AI and fine-tune your private data to build your own Copilots. This polyglot codebase relies on a multitude of microservices, implementing several [use cases](https://iappwksp.com/wksp/05-use-cases/) using our Copilot stack. It includes generative text and images for personalized financial coaching, summarization, and agent-like orchestration. Built on a cloud-native event-diven architecture (EDA) backbone, the design and codebase ensures enterprise-grade quality attributes such as availability, scalability, and maintainability. Embark on a journey to transform your applications into cutting-edge, intelligent systems with the self-guided workshop and discover the art of the possible. ### Partial Implementations Due to the rapid pace of advancements in foundation models, we are incrementally implementing use cases for Miyagi in the experiments folder. So far, we have the following implemented: 1. [MVP with Personalize (Synthesis via Semantic Kernel) and Chat on Azure Container Apps](https://agentmiyagi.com). 1. [Detailed breakdown and implementations](./services/README.md) 1. [Quickstart with RaG](./sandbox/usecases/rag/dotnet/Getting-started.ipynb) 1. [Agents with Assistants API](./agents/assistants-api/azure-openai/equity-analyst.ipynb) 1. [Agents with Autogen](./agents/README.md) 1. [VSCode extension for GitHub Copilot Agent](./sandbox/usecases/code-modernization/vscode-gh-copilot-extension/README.md) 1. [Miyagi ChatGPT Plugin](./services/chatgpt-plugin/python) 1. [Knowledge Graph memory using Langchain's entity cache](./sandbox/experiments/langchain/Memory_Usecases.ipynb) 1. [Qdrant vector store for embeddings via Langchain](./sandbox/experiments/langchain/qdrant_miyagi_example) 1. [MS Graph API intent invoked via Semantic Kernel's skills](./sandbox/experiments/semantic-kernel/ms-graph-chain) 1. [Miyagi prompt engineered chat interaction](./sandbox/experiments/langchain/chat) using LangChain's PromptTemplate 1. [Azure OpenAI GPT-3.5 basic flow](./sandbox/experiments/az-openai) 1. [GPT-3.5-turbo and Whisper-1 usage to transcribe audio and demonstrate few-shot example](./sandbox/experiments/gpt-3.5-turbo) 1. [DeepSpeed Chat](https://github.com/microsoft/DeepSpeedExamples/tree/master/applications/DeepSpeed-Chat) MiyagiGPT (BYO Weights w/ RLHF - Reinforcement Learning from Human Feedback) - coming soon ### Frontend Interaction with foundation models is more than chat. This sample shows a few use cases ![frontend](./assets/images/wip-ui.png) #### VSCode extension for GitHub Copilot Agent <p align="left"><img src="sandbox/usecases/code-modernization/vscode-gh-copilot-extension/demo.png" width=50% height=50% /></p> ### Architecture #### High-level logical architecture ![azure](./assets/images/wip-azure.png) #### Semantic Kernel Orchestration for Miyagi usecase ![sk-orchestration](./assets/images/sk-memory-orchestration.png) #### In-context learning flow ![round-trip](./assets/images/sk-round-trip.png) <p align="left"><img src="assets/images/embeddings.png" width=40% height=40% /></p> ### 30k foot view #### Typical AI Application <p align="left"><img src="assets/images/ai_application.png" width=60% height=60% /></p> #### AI Application in Azure <p align="left"><img src="assets/images/basic-arch.png" width=30% height=30% /></p> #### Prompt Flow ![prompt-flow](./assets/images/prompt-flow-basic.png) #### OSS Pre-trained Foundation Models ![aml-miyagi-dolly](./assets/images/aml-miyagi-dolly.png) ![aml-training](./assets/images/aml-finetune.png) #### Initial ideation for EDA + SK flow ![architecture](./assets/images/wip-architecture.png) ### Generative image use case architecture with Dreambooth This will be similar to [reddog](https://reddog-solutions.com) product [image generation use case](https://huggingface.co/thegovind/reddogpillmodel512). ![generative-image](./assets/images/wip-dreambooth.png) ## Tech Stack ### Copilot Stack ![copilot stack](./assets/images/copilot-stack.png) ### Services and capabilities - [Azure OpenAI](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/concepts/models) - [Semantic Kernel](https://github.com/microsoft/semantic-kernel) - [LangChain](https://python.langchain.com/docs/get_started/introduction) - [LlamaIndex](https://docs.llamaindex.ai/en/stable/) - [GitHub Copilot Agent](https://gh.io/copilot-partner-program) - [AI Studio](https://azure.microsoft.com/en-us/products/ai-studio) - [AI Search](https://azure.microsoft.com/en-us/products/ai-services/ai-search) - [AI Speech](https://azure.microsoft.com/en-us/products/ai-services/ai-speech) - [AzureML PromptFlow](https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/overview-what-is-prompt-flow?view=azureml-api-2) - [TypeChat](https://microsoft.github.io/TypeChat) - [Kernel-memory](https://github.com/microsoft/kernel-memory) - [AutoGen](https://github.com/microsoft/autogen) - [TaskWeaver](https://github.com/microsoft/TaskWeaver) - [Azure Functions](https://azure.microsoft.com/en-ca/products/functions/) - [APIM](https://learn.microsoft.com/en-us/azure/api-management/) - [Service Bus](https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-messaging-overview) - [Event Grid](https://learn.microsoft.com/en-us/azure/event-grid/overview) - [Logic Apps](https://learn.microsoft.com/en-us/azure/logic-apps/logic-apps-overview) - [AKS](https://azure.microsoft.com/en-us/products/kubernetes-service) / [ACA](https://azure.microsoft.com/en-us/products/container-apps) - [Cosmos DB](https://azure.microsoft.com/en-us/products/cosmos-db/) - [Github Actions](https://docs.github.com/en/actions) - [Azure Monitor](https://learn.microsoft.com/en-us/azure/azure-monitor/) - [Azure DB for PostgreSQL](https://azure.microsoft.com/en-us/products/postgresql) - [Azure Redis Cache](https://azure.microsoft.com/en-us/products/cache) - [Azure Storage](https://learn.microsoft.com/en-us/azure/storage/common/storage-introduction) ### Contributing This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com. When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA. This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [[email protected]](mailto:[email protected]) with any additional questions or comments. ### Disclaimer This software is provided for demonstration purposes only. It is not intended to be relied upon for any purpose. The creators of this software make no representations or warranties of any kind, express or implied, about the completeness, accuracy, reliability, suitability or availability with respect to the software or the information, products, services, or related graphics contained in the software for any purpose. Any reliance you place on such information is therefore strictly at your own risk. ### License This software is provided for demonstration purposes only. It is not intended to be relied upon for any purpose. The software is provided “as is” and without any warranties, express or implied. The software is not intended to be used for any commercial purpose. The software is provided solely for demonstration purposes and should not be used for any other purpose. The software is provided without any warranty of any kind, either express or implied, including, but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. The software is provided “as is” and without any warranty of any kind. The user assumes all risk and responsibility for the use of the software.

AI & Machine Learning AI Agents Vector Databases

752 Github Stars

Open Source

azure-openai-rag-workshop

<div align="center"> # 🤖 Azure OpenAI RAG workshop - Node.js version [![Open project in GitHub Codespaces](https://img.shields.io/badge/Codespaces-Open-blue?style=flat-square&logo=github)](https://codespaces.new/Azure-Samples/azure-openai-rag-workshop?hide_repo_select=true&ref=main&quickstart=true) [![Join Azure AI Community Discord](https://img.shields.io/badge/Discord-Azure_AI_Community-blue?style=flat-square&logo=discord&color=5865f2&logoColor=fff)](https://aka.ms/foundry/discord) ![Node version](https://img.shields.io/badge/Node.js->=20-3c873a?style=flat-square) [![Ollama + Mistral](https://img.shields.io/badge/Ollama-Mistral-ff7000?style=flat-square)](https://ollama.com/library/mistral) [![TypeScript](https://img.shields.io/badge/TypeScript-blue?style=flat-square&logo=typescript&logoColor=white)](https://www.typescriptlang.org) [![License](https://img.shields.io/badge/License-MIT-yellow?style=flat-square)](LICENSE) :star: If you like this sample, star it on GitHub — it helps a lot! [Overview](#overview) • [Run the sample](#run-the-sample) • [Other versions](#other-versions) • [References](#references) </div> This sample shows how to build an AI chat experience with Retrieval-Augmented Generation (RAG) using LangChain.js and OpenAI language models. The application is hosted on [Azure Static Web Apps](https://learn.microsoft.com/azure/static-web-apps/overview) and [Azure Container Apps](https://learn.microsoft.com/azure/container-apps/overview), with [Azure AI Search](https://learn.microsoft.com/azure/search/search-what-is-azure-search) as the vector database. You can use it as a starting point for building more complex AI applications. > [!IMPORTANT] > 👉 **Follow the [full-length workshop](https://aka.ms/ws/openai-rag)** to learn how we built this sample and how you can run and deploy it. ## Overview This sample uses [Fastify](https://fastify.dev) to create a [Node.js](https://nodejs.org/) service that leverage [OpenAI SDK](https://platform.openai.com/docs/libraries/) and [LangChain](https://js.langchain.com/) to build a chatbot that will answer questions based on a corpus of documents, with a website to interact with the API. This project is structured as monorepo, all packages source code is located under the `src/` folder. Here's the architecture of the application: <div align="center"> <img src="./docs/assets/architecture.png" alt="Architecture diagram" width="640px" /> </div> ## Run the sample You can use [GitHub Codespaces](https://github.com/features/codespaces) to work on this project directly from your browser: [![Open in GitHub Codespaces](https://img.shields.io/badge/Codespaces-Open-blue?style=flat-square&logo=github)](https://codespaces.new/Azure-Samples/azure-openai-rag-workshop?hide_repo_select=true&ref=main&quickstart=true) You can also use [Docker](https://www.docker.com/products/docker-desktop) and the [Dev Containers extension for VS Code](https://aka.ms/vscode/ext/devcontainer) to work locally using a ready-to-use dev environment: [![Open in Dev Containers](https://img.shields.io/static/v1?style=flat-square&label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/Azure-Samples/azure-openai-rag-workshop) If you prefer to install all the tools locally, you can follow these [setup instructions](https://aka.ms/ws?src=gh%3AAzure-Samples%2Fazure-openai-rag-workshop%2Fdocs%2Fworkshop.md&step=2#optional-working-locally-without-the-dev-container).  ### Azure prerequisites - **Azure account**. If you're new to Azure, [get an Azure account for free](https://azure.microsoft.com/free) to get free Azure credits to get started. If you're a student, you can also get free credits with [Azure for Students](https://aka.ms/azureforstudents). - **Azure subscription with access enabled for the Azure OpenAI service**. You can request access with [this form](https://aka.ms/oaiapply). - **Azure account permissions**: - Your Azure account must have `Microsoft.Authorization/roleAssignments/write` permissions, such as [Role Based Access Control Administrator](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#role-based-access-control-administrator-preview), [User Access Administrator](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#user-access-administrator), or [Owner](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#owner). If you don't have subscription-level permissions, you must be granted [RBAC](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#role-based-access-control-administrator-preview) for an existing resource group and [deploy to that existing group](docs/deploy_existing.md#resource-group). - Your Azure account also needs `Microsoft.Resources/deployments/write` permissions on the subscription level. ### Deploy the sample Open a terminal and run the following commands: ```bash azd auth login azd up ``` This commands will first ask you to log in into Azure. Then it will provison the Azure resources, package the services and deploy them to Azure. ### Clean up To clean up all the Azure resources created by this sample: 1. Run `azd down --purge` 2. When asked if you are sure you want to continue, enter `y` The resource group and all the resources will be deleted. ## Other versions This sample and workshop exists in different versions: - [**Node.js + Azure AI Search**](https://aka.ms/ws/openai-rag) - [**Node.js + Qdrant**](https://aka.ms/ws/openai-rag-qdrant) - [**Java/Quarkus + Qdrant**](https://aka.ms/ws/openai-rag-quarkus). ## References Here are some resources to learn more about the technologies used in this sample: - [LangChain.js documentation](https://js.langchain.com/) - [Generative AI For Beginners](https://github.com/microsoft/generative-ai-for-beginners) - [Azure OpenAI Service](https://learn.microsoft.com/azure/ai-services/openai/overview) You can also find [more Azure AI samples here](https://github.com/Azure-Samples/azureai-samples). This sample/workshop was based on the enterprise-ready sample **ChatGPT + Enterprise data with Azure OpenAI and AI Search**: - [JavaScript version](https://github.com/Azure-Samples/azure-search-openai-javascript) / [Serverless JavaScript version](https://github.com/Azure-Samples/serverless-chat-langchainjs) - [Python version](https://github.com/Azure-Samples/azure-search-openai-demo/) - [Java version](https://github.com/Azure-Samples/azure-search-openai-demo-java) - [C# version](https://github.com/Azure-Samples/azure-search-openai-demo-csharp) If you want to go further with more advanced use-cases, authentication, history and more, you should check it out! ## Getting Help If you get stuck or have any questions about building AI apps, join: [![Azure AI Foundry Discord](https://img.shields.io/badge/Discord-Azure_AI_Foundry_Community_Discord-blue?style=for-the-badge&logo=discord&color=5865f2&logoColor=fff)](https://aka.ms/foundry/discord) If you have product feedback or errors while building visit: [![Azure AI Foundry Developer Forum](https://img.shields.io/badge/GitHub-Azure_AI_Foundry_Developer_Forum-blue?style=for-the-badge&logo=github&color=000000&logoColor=fff)](https://aka.ms/foundry/forum) ## Contributing This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com. When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA. This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [[email protected]](mailto:[email protected]) with any additional questions or comments. ## Trademarks This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow [Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general). Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

Knowledge Bases & RAG PaaS & Self-hosting

150 Github Stars

Open Source

serverless-chat-langchainjs

<div align="center"> <img src="./packages/webapp/public/favicon.png" alt="" align="center" height="64" /> # Serverless AI Chat with RAG using LangChain.js [![Open project in GitHub Codespaces](https://img.shields.io/badge/Codespaces-Open-blue?style=flat-square&logo=github)](https://codespaces.new/Azure-Samples/serverless-chat-langchainjs?hide_repo_select=true&ref=main&quickstart=true) [![Join Azure AI Foundry Discord](https://img.shields.io/badge/Discord-Azure_AI_Community-blue?style=flat-square&logo=discord&color=5865f2&logoColor=fff)](https://aka.ms/foundry/discord) [![Official Learn documentation](https://img.shields.io/badge/Documentation-00a3ee?style=flat-square)](https://learn.microsoft.com/azure/developer/javascript/ai/get-started-app-chat-template-langchainjs) [![Watch to learn about RAG and this sample on YouTube](https://img.shields.io/badge/YouTube-d95652.svg?style=flat-square&logo=youtube)](https://www.youtube.com/watch?v=xkFOmx5yxIA&list=PLlrxD0HtieHi5ZpsHULPLxm839IrhmeDk&index=4) [![dev.to blog post walkthrough](https://img.shields.io/badge/Blog%20post-black?style=flat-square&logo=dev.to)](https://dev.to/azure/build-a-serverless-chatgpt-with-rag-using-langchainjs-3487) <br> [![Build Status](https://img.shields.io/github/actions/workflow/status/Azure-Samples/serverless-chat-langchainjs/build-test.yaml?style=flat-square&label=Build)](https://github.com/Azure-Samples/serverless-chat-langchainjs/actions) ![Node version](https://img.shields.io/badge/Node.js->=20-3c873a?style=flat-square) [![Ollama + Llama3.1](https://img.shields.io/badge/Ollama-Llama3.1-ff7000?style=flat-square)](https://ollama.com/library/llama3.1) [![TypeScript](https://img.shields.io/badge/TypeScript-blue?style=flat-square&logo=typescript&logoColor=white)](https://www.typescriptlang.org) [![License](https://img.shields.io/badge/License-MIT-yellow?style=flat-square)](LICENSE) :star: If you like this sample, star it on GitHub — it helps a lot! [Overview](#overview) • [Get started](#getting-started) • [Run the sample](#run-the-sample) • [Resources](#resources) • [FAQ](#faq) • [Troubleshooting](#troubleshooting) ![Animation showing the chat app in action](./docs/images/demo.gif) </div> This sample shows how to build a serverless AI chat experience with Retrieval-Augmented Generation using [LangChain.js](https://js.langchain.com/) and Azure. The application is hosted on [Azure Static Web Apps](https://learn.microsoft.com/azure/static-web-apps/overview) and [Azure Functions](https://learn.microsoft.com/azure/azure-functions/functions-overview?pivots=programming-language-javascript), with [Azure Cosmos DB for NoSQL](https://learn.microsoft.com/azure/cosmos-db/nosql/vector-) as the vector database. You can use it as a starting point for building more complex AI applications. > [!TIP] > You can test this application locally without any cost using [Ollama](https://ollama.com/). Follow the instructions in the [Local Development](#local-development) section to get started. ## Overview Building AI applications can be complex and time-consuming, but using LangChain.js and Azure serverless technologies allows to greatly simplify the process. This application is a chatbot that uses a set of enterprise documents to generate responses to user queries. We provide sample data to make this sample ready to try, but feel free to replace it with your own. We use a fictitious company called _Contoso Real Estate_, and the experience allows its customers to ask support questions about the usage of its products. The sample data includes a set of documents that describes its terms of service, privacy policy and a support guide. <div align="center"> <img src="./docs/images/architecture.drawio.png" alt="Application architecture" width="640px" /> </div> This application is made from multiple components: - A web app made with a single chat web component built with [Lit](https://lit.dev) and hosted on [Azure Static Web Apps](https://learn.microsoft.com/azure/static-web-apps/overview). The code is located in the `packages/webapp` folder. - A serverless API built with [Azure Functions](https://learn.microsoft.com/azure/azure-functions/functions-overview?pivots=programming-language-javascript) and using [LangChain.js](https://js.langchain.com/) to ingest the documents and generate responses to the user chat queries. The code is located in the `packages/api` folder. - A database to store chat sessions and the text extracted from the documents and the vectors generated by LangChain.js, using [Azure Cosmos DB for NoSQL](https://learn.microsoft.com/azure/cosmos-db/nosql/). - A file storage to store the source documents, using [Azure Blob Storage](https://learn.microsoft.com/azure/storage/blobs/storage-blobs-introduction). We use the [HTTP protocol for AI chat apps](https://aka.ms/chatprotocol) to communicate between the web app and the API. ## Features - **Serverless Architecture**: Utilizes Azure Functions and Azure Static Web Apps for a fully serverless deployment. - **Retrieval-Augmented Generation (RAG)**: Combines the power of Azure Cosmos DB and LangChain.js to provide relevant and accurate responses. - **Chat Sessions History**: Maintains a personal chat history for each user, allowing them to revisit previous conversations. - **Scalable and Cost-Effective**: Leverages Azure's serverless offerings to provide a scalable and cost-effective solution. - **Local Development**: Supports local development using Ollama for testing without any cloud costs. ## Getting started There are multiple ways to get started with this project. The quickest way is to use [GitHub Codespaces](#use-github-codespaces) that provides a preconfigured environment for you. Alternatively, you can [set up your local environment](#use-your-local-environment) following the instructions below. > [!IMPORTANT] > If you want to run this sample entirely locally using Ollama, you have to follow the instructions in the [local environment](#use-your-local-environment) section. ### Use your local environment You need to install following tools to work on your local machine: - [Node.js LTS](https://nodejs.org/download/) - [Azure Developer CLI](https://aka.ms/azure-dev/install) - [Git](https://git-scm.com/downloads) - [PowerShell 7+](https://github.com/powershell/powershell) _(for Windows users only)_ - **Important**: Ensure you can run `pwsh.exe` from a PowerShell command. If this fails, you likely need to upgrade PowerShell. - Instead of Powershell, you can also use Git Bash or WSL to run the Azure Developer CLI commands. - [Azure Functions Core Tools](https://learn.microsoft.com/azure/azure-functions/functions-run-local?tabs=macos%2Cisolated-process%2Cnode-v4%2Cpython-v2%2Chttp-trigger%2Ccontainer-apps&pivots=programming-language-javascript) _(should be installed automatically with NPM, only install manually if the API fails to start)_ Then you can get the project code: 1. [**Fork**](https://github.com/Azure-Samples/serverless-chat-langchainjs/fork) the project to create your own copy of this repository. 2. On your forked repository, select the **Code** button, then the **Local** tab, and copy the URL of your forked repository. <div align="center"> <img src="./docs/images/clone-url.png" alt="Screenshot showing how to copy the repository URL" width="400px" /> </div> 3. Open a terminal and run this command to clone the repo: <code> git clone <your-repo-url> </code> ### Use GitHub Codespaces You can run this project directly in your browser by using GitHub Codespaces, which will open a web-based VS Code: [![Open in GitHub Codespaces](https://img.shields.io/static/v1?style=for-the-badge&label=GitHub+Codespaces&message=Open&color=blue&logo=github)](https://codespaces.new/Azure-Samples/serverless-chat-langchainjs?hide_repo_select=true&ref&quickstart=true) ### Use a VSCode dev container A similar option to Codespaces is VS Code Dev Containers, that will open the project in your local VS Code instance using the [Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers). You will also need to have [Docker](https://www.docker.com/products/docker-desktop) installed on your machine to run the container. [![Open in Dev Containers](https://img.shields.io/static/v1?style=for-the-badge&label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/Azure-Samples/serverless-chat-langchainjs) ## Run the sample There are multiple ways to run this sample: locally using Ollama or Azure OpenAI models, or by deploying it to Azure. ### Deploy the sample to Azure #### Azure prerequisites - **Azure account**. If you're new to Azure, [get an Azure account for free](https://azure.microsoft.com/free) to get free Azure credits to get started. If you're a student, you can also get free credits with [Azure for Students](https://aka.ms/azureforstudents). - **Azure subscription with access enabled for the Azure OpenAI service**. You can request access with [this form](https://aka.ms/oaiapply). - **Azure account permissions**: - Your Azure account must have `Microsoft.Authorization/roleAssignments/write` permissions, such as [Role Based Access Control Administrator](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#role-based-access-control-administrator-preview), [User Access Administrator](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#user-access-administrator), or [Owner](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#owner). If you don't have subscription-level permissions, you must be granted [RBAC](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#role-based-access-control-administrator-preview) for an existing resource group and [deploy to that existing group](docs/deploy_existing.md#resource-group). - Your Azure account also needs `Microsoft.Resources/deployments/write` permissions on the subscription level. #### Cost estimation See the [cost estimation](./docs/cost.md) details for running this sample on Azure. #### Deploy the sample 1. Open a terminal and navigate to the root of the project. 2. Authenticate with Azure by running `azd auth login`. 3. Run `azd up` to deploy the application to Azure. This will provision Azure resources, deploy this sample, and build the search index based on the files found in the `./data` folder. - You will be prompted to select a base location for the resources. If you're unsure of which location to choose, select `eastus2`. - By default, the OpenAI resource will be deployed to `eastus2`. You can set a different location with `azd env set AZURE_OPENAI_RESOURCE_GROUP_LOCATION <location>`. Currently only a short list of locations is accepted. That location list is based on the [OpenAI model availability table](https://learn.microsoft.com/azure/ai-services/openai/concepts/models#standard-deployment-model-availability) and may become outdated as availability changes. The deployment process will take a few minutes. Once it's done, you'll see the URL of the web app in the terminal. <div align="center"> <img src="./docs/images/azd-up.png" alt="Screenshot of the azd up command result" width="600px" /> </div> You can now open the web app in your browser and start chatting with the bot. ##### Enhance security When deploying the sample in an enterprise context, you may want to enforce tighter security restrictions to protect your data and resources. See the [enhance security](./docs/enhance-security.md) guide for more information. #### Enable CI/CD If you want to enable Continuous Deployment for your forked repository, you need to configure the Azure pipeline first: 1. Open a terminal at the root of your forked project. 2. Authenticate with Azure by running `azd auth login`. 3. Run `azd pipeline config` to configure the required secrets and variables for connecting to Azure from GitHub Actions. - This command will set up the necessary Azure service principal and configure GitHub repository secrets. - Follow the prompts to complete the configuration. Once configured, the GitHub Actions workflow will automatically deploy your application to Azure whenever you push changes to the main branch. #### Clean up To clean up all the Azure resources created by this sample: 1. Run `azd down --purge` 2. When asked if you are sure you want to continue, enter `y` The resource group and all the resources will be deleted. ### Run the sample locally with Ollama If you have a machine with enough resources, you can run this sample entirely locally without using any cloud resources. To do that, you first have to install [Ollama](https://ollama.com) and then run the following commands to download the models on your machine: ```bash ollama pull llama3.1:latest ollama pull nomic-embed-text:latest ``` > [!NOTE] > The `llama3.1` model with download a few gigabytes of data, so it can take some time depending on your internet connection. After that you have to install the NPM dependencies: ```bash npm install ``` Then you can start the application by running the following command which will start the web app and the API locally: ```bash npm start ``` Then, open a new terminal running concurrently and run the following command to upload the PDF documents from the `/data` folder to the API: ```bash npm run upload:docs ``` This only has to be done once, unless you want to add more documents. You can now open the URL `http://localhost:8000` in your browser to start chatting with the bot. > [!NOTE] > While local models usually works well enough to answer the questions, sometimes they may not be able to follow perfectly the advanced formatting instructions for the citations and follow-up questions. This is expected, and a limitation of using smaller local models. ### Run the sample locally with Azure OpenAI models First you need to provision the Azure resources needed to run the sample. Follow the instructions in the [Deploy the sample to Azure](#deploy-the-sample-to-azure) section to deploy the sample to Azure, then you'll be able to run the sample locally using the deployed Azure resources. Once your deployment is complete, you should see a `.env` file in the `packages/api` folder. This file contains the environment variables needed to run the application using Azure resources. To run the sample, you can then use the same commands as for the Ollama setup. This will start the web app and the API locally: ```bash npm start ``` Open the URL `http://localhost:8000` in your browser to start chatting with the bot. Note that the documents are uploaded automatically when deploying the sample to Azure with `azd up`. > [!TIP] > You can switch back to using Ollama models by simply deleting the `packages/api/.env` file and starting the application again. To regenerate the `.env` file, you can run `azd env get-values > packages/api/.env`. ## Resources Here are some resources to learn more about the technologies used in this sample: - [LangChain.js documentation](https://js.langchain.com) - [Generative AI with JavaScript](https://github.com/microsoft/generative-ai-with-javascript) - [Generative AI For Beginners](https://github.com/microsoft/generative-ai-for-beginners) - [Azure OpenAI Service](https://learn.microsoft.com/azure/ai-services/openai/overview) - [Azure Cosmos DB for NoSQL](https://learn.microsoft.com/azure/cosmos-db/nosql/) - [Ask YouTube: LangChain.js + Azure Quickstart sample](https://github.com/Azure-Samples/langchainjs-quickstart-demo) - [Chat + Enterprise data with Azure OpenAI and Azure AI Search](https://github.com/Azure-Samples/azure-search-openai-javascript) - [Revolutionize your Enterprise Data with Chat: Next-gen Apps w/ Azure OpenAI and AI Search](https://aka.ms/entgptsearchblog) You can also find [more Azure AI samples here](https://github.com/Azure-Samples/azureai-samples). ## FAQ You can find answers to frequently asked questions in the [FAQ](./docs/faq.md). ## Troubleshooting If you have any issue when running or deploying this sample, please check the [troubleshooting guide](./docs/troubleshooting.md). If you can't find a solution to your problem, please [open an issue](https://github.com/Azure-Samples/serverless-chat-langchainjs/issues) in this repository. ## Guidance For more detailed guidance on how to use this sample, please refer to the [tutorial](./docs/tutorial/01-introduction.md). ## Getting Help If you get stuck or have any questions about building AI apps, join: [![Azure AI Foundry Discord](https://img.shields.io/badge/Discord-Azure_AI_Foundry_Community_Discord-blue?style=for-the-badge&logo=discord&color=5865f2&logoColor=fff)](https://aka.ms/foundry/discord) If you have product feedback or errors while building visit: [![Azure AI Foundry Developer Forum](https://img.shields.io/badge/GitHub-Azure_AI_Foundry_Developer_Forum-blue?style=for-the-badge&logo=github&color=000000&logoColor=fff)](https://aka.ms/foundry/forum) ## Contributing This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com. When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA. This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [[email protected]](mailto:[email protected]) with any additional questions or comments. ## Trademarks This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow [Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general). Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

Knowledge Bases & RAG FaaS & Serverless

858 Github Stars

Open Source

aisearch-openai-rag-audio

# VoiceRAG: An Application Pattern for RAG + Voice Using Azure AI Search and the GPT-4o Realtime API for Audio [![Open in GitHub Codespaces](https://img.shields.io/static/v1?style=for-the-badge&label=GitHub+Codespaces&message=Open&color=brightgreen&logo=github)](https://github.com/codespaces/new?hide_repo_select=true&ref=main&skip_quickstart=true&machine=basicLinux32gb&repo=860141324&devcontainer_path=.devcontainer%2Fdevcontainer.json&geo=WestUs2) [![Open in Dev Containers](https://img.shields.io/static/v1?style=for-the-badge&label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/Azure-Samples/aisearch-openai-rag-audio) This repo contains an example of how to implement RAG support in applications that use voice as their user interface, powered by the GPT-4o realtime API for audio. We describe the pattern in more detail in [this blog post](https://aka.ms/voicerag), and you can see this sample app in action in [this short video](https://youtu.be/vXJka8xZ9Ko). * [Features](#features) * [Architecture Diagram](#architecture-diagram) * [Getting Started](#getting-started) * [GitHub Codespaces](#github-codespaces) * [VS Code Dev Containers](#vs-code-dev-containers) * [Local environment](#local-environment) * [Deploying the app](#deploying-the-app) * [Development server](#development-server) * [Guidance](#guidance) * [Resources](#resources) * [Getting help](#getting-help) ## Features * **Voice interface**: The app uses the browser's microphone to capture voice input, and sends it to the backend where it is processed by the Azure OpenAI GPT-4o Realtime API. * **RAG (Retrieval Augmented Generation)**: The app uses the Azure AI Search service to answer questions about a knowledge base, and sends the retrieved documents to the GPT-4o Realtime API to generate a response. * **Audio output**: The app plays the response from the GPT-4o Realtime API as audio, using the browser's audio capabilities. * **Citations**: The app shows the search results that were used to generate the response. ### Architecture Diagram The `RTClient` in the frontend receives the audio input, sends that to the Python backend which uses an `RTMiddleTier` object to interface with the Azure OpenAI real-time API, and includes a tool for searching Azure AI Search. ![Diagram of real-time RAG pattern](docs/RTMTPattern.png) This repository includes infrastructure as code and a `Dockerfile` to deploy the app to Azure Container Apps, but it can also be run locally as long as Azure AI Search and Azure OpenAI services are configured. ## Getting Started You have a few options for getting started with this template. The quickest way to get started is [GitHub Codespaces](#github-codespaces), since it will setup all the tools for you, but you can also [set it up locally](#local-environment). You can also use a [VS Code dev container](#vs-code-dev-containers) ### GitHub Codespaces You can run this repo virtually by using GitHub Codespaces, which will open a web-based VS Code in your browser: [![Open in GitHub Codespaces](https://img.shields.io/static/v1?style=for-the-badge&label=GitHub+Codespaces&message=Open&color=brightgreen&logo=github)](https://github.com/codespaces/new?hide_repo_select=true&ref=main&skip_quickstart=true&machine=basicLinux32gb&repo=860141324&devcontainer_path=.devcontainer%2Fdevcontainer.json&geo=WestUs2) Once the codespace opens (this may take several minutes), open a new terminal and proceed to [deploy the app](#deploying-the-app). ### VS Code Dev Containers You can run the project in your local VS Code Dev Container using the [Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers): 1. Start Docker Desktop (install it if not already installed) 2. Open the project: [![Open in Dev Containers](https://img.shields.io/static/v1?style=for-the-badge&label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/azure-samples/aisearch-openai-rag-audio) 3. In the VS Code window that opens, once the project files show up (this may take several minutes), open a new terminal, and proceed to [deploying the app](#deploying-the-app). ### Local environment 1. Install the required tools: * [Azure Developer CLI](https://aka.ms/azure-dev/install) * [Node.js](https://nodejs.org/) * [Python >=3.11](https://www.python.org/downloads/) * **Important**: Python and the pip package manager must be in the path in Windows for the setup scripts to work. * **Important**: Ensure you can run `python --version` from console. On Ubuntu, you might need to run `sudo apt install python-is-python3` to link `python` to `python3`. * [Git](https://git-scm.com/downloads) * [Powershell](https://learn.microsoft.com/powershell/scripting/install/installing-powershell) - For Windows users only. 2. Clone the repo (`git clone https://github.com/Azure-Samples/aisearch-openai-rag-audio`) 3. Proceed to the next section to [deploy the app](#deploying-the-app). ## Deploying the app The steps below will provision Azure resources and deploy the application code to Azure Container Apps. 1. Login to your Azure account: ```shell azd auth login ``` For GitHub Codespaces users, if the previous command fails, try: ```shell azd auth login --use-device-code ``` 1. Create a new azd environment: ```shell azd env new ``` Enter a name that will be used for the resource group. This will create a new folder in the `.azure` folder, and set it as the active environment for any calls to `azd` going forward. 1. (Optional) This is the point where you can customize the deployment by setting azd environment variables, in order to [use existing services](docs/existing_services.md) or [customize the voice choice](docs/customizing_deploy.md). 1. Run this single command to provision the resources, deploy the code, and setup integrated vectorization for the sample data: ```shell azd up ```` * **Important**: Beware that the resources created by this command will incur immediate costs, primarily from the AI Search resource. These resources may accrue costs even if you interrupt the command before it is fully executed. You can run `azd down` or delete the resources manually to avoid unnecessary spending. * You will be prompted to select two locations, one for the majority of resources and one for the OpenAI resource, which is currently a short list. That location list is based on the [OpenAI model availability table](https://learn.microsoft.com/azure/ai-services/openai/concepts/models#global-standard-model-availability) and may become outdated as availability changes. 1. After the application has been successfully deployed you will see a URL printed to the console. Navigate to that URL to interact with the app in your browser. To try out the app, click the "Start conversation button", say "Hello", and then ask a question about your data like "What is the whistleblower policy for Contoso electronics?" You can also now run the app locally by following the instructions in [the next section](#development-server). ## Development server You can run this app locally using either the Azure services you provisioned by following the [deployment instructions](#deploying-the-app), or by pointing the local app at already [existing services](docs/existing_services.md). 1. If you deployed with `azd up`, you should see a `app/backend/.env` file with the necessary environment variables. 2. If did *not* use `azd up`, you will need to create `app/backend/.env` file with the following environment variables: ```shell AZURE_OPENAI_ENDPOINT=wss://<your instance name>.openai.azure.com AZURE_OPENAI_REALTIME_DEPLOYMENT=gpt-4o-realtime-preview AZURE_OPENAI_REALTIME_VOICE_CHOICE=<choose one: echo, alloy, shimmer> AZURE_OPENAI_API_KEY=<your api key> AZURE_SEARCH_ENDPOINT=https://<your service name>.search.windows.net AZURE_SEARCH_INDEX=<your index name> AZURE_SEARCH_API_KEY=<your api key> ``` To use Entra ID (your user when running locally, managed identity when deployed) simply don't set the keys. 3. Run this command to start the app: Windows: ```pwsh pwsh .\scripts\start.ps1 ``` Linux/Mac: ```bash ./scripts/start.sh ``` 4. The app is available on [http://localhost:8765](http://localhost:8765). Once the app is running, when you navigate to the URL above you should see the start screen of the app: ![app screenshot](docs/talktoyourdataapp.png) To try out the app, click the "Start conversation button", say "Hello", and then ask a question about your data like "What is the whistleblower policy for Contoso electronics?" ## Guidance ### Costs Pricing varies per region and usage, so it isn't possible to predict exact costs for your usage. However, you can try the [Azure pricing calculator](https://azure.com/e/a87a169b256e43c089015fda8182ca87) for the resources below. * Azure Container Apps: Consumption plan with 1 CPU core, 2.0 GB RAM. Pricing with Pay-as-You-Go. [Pricing](https://azure.microsoft.com/pricing/details/container-apps/) * Azure OpenAI: Standard tier, gpt-4o-realtime and text-embedding-3-large models. Pricing per 1K tokens used. [Pricing](https://azure.microsoft.com/pricing/details/cognitive-services/openai-service/) * Azure AI Search: Standard tier, 1 replica, free level of semantic search. Pricing per hour. [Pricing](https://azure.microsoft.com/pricing/details/search/) * Azure Blob Storage: Standard tier with ZRS (Zone-redundant storage). Pricing per storage and read operations. [Pricing](https://azure.microsoft.com/pricing/details/storage/blobs/) * Azure Monitor: Pay-as-you-go tier. Costs based on data ingested. [Pricing](https://azure.microsoft.com/pricing/details/monitor/) To reduce costs, you can switch to free SKUs for various services, but those SKUs have limitations. ⚠️ To avoid unnecessary costs, remember to take down your app if it's no longer in use, either by deleting the resource group in the Portal or running `azd down`. ### Security This template uses [Managed Identity](https://learn.microsoft.com/entra/identity/managed-identities-azure-resources/overview) to eliminate the need for developers to manage these credentials. Applications can use managed identities to obtain Microsoft Entra tokens without having to manage any credentials.To ensure best practices in your repo we recommend anyone creating solutions based on our templates ensure that the [Github secret scanning](https://docs.github.com/code-security/secret-scanning/about-secret-scanning) setting is enabled in your repos. ### Notes >Sample data: The PDF documents used in this demo contain information generated using a language model (Azure OpenAI Service). The information contained in these documents is only for demonstration purposes and does not reflect the opinions or beliefs of Microsoft. Microsoft makes no representations or warranties of any kind, express or implied, about the completeness, accuracy, reliability, suitability or availability with respect to the information contained in this document. All rights reserved to Microsoft. ## Resources * [Blog post: VoiceRAG](https://aka.ms/voicerag) * [Demo video: VoiceRAG](https://youtu.be/vXJka8xZ9Ko) * [Azure OpenAI Realtime Documentation](https://github.com/Azure-Samples/aoai-realtime-audio-sdk/) ### Getting help This is a sample built to demonstrate the capabilities of modern Generative AI apps and how they can be built in Azure. For help with deploying this sample, please post in [GitHub Issues](/issues). If you're a Microsoft employee, you can also post in [our Teams channel](https://aka.ms/azai-python-help). This repository is supported by the maintainers, _not_ by Microsoft Support, so please use the support mechanisms described above, and we will do our best to help you out. For general questions about developing AI solutions on Azure, join the Azure AI Foundry Developer Community: [![Azure AI Foundry Discord](https://img.shields.io/badge/Discord-Azure_AI_Foundry_Community_Discord-blue?style=for-the-badge&logo=discord&color=5865f2&logoColor=fff)](https://aka.ms/foundry/discord) [![Azure AI Foundry Developer Forum](https://img.shields.io/badge/GitHub-Azure_AI_Foundry_Developer_Forum-blue?style=for-the-badge&logo=github&color=000000&logoColor=fff)](https://aka.ms/foundry/forum)

Knowledge Bases & RAG Audio Editing & DAW

556 Github Stars

Open Source

azure-databricks-mlops-mlflow

--- page_type: sample ms.custom: - team=cse ms.contributors: - prdeb-12/21/2021 - anchugh-12/21/2021 languages: - python products: - azure-databricks - azure-blob-storage - azure-monitor --- # Azure Databricks MLOps using MLflow This is a template or sample for [MLOps](https://github.com/microsoft/mlops) for [Python](https://www.python.org) based source code in [Azure Databricks](https://docs.microsoft.com/en-us/azure/databricks/) using [MLflow](https://docs.microsoft.com/en-us/azure/databricks/applications/mlflow/) without using [MLflow Project](https://mlflow.org/docs/latest/projects.html#). This template provides the following features: - A way to run Python based MLOps without using [MLflow Project](https://mlflow.org/docs/latest/projects.html#), but still using [MLflow](https://docs.microsoft.com/en-us/azure/databricks/applications/mlflow/) for managing the end-to-end machine learning lifecycle. - Sample of machine learning source code structure along with Unit Test cases - Sample of MLOps code structure along with Unit Test cases - Demo setup to try on users subscription ## Problem Summary - This demonstrates deployment scenario of [Orchestrate MLOps on Azure Databricks using Databricks Notebook](https://docs.microsoft.com/en-us/azure/architecture/reference-architectures/ai/orchestrate-mlops-azure-databricks) ## Products/Technologies/Languages Used - Products & Technologies: - Azure Databricks - Azure Blob Storage - Azure Monitor - Languages: - Python ## Architecture ### Model Training ![Model Training](docs/images/model_training.png) ### Batch Scoring ![Batch Scoring](docs/images/batch_scoring.png) ## Individual Components - [ml_experiment](./ml_experiments/experiment_notebook.ipynb) - sample ML experiment notebook. - [ml_data](./ml_data/) - dummy data for sample model - [ml_ops](./ml_ops/) - sample MLOps code along with Unit Test cases, orchestrator, deployment setup. - [ml_source](./ml_source/) - sample ML code along with Unit Test cases - [Makefile](./Makefile) - for build, test in local environment - [requirements.txt](./requirements.txt) - python dependencies ## Getting Started ### Prerequisites - [Azure Databricks](https://docs.microsoft.com/en-us/azure/databricks/) workspace - [Azure Data Lake Storage Gen2](https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction) account - [Visual Studio Code](https://code.visualstudio.com/) in local environment for development - [Docker](https://www.docker.com/) in local environment for development ### Development 1. `git clone https://github.com/Azure-Samples/azure-databricks-mlops-mlflow.git` 2. `cd azure-databricks-mlops-mlflow` 3. Open cloned repository in Visual Studio Code [Remote Container](https://code.visualstudio.com/docs/remote/containers) 4. Open a [terminal](https://code.visualstudio.com/docs/remote/containers#_opening-a-terminal) in Remote Container from Visual Studio Code 5. `make install` to install sample packages (`taxi_fares` and `taxi_fares_mlops`) locally 6. `make test` to Unit Test the code locally ### Package 1. `make dist` to build wheel Ml and MLOps packages (`taxi_fares` and `taxi_fares_mlops`) locally ### Deployment 1. `make databricks-deploy-code` to deploy Databricks Orchestrator Notebooks, ML and MLOps Python wheel packages. If any code changes. 2. `make databricks-deploy-jobs` to deploy Databricks Jobs. If any changes in job specs. ### Run training and batch scoring 1. To trigger training, execute `make run-taxi-fares-model-training` 2. To trigger batch scoring, execute `make run-taxi-fares-batch-scoring` **NOTE:** for [deployment](#deployment) and [running](#run-training-and-batch-scoring) the Databricks environment should be created first, for creating a demo environment the [Demo](#demo) chapter can be followed. ### Observability Check Logs, create alerts. etc. in [Application Insights](https://docs.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview). Following are the few sample [Kusto Query](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/) to check logs, traces, exception, etc. - Check for Error, Info, Debug Logs Kusto Query for checking general logs for a specific MLflow experiment, filtered by `mlflow_experiment_id` ```kusto traces | extend mlflow_experiment_id = customDimensions.mlflow_experiment_id | where timestamp > ago(30m) | where mlflow_experiment_id == <mlflow experiment id> | limit 1000 ``` Kusto Query for checking general logs for a specific Databricks job execution filtered by `mlflow_experiment_id` and `mlflow_run_id` ```kusto traces | extend mlflow_run_id = customDimensions.mlflow_run_id | extend mlflow_experiment_id = customDimensions.mlflow_experiment_id | where timestamp > ago(30m) | where mlflow_experiment_id == <mlflow experiment id> | where mlflow_run_id == "<mlflow run id>" | limit 1000 ``` - Check for Exceptions Kusto Query for checking exception log if any ```kusto exceptions | where timestamp > ago(30m) | limit 1000 ``` - Check for duration of different stages in MLOps Sample Kusto Query for checking duration of different stages in MLOps ```kusto dependencies | where timestamp > ago(30m) | where cloud_RoleName == 'TaxiFares_Training' | limit 1000 ``` To correlate dependencies, exceptions and traces, `operation_Id` can be used a filter to above Kusto Queries. ## Demo 1. Create Databricks workspace, a storage account (Azure Data Lake Storage Gen2) and Application Insights 1. Create an [Azure Account](https://azure.microsoft.com/en-in/free/) 2. [Deploy resources](https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/deploy-portal#deploy-resources-from-custom-template) from [custom ARM template](ml_ops/deployment/arm_templates/databricks_and_storage.json) 2. Initialize Databricks (create cluster, base workspace, mlflow experiment, secret scope) 1. Get [Databricks CLI](https://docs.microsoft.com/en-us/azure/databricks/dev-tools/cli/) Host and Token 2. Authenticate Databricks CLI `make databricks-authenticate` 3. Execute `make databricks-init` 3. Create Azure Data Lake Storage Gen2 Container and upload data 1. [Create](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-portal#create-a-container) Azure Data Lake Storage Gen2 Container named - `taxifares` 2. [Upload](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-portal#upload-a-block-blob) as blob [taxi-fares data files](./ml_data/) into Azure Data Lake Storage Gen2 container named - `taxifares` 4. Put secrets to [Mount ADLS Gen2 Storage using Shared Access Key](https://docs.microsoft.com/en-gb/azure/databricks/data/data-sources/azure/azure-storage) 1. Get Azure Data Lake Storage Gen2 account name created in step 1 2. Get [Shared Key](https://docs.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key) for Azure Data Lake Storage Gen2 account 3. Execute `make databricks-secrets-put` to put secret in Databricks secret scope 5. Put Application Insights Key as a secret in Databricks secret scope (optional) 1. Get [Application Insights Key](https://docs.microsoft.com/en-us/azure/azure-monitor/app/create-new-resource#copy-the-instrumentation-key) created in step 1 2. Execute `make databricks-add-app-insights-key` to put secret in Databricks secret scope 6. Package and deploy into Databricks (Databricks Jobs, Orchestrator Notebooks, ML and MLOps Python wheel packages) 1. Execute `make deploy` 7. Run Databricks Jobs 1. To trigger training, execute `make run-taxifares-model-training` 2. To trigger batch scoring, execute `make run-taxifares-batch-scoring` 8. Expected results 1. Azure resources ![Azure resources](docs/images/result_azure_resources.png) 2. Databricks jobs ![Databricks jobs](docs/images/result_databricks_job.png) 3. Databricks mlflow experiment ![Databricks mlflow experiment](docs/images/result_mlflow_experiment.png) 4. Databricks mlflow model registry ![Databricks mlflow model registry](docs/images/result_mlflow_model_registry.png) 5. Output of batch scoring ![Output of batch scoring](docs/images/result_batch_scoring.png) ## Additional Details 1. [Continuous Integration (CI) & Continuous Deployment (CD)](docs/advance/cicd.md) 2. [Registered Models Stages and Transitioning](docs/advance/registered_model_stages.md) ## Related resources 1. [Azure Databricks](https://docs.microsoft.com/en-us/azure/databricks/) 2. [MLflow](https://docs.microsoft.com/en-us/azure/databricks/applications/mlflow/) 3. [MLflow Project](https://mlflow.org/docs/latest/projects.html#) 4. [Run MLflow Projects on Azure Databricks](https://docs.microsoft.com/en-us/azure/databricks/applications/mlflow/projects) 5. [Databricks Widgets](https://docs.microsoft.com/en-us/azure/databricks/dev-tools/databricks-utils#--widget-utilities) 6. [Databricks Notebook-scoped Python libraries](https://docs.microsoft.com/en-us/azure/databricks/libraries/notebooks-python-libraries) 7. [Databricks CLI](https://docs.microsoft.com/en-us/azure/databricks/dev-tools/cli/) 8. [Azure Data Lake Storage Gen2](https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction) 9. [Application Insights](https://docs.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview) 10. [Kusto Query Language](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/) ## Glossaries 1. [Application developer](https://docs.microsoft.com/en-us/azure/machine-learning/team-data-science-process/overview) : It is a role that work mainly towards operationalize of machine learning. 2. [Data scientist](https://docs.microsoft.com/en-us/azure/machine-learning/team-data-science-process/roles-tasks#structure-of-data-science-groups-and-teams) : It is a role to perform the data science parts of the project ## Contributors - [Julien Chomarat](https://github.com/jchomarat) - [Benjamin Guinebertière](https://github.com/benjguin) - [Ankit Sinha](https://github.com/ankitbko) - [Prabal Deb](https://github.com/prabdeb) - [Megha Patil](https://github.com/meghapatilcode) - [Srikantan Sankaran](https://github.com/ssrikantan) - [Frédéric Le Coquil](https://github.com/flecoqui) - [Anand Chugh](https://github.com/anandchugh)

AI & Machine Learning ML Frameworks

97 Github Stars

Open Source

podcast-synopsis-generation-openai

# Building A Pipeline For Podcast Synopsis Generation (and more) With Azure OpenAI ## Overview The process of podcast synopsis generation traditionally has required creators, producers, and/or writers to understand the content, and manually write the synopsis, summarizing the salient points and highlights of an episode. This approach is time-consuming and may involve brainstorming sessions that discuss the main themes, and refine the text until it effectively conveys the podcast's message and tone (e.g., a comedic podcast may feature humorous language or hyperbole in its summaries). This repository illustrates how to automate a large part of this manual and time-consuming process using the portfolio of services provided by [Azure Cognitive Services](https://azure.microsoft.com/en-us/products/cognitive-services), specifically [Azure Speech Service](https://azure.microsoft.com/en-us/products/cognitive-services/speech-services/) and [Azure OpenAI](https://azure.microsoft.com/en-us/products/cognitive-services/openai-service) are used for transcribing and generating the **synopsis**, **taglines**, **SEO keywords** and **translation** into multiple languages. Incorporating AI to automate this process does not eliminate the role of human creativity or the importance of human involvement. Instead, it enables a significant acceleration in time-to-market by harnessing the power of AI. The final validation and approval of content remains the responsibility of human specialists before publishing. ## Architecture ![architecture](./docs/media/high-level-arch.png) This repository implements a high-level architecture that will take the audio of a podcast (steps 1-3), transcribe the audio into text using Azure Speech Services (step 4), and generate a synopsis based on the transcription using Azure OpenAI GPT-3.5 (step 5). Note that the version of GPT model used can be easily upgraded to, for example GPT-4, or future versions. It consists of a React web app that allows a user to upload an audio file (podcast). The app uses an [Azure function (WebApiHttpTrigger)](/src/functions/WebApiHttpTrigger/) endpoint to upload the audio file to a storage account, i.e. the `raw-files` container. [Azure function (AudioFileUploadedBlobTrigger)](./src/functions/AudioFileUploadedBlobTrigger/) get triggered when a new file is added to the `raw-files` container. The function downloads the audio file and sends it to Azure Speech Services and transcribes the audio to text. It then saves the transcription to the same storage account, i.e. the `transcription` container. The last [Azure function (TranscriptionFileUploadedBlobTrigger)](./src/functions/TranscriptionFileUploadedBlobTrigger/), is triggered when new files are added to the `transcriptions` container. The function downloads the transcription file and sends it to four different Azure OpenAI methods that perform the following functions: 1. Generate synoposis 2. Generate tag lines 3. Generate SEO keywords 4. Generate synopsis translation in Mandarin Chinese, Brazilian Portuguese, German, and Latin American Spanish. It saves the results as separate text files in the storage account `open-ai-results` container. ## Getting Started 1. [Infrastructure Deployment and Configuration](./docs/1-infrastructure-deployment.md) 2. [Local Development Setup](./docs/2-local-development-setup.md) 3. [GitHub workflows](./docs/3-github-workflows.md) ## Related Publications and Demo - [Blog post with demo video](https://techcommunity.microsoft.com/t5/ai-cognitive-services-blog/automating-podcast-synopsis-generation-with-azure-openai-gpt/ba-p/3810308) for this solution. - [Prompt design examples](https://github.com/Azure/azure-openai-samples/tree/main/use_cases/generate_podcast_synopsis) for this use case. ## Contributing This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com. To contribute, please start by creating a self-assigned issue giving a high-level overview of what you'd like to do. Once any discussion there concludes, follow up with a PR. This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

AI Tools Podcast Tools

20 Github Stars

Software by azure-samples

miyagi

azure-openai-rag-workshop

serverless-chat-langchainjs

aisearch-openai-rag-audio

azure-databricks-mlops-mlflow

podcast-synopsis-generation-openai