Organizations are under increasing pressure to adopt AI tools that improve efficiency without sacrificing data security. Large language models (LLM) are impressive but using them with proprietary data raises legitimate concerns. Sending sensitive client information, pricing data, or internal intellectual property to a publicly available large language model is a non-starter for many enterprises.
My colleague, James Babb, wrote two blog posts highlighting some of these risks. (See: Shadow AI Risks and The Most Dangerous AI Users Are Your Best Employees – Insider Risk.)
To help mitigate some of those risks, Microsoft Foundry offers a way to use these powerful models while keeping your organization’s data safe within your own Microsoft Azure tenant. Microsoft does not use your prompts or completions to train or improve models. This means that internal documents stay within the same security perimeter as the rest of your Azure resources. (See: Data, privacy, and security for Azure Direct Models in Microsoft Foundry – Microsoft Foundry | Microsoft Learn for specific details.) Microsoft also announced their own Microsoft AI (MAI) LLMs at Build 2026. In addition to models from Anthropic and OpenAI, you will be able to access MAI models as well. Authentication follows the same patterns as other Azure services. You can use Managed Identity to authenticate your application to the Microsoft Foundry endpoint, eliminating the need to manage API keys in your code or configuration. The deployment integrates with Azure Role-Based Access Control (RBAC), so you can restrict who and what can call the models. If your organization already has Azure governance policies, logging, and monitoring in place, Microsoft Foundry fits into that existing framework rather than creating a parallel security surface.
Building a Tool
Combining Microsoft Foundry with a pattern called Retrieval Augmented Generation (RAG), developers can build tools that allow users to create AI-generated content using your organization’s own documents rather than relying solely on the model’s general training data while maintaining control over your data and how it is used. Personally, the way I learn is to build something on my own, even if it’s small.
For example, I’m currently building an application that generates professional documents for our team. The tool retrieves and searches similar past examples from our internal document library to guide the AI’s output. The result is generated content that matches our style, terminology, and level of detail rather than producing generic filler. This article covers how RAG works and what I learned while building the tool.
When I first considered building an AI tool that understood our internal content, I assumed I would need to train or fine-tune my own model. That felt like a significant undertaking. Training a model requires curating large datasets, managing compute resources, and developing expertise in machine learning that goes well beyond typical application development. For a team that wanted to ship a useful tool, not become a machine learning shop, that was a daunting barrier.
RAG eliminates that barrier entirely. Instead of training a model to know your content, you retrieve your content at runtime and hand it to a general-purpose model as context. The model does not need to be retrained when your documents change. You simply update the documents in your library, regenerate the embeddings, and the next query picks up the new content automatically. This was the insight that made the project feasible.
How Retrieval Augmented Generation Works
Large language models generate text based on patterns learned during training. They are remarkably capable, but they have a fundamental limitation: they only know what they were trained on. Ask a model to generate a professional document based on your team’s past work, and it will produce something plausible but generic. It has no knowledge of your past projects, your preferred structure, or the specific language your organization uses.
RAG solves this by adding a retrieval step before generation. Instead of asking the model to generate content from scratch, you first search your own document library for relevant examples, then include those examples in the prompt alongside the user request. The model uses the retrieved documents as context, producing output that is grounded in your actual content.
The RAG pattern has three components: Indexing, Retrieval, and Generation. I’ll discuss each of these components in more detail below.
Indexing
Before you can retrieve documents, you need to make them searchable. Traditional keyword search works for exact matches, but it misses semantic relationships. A document about “cloud infrastructure migration” should match a query about “moving workloads to Azure,” even though the words are different. To enable this kind of semantic search, each document is converted into a numerical representation called an embedding. An embedding is a high-dimensional vector that captures the meaning of the text. Documents with similar meaning have similar embeddings, and you can measure that similarity mathematically.
In the tool I built, I use Microsoft Foundry’s “text-embedding-3-small” model to generate embeddings for each document in the library. The documents are stored as markdown files in Azure Blob Storage, and the embedding index is loaded into memory at application startup. This keeps the architecture simple and avoids the need for a dedicated vector database, which would add cost and complexity for a document library of modest size, currently about 50 documents.
Retrieval
When a user requests generated content, the system converts the user’s input into an embedding using the same model. It then compares that embedding against every document embedding in the index using cosine similarity, a standard measure of how close two vectors are in direction. The top three most similar documents are selected as context for the generation step.
Generation
This is where RAG provides its value. Instead of hoping the model knows something relevant, you are explicitly providing it with your best examples. The model does not need to guess at your formatting conventions, your level of technical detail, or your preferred terminology. It also relies on your best examples as a reference.
The retrieved documents are injected into the system prompt alongside instructions that tell the model to use the examples as its primary source of style and content. The user’s specific request, including relevant context and details, is included in the user prompt. The model then generates content that follows the patterns established in the retrieved examples while tailoring the output to the specific request.
The generation step uses Microsoft Foundry’s chat completion API. The system prompt is explicit: use the provided examples as the primary content source, match their style and tone, and avoid generic output. This prompt engineering is critical. Without clear instructions, the model will default to its training data rather than the retrieved examples.
What I Learned
Partner with Microsoft experts you can trust
If it’s time to take that first step toward leveling up your organization’s security, get in touch with Ravenswood to start the conversation.
While building the tool that generates new content based on our existing content library, I learned a few things that I think would be helpful to share with others attempting the same thing, so I’ve listed some of my biggest learnings below.
Start Simple
My initial instinct was to build a full vector database with a dedicated search service. That turned out to be unnecessary. For a document library of a few dozen examples, an in-memory index loaded from a JSON file in blob storage works well. The search is fast, the architecture is straightforward, and there are fewer moving parts to maintain. A dedicated vector database makes sense at scale, but starting with the simplest approach that works lets you validate the concept before adding infrastructure.
Document Quality Matters More Than Model Sophistication
The quality of the retrieved examples has a larger impact on output quality than the choice of model or the tuning of parameters. A well-written example library produces consistently superior results. A poorly written library produces inconsistent results regardless of how sophisticated the model is. Investing time in curating and maintaining the example documents pays off more than any amount of prompt engineering.
The Retrieval Step Is What Makes It Trustworthy
With RAG, the model has real documents to reference, and the output is verifiable against those documents. This is what moves AI-generated content from a novelty to a practical tool. Stakeholders are more comfortable adopting AI when they can see the source material that informed the output.
Prompt Engineering Is Iterative
The first version of the system prompt produced mediocre results. It took several rounds of refinement to get the model to consistently prioritize the retrieved examples over its general training. Explicit instructions like “use the provided examples as your primary content source” made a measurable difference. Vague instructions like “consider the examples” did not.
Conclusion
RAG is not a complex concept. It is a straightforward pattern: search your documents, include the best matches in the prompt, and let the model do what it does well. Microsoft Foundry makes this feasible for organizations that cannot send data outside their security boundary. Your data stays in your tenant, and the integration follows the same authentication and governance patterns you already use for other Azure services.
For organizations considering AI adoption, a RAG tool built on Microsoft Foundry is a practical starting point. It delivers immediate value by grounding AI output in your own content, and it does so within a security model that your compliance and security teams can support.
