Tag: azure ai

Get Started with Foundry Local & AI Dev Gallery

Before we begin, let’s ask why people and organisations want to use local AI? The possible answers are:

  • In a restrictive environment, run the models offline and store sensitive data on the device or in a secure location.
  • Reduce inference costs. Utilise local models for low-complexity processing.
  • Implement a low-latency, real-time AI application.
  • Automate any tasks with these models without breaking the bank!
  • Use automation where there is no or limited internet access.

At the recent Build event, Microsoft unveiled a comprehensive set of capabilities designed to help developers implement local AI, including Windows AI Foundry and Foundry Local.

Foundry Local

Foundry Local is a local on-device AI inference solution developed by Microsoft as part of Azure AI Foundry. It enables developers and organisations to run small or large language models (S/LLMs) and other AI models directly on their own hardware, such as Windows devices.

What did not work

Reminder, this is still in Public Preview. When I tried the following terminal command, I received the error below. I have tried a few options but had no luck. If you are experiencing the same issue, try the steps from “What Worked”.

winget install Microsoft.FoundryLocal

What worked

To get things running, simply run the following commands in a PowerShell or Command Prompt window. The “FoundryLocal-x64-0.3.9267.43123.msix” file is nearly 1 GB in size. So it might take some time to download.

# Download the package and its dependency
$releaseUri = "https://github.com/microsoft/Foundry-Local/releases/download/v0.3.9267/FoundryLocal-x64-0.3.9267.43123.msix"
Invoke-WebRequest -Method Get -Uri $releaseUri -OutFile .\FoundryLocal.msix
$crtUri = "https://aka.ms/Microsoft.VCLibs.x64.14.00.Desktop.appx"
Invoke-WebRequest -Method Get -Uri $crtUri -OutFile .\VcLibs.appx

# Install the Foundry Local package
Add-AppxPackage .\FoundryLocal.msix -DependencyPath .\VcLibs.appx

Once installed, you can simply run the following command to see available models.

foundry model ls

Mindful of the file size of each model. You can download and run the model using the following command.

#foundry model run <model_name>
foundry model run phi-3.5-mini

Once the model is running, you can interact with the model. In the following example, I have asked the phi-3.5-mini model to translate “It is a beautiful day” in German, all within the Windows command prompt. This simple model can translate about 23 languages! You can find more details about that model here.

Some standard CLI commands are below. All Reference CLI can be found here. 

foundry --help

foundry model ls
foundry model run <model-name>
foundry service status

foundry cache list
foundry cache remove <model-name>
foundry cache cd <path>

AI Dev Gallery

Another great tool Microsoft recently released is the AI Dev Gallery: aka.ms/ai-dev-gallery. This tool enables developers to explore and integrate on-device AI functionality into their applications.

Learn more about the AI Dev Gallery: https://techcommunity.microsoft.com/blog/azuredevcommunityblog/getting-started-with-the-ai-dev-gallery/4354803

Below is a short demo of the AI Dev Gallery

Windows AI Foundry

On-device applications can also leverage the AI Backends features or AI APIs offered by Windows AI Foundry. Learn more about it here: https://learn.microsoft.com/en-us/windows/ai/overview

Reference links

  1. GitHub Foundry Local: https://github.com/microsoft/Foundry-Local/
  2. Learn more about Foundry Local: https://learn.microsoft.com/en-us/azure/ai-foundry/foundry-local/what-is-foundry-local
  3. Capability of PHI-3.5 SLM: https://techcommunity.microsoft.com/blog/azure-ai-services-blog/discover-the-new-multi-lingual-high-quality-phi-3-5-slms/4225280
  4. Learn more about the AI Dev Gallery: https://techcommunity.microsoft.com/blog/azuredevcommunityblog/getting-started-with-the-ai-dev-gallery/4354803
  5. Installer -AI Dev Gallery: https://apps.microsoft.com/detail/9n9pn1mm3bd5?hl=en-US&gl=AU
  6. Windows AI Foundry: https://learn.microsoft.com/en-us/windows/ai/overview

Global AI Bootcamp 2025 (Sydney)

Sydney’s AI community came together for an inspiring event at Microsoft Reactor Sydney, drawing over 120 passionate participants. The day was a dynamic mix of thought-provoking sessions and vibrant discussions.

We had a wide range of attendees, including data engineers, architects, business leaders, analysts, student university professors, etc. It was nice to see many familiar faces.

Model Agnostic AI Implementation

The topic of my presentation was “Model Agnostic AI Implementation.“
Synopsis: Choosing a particular AI Model or API can be time-consuming and challenging when designing and implementing an AI solution. Even Azure AI Foundry has over 1800+ AI models, which can be overwhelming. At a point in time, the model’s performance and benchmarking can also change significantly. What might look like the best option now can be superseded by another model in six months with better accuracy and lower transaction costs.
By implementing and extending the Azure AI Model Inference API, we can eliminate the need to select the perfect model upfront and avoid analysis paralysis. In the session, I will dive deep into this API abstraction to speed up the Model selection and implementation process. Later, if we want to change the model for specific scenarios, we will be able to do so without impacting the rest of the applications or systems.

If you are interested in the slides, here is the Link – slides from the presentation

Common Questions

AI Adoption and Strategy:

Several attendees were keen to understand the AI strategy and the steps. Below is the summarised version of what we have discussed. However, Microsoft Cloud Adoption Framerok has a detailed checklist and steps an organisation can follow: https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/scenarios/ai/

Benchmarking

During the session and 1:1 chat afterwards, many people had questions about benchmarking models, including how to independently verify models and where to find benchmarking information. As shown during the presentation, Azure AI Foundry has benchmarking of many popular models, and we can compare the quality/cost based on the tasks/skills.
You can create an account (free) and log in to explore the ranking: https://ai.azure.com/explore/benchmarks
You can learn more about benchmarking here: https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/model-benchmarks

There are countless other sites where you will find benchmarking information about different models. It will boil down to the specific use case you are trying to solve. Some similar sites are:

Learning and Career Path

Several students and technical folks were interested in how they could learn, sharpen their AI skills to be more industry-aligned and help them with AI-related jobs or projects. Some helpful links are below:

Community Posts

If you want to get in touch with some of the other speakers or organisers, visit these LinkedIn posts