Simulation for improvement

If you have been following along with my robot car guided by AI project you’ll know that I got it working with Gemini. I then switched to using Azure AI Foundry. Finally, I changed the code so that car ran most of the local decisions while the AI just made the major navigation decisions.

Each time navigation did improve, but I would have to observe the movements and tell the AI what issues I’d seen and then the Ai would go off and improve the code. I’d then test and update again, round and round in a loop. Each time there was a small incremental change in navigation ability but I still faced edge cases the existing algorithm struggled with.

I could have continued this update, observe, update loop infinitum, making constant incremental improvements. However, when I stepped back and though about how AI systems improve I realised that it was via repetitive learning in a simulated, rather than real environment.

I therefore asked Github Copilot to simulate the algorithm of the robot cars navigation and then test that via simulation. Any improvements learned via the simulation should then be applied back to the robot cars code. I then unleashed Github Copilot to complete this task, without asking for input, until it reached a near prefect navigation result in the simulator.

The result is this code:

https://github.com/directorcia/Azure/blob/master/Iot/LLM/llm-nav-max.ino

which I will say, although far from ‘perfect’ achieved much better results than the incremental ones I was obtaining simply by observational feedback.

The success I’ve had here using the concept of a simulation to test the algorithm and then continue to iterate to improve the code has made me think about what else I could apply this ‘simulation’ process to when other AI work I am performing.

Interesting.

Having the AI make make better decisions

After some Ai prompting it was suggested that my robot car navigation would improve if the car did more itself with local routines and only used the AI when it could not decide so I plugged away and ended up with this code:

https://github.com/directorcia/Azure/blob/master/Iot/LLM/llm-nav-max.ino

you will see:

An ESP32 robot car drives forward until its ultrasonic sensor (mounted on a pan servo) detects an obstacle. On each hazard event a two-tier decision pipeline fires:

1. LOCAL PLANNER – scores three candidate maneuvers (strafe, 90° turn, and backup+turn) using live L/F/R distance readings, front-clearance trend, and oscillation history. If the result is “obvious” (high confidence, low risk, no repeated-trap condition) it is executed immediately without any network round-trip.

2. FOUNDRY ARBITER – when the local answer is ambiguous, or a repeated-trap is detected, a structured prompt is posted to an Azure AI Foundry Responses API endpoint. The LLM names one candidate plan and returns a confidence score. The response is validated and sanitized; any failure falls back to the local plan.

After every maneuver the robot re-scans, records the front-clearance delta, and feeds plan quality (improved / worsened) back into the next decision cycle.

After a quick test, navigation does seem more intelligent, although it seems to make decisions a long way away and tends to get lost in wide open spaces with object far away. However, I think that if I nvested more time I could improve all that.

I have an additional idea on how I might improve the navigation in an upcoming article, but for now I think I’m pretty much done with the concept of navigation.

Connecting the robot car to Azure AI Foundry

If you have been following along you will know that I’ve connected a robot car to both a local LLM and cloud based Gemini. The last iteration is here:

https://blog.ciaopslabs.com/2026/07/18/controlling-a-robot-car-with-ai/

I decided that I should connect the car to Azure AI Foundry because there are so many more models available as well as everything else that comes with Azure.

So I set up a simple Foundry project

Step 1: Create an Azure AI Foundry Project

Browse to Azure AI Foundry:
Azure AI Foundry
Sign in with your Azure account.
Select:
- Create Project
- Project Name:
```
RobotNavigation
```
Create or select:
- Azure Subscription
- Resource Group
- Azure AI Services Resource
Wait for deployment to complete.

Step 2: Deploy a Model

Within your Foundry project:

Open:
```
Model Catalog
```
Deploy:
- GPT-5-mini
- GPT-5.1-mini
- GPT-4.1-mini

For a robot car:

GPT-5-mini

is usually sufficient and inexpensive.

Step 3: Obtain Connection Details

From Foundry copy:

Endpoint URL

API Key

Model Name

C++

#define FOUNDRY_RESPONSES_URL \

“https://robot-navigation-resource.services.ai.azure.com/openai/v1/responses”

#define FOUNDRY_MODEL “gpt-5-mini”

The recommendation was the use the gpt-5-mini model, so I plugged it into the existing code, made a few improvements and ended up with this:

https://github.com/directorcia/Azure/blob/master/Iot/LLM/llm-foundry.ino

My observation is that the navigate is generally better but the delays are longer when it has to ‘thinlk’ (aka go to the LLM). This has to do with the size of model, basically gpt-5-mini vs gemini3-flash.

So, the lesson here is I need the smallest possible model for the job. For now I’ll stick with gpt-5-mini.

So more research indicates that I shoudl probably offload more processes to the local device and only the LLM at a much higher level. A better plan seems to bei instead of asking the LLM to invent a manoeuvre plan, ask it to choose from local candidate plans you already created.

So let me go and try that now.

Controlling a robot car with AI

After having AI show my fortune, the next project was having AI navigate my robot car.

To keep things simple I constrained movement to a 5 x 5 grid ( X = 0 – 4 and Y = 0 – 4) and the car could only move N, S, E or W one cell at a time. To get the algorithm right and test this with Ai before actually applying it to the car I simulated the result on the OLED screen I had previously configured.

With that all working I upgraded the code to run with the robot car and you’ll find it here:

https://github.com/directorcia/Azure/blob/master/Iot/LLM/llm-cellmove.ino

The main issue I found was more mechanical with the robot car wheels dragging and not being very precise, so the car woudl easily wander off in different directions. This has more to do with teh quality of the motors and wheels as well as the friction encountered when the wheels start moving. However, aside from that the test worked successfully and the car cycled up the grid and then back down.

Next, I wanted the AI to actually assist with the navigation of the car. I went through plenty of iterations with this. The most important change is that I moved from using a local AI to using Gemini via API calls. The main reason for this was simply speed. As the prompts became larger the local AI model struggled to return the results to the car in enough time to implement effective navigation. I had also wanted to integrate large cloud based LLM so here was the opportunity, so I hooked up Gemini.

The robot car also has an ultrasonic senor connected to a motor at the front so I could sweep it left and right as well for better object detection. However, initially I kept it simple and all on teh car by just using the ultrasonic sensor to detect hazards and try to makeover around them. The code for that is here:

https://github.com/directorcia/Azure/blob/master/Iot/LLM/llm-sweep.ino

I then upgraded the code to integrate the LLM into the navigation process by making decisions on which direction to turn. That code is here:

https://github.com/directorcia/Azure/blob/master/Iot/LLM/llm-sweep.ino

One issues I ran into, that wasted a lot of my time and was totally my own fault, was when I started having issues with the wheel moving the car forward. I blamed the code but in fact, again it was a hardware issue, being the battery charge had become too low to actually drive the wheels acceptably. It is interesting at how quickly that car now actually drains power when fully running.

With the power issues resolved I upgraded the code a number of times to allow the LLM a much higher level of navigation control. You’ll find the final result here:

https://github.com/directorcia/Azure/blob/master/Iot/LLM/llm-assist-nav.ino

The end result of all these experiments is that I have learned that in the full configuration the car now burns a lot of power as it moves, turns the sensor, communicates over wifi and more. all the changes I made to the code would make the car slightly less likely to crash into objects on the floor but a lot more though needs to go into ‘crash free’ navigation. The obvious improvement is to add more sensors to provide the LLM with more information to make better decisions. I also found that the wheel on the car are not precise enough and don’t really provide the best grip. This means they tend to be slow to engage and lag cause the car to veer.

I think all of these can be solved iteratively over time and I am confident that I can get to a situation that allows the robot car to move pretty much crash free around the floor just like a robot vacuum can already. However, the time required is probably not something that I’m willing to invest in just now to get a little incrementally better. I’m happy that my ‘proof of concept’ when it comes to navigation with LLMs works. I think it is time to move onto the next project.

Having the device show my fortune

After getting the ESP32 talking to the local LLM the next stage was to do something more than just flashing an LED. I decided that I’d use the LLM to produce a ‘fortune’ for me and then display that on an OLED screen I’d connect to the ESP32.

The OLED screen in question was this White I2C OLED display (SSD1306).

White I2C OLED display (SSD1306)

To use this OLED you need the Adafruit_SSD1306 library.

Here is the prompt being sent to the local LLM:

You are a mystical fortune teller. Give one short fortune. Maximum 12 words. No introduction. No quotes.

The result from the LLM is then displayed on the OLED screen which is connected to the ESP32-C3-DevKitM-1 via GPIO6 and GPIO7 acting as SDA and SCL communication ports. I also left the external LED on GPIO4, from the last project, as well to aid troubleshooting.

The code is here:

https://github.com/directorcia/Azure/blob/master/Iot/LLM/llm-fortune.ino

and the results look like:

Video URL = ESP32 displaying results from local LLM

Getting device talking to LLM

With the LLM now up and working on a separate device on my LAN, the next step is to test it remotely to ensure that it works. For this I used the following simple PowerShell on a remote machine

which you will find here:

https://github.com/directorcia/Azure/blob/master/Iot/LLM/echo-ping.ps1

Next, I ran the following PowerShell script:

https://github.com/directorcia/Azure/blob/master/Iot/LLM/echo-test.ps1

which simply runs a standard prompt of;

“Reply with exactly: Hello from Ollama”

and then ensure that I get that reply back from the remote LLM server. This means I have communications to the actual server as well as the LLM.

With all the remote communications confirmed, the next step was to get a device talking to the LLM. For this I had a ESP32-C3-DevKitM-1 hanging around.

The main benefit of this device is that it has inbuilt Wifi. I connected up a LED and resistor to GPIO4 like so:

I then used this code in the device:

https://github.com/directorcia/Azure/blob/master/Iot/LLM/llm-flash.ino

to prompt the LLM for a number of flashes from 1 – 4, which the device would then complete that on the LED. I could also monitor the progress using the terminal, which would look like:

Connecting to Wi-Fi…..
Wi-Fi connected. IP: 192.168.1.42
Requested flash count: 3
Sending request to Ollama (attempt 1)…
HTTP status: 200
Ollama reply: {“flash”:3}

Initially I found that the LLM was returning the same number of flashes, so I needed to adjust the prompting to get some variation. The good news is that I got it all working and the resultant code is above.

So now I have successfully gotten a device talking to a local LLM. I’ll be expanding on this in upcoming articles but very happy that have this basic configuration all working now.