I finally gave local AI a real shot — and it surprised me

For a long time, I’ve been hesitant to actually use AI in my daily routine. It’s not because I don’t see the appeal — I do. The problem has always been trust. The idea of handing my most personal data over to some cloud service, where it gets processed on servers I don’t control and might be used for things I never intended, has been a major mental block for me. But the technology has been shifting. It’s becoming more and more realistic to run powerful AI models locally, right on your own computer, without sending anything to the cloud. That changes the equation.

So, I’ve finally decided to take the plunge. I’m starting to actually use local AI on a regular basis, and I’m documenting what happens. I’m not an AI expert by any means. This is going to be a learning experience for me, and honestly, there’s a good chance I stumble along the way. But I figure if I’m this skeptical and privacy-conscious, there are probably a lot of other people who feel the same way. Maybe my journey can help you decide whether it’s worth your time and money.

The timing feels right. Apple has been pushing its new Mac desktops as capable local AI machines, and there’s a whole wave of Windows laptops and desktops with Nvidia’s new RTX Spark hardware landing soon. Some of those machines come with up to 128GB of RAM and are specifically designed to run agentic AI. Local AI and AI agents are having a moment. I get why. The idea of having a little digital assistant that lives entirely on the box on my desk — one that answers only to me, not to OpenAI, Google, Microsoft, or Anthropic — has a different kind of appeal than subscribing to a chatbot or trusting a cloud service. Don’t get me wrong. I’m not going to have it write articles for me, edit my work, or generate ugly fliers. But with privacy on my side, I thought maybe I could finally find some genuinely useful busywork to trust it with.

As part of my ongoing testing of the M5 Ultra Mac Studio, plus some experiments on other Mac and Windows systems, I’ve installed a few local large language models and started tinkering. Here’s how it’s going so far.

Getting started with Hermes and Qwen

I started by installing the open-source Hermes Agent on the Mac Studio. Hermes is a self-hosted AI agent desktop app that works on macOS, Windows, and Linux. It’s completely free to use if you stick with local LLMs. My Mac Studio has 256GB of unified memory, so I can run just about any model I want. That’s both exciting and overwhelming. There are more models available than I can count, and many of them have specialized uses. As a general rule, models with more billions of parameters are capable of doing more than smaller ones with only a few billion. Since I don’t have to worry about paying per token when running a model locally, I decided to start big.

Using Hermes’ straightforward onboarding interface and its built-in model picker, I chose Qwen 3.8 Flash Next — a 125-billion parameter model that weighs in at around 105GB. I also plan to spend time with much smaller Qwen models on other devices, including an M6 Mac Mini, an M5 MacBook Air, an Asus TUF Gaming A14 with an AMD Strix Halo processor, and eventually an RTX Spark machine. If you have your own favorite local models and hardware, I’d love to hear about them in the comments.

The setup process was surprisingly smooth. It didn’t take long to get Hermes and Qwen up and running, and I even connected it to a Telegram bot so I can control it from my phone. That part felt great. But then I hit my usual nemesis when it comes to AI: the text box. There I was, staring at a blank prompt, thinking, “What do I actually do with this thing?”

A humble start: the daily briefing

To get things moving, I followed some helpful advice I found on YouTube. The suggestion was simple: start small. I asked Hermes to create a daily morning briefing for me using a scheduled cron job. It scans my email and calendar, tells me anything that needs immediate attention, and gives me a short weather report. Honestly, it’s not the most useful thing in the world, and you certainly don’t need a $12,000 computer to build something like it. But it’s a starting point, and it’s something I can develop over time.

There were some hiccups along the way. The briefing kept failing to generate at first. I couldn’t figure out why until I realized that macOS was asleep when the job was supposed to fire at 7:30 in the morning. Duh. Once I fixed that, it started working like clockwork. Now I’m trying to think of more information I can feed into it to make the results actually helpful.

Sorting hundreds of Steam games

The next task was more genuinely useful. I have more than 400 games on Steam, and I’ve been meaning to reorganize my library for a while. Steam doesn’t categorize your games for you. You have to do it manually, one game at a time, which is tedious and easy to put off forever. This seemed like exactly the kind of busywork an AI agent could handle.

I asked Hermes, “Can you reorganize my Steam library?” It was able to see most of my games by simply looking at the already installed Steam client on my computer. It quickly came up with several organization options. I decided to have it sort everything by genre, while preserving the custom categories I already made — things like favorites, co-op titles, party games, and games I play with my wife.

Of course, giving an AI agent the ability to take actions on your machine means giving it some permissions. In this case, I had to register a Steam Web API key and hand it over to Hermes. Once it had access, it only took a few minutes to churn through all those games and sort them into categories. The final result was actually pretty good. My Steam library is now clean and easy to browse. And once it was done, I revoked the API key, so I don’t have to worry about that access lingering.

Keeping sensitive data on my machine

Another simple but valuable thing I did was data analysis. I had Hermes look at some financial records and crunch numbers for me. I also asked it to create a spec comparison spreadsheet for a new laptop I was considering. These are small tasks, but they’re both things I would not trust a cloud service with. I don’t want to upload sensitive financial data to some remote server. And in the case of the laptop, that information was under embargo, so keeping it on my own computer was absolutely essential. That’s the whole point of local AI for me. If I had to send that data anywhere else, I wouldn’t have used the tool at all. The fact that everything stayed on my machine made the difference between using it and skipping it.

Automating benchmark tests

The bigger project I’m working on now is automating a series of laptop benchmark tests. I run all of our benchmarks manually, and it’s incredibly time-consuming. A typical laptop needs to run a dozen or so different tests, and each one has to be run three times to take the average. Tending to that process for hours is exhausting, and I’d love to automate at least some of it.

So far, that’s been a work in progress. Even walking Hermes through all the required procedures and getting it to write Python scripts for me has been something of an endeavor. It’s capable of producing code, but it still needs a lot of hand-holding. I’m hoping to make some positive headway, but I’m not there yet.

Early lessons and limitations

It’s still early days with Hermes, and I’m hoping this is just the tip of the iceberg when it comes to useful things I can get it to do. But I’ve also learned that even powerful local models don’t magically turn into a flawless assistant that can do anything. Things still break. My daily briefing has already failed multiple times, and I’ve had to troubleshoot more than I expected.

Still, I view Hermes and local AI as tools — computer programs that run on my computer. I don’t want a digital friend. I’m not saying “thank you” or “please,” and I’m definitely not typing “sorry” when I change my mind about something. I’m as stone-cold and indifferent when ordering it around as I am when clicking a mouse through an operating system. It’s a program, not a person.

I’m also trying to remain vigilant and cautious as I continue testing and learning. Local AI has real potential, and the privacy benefits are genuinely compelling. But it’s still a tool, and it still has limits. I’m curious to see where this journey takes me, and I’ll keep posting about what I find along the way.