How to setup Ollama models
Some time ago I partially switched from using cloud based managed LLMs to local ones. The reasons were few:
Running models on my own hardware.
I wanted to utilize my Macbook Pro M2's unified memory (memory shared between the CPU and GPU) which is fantastic for running LLMs. Hats off to Apple for managing to make Apple Silicon products so power-efficient, fast and in many cases not so expensive hardware. This includes for example the latest Mac mini M4 which could be an interesting choice for running LLMs in your homelab. I also found framework called EXO that allows you to interconnect different devices (e.g. MacBook, Mini PC, Raspberry Pi, RTX GPU running on Windows, etc.) together to create a cluster capable of running even the biggest available open-source LLMs.
I think the exclusivity of owning Apple hardware is slowly fading away as they are releasing more and more affordable hardware with great performance and power efficiency. In AI related cases in particular, it feels like you are not overpaying just because of the brand but you are actually getting a good value for your money. Or at least I hope they continue this trend.
Running models offline.
Another reason was having an ability to run a LLM without needing internet connection. Whenever I’m travelling by train through the zones that don’t have network connection (and probably never will) or when I have an inconvenience of having to use public Wi-Fi (which I never do), I can quickly spin up local LLM and use it.
Having more control over the models.
This includes being able to choose the model size and parameters. Not all models are equally good at different tasks, sometimes you need a smaller and faster model.