Running LLMs locally with ollama

ChatGPT has become indispensable for a lot of technology professionals. For some, it has replaced search engines as a means of gaining knowledge and insights. For a number of possible reasons, you may want to run this tech locally.

I recently came across ollama and it piqued my interest.

Pre requisites #

Running LLMs may not work on every system. I’ve successfully been able to run ollama with llama3.1 model on a Macbook Pro M3 with 18 GB RAM as well as on a Fedora 41 laptop with Ryzen 7 and 64 GB RAM. When trying multiple models, memory was definitely a limiting factor and the mac started to use quite a bit of swap.

Installing ollama #

Installation of ollama depends on your device’s operating system.

For macOS, download the zip from the ollama download page, extract it and move the extracted Ollama.app to the Applications folder.

For Windows, simply download the exe from the ollama download page and execute it.

For Linux, execute the following:

curl -fsSL https://ollama.com/install.sh | sh

This will install ollama in /usr/local/bin.

Running ollama #

The most popular LLM is llama3. llama3.1 is a close second and is most likely what you want. You can run it by simply typing in ollama run llama3.1. You will get a chat like prompt as soon as it starts. Ask it some random stuff and see what it has to offer.

$ ollama run llama3.1
>>> What is the weather like in the United Kingdom?
The weather in the United Kingdom (UK) is known for being quite changeable and unpredictable, even at the same time of day...
>>> Send a message (/? for help)

You now have an offline LLM that can answer more questions than most humans.