/
How to install a local LLM with Ollama

How to install a local LLM with Ollama

Why Install a Local LLM?

Privacy & Security

  • Local data: When you run an LLM locally, all the data you input and the model's output stays on your computer.

  • Reduced risk of data breaches: Since nothing is transmitted to an external server, there's no exposure if a third-party provider is compromised.

Cost Saving

  • No monthly charge: Open-access LLMs are free to run once installed.

  • No API usage fees: Cloud-based LLM APIs typically charge per token processed; running locally eliminates this cost entirely.

  • One-time hardware investment: As long as your computer can handle the installation, no additional hardware is required.

Performance & Control

  • Faster response time: Since the LLM runs locally, there's no network latency.

  • Customization and fine-tuning: You have full control over the model and can customize it (this requires technical expertise).

  • Offline access: You can use the LLM without an internet connection.

Enhanced Experimentation & Learning

  • Hands-on learning: Running a local LLM is a great way to practice and learn how models work.

  • Research & development: A local setup offers great flexibility for experimentation and development.

How to Install

This guide uses Ollama, a free, open-source tool that makes it simple to download and run LLMs on your own computer.

Before you start: check your system requirements

Model size determines how much RAM and disk space you'll need. As a rough guide:

Model size

RAM needed

Disk space

Model size

RAM needed

Disk space

~7–9B parameters

8 GB minimum

~5 GB

~13B parameters

16 GB minimum

~8 GB

30B+ parameters

32 GB+

20 GB+

If you're unsure, start with a smaller model (7–9B) and size up only if your computer handles it comfortably.

Step 1: Download Ollama

Go to the Ollama GitHub page and select the installer for your operating system (macOS, Windows, or Linux).

Step 2: Run the installer

Follow the on-screen instructions to complete installation.

  • macOS/Linux: Open Terminal.

  • Windows: Open Command Prompt or PowerShell.

Step 3: Choose a model to run

Ollama hosts a library of open-access models you can browse at ollama.com/library. A few good starting points:

Model

Good for

Approx. size

Model

Good for

Approx. size

llama3

General-purpose use

~4.7 GB

gemma3

Lightweight, efficient everyday use

~5 GB

mistral

Strong performance for its size

~4.1 GB

Step 4: Install and run the model

In your terminal, type the run command for your chosen model and press Enter. For example:

ollama run gemma3

This will automatically download and install the model the first time you run it. Installation size varies by model — double-check the table above to confirm you have enough RAM and disk space before proceeding.

Step 5: Confirm it's working

Once installation finishes, you'll see a prompt inside the terminal. Type a test message (e.g., "Hello, can you introduce yourself?") and press Enter. If the model responds, you're up and running.

Step 6: Chat with your local LLM

You can now use the model directly in the terminal just like you would with ChatGPT in a browser.

Useful commands

  • Exit the chat: type /bye or press Ctrl+D

  • List installed models: ollama list

  • Remove a model to free up space: ollama rm <model-name>

Troubleshooting

If installation fails or you run into errors, check the Ollama GitHub issues page or its documentation for common fixes.