home_stack/scripts/README-models.md

Model Setup Scripts

This directory contains scripts to download and set up AI models for the Ollama service.

Available Scripts

1. pull-deepseek-model.sh

Main script that downloads the DeepSeek-R1-0528-Qwen3-8B model in GGUF format from Hugging Face and loads it into Ollama.

Features:

• Downloads the Q4_K_XL quantized version (optimal balance of quality and size)
• Creates a proper Modelfile with appropriate templates and parameters
• Loads the model into Ollama with the name deepseek-r1-qwen3-8b
• Includes verification and testing steps

Requirements:

• huggingface-hub Python package (will be installed automatically if missing)
• Ollama container running
• Internet connection

Usage:

# Using Makefile (recommended)
make setup-ollama          # Start Ollama and pull model automatically
make pull-deepseek-model   # Pull model only (requires Ollama to be running)

# Or run directly
./scripts/pull-deepseek-model.sh

2. pull-model-simple.sh

Alternative script that attempts to pull models using ollama pull command.

Usage:

./scripts/pull-model-simple.sh

Makefile Targets

The main Makefile includes these Ollama-related targets:

• make ollama-up - Start only the Ollama container
• make ollama-down - Stop only the Ollama container
• make ollama-logs - View Ollama container logs
• make pull-deepseek-model - Download and load DeepSeek model
• make setup-ollama - Complete setup (start Ollama + pull model)

Model Information

DeepSeek-R1-0528-Qwen3-8B-GGUF

• Source: unsloth/DeepSeek-R1-0528-Qwen3-8B-GGUF
• Quantization: Q4_K_XL (4-bit quantization, extra large)
• Size: ~5.5GB
• Context Length: 32,768 tokens
• Use Case: General conversation, reasoning, coding assistance

Quick Start

1. Start the entire stack:

   make up
   

2. Or start just Ollama and set up the model:

   make setup-ollama
   

3. Use the model:

   # Via Docker
   docker exec ollama-server ollama run deepseek-r1-qwen3-8b "Hello, how are you?"
   
   # Via API
   curl http://localhost:11434/api/generate -d '{
     "model": "deepseek-r1-qwen3-8b",
     "prompt": "Hello, how are you?",
     "stream": false
   }'
   

Troubleshooting

1. Script fails with "Ollama not running":

   • Make sure Ollama container is started: make ollama-up
   • Wait a few seconds for the service to be ready

2. Download fails:

   • Check internet connection
   • Verify Hugging Face Hub access
   • Try installing huggingface-hub manually: pip install huggingface-hub

3. Model loading fails:

   • Check if the GGUF file was downloaded completely
   • Verify the Modelfile syntax
   • Check Ollama container logs: make ollama-logs

4. Out of disk space:

   • The model file is ~5.5GB, ensure sufficient free space
   • Consider using a smaller quantization (modify the script)

Customization

You can modify the scripts to download different models or quantizations:

1. Change model variant: Edit MODEL_FILE in pull-deepseek-model.sh
2. Adjust parameters: Modify the Modelfile template in the script
3. Use different model: Change MODEL_REPO to any GGUF model on Hugging Face

File Locations

• Models: ./ai_stack/ollama/models/
• Modelfiles: ./ai_stack/ollama/models/Modelfile.*
• Ollama data: Docker volume ollama_data