{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2vQXvnUUsTzI"
      },
      "source": [
        "To run this, press \"*Runtime*\" and press \"*Run all*\" on a **free** Tesla T4 Google Colab instance!\n",
        "<div class=\"align-center\">\n",
        "<a href=\"https://unsloth.ai/\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png\" width=\"115\"></a>\n",
        "<a href=\"https://discord.gg/unsloth\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/Discord button.png\" width=\"145\"></a>\n",
        "<a href=\"https://unsloth.ai/docs/\"><img src=\"https://github.com/unslothai/unsloth/blob/main/images/documentation%20green%20button.png?raw=true\" width=\"125\"></a> Join Discord if you need help + ⭐ <i>Star us on <a href=\"https://github.com/unslothai/unsloth\">Github</a> </i> ⭐\n",
        "</div>\n",
        "\n",
        "To install Unsloth on your local device, follow [our guide](https://unsloth.ai/docs/get-started/install). This notebook is licensed [LGPL-3.0](https://github.com/unslothai/notebooks?tab=LGPL-3.0-1-ov-file#readme).\n",
        "\n",
        "You will learn how to do [data prep](#Data), how to [train](#Train), how to [run the model](#Inference), & how to save it"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7j01DfVgsTzJ"
      },
      "source": [
        "### News"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6dT42nHksTzJ"
      },
      "source": [
        "Introducing **Unsloth Studio** - a new open source, no-code web UI to train and run LLMs. [Blog](https://unsloth.ai/docs/new/studio) • [Notebook](https://colab.research.google.com/github/unslothai/unsloth/blob/main/studio/Unsloth_Studio_Colab.ipynb)\n",
        "\n",
        "<table><tr>\n",
        "<td align=\"center\"><a href=\"https://unsloth.ai/docs/new/studio\"><img src=\"https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F~%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FxV1PO5DbF3ksB51nE2Tw%252Fmore%2520cropped%2520ui%2520for%2520homepage.png%3Falt%3Dmedia%26token%3Df75942c9-3d8d-4b59-8ba2-1a4a38de1b86&width=376&dpr=3&quality=100&sign=a663c397&sv=2\" width=\"200\" height=\"120\" alt=\"Unsloth Studio Training UI\"></a><br><sub><b>Train models</b> — no code needed</sub></td>\n",
        "<td align=\"center\"><a href=\"https://unsloth.ai/docs/new/studio\"><img src=\"https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F~%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FRCnTAZ6Uh88DIlU3g0Ij%252Fmainpage%2520unsloth.png%3Falt%3Dmedia%26token%3D837c96b6-bd09-4e81-bc76-fa50421e9bfb&width=376&dpr=3&quality=100&sign=c1a39da1&sv=2\" width=\"200\" height=\"120\" alt=\"Unsloth Studio Chat UI\"></a><br><sub><b>Run GGUF models</b> on Mac, Windows & Linux</sub></td>\n",
        "</tr></table>\n",
        "\n",
        "Train MoEs - DeepSeek, GLM, Qwen and gpt-oss 12x faster with 35% less VRAM. [Blog](https://unsloth.ai/docs/new/faster-moe)\n",
        "\n",
        "Ultra Long-Context Reinforcement Learning is here with 7x more context windows! [Blog](https://unsloth.ai/docs/new/grpo-long-context)\n",
        "\n",
        "New in Reinforcement Learning: [FP8 RL](https://unsloth.ai/docs/new/fp8-reinforcement-learning) • [Vision RL](https://unsloth.ai/docs/new/vision-reinforcement-learning-vlm-rl) • [Standby](https://unsloth.ai/docs/basics/memory-efficient-rl) • [gpt-oss RL](https://unsloth.ai/docs/new/gpt-oss-reinforcement-learning)\n",
        "\n",
        "Visit our docs for all our [model uploads](https://unsloth.ai/docs/get-started/unsloth-model-catalog) and [notebooks](https://unsloth.ai/docs/get-started/unsloth-notebooks)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K7fgQkATsTzK"
      },
      "source": [
        "### Installation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vA7IKFdUsTzK"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "import os, re\n",
        "if \"COLAB_\" not in \"\".join(os.environ.keys()):\n",
        "    !pip install unsloth  # Do this in local & cloud setups\n",
        "else:\n",
        "    import torch; v = re.match(r'[\\d]{1,}\\.[\\d]{1,}', str(torch.__version__)).group(0)\n",
        "    xformers = 'xformers==' + {'2.10':'0.0.34','2.9':'0.0.33.post1','2.8':'0.0.32.post2'}.get(v, \"0.0.34\")\n",
        "    !pip install sentencepiece protobuf \"datasets==4.3.0\" \"huggingface_hub>=0.34.0\" hf_transfer\n",
        "    !pip install --no-deps unsloth_zoo bitsandbytes accelerate {xformers} peft trl triton unsloth\n",
        "!pip install --no-deps git+https://github.com/huggingface/transformers.git\n",
        "!pip install torchcodec\n",
        "import torch; torch._dynamo.config.recompile_limit = 64;"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Mp4i13PHsTzK"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "!pip install --no-deps --upgrade timm # For Gemma 4 vision/audio"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GFOEZbP7ONMs"
      },
      "source": [
        "### Unsloth"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QmUBVEnvCDJv"
      },
      "outputs": [],
      "source": [
        "from unsloth import FastVisionModel # FastLanguageModel for LLMs\n",
        "import torch\n",
        "\n",
        "# 4bit pre quantized models we support for 4x faster downloading + no OOMs.\n",
        "fourbit_models = [\n",
        "    # Gemma 4 models\n",
        "    \"unsloth/gemma-4-E2B-it\",\n",
        "    \"unsloth/gemma-4-E2B\",\n",
        "    \"unsloth/gemma-4-31B-it\",\n",
        "    \"unsloth/gemma-4-E4B\",\n",
        "    \"unsloth/gemma-4-31B-it\",\n",
        "    \"unsloth/gemma-4-31B\",\n",
        "    \"unsloth/gemma-4-26B-A4B-it\",\n",
        "    \"unsloth/gemma-4-26B-A4B\",\n",
        "] # More models at https://huggingface.co/unsloth\n",
        "\n",
        "model, processor = FastVisionModel.from_pretrained(\n",
        "    \"unsloth/gemma-4-E2B-it\",\n",
        "    load_in_4bit = True, # Use 4bit to reduce memory use. False for 16bit LoRA.\n",
        "    use_gradient_checkpointing = \"unsloth\", # True or \"unsloth\" for long context\n",
        ")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SXd9bTZd1aaL"
      },
      "source": [
        "We now add LoRA adapters for parameter efficient fine-tuning, allowing us to train only 1% of all model parameters efficiently.\n",
        "\n",
        "**[NEW]** We also support fine-tuning only the vision component, only the language component, or both. Additionally, you can choose to fine-tune the attention modules, the MLP layers, or both!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6bZsfBuZDeCL"
      },
      "outputs": [],
      "source": [
        "model = FastVisionModel.get_peft_model(\n",
        "    model,\n",
        "    finetune_vision_layers     = True, # False if not finetuning vision layers\n",
        "    finetune_language_layers   = True, # False if not finetuning language layers\n",
        "    finetune_attention_modules = True, # False if not finetuning attention layers\n",
        "    finetune_mlp_modules       = True, # False if not finetuning MLP layers\n",
        "\n",
        "    r = 32,                           # The larger, the higher the accuracy, but might overfit\n",
        "    lora_alpha = 32,                  # Recommended alpha == r at least\n",
        "    lora_dropout = 0,\n",
        "    bias = \"none\",\n",
        "    random_state = 3407,\n",
        "    use_rslora = False,               # We support rank stabilized LoRA\n",
        "    loftq_config = None,               # And LoftQ\n",
        "    target_modules = \"all-linear\",    # Optional now! Can specify a list if needed\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "Ydo-UuV92Xei"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vITh0KVJ10qX"
      },
      "source": [
        "<a name=\"Data\"></a>\n",
        "### Data Prep\n",
        "We'll use a sampled dataset of handwritten math formulas. The objective is to convert these images into a computer-readable format—specifically LaTeX—so they can be rendered. This is particularly useful for complex expressions.\n",
        "\n",
        "You can access the dataset [here](https://huggingface.co/datasets/unsloth/LaTeX_OCR). The full dataset is [here](https://huggingface.co/datasets/linxy/LaTeX_OCR)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LjY75GoYUCB8"
      },
      "outputs": [],
      "source": [
        "from datasets import load_dataset\n",
        "dataset = load_dataset(\"unsloth/LaTeX_OCR\", split = \"train\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W1W2Qhsz6rUT"
      },
      "source": [
        "Let's take an overview of the dataset. We'll examine the second image and its corresponding caption."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bfcSGwIb6p_R"
      },
      "outputs": [],
      "source": [
        "dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uOLWY2936t1n"
      },
      "outputs": [],
      "source": [
        "dataset[2][\"image\"]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lXjfJr4W6z8P"
      },
      "outputs": [],
      "source": [
        "dataset[2][\"text\"]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rKHxfZua1CrS"
      },
      "source": [
        "We can also render LaTeX directly in the browser!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nPopsxAC1CrS"
      },
      "outputs": [],
      "source": [
        "from IPython.display import display, Math, Latex\n",
        "\n",
        "latex = dataset[3][\"text\"]\n",
        "display(Math(latex))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K9CBpiISFa6C"
      },
      "source": [
        "To format the dataset, all vision fine-tuning tasks should follow this format:\n",
        "\n",
        "```python\n",
        "[\n",
        "    {\n",
        "        \"role\": \"user\",\n",
        "        \"content\": [\n",
        "            {\"type\": \"text\", \"text\": instruction},\n",
        "            {\"type\": \"image\", \"image\": sample[\"image\"]},\n",
        "        ],\n",
        "    },\n",
        "    {\n",
        "        \"role\": \"user\",\n",
        "        \"content\": [\n",
        "            {\"type\": \"text\", \"text\": instruction},\n",
        "            {\"type\": \"image\", \"image\": sample[\"image\"]},\n",
        "        ],\n",
        "    },\n",
        "]\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oPXzJZzHEgXe"
      },
      "outputs": [],
      "source": [
        "instruction = \"Write the LaTeX representation for this image.\"\n",
        "\n",
        "def convert_to_conversation(sample):\n",
        "    conversation = [\n",
        "        {\n",
        "            \"role\": \"user\",\n",
        "            \"content\": [\n",
        "                {\"type\": \"text\", \"text\": instruction},\n",
        "                {\"type\": \"image\", \"image\": sample[\"image\"]},\n",
        "            ],\n",
        "        },\n",
        "        {\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": sample[\"text\"]}]},\n",
        "    ]\n",
        "    return {\"messages\": conversation}\n",
        "pass"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FY-9u-OD6_gE"
      },
      "source": [
        "Let's convert the dataset into the \"correct\" format for finetuning:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gFW2qXIr7Ezy"
      },
      "outputs": [],
      "source": [
        "converted_dataset = [convert_to_conversation(sample) for sample in dataset]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ndDUB23CGAC5"
      },
      "source": [
        "The first example is now structured like below:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gGFzmplrEy9I"
      },
      "outputs": [],
      "source": [
        "converted_dataset[0]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MsRPBIb0JJ6c"
      },
      "source": [
        "Lets take the Gemma 4 instruction chat template and use it in our base model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "exoDVEvmJN-6"
      },
      "outputs": [],
      "source": [
        "from unsloth import get_chat_template\n",
        "\n",
        "processor = get_chat_template(\n",
        "    processor,\n",
        "    \"gemma-4\"\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FecKS-dA82f5"
      },
      "source": [
        "Before fine-tuning, let us evaluate the base model's performance. We do not expect strong results, as it has not encountered this chat template before."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vcat4UxA81vr"
      },
      "outputs": [],
      "source": [
        "FastVisionModel.for_inference(model)  # Enable for inference!\n",
        "\n",
        "image = dataset[2][\"image\"]\n",
        "instruction = \"Write the LaTeX representation for this image.\"\n",
        "\n",
        "messages = [\n",
        "    {\n",
        "        \"role\": \"user\",\n",
        "        \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": instruction}],\n",
        "    }\n",
        "]\n",
        "input_text = processor.apply_chat_template(messages, add_generation_prompt = True)\n",
        "inputs = processor(\n",
        "    image,\n",
        "    input_text,\n",
        "    add_special_tokens = False,\n",
        "    return_tensors = \"pt\",\n",
        ").to(\"cuda\")\n",
        "\n",
        "from transformers import TextStreamer\n",
        "\n",
        "text_streamer = TextStreamer(processor, skip_prompt = True)\n",
        "result = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 128,\n",
        "                        use_cache = False, temperature = 1.0, top_p = 0.95, top_k = 64)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FeAiMlQ71CrS"
      },
      "source": [
        "You can see it's absolutely terrible! It doesn't follow instructions at all"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "idAEIeSQ3xdS"
      },
      "source": [
        "<a name=\"Train\"></a>\n",
        "### Train the model\n",
        "Now let's train our model. We do 60 steps to speed things up, but you can set `num_train_epochs=1` for a full run, and turn off `max_steps=None`. We also support `DPOTrainer` and `GRPOTrainer` for reinforcement learning!!\n",
        "\n",
        "We use our new `UnslothVisionDataCollator` which will help in our vision finetuning setup."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "95_Nn-89DhsL"
      },
      "outputs": [],
      "source": [
        "from unsloth.trainer import UnslothVisionDataCollator\n",
        "from trl import SFTTrainer, SFTConfig\n",
        "\n",
        "FastVisionModel.for_training(model) # Enable for training!\n",
        "\n",
        "trainer = SFTTrainer(\n",
        "    model = model,\n",
        "    train_dataset = converted_dataset,\n",
        "    processing_class = processor.tokenizer,\n",
        "    data_collator = UnslothVisionDataCollator(model, processor),\n",
        "    args = SFTConfig(\n",
        "        per_device_train_batch_size = 1,\n",
        "        gradient_accumulation_steps = 4,\n",
        "        gradient_checkpointing = True,\n",
        "\n",
        "        # use reentrant checkpointing\n",
        "        gradient_checkpointing_kwargs = {\"use_reentrant\": False},\n",
        "        max_grad_norm = 0.3,              # max gradient norm based on QLoRA paper\n",
        "        warmup_ratio = 0.03,\n",
        "        max_steps = 60,\n",
        "        #num_train_epochs = 2,          # Set this instead of max_steps for full training runs\n",
        "        learning_rate = 2e-4,\n",
        "        logging_steps = 1,\n",
        "        save_strategy = \"steps\",\n",
        "        optim = \"adamw_torch_fused\",\n",
        "        weight_decay = 0.001,\n",
        "        lr_scheduler_type = \"cosine\",\n",
        "        seed = 3407,\n",
        "        output_dir = \"outputs\",\n",
        "        report_to = \"none\",             # For Weights and Biases\n",
        "\n",
        "        # You MUST put the below items for vision finetuning:\n",
        "        remove_unused_columns = False,\n",
        "        dataset_text_field = \"\",\n",
        "        dataset_kwargs = {\"skip_prepare_dataset\": True},\n",
        "        max_length = 2048,\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "2ejIt2xSNKKp"
      },
      "outputs": [],
      "source": [
        "# @title Show current memory stats\n",
        "gpu_stats = torch.cuda.get_device_properties(0)\n",
        "start_gpu_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n",
        "max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)\n",
        "print(f\"GPU = {gpu_stats.name}. Max memory = {max_memory} GB.\")\n",
        "print(f\"{start_gpu_memory} GB of memory reserved.\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yqxqAZ7KJ4oL"
      },
      "outputs": [],
      "source": [
        "trainer_stats = trainer.train()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "pCqnaKmlO1U9"
      },
      "outputs": [],
      "source": [
        "# @title Show final memory and time stats\n",
        "used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n",
        "used_memory_for_lora = round(used_memory - start_gpu_memory, 3)\n",
        "used_percentage = round(used_memory / max_memory * 100, 3)\n",
        "lora_percentage = round(used_memory_for_lora / max_memory * 100, 3)\n",
        "print(f\"{trainer_stats.metrics['train_runtime']} seconds used for training.\")\n",
        "print(\n",
        "    f\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\"\n",
        ")\n",
        "print(f\"Peak reserved memory = {used_memory} GB.\")\n",
        "print(f\"Peak reserved memory for training = {used_memory_for_lora} GB.\")\n",
        "print(f\"Peak reserved memory % of max memory = {used_percentage} %.\")\n",
        "print(f\"Peak reserved memory for training % of max memory = {lora_percentage} %.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ekOmTR1hSNcr"
      },
      "source": [
        "<a name=\"Inference\"></a>\n",
        "### Inference\n",
        "Let's run the model! You can modify the instruction and input—just leave the output blank.\n",
        "\n",
        "We'll use the best hyperparameters for inference on Gemma: `top_p=0.95`, `top_k=64`, and `temperature=1.0`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kR3gIAX-SM2q"
      },
      "outputs": [],
      "source": [
        "FastVisionModel.for_inference(model)  # Enable for inference!\n",
        "\n",
        "image = dataset[10][\"image\"]\n",
        "instruction = \"Write the LaTeX representation for this image.\"\n",
        "\n",
        "messages = [\n",
        "    {\n",
        "        \"role\": \"user\",\n",
        "        \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": instruction}],\n",
        "    }\n",
        "]\n",
        "\n",
        "input_text = processor.apply_chat_template(messages, add_generation_prompt = True)\n",
        "\n",
        "inputs = processor(\n",
        "    image,\n",
        "    input_text,\n",
        "    add_special_tokens = False,\n",
        "    return_tensors = \"pt\",\n",
        ").to(\"cuda\")\n",
        "\n",
        "from transformers import TextStreamer\n",
        "\n",
        "text_streamer = TextStreamer(processor, skip_prompt = True)\n",
        "result = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 128,\n",
        "                        use_cache = False, temperature = 1.0, top_p = 0.95, top_k = 64)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uMuVrWbjAzhc"
      },
      "source": [
        "<a name=\"Save\"></a>\n",
        "### Saving, loading finetuned models\n",
        "To save the final model as LoRA adapters, use Hugging Face’s `push_to_hub` for online saving, or `save_pretrained` for local storage.\n",
        "\n",
        "**[NOTE]** This ONLY saves the LoRA adapters, and not the full model. To save to 16bit or GGUF, scroll down!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "upcOlWe7A1vc"
      },
      "outputs": [],
      "source": [
        "model.save_pretrained(\"gemma_4_lora\")  # Local saving\n",
        "processor.save_pretrained(\"gemma_4_lora\")\n",
        "# model.push_to_hub(\"your_name/gemma_4_lora\", token = \"YOUR_HF_TOKEN\") # Online saving\n",
        "# processor.push_to_hub(\"your_name/gemma_4_lora\", token = \"YOUR_HF_TOKEN\") # Online saving"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AEEcJ4qfC7Lp"
      },
      "source": [
        "Now if you want to load the LoRA adapters we just saved for inference, set `False` to `True`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MKX_XKs_BNZR"
      },
      "outputs": [],
      "source": [
        "if False:\n",
        "    from unsloth import FastVisionModel\n",
        "\n",
        "    model, processor = FastVisionModel.from_pretrained(\n",
        "        model_name = \"gemma_4_lora\",  # YOUR MODEL YOU USED FOR TRAINING\n",
        "        load_in_4bit = True,  # Set to False for 16bit LoRA\n",
        "    )\n",
        "    FastVisionModel.for_inference(model)  # Enable for inference!\n",
        "\n",
        "FastVisionModel.for_inference(model)  # Enable for inference!\n",
        "\n",
        "sample = dataset[1]\n",
        "image = sample[\"image\"].convert(\"RGB\")\n",
        "messages = [\n",
        "    {\n",
        "        \"role\": \"user\",\n",
        "        \"content\": [\n",
        "            {\n",
        "                \"type\": \"text\",\n",
        "                \"text\": sample[\"text\"],\n",
        "            },\n",
        "            {\n",
        "                \"type\": \"image\",\n",
        "            },\n",
        "        ],\n",
        "    },\n",
        "]\n",
        "input_text = processor.apply_chat_template(messages, add_generation_prompt = True)\n",
        "inputs = processor(\n",
        "    image,\n",
        "    input_text,\n",
        "    add_special_tokens = False,\n",
        "    return_tensors = \"pt\",\n",
        ").to(\"cuda\")\n",
        "\n",
        "from transformers import TextStreamer\n",
        "\n",
        "text_streamer = TextStreamer(processor.tokenizer, skip_prompt = True)\n",
        "_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 128,\n",
        "                   use_cache = False, temperature = 1.0, top_p = 0.95, top_k = 64)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f422JgM9sdVT"
      },
      "source": [
        "### Saving to float16 for VLLM\n",
        "\n",
        "We also support saving to `float16` directly. Select `merged_16bit` for float16. Use `push_to_hub_merged` to upload to your Hugging Face account! You can go to https://huggingface.co/settings/tokens for your personal tokens. See [our docs](https://unsloth.ai/docs/basics/inference-and-deployment) for more deployment options."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iHjt_SMYsd3P"
      },
      "outputs": [],
      "source": [
        "# Select ONLY 1 to save! (Both not needed!)\n",
        "\n",
        "# Save locally to 16bit\n",
        "if False: model.save_pretrained_merged(\"unsloth_finetune\", processor,)\n",
        "\n",
        "# To export and save to your Hugging Face account\n",
        "if False: model.push_to_hub_merged(\"YOUR_USERNAME/unsloth_finetune\", processor, token = \"YOUR_HF_TOKEN\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TSjNVDCYv-yr"
      },
      "source": [
        "And we're done! If you have any questions on Unsloth, we have a [Discord](https://discord.gg/unsloth) channel! If you find any bugs or want to keep updated with the latest LLM stuff, or need help, join projects etc, feel free to join our Discord!\n",
        "\n",
        "Some other resources:\n",
        "1. Train your own reasoning model - Llama GRPO notebook [Free Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-GRPO.ipynb)\n",
        "2. Saving finetunes to Ollama. [Free notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_(8B)-Ollama.ipynb)\n",
        "3. Llama 3.2 Vision finetuning - Radiography use case. [Free Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(11B)-Vision.ipynb)\n",
        "4. See notebooks for DPO, ORPO, Continued pretraining, conversational finetuning and more on our [documentation](https://unsloth.ai/docs/get-started/unsloth-notebooks)!\n",
        "\n",
        "<div class=\"align-center\">\n",
        "  <a href=\"https://unsloth.ai\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png\" width=\"115\"></a>\n",
        "  <a href=\"https://discord.gg/unsloth\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/Discord.png\" width=\"145\"></a>\n",
        "  <a href=\"https://unsloth.ai/docs/\"><img src=\"https://github.com/unslothai/unsloth/blob/main/images/documentation%20green%20button.png?raw=true\" width=\"125\"></a>\n",
        "\n",
        "  Join Discord if you need help + ⭐️ <i>Star us on <a href=\"https://github.com/unslothai/unsloth\">Github</a> </i> ⭐️\n",
        "</div>\n",
        "\n",
        "  This notebook and all Unsloth notebooks are licensed [LGPL-3.0](https://github.com/unslothai/notebooks?tab=LGPL-3.0-1-ov-file#readme)."
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.13.3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}