Skip to content

Label Generation

PhotoPrism’s built-in image classification runs a fixed-vocabulary model on ONNX; Label Models compares the available models and explains how to select one or add your own. With the Ollama and OpenAI integrations, you can also generate labels via multimodal LLMs.

Ollama Setup Guide

Follow the steps in our User Guide to connect PhotoPrism directly to an Ollama instance and replace (or augment) the default built-in classifier with labels generated by a vision-capable LLM.

Learn more ›

Configuration Tips

PhotoPrism evaluates models from the bottom of the list up, so placing the Ollama entries after the others ensures Ollama is chosen first while the others remain available as fallback options.

Requests to the OpenAI API (api.openai.com) or Ollama Cloud (ollama.com) are not sent without an API key: they fail with missing api key, and a warning naming the model is written to the system log once. Set Service.Key, or OPENAI_API_KEY / OLLAMA_API_KEY.

Ollama-generated captions and labels are stored with the ollama metadata source automatically, so you do not need to request a specific source field in the schema or pass --source to the CLI unless you want to override the default.

Prompt Localization

To generate output in other languages, keep the base instructions in English and add the desired language (e.g., "Respond in German"). This method works for both caption and label prompts.

Verify labels separately from captions. A model can honor the requested language for captions and silently ignore it for labels, with no error and nothing in the log. Learn more ›

Label Behavior Worth Knowing

Two behaviors affect anyone writing or tuning a label prompt:

  • The built-in prompt omits a label count on purpose. A short list of high-confidence labels is more useful and cheaper than a long one — the count multiplies through the database, the API response, and the UI that fetches and renders them — and models that interpret an image poorly mostly add noise when pushed for more. Not every model honors a count instruction anyway. What the benchmark shows is how differently models answer that prompt: hosted models volunteer seven to twelve labels per image, models that fit in 8 GB return one to four. Asking for a range of 8-15 multiplies the set by 1.9-3.5x and raises subject coverage, and it costs roughly double the label latency plus a higher share of multi-word names on every model not already at zero (qwen3-vl:4b-instruct 0.0% → 4.2%, qwen3.5:4b 1.7% → 3.5%, minicpm-v4.5:8b 5.6% → 8.8%) — and those are discarded by normalization. Treat the count as a per-model tuning knob you measure, not a default to fix. Context in photoprism#5774.
  • Whether a multi-word label name survives depends on the model's Normalize mode. Under single-word — the default for self-hosted models — a phrase is reduced to the first token that resolves against the label vocabulary, which is usually not the subject: ferris wheel is stored as Ferris and amusement park as Park. Under phrase, the default for hosted models, the compound is kept and matched as a whole, so sea lions becomes Sea Lion. Either instruct the model to return single-word nouns, or set Normalize: phrase and pair it with a system prompt that permits phrases — the mismatch, not the mode, is what loses subjects. A name containing no Latin letters is exempt: the vocabulary is English, so tokenizing it could resolve nothing, and it is kept whole in every mode. Learn more ›

NSFW Detection Through Labels

PHOTOPRISM_NSFW_MODEL selects how NSFW content is detected: auto (default) runs the dedicated detector chosen in vision.yml, none turns detection off, and labels uses the NSFW fields returned by an Ollama or OpenAI labels model instead. Label-derived NSFW applies only in labels mode and only with PHOTOPRISM_DETECT_NSFW=true. The variable is declared in internal/ai/vision/config.go and assigned in internal/config/config_vision.go:

vision.DetectNSFWLabels = c.DetectNSFWLabels() // DetectNSFW() && nsfw-model == "labels"

When DetectNSFWLabels is true, the engine builders in internal/ai/vision/engine_ollama.go and engine_openai.go swap their default user prompts for LabelPromptNSFW, and the JSON schema generators (SchemaLabels(includeNSFW=true)) add the nsfw and nsfw_confidence fields. When it is false, the prompt and schema describe only name, confidence, and topicality, so the LLM response cannot trigger NSFW flagging, even if a custom prompt asks for the fields.

Downstream, the vision worker (internal/workers/vision.go) applies the label verdict in labelsPrivateFlag, and only in labels mode:

if private || !conf.DetectNSFWLabels() || vision.Config == nil {
    return private, false
}
if labels.IsNSFW(vision.Config.Thresholds.GetNSFW()) {
    return true, true
}

The index pipeline (internal/photoprism/index_mediafile.go) reaches the same outcome for photos that are new to the index, and falls back to the dedicated detector when the labels say nothing and dedicated detection is enabled:

isNSFW = labelsMarkNSFW(labels, o.DetectNSFWLabels)
...
if !photoExists {
    if isNSFW {
        photo.PhotoPrivate = true
    } else if o.DetectNsfw {
        if result := m.DetectNSFW(); result.IsUnsafe() {
            photo.PhotoPrivate = true
        }
    }
}

Label-derived flags use Thresholds.NSFW (default 75). The dedicated ModelTypeNsfw detector (a built-in ONNX model by default) uses Thresholds.NSFWIndex and Thresholds.NSFWUpload, which default to the detector's calibrated threshold; neither falls back to NSFW. In labels mode, no dedicated detector is loaded and uploads are not screened.

An nsfw entry in vision.yml that uses the Ollama or OpenAI request format returns no NSFW scores, so PhotoPrism replaces it with the built-in detector when the file is loaded, keeping its Run setting, and writes a warning to the system log. Use a labels model with PHOTOPRISM_NSFW_MODEL=labels to get NSFW flags from these engines.

The user-facing matrix and threshold details are in NSFW Detection.

Troubleshooting

Verify Active Configuration

docker compose exec photoprism photoprism vision ls

Ensure the output lists your Ollama label model. If the custom entry is missing, double-check the YAML indentation, file name and location (storage/config/vision.yml), or environment overrides.

Schema or JSON Errors

If PhotoPrism logs vision: invalid label payload from ollama, the model returned data that didn’t match the expected structure. Confirm that:

  • The adapter injected schema instructions (keep System/Prompt intact or reuse the defaults).
  • The model is not emitting a reasoning block — thinking (reasoning) models such as the Qwen3.5 family and qwen3-vl:* prepend their reasoning to the JSON when reasoning is enabled. Set Service.Think: "false" on the model to disable it — PhotoPrism disables Ollama reasoning by default in releases after 260601, so this affects 260601 and earlier or configs that re-enabled it (see Ollama Models).

Valid JSON, but No Labels

A well-formed response carrying an empty labels array is a different failure, and checking the schema will not help. Models trained for grounded detection rather than classification do this consistently on ordinary photos — they answer correctly that there is nothing to detect, which is not what a label prompt is asking for. medgemma* behaved this way on every image in our benchmark.

The remedy is to change model rather than the prompt or schema. Confirm it first with a single trace run — an empty array in the response, with no parse error in the log, points here rather than at Schema or JSON Errors:

photoprism --log-level=trace vision run -m labels --count 1 --force

Latency & Timeouts

Structured responses introduce additional parsing overhead. If you encounter timeouts:

  • Increase the global service timeout (e.g., ServiceTimeout in advanced deployments) if needed.
  • Reduce image resolution (Resolution: 500) or use smaller models.
  • Keep Options.Temperature low to encourage deterministic output.

Lowering Resolution is worth trying, but it is not the largest lever. What an image costs in prompt tokens is mostly a property of the model's vision encoder, not of the thumbnail: the same 720 px picture cost 208 prompt tokens on one model and 1,182 on another in our benchmark. Switching model can therefore cut prefill time more than reducing resolution does, and it is worth comparing the two before settling for smaller thumbnails.

GPU Considerations

When Ollama uses GPUs, long-running sessions might degrade over time due to VRAM fragmentation. Restart the Ollama container to recover performance:

docker compose stop ollama
docker compose up -d ollama

stop restarts the existing container; docker compose down ollama works too, but removes and recreates it, which is more than a VRAM reset needs.