Skip to content

Label Generation

PhotoPrism’s built-in image classification relies on TensorFlow models such as Nasnet. With the new Ollama integration, you can generate labels via multimodal LLMs.

Ollama Setup Guide

Follow the steps in our User Guide to connect PhotoPrism directly to an Ollama instance and replace (or augment) the default Nasnet classifier with labels generated by a vision-capable LLM.

Learn more ›

Configuration Tips

PhotoPrism evaluates models from the bottom of the list up, so placing the Ollama entries after the others ensures Ollama is chosen first while the others remain available as fallback options.

Ollama-generated captions and labels are stored with the ollama metadata source automatically, so you do not need to request a specific source field in the schema or pass --source to the CLI unless you want to override the default.

Prompt Localization

To generate output in other languages, keep the base instructions in English and add the desired language (e.g., "Respond in German"). This method works for both caption and label prompts.

Verify labels separately from captions. A model can honor the requested language for captions and silently ignore it for labels, with no error and nothing in the log. Learn more ›

Label Behavior Worth Knowing

Two behaviors affect anyone writing or tuning a label prompt:

  • The built-in prompt omits a label count on purpose. A short list of high-confidence labels is more useful and cheaper than a long one — the count multiplies through the database, the API response, and the UI that fetches and renders them — and models that interpret an image poorly mostly add noise when pushed for more. Not every model honors a count instruction anyway. What the benchmark shows is how differently models answer that prompt: hosted models volunteer seven to twelve labels per image, models that fit in 8 GB return one to four. Asking for a range of 8-15 multiplies the set by 1.9-3.5x and raises subject coverage, and it costs roughly double the label latency plus a higher share of multi-word names on every model not already at zero (qwen3-vl:4b-instruct 0.0% → 4.2%, qwen3.5:4b 1.7% → 3.5%, minicpm-v4.5:8b 5.6% → 8.8%) — and those are discarded by normalization. Treat the count as a per-model tuning knob you measure, not a default to fix. Context in photoprism#5774.
  • Whether a multi-word label name survives depends on the model's Normalize mode. Under single-word — the default for self-hosted models — a phrase is reduced to the first token that resolves against the label vocabulary, which is usually not the subject: ferris wheel is stored as Ferris and amusement park as Park. Under phrase, the default for hosted models, the compound is kept and matched as a whole, so sea lions becomes Sea Lion. Either instruct the model to return single-word nouns, or set Normalize: phrase and pair it with a system prompt that permits phrases — the mismatch, not the mode, is what loses subjects. A name containing no Latin letters is exempt: the vocabulary is English, so tokenizing it could resolve nothing, and it is kept whole in every mode. Learn more ›

NSFW Detection Through Labels

When an Ollama or OpenAI model is wired up for Type: labels, PhotoPrism can ask it to return NSFW classification alongside the regular label fields. The variable is declared in internal/ai/vision/config.go and assigned during configuration in internal/config/config.go:

vision.DetectNSFWLabels = c.DetectNSFW() && c.Experimental()

When DetectNSFWLabels is true, the engine builders in internal/ai/vision/engine_ollama.go and engine_openai.go swap their default user prompts for LabelPromptNSFW, and the JSON schema generators (SchemaLabels(includeNSFW=true)) add the nsfw and nsfw_confidence fields. When it is false, the prompt and schema describe only name, confidence, and topicality, so the LLM response cannot trigger NSFW flagging.

Downstream, the vision worker (internal/workers/vision.go) guards the labels-based NSFW promotion with conf.DetectNSFW():

if w.conf.DetectNSFW() && !m.PhotoPrivate {
    if labels.IsNSFW(vision.Config.Thresholds.GetNSFW()) {
        m.PhotoPrivate = true
    }
}

The index pipeline (internal/photoprism/index_mediafile.go) reaches the same outcome by a different route: it reads the label verdict into a local first and applies it only for photos that are new to the index, falling back to the file-level check when the labels say nothing.

isNSFW = labels.IsNSFW(vision.Config.Thresholds.GetNSFW())
...
if !photoExists {
    if isNSFW {
        photo.PhotoPrivate = true
    } else if o.DetectNsfw {
        photo.PhotoPrivate = m.DetectNSFW()
    }
}

The dedicated ModelTypeNsfw entry (TensorFlow by default, overridable in vision.yml) is a separate inference pass that only runs when DetectNSFW is true and the caller includes nsfw in the active model list (--models labels,nsfw for the CLI; the scheduler picks it up from VisionModelShouldRun automatically).

The user-facing matrix and threshold details are in NSFW Detection.

Troubleshooting

Verify Active Configuration

docker compose exec photoprism photoprism vision ls

Ensure the output lists both Nasnet (default) and your Ollama label model. If the custom entry is missing, double-check the YAML indentation, file name (vision.yml, not .yaml), or environment overrides.

Schema or JSON Errors

If PhotoPrism logs vision: invalid label payload from ollama, the model returned data that didn’t match the expected structure. Confirm that:

  • The adapter injected schema instructions (keep System/Prompt intact or reuse the defaults).
  • The model is not emitting a reasoning block — thinking (reasoning) models such as the Qwen3.5 family and qwen3-vl:* prepend their reasoning to the JSON when reasoning is enabled. Set Service.Think: "false" on the model to disable it — PhotoPrism disables Ollama reasoning by default in releases after 260601, so this affects 260601 and earlier or configs that re-enabled it (see Ollama Models).

PhotoPrism may fall back to the existing TensorFlow Nasnet model when the Ollama response cannot be parsed.

Valid JSON, but No Labels

A well-formed response carrying an empty labels array is a different failure, and checking the schema will not help. Models trained for grounded detection rather than classification do this consistently on ordinary photos — they answer correctly that there is nothing to detect, which is not what a label prompt is asking for. medgemma* behaved this way on every image in our benchmark.

The remedy is to change model rather than the prompt or schema. Confirm it first with a single trace run — an empty array in the response, with no parse error in the log, points here rather than at Schema or JSON Errors:

photoprism --log-level=trace vision run -m labels --count 1 --force

Latency & Timeouts

Structured responses introduce additional parsing overhead. If you encounter timeouts:

  • Increase the global service timeout (e.g., ServiceTimeout in advanced deployments) if needed.
  • Reduce image resolution (Resolution: 500) or use smaller models.
  • Keep Options.Temperature low to encourage deterministic output.

Lowering Resolution is worth trying, but it is not the largest lever. What an image costs in prompt tokens is mostly a property of the model's vision encoder, not of the thumbnail: the same 720 px picture cost 208 prompt tokens on one model and 1,182 on another in our benchmark. Switching model can therefore cut prefill time more than reducing resolution does, and it is worth comparing the two before settling for smaller thumbnails.

GPU Considerations

When Ollama uses GPUs, long-running sessions might degrade over time due to VRAM fragmentation. Restart the Ollama container to recover performance:

docker compose stop ollama
docker compose up -d ollama

stop restarts the existing container; docker compose down ollama works too, but removes and recreates it, which is more than a VRAM reset needs.