Using AI Models

As an addition to the built-in models, PhotoPrism lets you generate captions and labels with Ollama and the OpenAI API. Our step-by-step guides explain how to set them up and provide tested configuration examples you can use as a starting point.

Learn more ›

Model Engines

PhotoPrism currently supports the following runtimes and services:

Engine	Resolution	Runs	Best For
ONNX	224 px	Built-in	Fast, offline default models for labels and NSFW detection
ONNX	720 px	Built-in	Face detection and embeddings
Ollama	720 px	Self-Hosted	Good for generating quality captions & labels; a server with GPU is recommended
OpenAI API	720 px	Cloud	Highest quality captions & labels, also suitable for users without a GPU; requires API key and network access

Performance

• Built-in Models: Our built-in models run on ONNX and generally perform well on all types of hardware. With an NVIDIA graphics card, they can also use the GPU.
• Ollama: Generating labels for an image on an NVIDIA RTX 4060 usually takes 1-4 seconds. The exact time varies depending on the model used and the number of labels generated.
• OpenAI: Processing one image takes about 3 seconds, though this can vary by model, region, and demand.

Two things behind those numbers are worth knowing before you tune for speed:

• What an image costs in prompt tokens is a property of the model, not of the thumbnail. The same 720 px picture cost 208 prompt tokens on one model and 1,182 on another in our benchmark — a fivefold difference for identical input, caused by the model's vision encoder. Switching model can therefore reduce time-to-first-token more than lowering Resolution does.
• Hosted-model timings are approximate. Repeating the same benchmark a few hours apart moved one cloud model's label latency by more than sixfold while its output barely changed. Self-hosted timings reproduced closely across the same pair of runs. Choose a hosted model on output quality, and measure latency when it matters to you rather than trusting a published figure.

Without GPU acceleration, Ollama models will be significantly slower, taking anywhere from 10 seconds to over a minute to complete. This may be acceptable if you only want to process a few pictures or are willing to wait.

Image Classification

Labels are generated by a built-in ONNX model unless your vision.yml configures a different labels model, for example one served by Ollama. To turn off label generation, set PHOTOPRISM_LABELS_MODEL to "none". This disables every labels model, including those served by Ollama or the OpenAI API, while existing labels are kept and can still be edited. The previous option PHOTOPRISM_DISABLE_CLASSIFICATION is deprecated.

Learn more ›

Face Recognition

Faces are handled by built-in models that need no setup: YuNet locates them, and SFace turns each one into a vector that can be compared with others. Both run on ONNX. Libraries created before SFace became available keep FaceNet, which runs on TensorFlow, until they are migrated to the current model.

AI Models > Face Recognition documents the config options and the upgrade procedure, and Troubleshooting > Face Recognition provides checklists for faces that are missing, grouped incorrectly, or slow to tag.

vision.yml Reference

Custom AI engines, models, and run modes can be specified in a vision.yml file located in your config directory (default: storage/config). The file defines a list of models and thresholds to be used, e.g.:

If PhotoPrism can’t read your config file, make sure the file exists at the config path configured for your instance. Older installations may use storage/settings.

Run docker compose exec photoprism photoprism show config | grep config-path to find out what's your configured config path.

Models:
- Type: caption
  Model: gemma4:latest
  Engine: ollama
  Run: auto
  Options:
    Temperature: 0.05
  Service:
    Uri: http://ollama:11434/api/generate
    Think: "false"
- Type: labels
  Model: qwen3.5:4b
  Engine: ollama
  Service:
    Uri: http://ollama:11434/api/generate
    Think: "false"
Thresholds:
  Confidence: 10
  Topicality: 0
  NSFW: 75

If a model type is omitted, PhotoPrism will use the built-in defaults for labels, nsfw, face, or caption. The optional Thresholds block can be used to filter out labels with a low probability or to adjust the NSFW thresholds.

Field	Default	Notes
Type (required)	—	labels, caption, face, nsfw. Drives routing & scheduling.
Model	""	Model identifier in the format <name>:<version>.
Name	derived from Model	Model name.
Version	latest (non-OpenAI)	Model version, not used by OpenAI.
Engine	inferred from service/alias	Aliases set formats, file scheme, resolution. Explicit Service values still win.
Run	auto	See Run modes table below.
Default	false	Select the built-in model for a type.
Disabled	false	Registered but inactive.
Resolution	model-specific / 720 (Ollama/OpenAI)	Thumbnail edge in px; built-in models use the size they were trained for.
System / Prompt	engine defaults / empty	Override prompts per model.
Format	""	Response hint (json, text, markdown).
Schema / SchemaFile	engine defaults / empty	Inline vs file JSON schema (labels).
Normalize2	engine default	Label name normalization; single-word, phrase, or false. Labels models only.
Options	engine defaults / empty	Sampling/settings merged with engine defaults.
Service	engine defaults / empty	Remote endpoint config (see below).

Run Modes

Value	When it runs	Recommended use
auto	Built-in labels and NSFW models during index; others via metadata/schedule	Leave as-is for most setups.
manual	Only when explicitly invoked (CLI/API)	Experiments and diagnostics.
on-index	During indexing + manual	Fast built-in models only.
newly-indexed	Metadata worker after indexing + manual	External/Ollama/OpenAI without slowing import.
on-demand	Manual, metadata worker, and scheduled jobs	Broad coverage without index path.
on-schedule	Scheduled jobs + manual	Nightly/cron-style runs.
always	Indexing, metadata, scheduled, manual	High-priority models; watch resource use.
never	Never executes	Keep definition without running it.

For performance reasons, on-index is only supported for built-in local models.

Options

Adjusts model parameters, such as temperature and top-p, as well as other constraints, when using Ollama or OpenAI:

Option	Engines	Default	Description
Temperature	Ollama, OpenAI	engine default	Controls randomness with a value between 0.01 and 2.0; not used for OpenAI's GPT-5 and GPT-6.
TopK	Ollama	engine default	Limits sampling to the top K tokens to reduce rare or noisy outputs.
TopP	Ollama, OpenAI	engine default	Nucleus sampling; keeps the smallest token set whose cumulative probability ≥ p; not used for OpenAI's GPT-5 and GPT-6.
MinP	Ollama	engine default	Drops tokens whose probability mass is below p, trimming the long tail.
TypicalP	Ollama	engine default	Keeps tokens with typicality under the threshold; combine with TopP/MinP for flow.
TfsZ	Ollama	engine default	Tail free sampling parameter; lower values reduce repetition.
Seed	Ollama	random per run	Fix for reproducible outputs; unset for more variety between runs.
NumKeep	Ollama	engine default	How many tokens to keep from the prompt before sampling starts.
RepeatLastN	Ollama	engine default	Number of recent tokens considered for repetition penalties.
RepeatPenalty	Ollama	engine default	Multiplier >1 discourages repeating the same tokens or phrases.
PresencePenalty	OpenAI	engine default	Increases the likelihood of introducing new tokens by penalizing existing ones.
FrequencyPenalty	OpenAI	engine default	Penalizes tokens in proportion to their frequency so far.
PenalizeNewline	Ollama	engine default	Whether to apply repetition penalties to newline tokens.
Stop	Ollama, OpenAI	engine default	Array of stop sequences (e.g., ["\\n\\n"]).
Mirostat	Ollama	engine default	Enables Mirostat sampling (0 off, 1/2 modes).
MirostatTau	Ollama	engine default	Controls surprise target for Mirostat sampling.
MirostatEta	Ollama	engine default	Learning rate for Mirostat adaptation.
NumPredict	Ollama	engine default	Ollama-specific max output tokens; synonymous intent with MaxOutputTokens.
MaxOutputTokens	Ollama, OpenAI	engine default	Upper bound on generated tokens; adapters raise low values to defaults.
ForceJson	Ollama, OpenAI	engine default	Forces structured output when enabled.
SchemaVersion	Ollama, OpenAI	derived from schema	Override when coordinating schema migrations.
CombineOutputs	OpenAI	engine default	Controls whether multi-output models combine results automatically.
Detail	OpenAI	engine default	Controls OpenAI vision detail level (low, high, auto).
NumCtx	Ollama, OpenAI	engine default	Context window length (tokens).
NumThread	Ollama	runtime auto	Caps CPU threads for local engines.
NumBatch	Ollama	engine default	Batch size for prompt processing.
NumGpu	Ollama	engine default	Number of GPUs to distribute work across.
MainGpu	Ollama	engine default	Primary GPU index when multiple GPUs are present.
LowVram	Ollama	engine default	Enable VRAM-saving mode; may reduce performance.
VocabOnly	Ollama	engine default	Load vocabulary only for quick metadata inspection.
UseMmap	Ollama	engine default	Memory map model weights instead of fully loading them.
UseMlock	Ollama	engine default	Lock model weights in RAM to reduce paging.
Numa	Ollama	engine default	Enable NUMA-aware allocations when available.

Service

Configures the endpoint URL, method, format, and authentication for Ollama, OpenAI, and other engines that perform remote HTTP requests:

Field	Default	Notes
Uri	engine default	Service endpoint URL, used exactly as specified, so it must include the full path (e.g. /api/generate). Empty for local models. Note this differs from OLLAMA_BASE_URL, which is a base that the engine appends the path to.
Method	POST	Override only if provider needs it.
Key	""	Bearer token; supports env expansion (OpenAI: OPENAI_API_KEY, Ollama: OLLAMA_API_KEY1).
Username / Password	""	Injected as basic auth when Uri lacks userinfo.
Model	""	Endpoint-specific override; wins over model/name.
Org / Project	""	Organization / Project ID when using OpenAI.
Think	"" (Ollama: "false")	Ollama reasoning hint. The Ollama engine defaults it to "false" (reasoning off) so thinking models (Qwen3.5, qwen3-vl:*, …) don't leak reasoning into captions/labels; re-enable with "true". Quoted "true"/"false" are sent as JSON booleans.
Tier	""	OpenAI service tier sent as service_tier (e.g. flex for cheaper, slower processing). OpenAI-only; supports ${ENV} expansion.
RequestFormat / ResponseFormat	engine default	Explicit values win over engine defaults.
FileScheme	engine default	Controls image transport e.g. data or base64.
Disabled	false	Disables the endpoint without removing the model.

Authentication: Service.Key, Username, and Password support ${ENV_VAR} expansion of any variable. Service.Model only expands variables whose names end in _MODEL, e.g. ${OLLAMA_MODEL}, and Service.Uri only those ending in _URL, _URI, or _HOST, e.g. ${OLLAMA_BASE_URL}/api/generate; other variables in these two fields are not expanded and a warning is logged. Service.Key sets Authorization: Bearer <token>; Username/Password injects HTTP basic authentication into the service URI when it is not already present. When Service.Key is empty, PhotoPrism defaults to OPENAI_API_KEY (OpenAI engine) or OLLAMA_API_KEY¹ (Ollama engine), also honoring their _FILE counterparts. Requests to the OpenAI API (api.openai.com) or Ollama Cloud (ollama.com) are not sent without a key.

---

1. Available since the March 5, 2026 release. ↩↩

2. Available since the September 19, 2026 release. ↩