#Queues for k0smos

This is the canonical operator guide for the k0smos queue subsystem.

Shipped answer: one tenant normally needs two worker processes: one for the default logical queue and one for notifications. k0smos has one queue subsystem and two alternative transports. Changing from database to Redis does not create another logical queue or require another worker.

Queue workers capture the tenant theme, default locale and active modules at startup. After a tenant theme or default-locale change, gracefully restart its workers so subsequent email and jobs use the new effective snapshot. In-flight jobs finish with the snapshot they started with.

Run the read-only diagnostic before changing a deployment:

php bin/console queue:topology --host=k0smos.example.com
php bin/console queue:topology --host=k0smos.example.com --format=json

--host selects the tenant to inspect. --format=json only changes standard output from the human-readable table to a machine-readable document; it does not select or contact a server. Use it from deployment scripts, CI, monitoring, or configuration management to read the expected process count, systemd unit names, heartbeat state, and diagnostics without parsing terminal tables.

The command reads the selected tenant configuration, active modules, DB-backed notification settings, producer metadata, and worker heartbeats. It does not expose passwords, tokens, payloads, or other secrets.

#Worker Count Rule

Run one process for every distinct effective logical queue that has an asynchronous producer:

worker processes per tenant = distinct logical queue names with async producers

Several job or message types sharing one queue still need only one process. Two slots consuming the same busy queue are two processes for capacity, not a third logical queue.

#Shipped topology

The shipped total is therefore two processes per tenant. If notification email is disabled in DB settings, its producer is inactive and the topology command does not require that lane.

#When another queue is worth its cost

Prefer default plus notifications until monitoring shows a concrete isolation need. A custom name without a producer is a configuration error, not useful capacity.

#Do Not Confuse These Four Layers

Never pass a job type such as queue.async_message, notification.email.send, or deadline.check_reminders to queue:work --queue. The option expects a logical queue name.

#Transport Decision

Redis failover keeps new work operable during an outage; it does not migrate jobs that already exist only in Redis into SQL. Both modes require queue migrations because Redis mode uses the SQL fallback.

Redis provisioning — tenant configuration keys, systemd unit ordering, key namespace, instance requirements, and the transport-mismatch failure mode — is in the systemd worker guide.

#Authoritative Producer Inventory

QueueTopologyRegistry is the runtime source for producer metadata used by queue:topology. The registry is filtered by active modules and mutable settings, so documentation is not the deployment source of truth.

Typed messages are serialized into the transport job type queue.async_message, then QueuedMessageJobHandler resolves the typed handler. Legacy/raw-compatible job types remain consumable:

• front.contact_form.send_mail
• notification.email.send
• deadline.generate_recurring_ticket
• deadline.check_reminders

Those compatibility types do not add logical queues or processes. Low-level callers that bypass the typed producer guards can still leave raw jobs pending when no worker consumes the selected logical queue; use queue:topology and heartbeat monitoring to prevent that configuration.

#Effective Queue-name Sources

Logical names remain configurable and must match [A-Za-z0-9._-]+. QueueName provides central validation, while QueueTopologyRegistry detects custom names with no consumers and divergences from the shipped AI/notification lanes.

#Topology Diagnostics

Text output answers the operator question directly. JSON is stable for deployment automation and contains:

• selected/primary/fallback transport;
• effective logical queues and active producers;
• expected process count;
• expected worker id and systemd instance for each queue;
• current active heartbeat identifiers without payloads/secrets;
• missing workers;
• duplicate worker ids used on different queues;
• global worker-id overrides that would make a multi-queue deployment ambiguous;
• custom worker-default names without discovered producers;
• custom AI/notification queues and invalid configuration values.

Example worker tuple:

tenant:          k0smos.example.com
logical queue:   default
slot:            1
worker id:       k0smos.example.com.default.1
systemd unit:    k0smos-queue-worker@k0smos.example.com-default-1.service

The old tenant.slot generated id is retired because two queues using slot 1 produced ambiguous logs and heartbeat identifiers.

#Start And Supervise Workers

Direct CLI use:

TENANT_ENV=k0smos.example.com php bin/console queue:work --queue=default --worker-slot=1
TENANT_ENV=k0smos.example.com php bin/console queue:work --queue=notifications --worker-slot=1

Copy-ready systemd units, environment files, a two-process setup, scaling with slot 2, deployment order, monitoring, and a per-tenant target are documented in the systemd worker guide.

SIGTERM/SIGINT stop new reservations and let the current handler return. Keep handlers idempotent: retries, stale reservation reclaim, and process termination provide at-least-once rather than exactly-once delivery.

#Failure And Dead-letter Behavior

• Worker exceptions retry while attempts remain.
• Exhausted DBAL jobs stay as queue_jobs.status = 'failed'.
• Exhausted Redis jobs stay in the Redis failed structures.
• Definitive worker failures create in-app admin notifications.
• Producer-side synchronous fallbacks create throttled in-app admin notifications.
• Processed jobs remain log-only by default.
• Retry/delete dead-letter administration is a separate operational feature; queue:topology is intentionally read-only.

#Glossary

Transport — storage/availability adapter behind the queue contracts.

Logical queue — named worker lane selected by queue:work --queue.

Job type — low-level payload category resolved by JobHandlerRegistry.

Typed message — serializable application message implementing QueueMessageInterface.

Handler — application service that performs the job/message side effect.

Heartbeat — last-seen record for a worker id on one logical queue.

Synchronous fallback — producer executes the same handler in its current process because no worker is active or async dispatch failed.

Retry — a failed reservation returned to pending work while attempts remain.

Dead letter — permanently failed job retained for diagnosis after retry exhaustion.

#Environment-dependent Smoke Checks

These require real Redis, real workers, or an isolated container/tenant, so they are not part of the automated suite. Run them in an isolated environment only — never against a runtime tenant database. If a run diverges from the behavior documented here, update README.md, doc/public/en/complete.md, this guide, and the affected changelog before closing the check.

• Redis queue round-trip integration (skippable when Redis is unavailable): dispatch, reserve, retry, delayed job, and dead-letter.
• Deadline command on an isolated container/tenant: with no active worker both handlers run inline and log queue.sync_fallback; with a worker active the work is dispatched asynchronously.
• queue.driver=database with an active worker: advanced AI chat and attachment upload work end to end.
• queue.driver=database with no worker: advanced AI chat and attachment upload answer with a controlled 503, never a raw 500.
• queue.driver=redis with Redis stopped mid-session: AI registries and the queue transport fail over to SQL with a logged warning and the chat continues.
• Deadline CLI on an isolated tenant with no worker: TENANT_ENV=k0smos.example.com php bin/console k0smos:deadline:schedule processes reminders/recurrences inline and exits 0.

#Rebuild The Source Inventory

The former static occurrence appendix was retired so it cannot drift into a second architecture document. Generate audit evidence on demand under reports/ instead of committing another operator guide:

mkdir -p reports
rg -n "MessageBusInterface|QueueMessageInterface|MessageDispatchOptions|QueueManagerInterface|QueueWorkerHeartbeatInterface|dispatchOrRun|queue:work|queue\\.driver|queue_jobs|queue_worker_heartbeats" \
  src modules config tests -g '*.php' -g '*.md' -g '*.json' -g '*.service' \
  > reports/queue-occurrences.txt

Stable architectural entry points:

• src/Domain/Queue/
• src/Application/Queue/QueueTopologyRegistry.php
• src/Application/Queue/QueueWorker.php
• src/Infrastructure/Queue/
• src/Command/QueueWorkCommand.php
• src/Command/QueueTopologyCommand.php
• src/Container/Definition/QueueDefinitions.php

#Committed content updates

Canonical Ecommerce/Blog/Page changes first write SQL intent in their source transaction. The content.changes topology entry is drained by the default worker, at most 100 deliveries per five-second poll, with independent consumer checkpoints. Worker absence leaves updates queued; it does not run search providers inside the source HTTP request. Apply maintenance and active-module migrations before enabling these producer changes.

TENANT_ENV=k0smos.example.com php bin/console content:changes
TENANT_ENV=k0smos.example.com php bin/console content:changes --dispatch --limit=100
TENANT_ENV=k0smos.example.com php bin/console content:changes --retry --subscriber=discovery.freshness --limit=100
TENANT_ENV=k0smos.example.com php bin/console content:changes --reconcile=blog:article --limit=100

Recovery advances one durable cursor per source (ecommerce:product, blog:article, page:page when active). Repeat the recovery command until its JSON reports complete: true, then drain delivery. Use it after missed events, source imports or consumer activation. Each pass replays current IDs, including unpublished records, then tombstones missing tracked IDs. It never changes source content. Five failures quarantine a delivery; explicit bounded retry preserves active worker leases. Discovery offers the same operations for its subscriber in the backoffice; canonical replay additionally checks the source owner's edit permission. The core contract and external indexing limitations are in src/Application/ContentChange/AI.ContentChange.md.

#Discovery maintenance

Handler registration follows active owners: Ai reply/attachment consumers require Ai, native payment webhook handling requires Ecommerce, and Discovery consumers require Discovery. A tenant without Ecommerce can resolve the shared default worker without catalog/payment checkout dependencies. The isolated 20-combination Ai/Discovery/source matrix checks handler, job registry and worker resolution.

Discovery's delivery tests also execute registered bounded CLI actions and a message through the default SQL queue's serialization/reserve/handle/ack cycle. That queue case verifies an unavailable Media source fails safely; successful binary extraction is covered separately. Psapi/Wpapi import fixtures verify committed outbox delivery creates pending Discovery work while queue dispatch is unavailable and preserves local editorial ownership. Those checks do not measure production queue delay or a complete import-to-vector round trip.

When Discovery is active, discovery.maintenance queues identity-only coalesced wakeups on default. A handler advances one bounded SQL-owned phase and may schedule a continuation, up to 256 steps. Phase ownership and child work use conditional leases; duplicate wakeups cannot execute the same claim. External embedding/Qdrant calls and Office/PDF parsing never run inside an owner SQL transaction. Document jobs remain on the existing discovery.documents producer; maintenance redispatches one due current job after lost delivery, without parsing inside the maintenance step.

Schedule discovery:work --advance --steps=100 --seconds=45 periodically in addition to the default worker. It recovers lost wakeups, future retries, expiry/retention and recurring immutable-point deletion. The time bound prevents starting another step; a step already in progress retains its own provider timeout. Pausing stops new projection work but retains cleanup, and retired generations remain in the recovery keyset. Exhausted projections require reconciliation; expired final document attempts remain available for explicit retry. These operations never automatically switch a serving generation or approve AI relations.