#k0smos Queue Workers With systemd

These examples run one long-lived queue:work process for one tenant/logical-queue/slot tuple. The tuple is encoded in both the systemd instance and its environment filename:

k0smos-queue-worker@<tenant>-<queue>-<slot>.service
/etc/k0smos/queue-worker-<tenant>-<queue>-<slot>.env

For the shipped topology, one tenant normally needs two processes:

Run php bin/console queue:topology --host=k0smos.example.com before creating units. It reads tenant configuration and DB-backed notification settings and reports the exact effective queues. A transport change between database and Redis does not add a process.

Transport-specific provisioning is in Redis Transport, and the tuning a PrestaShop catalog import needs is in PrestaShop Imports Through Psapi. The subsystem itself — worker-count rule, producers, failure semantics — is canonical in the queue operations guide.

#Prerequisites

• Deploy the application to /opt/k0smos, or adjust WorkingDirectory and the paths in the unit.
• Verify /usr/bin/php is the intended production CLI binary and includes pcntl on Linux.
• Install Composer dependencies before starting workers.
• Run tenant migrations first. DBAL and Redis failover require queue_jobs and queue_worker_heartbeats.
• Run every worker as the same Unix user as the PHP-FPM/FrankenPHP pool that serves the tenant. The shipped unit uses www-data:www-data because that is the reference pool; when the pool runs as another user, change User=, Group= and UMask= in the unit to match it instead of sharing files between two users. See Worker User Must Match The Web Process.
• Give that user read access to the release and write access to the tenant runtime paths (var/, log/, tmp/, uploads/storage, and SQLite files when SQLite is used). Keep environment files root-owned, group-readable by the worker user, and mode 0640 or stricter.
• Ensure Redis is reachable only when queue.driver=redis; database is the shipped transport and does not require Redis. See Redis Transport for the tenant keys, unit ordering, and instance requirements.
• Ensure the configured Python runtime exists when a worker consumes advanced AI work.

#Redis Transport

Redis replaces the storage behind the same queue contracts. It does not add a logical queue, rename a unit, or change the process count: the tenant still runs one instance per effective logical queue. Run queue:topology after switching and expect an unchanged unit list with a redis primary and a database fallback transport.

#Enable The Transport

Set module_config.queue.driver to redis in config/tenants/k0smos.example.com.json and fill the redis block:

"queue": {
  "name": "default",
  "driver": "redis",
  "redis": {
    "host": "${K0SMOS_REDIS_HOST:-127.0.0.1}",
    "port": "${K0SMOS_REDIS_PORT:-6379}",
    "db": "${K0SMOS_QUEUE_REDIS_DB:-0}",
    "password": "${K0SMOS_REDIS_PASSWORD:-}",
    "prefix": "k0smos:queue",
    "timeout": 2.0
  },
  "max_attempts": 3
}

The transport uses the redis PHP extension, which is a hard Composer requirement. Confirm the production CLI binary loads it before enabling the driver:

/usr/bin/php -m | grep -x redis

#Both Sides Must Resolve The Same Transport

Tenant JSON placeholders read getenv(), then $_ENV, then $_SERVER, and the resolved configuration is cached per environment fingerprint. A worker and a web process that see different values for K0SMOS_QUEUE_DRIVER, K0SMOS_REDIS_HOST, K0SMOS_QUEUE_REDIS_DB, or K0SMOS_QUEUE_PREFIX resolve different transports without raising an error: the producer writes to one store while the worker consumes the other, the producer finds no heartbeat, and every dispatch silently degrades to its synchronous fallback.

Export the same variables in both places — the worker environment file and the PHP-FPM/FrankenPHP service environment — or write the values literally in tenant JSON and keep only the password in the environment.

#Order The Units After Redis

Do not edit the shipped template unit. Add a drop-in when Redis runs on the same host:

sudo install -d -m 0755 /etc/systemd/system/k0smos-queue-worker@.service.d
sudo tee /etc/systemd/system/k0smos-queue-worker@.service.d/redis.conf >/dev/null <<'EOF'
[Unit]
After=redis-server.service
Wants=redis-server.service
EOF
sudo systemctl daemon-reload

Wants= is the right default because the DBAL failover keeps the worker useful during a Redis outage. Use Requires= only when a worker must never run without Redis. When Redis is on another host, keep the shipped After=network-online.target and rely on failover plus Restart=always instead of a host-local dependency.

#Key Namespace

The effective prefix is the configured prefix plus the tenant id from the tenant JSON, so tenant id 7 with the default prefix produces k0smos:queue:7. Per logical queue:

Two tenants sharing one Redis database are separated only by that tenant id segment. Give every tenant a distinct id, or a distinct redis.db or prefix, before pointing a second tenant at the same server.

Read-only inspection while a worker runs:

redis-cli -n 0 llen k0smos:queue:7:default:pending
redis-cli -n 0 llen k0smos:queue:7:default:processing
redis-cli -n 0 zrange k0smos:queue:7:workers:default 0 -1 WITHSCORES
redis-cli -n 0 hlen k0smos:queue:7:default:failed:jobs

Job payloads carry application data. Treat redis-cli output as production data and redact it before it reaches a ticket.

#Instance Hygiene

• Use a database or an instance dedicated to queues. If the same Redis also serves caches with an eviction policy such as allkeys-lru, queue keys are evictable and jobs disappear without an error. maxmemory-policy noeviction is the correct setting for a queue store.
• Jobs in Redis are exactly as durable as the Redis instance. Enable RDB or AOF persistence, or accept that a Redis restart loses everything that was never written to the DBAL fallback.
• Bind Redis to the private interface and set requirepass. The queue transport authenticates but does not encrypt; keep the hop inside a trusted network.

#Failover Behavior

In redis mode the transport is wrapped in a failover manager: dispatch, reserve, reclaim, and heartbeat operations fall back to DBAL when the Redis connection or the operation fails, and the failure is logged. Consequences:

• Queue migrations are mandatory. queue_jobs and queue_worker_heartbeats are the live fallback store, not unused schema.
• An outage does not migrate jobs that already exist only in Redis; they become visible again when Redis returns.
• After an outage, work can exist in both stores at once. Keep the workers running until both drain, and read the primary/fallback pair reported by queue:topology.

#Install The Unit And Environment Files

sudo cp doc/example/systemd/k0smos-queue-worker@.service /etc/systemd/system/
sudo install -d -m 0750 /etc/k0smos
sudo install -m 0640 -o root -g www-data \
  doc/example/systemd/k0smos-queue-worker.env.example \
  /etc/k0smos/queue-worker-k0smos.example.com-default-1.env
sudo install -m 0640 -o root -g www-data \
  doc/example/systemd/k0smos-queue-worker.env.example \
  /etc/k0smos/queue-worker-k0smos.example.com-notifications-1.env

Set the tuple explicitly in each file:

# queue-worker-k0smos.example.com-default-1.env
TENANT_ENV=k0smos.example.com
QUEUE_NAME=default
QUEUE_WORKER_SLOT=1

# queue-worker-k0smos.example.com-notifications-1.env
TENANT_ENV=k0smos.example.com
QUEUE_NAME=notifications
QUEUE_WORKER_SLOT=1

Leave QUEUE_WORKER_ID unset unless an external naming policy requires it. k0smos then generates k0smos.example.com.default.1 and k0smos.example.com.notifications.1, so logs and heartbeat rows remain unambiguous.

#Worker User Must Match The Web Process

A tenant's runtime files are created by whichever process touches them first: the PHP-FPM/FrankenPHP pool on a web request, or the shell that runs bin/console for migrations and imports. On SQLite tenants that includes the database file itself. If the queue worker runs as a different Unix user, even one in the same group, the following happens:

1. The pool or shell creates var/db/<id>.sqlite with its own umask, usually 022, so the file is 0644: the group can read it but not write it.
2. The worker writes on its first loop (heartbeat row, job reservation). SQLite answers attempt to write a readonly database, the process exits with status 1 in under a second.
3. systemd restarts it until StartLimitBurst is exhausted, then leaves both units failed. The per-tenant target still reports active, because a target is active once it has been reached and does not track the units it lists in Wants=.
4. queue:topology --host=<tenant> reports missing_worker for every logical queue, and every producer silently falls back to synchronous execution.

The same command started by hand from the owning user's shell works, which is the tell-tale sign: the configuration is right and only the identity differs.

#Preferred remedy: one user for the whole tenant runtime

Set the unit to the user and group of the pool that serves the tenant. Use a drop-in so the shipped unit file stays untouched:

# /etc/systemd/system/k0smos-queue-worker@.service.d/user.conf
[Service]
User=deploy
Group=www-data
UMask=0002

Place the drop-in under k0smos-queue-worker@k0smos.example.com-default-1.service.d/ instead when only one tenant differs. Then reload and clear the exhausted restart counter:

sudo systemctl daemon-reload
sudo systemctl reset-failed 'k0smos-queue-worker@k0smos.example.com-*'
sudo systemctl restart k0smos-queue-workers-k0smos.example.com.target
php bin/console queue:topology --host=k0smos.example.com

Confirm the identities match:

ps -o user= -C php-fpm8.5 | sort -u
systemctl show 'k0smos-queue-worker@k0smos.example.com-default-1.service' -p User
ls -l var/db/*.sqlite

On a production host the pool should be an unprivileged service user such as www-data; align the worker to it. On a shared development host where the pool already runs as the developer's account, aligning the worker to that account adds no exposure that the pool does not already have.

#Fallback: two users with a group-writable regime

Keeping distinct users for the pool and the worker is supported only when every writer honours a group-writable regime, and it stays fragile because one missing piece reproduces the failure on the next new file:

• setgid on every tenant runtime directory (chmod g+s var var/db var/cache var/media), so new files inherit the shared group;
• umask 002 in the pool configuration and in every interactive shell that runs bin/console;
• UMask=0002 in the unit: the shipped 0027 makes files created by the worker unwritable by the other user, which is the same problem in reverse;
• a one-off chmod g+w on existing SQLite files and their -wal/-shm companions;
• the tenant JSON under config/tenants/, the .env* files and the release checkout readable by the shared group.

Prefer the single-user remedy unless a policy explicitly requires separate identities.

#Enable The Normal Two-process Topology

sudo systemctl daemon-reload
sudo systemctl enable --now \
  k0smos-queue-worker@k0smos.example.com-default-1.service \
  k0smos-queue-worker@k0smos.example.com-notifications-1.service

Add one systemd instance for every distinct effective logical queue that has an asynchronous producer. Multiple message/job types on the same queue do not add units. A custom ai, imports, or notification queue does add a unit when it is distinct from the existing names.

#Scale One Logical Queue

To run two default consumers, copy the environment file, set slot 2, and enable a second tuple. Do not reuse slot 1.

sudo cp \
  /etc/k0smos/queue-worker-k0smos.example.com-default-1.env \
  /etc/k0smos/queue-worker-k0smos.example.com-default-2.env
sudo sed -i 's/QUEUE_WORKER_SLOT=1/QUEUE_WORKER_SLOT=2/' \
  /etc/k0smos/queue-worker-k0smos.example.com-default-2.env
sudo systemctl enable --now \
  k0smos-queue-worker@k0smos.example.com-default-2.service

The expected generated worker id is k0smos.example.com.default.2.

#Optional Per-tenant Target

Copy k0smos-queue-workers-k0smos.example.com.target to /etc/systemd/system/, rename it for the real tenant when necessary, edit its Wants= list to match queue:topology, then enable it:

sudo systemctl daemon-reload
sudo systemctl enable --now \
  k0smos-queue-workers-k0smos.example.com.target

A target is preferable when deployment automation should start, stop, and inspect the tenant's complete worker set as one unit. Enabling instances directly is simpler for a single queue or independently managed capacity.

#PrestaShop Imports Through Psapi

PrestaShop imports are the heaviest workload the shipped topology carries and the usual reason to revisit these units. They have no lane of their own.

A dedicated imports queue is a code change, not a setting: queue.name does not rename a producer that dispatches to default explicitly.

#The Back Office Decides Async Or Sync At Launch Time

POST /api/admin/psapi/import dispatches to the queue only when the default lane has a heartbeat newer than 180 seconds. Otherwise it logs queue.sync_fallback, raises a back-office warning, and runs the import inside the web request, where a PHP or proxy timeout can kill it mid-run.

Check both before launching a large import:

php bin/console queue:topology --host=k0smos.example.com
systemctl is-active k0smos-queue-worker@k0smos.example.com-default-1.service

/admin/psapi/import-status reports the mode the run actually used. Treat a sync production run as a worker outage, not as a variation.

#A Busy Worker Is An Invisible Worker

The worker writes its heartbeat around reservation and completion, not during a handler. A single default worker spending twenty minutes inside a PrestaShop import therefore has a heartbeat older than 180 seconds for most of that time, and every other default producer degrades while the import runs: contact mail, payment webhooks, and Deadline messages execute synchronously in their caller, and advanced AI — which rejects instead of falling back — answers 503. notifications is a separate lane and is unaffected.

For a tenant that imports during business hours, run a second default slot so one worker always stays free:

sudo cp \
  /etc/k0smos/queue-worker-k0smos.example.com-default-1.env \
  /etc/k0smos/queue-worker-k0smos.example.com-default-2.env
sudo sed -i 's/QUEUE_WORKER_SLOT=1/QUEUE_WORKER_SLOT=2/' \
  /etc/k0smos/queue-worker-k0smos.example.com-default-2.env
sudo systemctl enable --now \
  k0smos-queue-worker@k0smos.example.com-default-2.service

A second slot also introduces reclaim, so read the next section before enabling it.

#Tune The Three Timers That Interrupt An Import

Reclaim is based on reservation age, not on worker liveness, and a worker busy inside a handler never reclaims anything itself. With one slot the in-flight import is safe. With two or more slots and reclaim_after below the import duration, the sibling worker re-delivers the job: the import does not run twice — the job store refuses to claim a run updated within the last 900 seconds — but the delivery consumes an attempt and removes the queue job while the original is still working, so that run loses its retry.

Set every timer above the longest expected run:

sudo install -d -m 0755 /etc/systemd/system/k0smos-queue-worker@.service.d
sudo tee /etc/systemd/system/k0smos-queue-worker@.service.d/imports.conf >/dev/null <<'EOF'
[Service]
# Let a running import finish before systemd escalates to SIGKILL.
TimeoutStopSec=3600s
# Catalog imports hold more per-process memory than mail or webhook jobs.
ExecStart=
ExecStart=/usr/bin/php -d memory_limit=512M /opt/k0smos/bin/console queue:work
EOF
sudo systemctl daemon-reload
sudo systemctl restart k0smos-queue-worker@k0smos.example.com-default-1.service

The empty ExecStart= is required: without it systemd appends a second command instead of replacing the shipped one.

Raise the reclaim window in the same tuple's environment file:

QUEUE_RECLAIM_AFTER_SECONDS=3600

A drop-in under k0smos-queue-worker@.service.d/ applies to every instance of the template. Use the per-instance directory k0smos-queue-worker@k0smos.example.com-default-1.service.d/ when only the import lane needs the longer timeout.

Also review QUEUE_MAX_JOBS and QUEUE_MAX_IDLE_SECONDS. They recycle a worker after N jobs or N idle seconds and are a deliberate memory-hygiene measure; they never interrupt a running handler, because both are evaluated between jobs.

#Watch A Running Import

sudo journalctl -u k0smos-queue-worker@k0smos.example.com-default-1.service -f
sudo journalctl -u k0smos-queue-worker@k0smos.example.com-default-1.service \
  --since "1 hour ago" | grep -E 'psapi\.import_failed|queue\.sync_fallback'
TENANT_ENV=k0smos.example.com php bin/console k0smos:psapi:diagnostics

The journal shows dispatch and failure events. The durable psapi_import_runs and psapi_import_run_entities rows — surfaced by /admin/psapi/import-status — remain authoritative for status, counters, and checkpoints.

#Very Large First Imports

Run the first full catalog import as a foreground CLI job rather than a back-office launch, so it never occupies the shared default lane:

TENANT_ENV=k0smos.example.com php bin/console k0smos:psapi:import --dry-run
TENANT_ENV=k0smos.example.com php bin/console k0smos:psapi:import --profile=core

Use --entity=products --since=2026-09-01 for a bounded rerun. Leave the workers running while the CLI import proceeds: they keep serving mail, webhooks, and notifications.

#Deploy, Restart, And Monitor

Recommended deployment order:

1. Put the web application into the deployment's maintenance/drain mode when required.
2. Stop or gracefully restart workers before replacing code that changes message classes or handlers.
3. Deploy code and Composer dependencies.
4. Run tenant migrations.
5. Run queue:topology --format=json and reconcile environment files/units.
6. Restart the target or the affected instances, then restore web traffic.

sudo systemctl daemon-reload
sudo systemctl restart \
  k0smos-queue-worker@k0smos.example.com-default-1.service \
  k0smos-queue-worker@k0smos.example.com-notifications-1.service
sudo systemctl status \
  k0smos-queue-worker@k0smos.example.com-default-1.service
sudo journalctl \
  -u k0smos-queue-worker@k0smos.example.com-default-1.service -f

systemctl stop sends SIGTERM. The worker stops reserving new jobs and exits after its current handler returns. TimeoutStopSec=180s is the final bound; work exceeding it may be killed and later reclaimed according to the queue's stale-job settings. Keep handlers idempotent because retry and crash recovery provide at-least-once delivery.

Before restarting a default worker, check that no import is running. A systemctl restart during a PrestaShop import interrupts it at TimeoutStopSec, and the run row stays running until the job store's 900-second stale window allows a new claim.

#Troubleshooting

Dead-letter inspection and retry are a separate operational feature; queue:topology is read-only by design.