Agent operations

This runbook configures a server and a remote agent. Paired runtime traffic is outbound; initial pairing uses a temporary agent listener. It covers the server listener, admin pairing, agent installation, routing, maintenance, and recovery.

Execution location in step logs

Each accepted remote step records the agent name and ID before script output. Each execution attempt records the physical hostname, execution mode, interpreter, and attempt number. Container attempts also record the runtime, container name, and image. Server containers record the image reference. Agent containers record the resolved image ID.

Retries and fan-out retain separate entries. Each remote log entry includes the authenticated agent ID. This ID distinguishes interleaved fan-out output after secret redaction. Embedded line separators appear as escaped text.

Container execution uses the physical hostname reported by Docker or Podman. Host execution uses this runtime information when available. Native host agents otherwise use the operating system hostname. Unavailable physical hostnames appear as unknown.

For a containerized host agent without runtime information, set DURPDEPLOY_AGENT_HOSTNAME to the physical machine hostname. This setting overrides automatic hostname detection. The setting does not grant runtime socket access.

DURPDEPLOY_AGENT_HOSTNAME=app-server-02 make dev-agent

Read these entries in live step logs, saved logs, public log APIs, and log downloads. Older agents can provide their identity without hostname or container details. Historical logs retain their original information.

Fleet maintenance and health

Delete removes the agent’s server registration, pairing credentials, and configuration and prevents further access. Both the list and detail page ask for confirmation. Deployments, snapshots, logs, and step attempts remain; their agent references are cleared. Agent-owned remote protocol state is removed. Deletion is refused while execution or cleanup is unconfirmed or remote logs remain buffered. Drain the agent, let its work finish, and reconcile cleanup and log delivery before deleting it. A refused delete leaves the agent unchanged. Previously revoked agents can also be deleted when no unresolved work remains.

Administrators can call DELETE /api/v1/admin/agents/{id} with an empty body or {}. Success returns 204; missing or deleted agents return 404; unresolved remote work or buffered logs return 409. Successful deletion records delete_agent in the audit log. The existing POST /api/v1/admin/agents/{id}/revoke remains available for revoking access while retaining the inventory entry and re-pairing option. The Revoke button remains available for an immediate access stop when unresolved work prevents deletion. The same installed agent can pair again as a new registration with the same identity fingerprint. The server retains no tombstone or duplicate-identity block. Pair it from the list as usual; it receives a new agent ID and default configuration. Previous deployments stay detached from the new registration. To generate a fresh code, stop the idle agent, move only state.json out of its private state directory, and restart it. Keep its identity certificate and key. Do this only after execution, cleanup, and log delivery have been reconciled. The agent’s bootstrap listener validates its current pairing code; an incorrect code does not restore the deleted registration.

Admins manage the fleet at /admin/agents. Drain stops new claims and keeps the agent authenticated. Claims issued before Drain can still start, send logs and heartbeats, and finish. Waiting work stays queued; later remote steps also wait until Resume. Drain state survives server restarts. Waiting on a drained target does not consume its step timeout; issued work keeps its normal timeout. A failed or timed-out sibling still fails the step. Drain does not cancel, revoke, or transfer work to another agent.

Deployments queued for remote work show Waiting for agents when no claim has been issued. The message clears automatically when an agent takes the work or the deployment finishes. The status API exposes waiting_for_agents.

After Drain, status is draining while issued work remains, then drained when no issued work remains. Refresh the page after work finishes to see the updated status. Queued work still waits for Resume in either state.

Administrative status (active, draining, drained, disabled, revoked, or pending) is separate from heartbeat health. Active agents become stale after 120 seconds and offline after 600 seconds without contact. An agent that has never reported is initially unknown and uses its latest completed pairing time for these deadlines, including after re-pairing. Draining agents continue to report health. Disabled and revoked agents do not generate health alerts.

The admin API exposes this display state as administrative_status. The existing status field remains the base registration state; draining: true means maintenance is enabled in both draining and drained states.

The detail page and admin API show issued work, waiting work, the last successful deployment, and the last failed/lost/unconfirmed result. Follow the deployment link for redacted logs. Error summaries use persisted outcome codes; richer diagnostics require agent follow-up #10.

Compatibility shows the observed agent/1, agent/2, or agent/3 protocol. The fleet also shows the server build identity and the pinned agent module version. Agent-reported software versions remain unverified. The web page shows Not reported when the version is missing. Read software versioning for coordinated server and agent diagnostics. This view does not upgrade agents.

Stale, offline, and recovered transitions use the existing notification bus and global notification settings. Maintenance persists the last health state to suppress repeated alerts, including after restart. Delivery is best effort; it does not replay failed notification deliveries. Alert delivery runs outside claim maintenance, so slow notification channels do not delay lease recovery. Delivery is ordered through a bounded in-memory queue; a full queue skips new alerts and logs a warning, without retrying them. Alerts identify the agent and link to its admin detail page. Drain/Resume and capability/environment routing-label changes have audit actions and include the agent ID plus the changed label or environment ID. Viewer and deployer roles cannot access the admin fleet pages or API.

Two storage boundaries

The server owns the DurpDeploy database. SQLite, its WAL and SHM files, projects, releases, deployments, variables, audit records, and agent records stay on the server. The agent never receives the SQLite database, DURPDEPLOY_SECRET_KEY, or the server’s control-plane state directory.

The agent has no database. Its private state directory contains the agent identity certificate and key, paired server identity state, and durable claim recovery state, including container cleanup metadata and encrypted terminal reports awaiting acknowledgement. Keep that directory private and back it up only if preserving the enrolled identity is intentional.

Agents initiate runtime connections, but pairing needs a temporary unpaired agent callback listener. After code and fingerprint confirmation, the server sends one server-init callback. After the completion acknowledgement, that callback listener closes. The paired agent has no persistent inbound listener, and remote dispatch does not use SSH.

Transport and ports

There are two separate HTTPS paths:

• Caddy serves the browser and JSON API HTTPS origin on port 443. Caddy obtains the public certificate from Let’s Encrypt and reverse proxies to the normal application listener on port 8080.
• The dedicated agent listener uses port 23187, or another operator-selected address, directly on the DurpDeploy process. It uses TLS 1.3, self-signed Ed25519 identities, and direct SHA-256 certificate fingerprint pinning. It is not routed through Caddy and must not be terminated by Caddy.

Only the agent network must have access to the dedicated listener. Allow inbound TCP 23187 from the agent hosts or agent network. Allow ports 80 and 443 for Caddy. Keep application port 8080 private. Agents need outbound TCP access to the server’s 23187 address. They do not need an inbound agent port.

The direct mTLS listener enforces a 5-second header timeout, a 30-second whole-request read timeout, a 60-second write timeout, and a 60-second idle keep-alive timeout. Read deadlines are absolute: slowly sending bytes does not extend them. These limits do not replace protocol body-size limits or network firewall rules. The normal 25-second agent long poll fits inside the write budget, including the request read. A connection can close on a timeout without an HTTP error response; agents must use their normal retry and reconciliation flow rather than assuming a write failed before commit. See HTTP connection limits for the separate browser/API streaming and reverse-proxy policy.

Configure the server listener

Configure the direct listener through the server environment file. make dev enables it automatically for local container development with port 23187, public URL https://host.containers.internal:23187, and an identity below the ignored tmp/ directory. Set any of the variables explicitly to override that default. make dev creates a missing local identity with the public origin hostname as the self-signed certificate SAN and preserves an existing identity. For a local foreground run without make dev, no listener setup is required:

go run ./cmd/server

Startup creates the identity automatically. To provision it before starting the server, run go run ./cmd/server dev-agent-identity with any desired listener overrides set.

The public URL is the direct server mTLS endpoint, not the Caddy/Let’s Encrypt browser or API endpoint. The agent is outbound-only and has no database. Its state directory is not server storage.

The server listener is always active. A bare binary defaults to 0.0.0.0:23187, https://localhost:23187, and .agent-identity, creating the identity on first start. Production installations must override the public URL and identity directory with externally reachable, persistent values:

DURPDEPLOY_AGENT_LISTEN_ADDR=0.0.0.0:23187
DURPDEPLOY_AGENT_PUBLIC_URL=https://<agent-control-host>
DURPDEPLOY_AGENT_IDENTITY_DIR=/var/lib/durpdeploy/agent-identity

DURPDEPLOY_AGENT_PUBLIC_URL must be an HTTPS origin with no path, query, or fragment. Its hostname is placed in the self-signed certificate SAN. Pairing persists the direct listener URL for the agent. Operators do not enter it on the agent host. Each variable can be overridden independently; an omitted variable keeps its default.

For Compose, apply listener overrides in the app service’s environment block in compose.yml; these entries take precedence over compose.app.env. Recreate the app with docker compose up -d app after changing them. Keep the identity volume across restarts, owned by the app user and mode 0700 or stricter. Do not copy its private key to an agent. Check the listener and server logs before pairing:

docker compose logs --tail=100 app
sudo ss -ltn '( sport = :23187 )'

The stock Compose files publish port 23187 directly and preserve the server identity in a private volume. Restrict that port to the necessary agent network. Do not put the server identity directory in the agent service.

Admin UI pairing workflow

Use an administrator browser session at the Caddy URL. The agent pages are admin-only.

1. Start the local agent listener. Open Admin, Agents, then enter a display name, agent address, and the short-lived pairing code. The address can omit https://; the server adds it automatically. The server creates the stable Agent ID.
2. Compare the fingerprint on the confirmation page with the fingerprint shown by the agent through a trusted channel. Approve only when they match. Deny returns to the agent list without creating the agent. Server-init (/agent/v1/pairings/server-init) uses the approved value plus the server-held code and pinned endpoint to finalize pairing. The values are console-only and cannot be retrieved later. You can use the code one time. You cannot retrieve it later. Never put it in source control, tickets, chat, shell history, or logs.
3. Change the display name or add capability and environment labels on the agent details page, then verify its heartbeat. The Agent ID does not change.

Environment and capability labels route remote steps. An active, paired agent matches when it has the deployment’s environment label and every capability label required by the step. Capability matching is case-insensitive. An environment label is a routing input, not an authorization boundary.

Each matching agent receives its own copy of the remote step. The deployment waits for every copy to succeed before it continues. If no agent matches, the step fails immediately; DurpDeploy does not run it locally.

Examples:

• One agent: a production deployment contains a remote step requiring linux. One active, paired agent has the production environment label and the linux capability label, so that agent receives the step.
• Fan-out: two active, paired agents have both labels. Both receive the step, and the deployment continues only after both copies succeed.
• No match: the agents have the wrong environment, lack linux, are not active, or are not paired. The step fails without a local fallback.

Agent start and pairing

Start an unpaired agent with only local inputs:

DURPDEPLOY_AGENT_STATE_DIR=/var/lib/durpdeploy-agent
DURPDEPLOY_AGENT_LISTEN_ADDR=0.0.0.0:23187
DURPDEPLOY_AGENT_VERSION=<agent-version>

DURPDEPLOY_AGENT_STATE_DIR defaults to the platform configuration directory, but production services set it explicitly. DURPDEPLOY_AGENT_LISTEN_ADDR is needed only while the local pairing listener is open. DURPDEPLOY_AGENT_VERSION is sent in heartbeats after pairing. Host agents use agent/2; enabling container execution selects agent/3. The server also accepts legacy agent/1 polls as Bash-only. There is no protocol variable.

The first run prints a short-lived pairing code and agent fingerprint. Enter the code in the authenticated admin pairing flow, compare the displayed fingerprint, and approve before the code expires. Do not put the pairing code, fingerprint, endpoint, or private key in documentation, tickets, shell history, or logs. Pairing persists the agent identity, pull URL, server pins, and agent ID in the private state directory.

Pairing recovery checks the registration again after the completion exchange. If deletion or revocation finishes while it waits on the agent, the API returns 409 and the web flow reports a conflict instead of a stale paired success. If the completion acknowledgement times out after activation commits, a separate bounded registration check still determines paired success or conflict.

After pairing, restart with DURPDEPLOY_AGENT_STATE_DIR and DURPDEPLOY_AGENT_VERSION. Do not supply a server URL, certificate, fingerprint, token, or agent ID manually. Normal work is outbound polling, heartbeats, log uploads, and result or cancellation acknowledgements. The agent persists pairing identity and encrypted terminal reports in its private state directory. It preserves the claim recovery marker until the result is acknowledged and owned containers are reconciled. Never delete this state to clear a cleanup warning.

For opt-in container execution, set:

DURPDEPLOY_AGENT_CONTAINER_ENABLED=true
DURPDEPLOY_AGENT_CONTAINER_RUNTIME=podman
DURPDEPLOY_AGENT_CONTAINER_SOCKET=unix:///run/user/10001/podman/podman.sock

Use docker with its local Unix socket instead where appropriate. The runtime must enforce the agent’s CPU, memory, PID, seccomp, and filesystem limits. For an agent in a container, mount the local socket into that agent and set the socket URL to the mounted path. Container-mode workloads receive no runtime socket. Host-mode scripts run inside the supervisor and inherit socket access and the operator’s runtime authority; use only scripts trusted with that account. Do not grant socket permissions automatically or configure TCP/SSH endpoints. See the pinned agent’s container setup.

Remote steps use agent_execution_mode: "host" by default. Select "container" and an image to opt in. Labels still route the step, and only agents reporting a ready runtime and the container interpreter can claim it. The admin detail API and page show the last reported capabilities.

Upgrade the server before enabling v3 agents. An agent downgrade clears v3 capabilities and fails incompatible waiting work; it does not restart issued work. Do not downgrade the server while container work or cleanup is pending. If cleanup is uncertain, the deployment remains cleanup_unconfirmed and blocks its environment. Restore runtime access and let the same paired agent reconcile and send a ready v3 poll. Confirmation releases the queue and permits retry while preserving the original outcome. If maintenance already marked a claim lost or cancellation unconfirmed, the agent can replay its durable terminal report using the original claim token. A late success does not change the failed deployment to success. A late cleanup failure still requires the ready v3 poll before the queue can advance. Never re-pair or remove state as a substitute for cleanup.

Development container

For a populated demo with both host and container steps, use make demo. For the standalone development agent, explicitly pass the existing local Unix socket to enable container execution, for example:

make dev-agent DEV_AGENT_RUNTIME_SOCKET="$(podman info --format '{{.Host.RemoteSocket.Path}}')"

The socket must already be active and accessible; this does not change its permissions. Rootless Podman uses keep-id:uid=10001,gid=10001 and adjusts only the private named state volume’s ownership via :U, preserving pairing keys and state during a stopped upgrade. Continue using the same profile for that retained volume. Docker adds the existing socket’s group. The socket is mounted in the supervisor; container-mode workloads receive no socket. Host-mode scripts run inside the supervisor and inherit its runtime authority. The supervisor’s SELinux label separation is disabled for that socket mount; workload restrictions remain enforced. Ordinary make dev-agent remains host-only and does not mount an engine socket. Finish/cancel workloads and confirm cleanup before make dev-agent-down; do not reset pairing to clear a cleanup warning.

With make dev running in another terminal, start the published agent image:

make dev-agent

This pulls harbor.durp.info/github/developerdurp/durpdeploy-agent:4110429 and runs it in the foreground using Docker or Podman. In Admin > Agents > Pair agent, enter https://127.0.0.1:23188, the terminal’s pairing code, and fingerprint, then compare the fingerprint and approve. The development server advertises https://host.containers.internal:23187, which the container can reach. The server’s agent listener must be reachable from the container; make dev sets its listen address to 0.0.0.0:23187 by default.

The published development image pin can differ from the agent module revision under review. The agent reports the image commit tag as its version. The pinned 4110429 image supports staging transfer and release package jobs. It provides the artifact/1 and staging/1 capabilities required for package jobs.

make dev reloads the server when watched Go, templ, or SQL files change. Server restart recovery cancels in-flight remote steps; an agent can finish its script before it receives that cancellation. Avoid source edits during deployment tests, then rerun any deployment interrupted by a reload.

Stop with Ctrl-C, or from another terminal:

make dev-agent-down

The container is removed, but its private durpdeploy-dev-agent-state volume retains the paired identity for the next run. To stop and delete that identity:

make dev-agent-reset

Reset requires pairing again. Revoke the old agent in the admin UI when discarding it. Pairing output is terminal-only; container logging is disabled. The image supplies Bash; this image does not install Python or PowerShell.

Override the image, host pairing port, or container name when needed:

make dev-agent DEV_AGENT_IMAGE=harbor.durp.info/github/developerdurp/durpdeploy-agent:4110429 DEV_AGENT_PORT=10945

DEV_AGENT_CONTAINER also determines the default state-volume name. Supply the same overrides to the down/reset targets. Set DEV_CONTAINER_ENGINE=podman or DEV_CONTAINER_ENGINE=docker to select an engine explicitly. This target does not start or modify the server, and does not mount its database or secrets.

Agent execution boundary

Host-mode agent execution does not use a per-step chroot. The container or systemd service is the filesystem and cgroup boundary, and the operator or user is responsible for every deployment script they run there, including its contents, the secrets supplied to it, its network access, and all effects available inside the agent container. A read-only root filesystem does not stop a script from reading files that are visible in the container or exfiltrating secrets supplied to it.

The default socket-free agent container contract is deliberately limited and explicit:

• The agent process and selected interpreter run as the preselected unprivileged service UID 10001; neither process has Linux capabilities.
• The root filesystem is read-only. Writable locations are private to the container, primarily the agent state directory and a private /tmp tmpfs.
• The container has no host or control-plane database, server secret, control-plane state directory, Docker socket, or arbitrary host filesystem mount. Its private state volume is the only operator-provided data path.
• Linux capabilities are dropped by default, and NoNewPrivs is enabled.
• CPU, memory, and process-count limits are enforced on the service cgroup.
• No host cgroup tree, host filesystem, or server data is mounted into the agent container.

An unprivileged agent cannot change to a separate runner UID without SETUID/SETGID. Those capabilities are intentionally absent. Scripts therefore share the agent UID and can read or change its private state volume, including the paired identity. Use one agent boundary per trusted script domain, and re-pair the agent if a script may have altered that state. The separate host or container still prevents access to control-plane state and arbitrary host data in the default socket-free profile. With the opt-in runtime socket, host-mode scripts can use the runtime API to access host data with the operator’s authority. The filesystem restrictions do not contain that access. Container-mode steps run in separate restricted containers without that socket or the supervisor’s private state volume.

These limits do not restrict host-mode network destinations or prevent use of supplied secrets. Prefer remote hosts for production. Co-location requires private mounts and trusted scripts.

The standalone agent’s systemd unit supplies read-only and private mount boundaries. A foreground agent lacks these boundaries and is reserved for initial pairing. Server steps still require containers, including development runs. See server execution.

Direct binary installation

The agent lives in the standalone DeveloperDurp/durpdeploy-agent repository. Install Git, GNU Make, jq, and Go 1.25.7 or newer, then build the agent from the version required by the DurpDeploy server release. In a trusted server checkout, record the pinned version, authenticated module checksum, and source commit:

set -euo pipefail
AGENT_MODULE=github.com/DeveloperDurp/durpdeploy-agent
AGENT_VERSION=$(go list -m -f '{{.Version}}' "$AGENT_MODULE")
AGENT_SUM=$(awk -v module="$AGENT_MODULE" -v version="$AGENT_VERSION" \
  '$1 == module && $2 == version { print $3 }' go.sum)
AGENT_COMMIT=$(go mod download -json "$AGENT_MODULE@$AGENT_VERSION" \
  | jq -e -r '.Origin.Hash // empty')
test -n "$AGENT_VERSION"
test -n "$AGENT_SUM"
test -n "$AGENT_COMMIT"
printf 'AGENT_VERSION=%s\nAGENT_SUM=%s\nAGENT_COMMIT=%s\n' \
  "$AGENT_VERSION" "$AGENT_SUM" "$AGENT_COMMIT"

Transfer those three values to the agent host through the same trusted channel as the server release. Verify the module downloaded there against the server’s checksum and commit before checking out the exact commit. These comparisons fail closed if a tag was rewritten:

set -euo pipefail
AGENT_MODULE=github.com/DeveloperDurp/durpdeploy-agent
AGENT_VERSION=v0.1.2-0.20261017113854-411042912a65 # use the value recorded from the server checkout
AGENT_SUM='h1:...' # use the value recorded from the server checkout
AGENT_COMMIT=... # use the value recorded from the server checkout
AGENT_DOWNLOAD=$(go mod download -json "$AGENT_MODULE@$AGENT_VERSION")
DOWNLOADED_SUM=$(printf '%s' "$AGENT_DOWNLOAD" | jq -e -r '.Sum // empty')
DOWNLOADED_COMMIT=$(printf '%s' "$AGENT_DOWNLOAD" \
  | jq -e -r '.Origin.Hash // empty')
test -n "$AGENT_VERSION"
test -n "$AGENT_SUM"
test -n "$AGENT_COMMIT"
test "$DOWNLOADED_SUM" = "$AGENT_SUM"
test "$DOWNLOADED_COMMIT" = "$AGENT_COMMIT"
git clone https://github.com/DeveloperDurp/durpdeploy-agent.git
cd durpdeploy-agent
git checkout --detach "$AGENT_COMMIT"
test "$(git rev-parse HEAD)" = "$AGENT_COMMIT"
make build

BUILD_INFO=$(go version -m ./durpdeploy-agent)
printf '%s\n' "$BUILD_INFO"
printf '%s\n' "$BUILD_INFO" | grep -Fq "vcs.revision=$AGENT_COMMIT"
printf '%s\n' "$BUILD_INFO" | grep -Fq 'vcs.modified=false'
SOURCE_SHA=$(sha256sum ./durpdeploy-agent | awk '{ print $1 }')
sudo install -o root -g root -m 0755 ./durpdeploy-agent /usr/local/bin/durpdeploy-agent
INSTALLED_SHA=$(sha256sum /usr/local/bin/durpdeploy-agent | awk '{ print $1 }')
test "$SOURCE_SHA" = "$INSTALLED_SHA"
printf 'installed sha256: %s\n' "$INSTALLED_SHA"

The block stops at the first failed check. Building cmd/agent in the standalone repository does not create or open a database. Keep the printed digest with the deployment record so the installed artifact can be checked later.

The server accepts agent/1 for legacy Bash-only host agents, agent/2 for host interpreter capabilities, and agent/3 for remote containers. Upgrade the server first, then enable container agents. Prefer the exact agent revision pinned by the server. This change pins the reviewed PR revision; the agent tag is a later release step.

Create the service account and private state directory:

sudo useradd --system --home-dir /var/lib/durpdeploy-agent \
  --shell /usr/sbin/nologin durpdeploy-agent
sudo install -d -o durpdeploy-agent -g durpdeploy-agent -m 0700 \
  /var/lib/durpdeploy-agent
sudo install -m 0600 /tmp/durpdeploy-agent.env /etc/durpdeploy-agent.env

For first pairing, a foreground run can use a protected environment file owned by the agent user:

sudo install -o durpdeploy-agent -g durpdeploy-agent -m 0600 \
  /tmp/durpdeploy-agent.env /var/lib/durpdeploy-agent/agent.env
sudo -u durpdeploy-agent sh -c \
  'set -a; . /var/lib/durpdeploy-agent/agent.env; exec /usr/local/bin/durpdeploy-agent'

The systemd procedure below is preferred for production. Its EnvironmentFile=/etc/durpdeploy-agent.env is read by systemd, while the service process receives the values without putting them in shell history.

Docker or Podman Compose

The standalone agent repository’s Compose service is a co-located compatibility and validation path. It is not a server sidecar or a production placement recommendation. Production agents should run remotely on the host where the deployment commands belong.

Create compose.agent.env with the same local variables, use mode 0600, and do not include a credential-shaped literal:

DURPDEPLOY_AGENT_LISTEN_ADDR=0.0.0.0:23187
DURPDEPLOY_AGENT_VERSION=<agent-version>

Docker Compose:

chmod 0600 compose.agent.env
docker compose -f /path/to/durpdeploy-agent/compose.yml up -d --build agent
docker compose -f /path/to/durpdeploy-agent/compose.yml ps agent
docker compose -f /path/to/durpdeploy-agent/compose.yml logs -f agent

Podman Compose:

chmod 0600 compose.agent.env
podman compose -f /path/to/durpdeploy-agent/compose.yml up -d --build agent
podman compose -f /path/to/durpdeploy-agent/compose.yml ps agent
podman compose -f /path/to/durpdeploy-agent/compose.yml logs -f agent

The default socket-free service mounts one private volume at /var/lib/durpdeploy-agent and a private /tmp. It has no /data mount, server secret, Docker socket, host network, host cgroup mount, or persistent inbound listener. Its root is read-only and its CPU, memory, and process count are limited. The state volume contains agent identity, not SQLite or a server backup.

systemd installation and operations

Install the unit from your standalone durpdeploy-agent checkout after installing its binary and environment file. This unit belongs to the agent repository, not the DurpDeploy server repository:

sudo install -m 0644 /path/to/durpdeploy-agent/systemd/durpdeploy-agent.service \
  /etc/systemd/system/durpdeploy-agent.service
sudo systemctl daemon-reload
sudo systemctl enable --now durpdeploy-agent
sudo systemctl status durpdeploy-agent --no-pager

The unit runs the agent and selected interpreter as durpdeploy-agent, sets the state directory, uses /etc/durpdeploy-agent.env, applies a private UMask=0077, and permits writes only to the agent state directory. It also applies NoNewPrivileges, private mounts and /tmp, and service cgroup limits. Keep both /etc/durpdeploy-agent.env and the state directory inaccessible to other users:

sudo chown root:root /etc/durpdeploy-agent.env
sudo chmod 0600 /etc/durpdeploy-agent.env
sudo chown -R durpdeploy-agent:durpdeploy-agent /var/lib/durpdeploy-agent
sudo chmod 0700 /var/lib/durpdeploy-agent

Operations:

sudo systemctl start durpdeploy-agent
sudo systemctl status durpdeploy-agent --no-pager
sudo journalctl -u durpdeploy-agent -f
sudo systemctl restart durpdeploy-agent
sudo systemctl stop durpdeploy-agent

After a successful pairing, keep the state directory across an ordinary restart. The agent resumes its pinned outbound connection without manual server settings.

Verify before deploying

sudo systemctl is-active durpdeploy-agent
sudo journalctl -u durpdeploy-agent -n 50 --no-pager

In the UI, confirm the agent is active and its heartbeat is current. Start a small non-production deployment first. A remote step must show every matching agent run, not local execution.

Troubleshooting

404 or a stale binary

The dedicated listener has its own routes in /agent/v1. The browser and API routes are not valid substitutes. A 404 often means the agent URL points at Caddy or port 443 instead of the direct listener. A stale server binary can also cause a 404. Confirm that the server has all three listener variables. Use ss -ltn to check port 23187. Verify the installed binary. Restart the server. In a checkout of the standalone agent repository, rebuild with make build, verify the binary as described above, and install it again when its behavior does not match the checkout.

Wrong fingerprint

The agent rejects an unpinned or changed certificate. For an unexpected change, stop the agent. Investigate the server identity through a trusted channel. Then do the controlled re-pair or planned rotation procedure below. Do not use trust-all TLS or accept a fingerprint copied from an untrusted connection.

Expired or reused pairing code

Pairing codes expire after 10 minutes and are consumed once. Restart the unpaired local listener to obtain a fresh code. For an already active agent, revoke and re-pair it first.

No match

Check that the paired agent has the expected capability and environment labels and is reporting a healthy heartbeat. The agent must be active and paired, its environment label must match the deployment environment, and it must have every capability label required by the remote step. A no-match step fails without running locally.

Revoked agent

Revocation blocks the old identity. Stop the old process, confirm the agent is revoked in Admin, Agents, use Re-pair, and pair the replacement with a controlled state directory. Do not restore the old private key onto an unrelated host.

Lost agent

Stop the service and inspect the agent and server journals. Started work that misses heartbeats for 45 seconds becomes lost and is not automatically replayed. Review the original deployment, fix the host, and create an explicit new deployment.

Cancel unconfirmed

The server keeps the deployment running until the agent acknowledges a cancel. If the 30-second acknowledgement deadline expires, routing becomes cancel_unconfirmed. Treat the outcome as uncertain, inspect the host and logs, and create a new deployment only after confirming what happened. Do not retry the original deployment or assume cancellation succeeded.

Revocation, re-pairing, and fingerprint rotation

For a lost or compromised host:

1. Stop the agent service and isolate the host.
2. In Admin, Agents, revoke the agent.
3. Review redacted agent events, deployment history, and the audit log.
4. Prepare a replacement host with a new private state directory.
5. Use Re-pair on the revoked agent and pair the replacement. Update only the replacement local listener configuration.

For planned server certificate rotation, keep the current server identity and fingerprint working first. The server can advertise the old and pending pins through an authenticated heartbeat. Agents stage [old, pending], connect to the pending certificate, and remove the old pin only after that verified connection succeeds. Do not delete the old server identity or pin until all agents have promoted the pending pin. If rotation is interrupted, restore the old listener identity and complete the staged process rather than replacing the state file by guesswork.

Upgrades, rollback, and backup scope

Migrate the dedicated agent port

The default dedicated listener port changes from 10943 to 23187. The web listener ports do not change. Existing paired agents retain their stored server endpoint. Changing DURPDEPLOY_AGENT_PUBLIC_URL affects new pairings only.

1. Drain agents and wait for their current deployments to finish.
2. Back up the database, server encryption key, server identity directory, and agent state directories.
3. Preserve these files and their permissions during the upgrade.
4. Set DURPDEPLOY_AGENT_LISTEN_ADDR=0.0.0.0:23187.
5. Set DURPDEPLOY_AGENT_PUBLIC_URL=https://YOUR_EXISTING_HOST:23187 for new pairings.
6. Allow inbound TCP 23187 from agent networks and outbound TCP 23187 from agent hosts.
7. Update container mappings to 23187:23187 for new installations.
8. For existing agents, retain external port 10943 with an additional 10943:23187 mapping or equivalent TCP forwarding.
9. Preserve direct TLS passthrough and the existing server certificate.
10. Restart the server and agents with their existing state directories.
11. Verify polling, a test deployment, and streamed logs before resuming normal work.

For Helm, set agent.port=23187 and agent.listenAddress=0.0.0.0:23187. Set agent.publicURL to the reachable agent endpoint. Preserve the identity PVC and an external forwarding path for existing endpoints.

Do not remove the old mapping or firewall access while paired agents still use it. The current agent stores its endpoint in private pairing state. Use an agent-supported endpoint migration procedure before removing the compatibility path. Do not delete state, reset identities, change pins, or edit pairing records to change a port. Alternatively, retain 10943 through explicit listener and public URL overrides until coordinated migration is available.

Local development publishes agent pairing on host port 23188 and container port 23187. The pinned development image accepts the explicit listener setting without an image upgrade. Override DEV_AGENT_PORT if the host port is already in use. Coordinate standalone agent defaults with agent issue #23.

Upgrade the server and agent to compatible releases together. For an agent, check out the standalone version pinned by the server, build and verify a new durpdeploy-agent, install it over the binary, and restart through the normal controlled pairing procedure. Preserve the state directory across an protocol-compatible upgrade. If rollback is necessary, stop the service, install the previous verified binary, and restore the matching known-good configuration. Do not delete pins or identity files during an ordinary binary rollback.

Back up the server SQLite database and the matching server encryption key as a pair. Litestream or SQLite backup procedures cover server database state. Back up an agent state directory only to preserve that exact paired identity. It is not a database, and restoring it to another host transfers the identity.

Safety rules

Do not route port 23187 through Caddy. Do not use a central CA, trust-on-first- use, SSH, a trust-all TLS setting, inbound agent ports, or a shared agent state directory. Never give an agent the server database, server secret key, or control-plane state directory. The default profile is socket-free. Opt-in container execution exposes a local runtime socket to the supervisor only; host-mode scripts then inherit runtime authority. Workload containers get no socket.

For protocol details and fixed timing limits, see agent-protocol.md. For server deployment and backup operations, see deploy.md and backup-restore.md.