# Continuous monitoring and HTTP endpoints

Choose **serve mode** when you want monitoring to keep running and expose HTTP
status or Prometheus metrics. For a command that runs once and exits, including
scheduled jobs, use [simple mode](simple.md).

Install version 0.0.56 or newer using the
[installation instructions](https://github.com/decryptus/monit-docker#installation). The process needs Docker
access. Prometheus and Grafana are optional: you can use the HTTP endpoints alone.

For all three services with the dashboard already loaded, use the
[Docker Compose quickstart](compose.md).

`serve` uses the same monitoring engine as `monit` and `cron`. It runs an
immediate cycle, then waits 30 seconds after each completed cycle by default.
Cycles are sequential, including when collection takes longer than the interval.
It adds no runtime dependencies. Read endpoints use an in-memory cache; they never collect Docker statistics or
execute an action. The optional manual-action endpoint queues work for the
monitoring scheduler.

## Start a local monitor

In a terminal, start monitoring all containers and leave the command running:

```sh
monit-docker serve --interval 30
```

In a **second terminal**, check the endpoints:

```sh
curl http://127.0.0.1:9808/healthz
curl -i http://127.0.0.1:9808/readyz
curl http://127.0.0.1:9808/v1/status
curl http://127.0.0.1:9808/metrics
```

Wait for `/readyz` to return HTTP 200 and `{"ready": true}`. A 503 response
means that no successful fresh result is available yet. At least one container
must match. If it stays unready, inspect `/v1/status` and the logs, then see
[troubleshooting](troubleshooting.md). Stop the monitor with Ctrl+C.

To select particular containers, use for example
`monit-docker --name 'web*' serve --interval 30`; replace the pattern with names
that exist on your Docker daemon.

Global configuration and selectors precede `serve`. The default listen address
is `127.0.0.1` and port is `9808`. `--bind` accepts an IPv4 address and `--port`
accepts 1–65535. Without `--cmd`, the process observes containers only and needs
no state file. All resources are collected by default; repeated `--rsc` options
select a subset, for example `--rsc cpu_percent --rsc mem_usage`.

### Configure serve in YAML

Version 1.3.0 accepts a top-level `serve` section in
`monit-docker.yml`. For example:

```yaml
serve:
  bind: 127.0.0.1
  port: 9810
  interval: 60
  rsc: [status, health]
  status_token_file: /run/secrets/status
```

Start this configuration with `monit-docker serve`, or select a file with
`monit-docker -c /path/to/monit-docker.yml serve`. An explicit CLI option
overrides that YAML field, even when the argument equals the built-in default.
For example, `serve --port 9808 --rsc pid` uses port 9808, collects only PID,
and keeps the configured interval and status token file.

Use the long flag name with underscores in place of hyphens. Use YAML
booleans for switches and ordered lists for `rsc` and `cmd` (including
conditional rules). Each CLI list replaces the whole corresponding YAML
list. To turn off a YAML switch for one launch, use `--no-allow-actions`,
`--no-allow-maintenance` or `--no-trust-proxy-user`. Keep only file paths in the five token settings. `--dry-run` is
CLI-only. Global options still precede the subcommand. Named scenarios
(`monit-docker run NAME`) use their own configuration. The
[configuration reference](config-cli-contract.md) lists the available settings.

| Purpose | Keys under `serve:` |
|---|---|
| HTTP listener | `bind`, `port` |
| Collection | `interval`, `stale_after`, `rsc` |
| Automatic actions | `cmd`, `state_file`, `cooldown`, `trigger_after`, `max_gap`, `max_restarts` |
| Manual actions and proxy identity | `allow_actions`, `allow_maintenance`, `action_origin`, `action_cooldown`, `trust_proxy_user` |
| Secret file paths | `action_token_file`, `audit_read_token_file`, `notification_token_file`, `metrics_token_file`, `status_token_file` |

To protect the cached status (container names, IDs and measurements), start the
agent with `monit-docker serve --status-token-file /run/secrets/status`.
Use a dedicated file containing 64 lowercase hexadecimal characters and an
optional final newline. The agent loads the file at startup. GET and HEAD
require a single `Authorization: Bearer TOKEN` header when enabled. Missing
or incorrect credentials return HTTP 403. Health and readiness probes stay
available; Prometheus uses its own `--metrics-token-file`.

For example, read status through a local connection or a trusted HTTPS endpoint
without passing the secret as a command-line argument:

```sh
{ printf 'Authorization: Bearer '; tr -d '\n' < /run/secrets/status; printf '\n'; } |
  curl --fail --silent --show-error --header @- "$AGENT_URL/v1/status"
```

When using the web UI, configure its authenticated reverse proxy to inject the
status bearer on the upstream `/v1/status` location. The stock Nginx
configuration clears the upstream Authorization header. Replace that directive
for this location only and keep the secret on the proxy, outside browser code.

The optional
[separate UI component](ui.md) supplies an authenticated Nginx frontend. Manual
start/stop/restart is disabled by default; enabling its API requires explicit
configuration, a proxy secret, an allowed HTTPS origin and persistent state.
Keep it on loopback or a trusted private network. To access it from elsewhere,
place an authenticated TLS reverse proxy in front; setting `--bind 0.0.0.0`
explicitly exposes the endpoint data on every IPv4 interface. Container names,
IDs, statuses and measurements are visible. Configuration, command text and raw
exception messages are not exposed by the API.

## Run serve in Docker

For a local Docker Engine using `/var/run/docker.sock`:

```sh
docker run --rm --name monit-docker-serve \
  -p 127.0.0.1:9808:9808 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  decryptus/monit-docker:0.0.57 monit-docker serve --bind 0.0.0.0
```

Leave this running and use the same `curl` checks from a second terminal on the
host. The listener binds inside the container, while the published host port is
restricted to loopback. Docker socket access allows control of that daemon.
This example observes only; if you add rules, also mount a persistent state
directory and set `--state-file` inside it.

## Add history and charts

1. Configure [Prometheus scraping](metrics.md#prometheus-configuration) and check
   that `up{job="monit-docker"}` and `monit_docker_ready{job="monit-docker"}` are 1.
   The supplied localhost target works when Prometheus runs on the host alongside
   the native process or the Docker example's published port.
2. Add Prometheus as a Grafana data source and [import the example dashboard](grafana.md).
3. Select the data source and filters. Wait for multiple scrapes before expecting
   network/I/O rates or action rates to appear.

If Prometheus runs in a separate container, its loopback is different. Use a
reachable private address or shared container network and adjust the scrape
target accordingly. Prometheus keeps the history; `serve` keeps only its latest
result in memory.

## Optional remediation

```sh
monit-docker --name 'web*' serve --interval 30 \
  --state-file /var/lib/monit-docker/web.json --cooldown 300 \
  --dry-run --cmd-if 'mem_percent > 90 ? restart'
```

For sustained conditions, add [`--trigger-after` and `--max-gap`](trigger-delay.md);
they use the same persistent tracking as cron.

Rules require an explicit `--state-file`, including in dry run. Review behavior,
then remove `--dry-run` to execute actions. Each cycle acquires the same state
lock and reserves the same persistent cooldowns as [cron](cron.md). The lock is
released between cycles. Using the same file lets cron and serve coordinate;
separate files do not coordinate actions. Lock contention (117), state failures
(118), Docker errors or action failures mark that cycle as failed. The process
remains alive and retries on the next cycle. The first failed action stops evaluation
of the remaining containers. Until a complete cycle succeeds, `/readyz` returns
503 and `/v1/status` exposes an empty container list. The successful actions of a partial
cycle are not rolled back, and their reservations remain on disk.

Automatic Docker restarts default to three attempts per container ID across all
rules in the state file, including failed attempts. Configure `--max-restarts`
and explicitly rearm after intervention with `restart-reset`; see
[restart limits](restart-limit.md). Exhausting the budget skips affected rules
with `restart-limit` while monitoring continues. It is not a failed cycle.

Dry-run reservations are simulated within each cycle and discarded afterwards.
Action-decision counters appear in the API and metrics; individual command text
is not served. Exec failures retain the agent error code 116 in this long-running
mode; there is no process-exit propagation option.

## Cache, freshness and lifecycle

The latest successful cycle's snapshots are held in a synchronized memory cache.
They are replaced as a complete set; requests never see half of a cycle. The
cache holds no history and is empty after a restart. Cooldown reservations and optional [trigger observations](trigger-delay.md)
are persisted; Prometheus can store time-series history externally.

`--stale-after` defaults to `max(90, 3 * interval)` seconds and must be at least
the interval. Freshness uses a monotonic clock, so system clock corrections do
not make old measurements fresh. Timestamps in the API use Unix wall-clock
seconds. Cache age is measured from successful cycle completion, not separately
for each container's sampling time.

Before the first success, after a failed cycle, or when data expires,
`ready` is false and the current `containers` array is empty and per-container metrics are omitted.
After a failure, the UI can still show the last complete snapshot in a dated,
read-only section. The failure notice names the action and container when available. Process counters and the last-success timestamp remain available.
During a cycle, the previous result remains available while it is still fresh.
The HTTP listener remains responsive during a slow or blocked Docker call.

SIGINT and SIGTERM request shutdown: finish the current cycle, stop scheduling,
close the HTTP socket and exit successfully. Existing Docker client timeouts
still apply; there is no cycle-wide deadline or forced cancellation. A hung
Docker call can therefore delay shutdown. HTTPdis dispatches the API routes;
Sonicprobe provides eight request workers and a pending queue of eight requests.
Active requests have five-second socket timeouts. A full queue applies backpressure
to the accept loop; shutdown discards queued connections. Use a
reverse proxy for public-facing HTTP limits and supervision for process recovery.
Configuration is loaded at startup; restart the process after changing it.

Route declarations and JSON/Prometheus serialization live in the HTTP adapter;
the monitoring core does not depend on HTTPdis. HTTPdis is loaded only in `serve`
mode, and Nginx serves the optional UI separately. HTTPdis has a process-global
route registry: run this agent in its own process, rather than embedding it in
another HTTPdis/DWho application. The adapter requires HTTPdis 0.6.28 or later
and Sonicprobe 0.3.53 or later; the Docker image includes their libmagic runtime.

## HTTP contract

The [HTTP API and metrics baseline](http-api-contract.md) documents exact field
types, capability discovery, request authentication, errors, journal cursors and
tested `curl` examples for the private agent. The summary below covers common use.

| Endpoint | Success / failure | Purpose |
| --- | --- | --- |
| `/healthz` | 200 while HTTP is responsive | `{"alive": true}`; does not claim Docker is healthy |
| `/readyz` | 200 ready, otherwise 503 | `{"ready": true}` or `{"ready": false}` |
| `/v1/status` | 200, including degraded states | Cached status described below |
| `/metrics` | 200, including degraded states | [Available Prometheus metrics](metrics.md) |

GET and HEAD are supported on these read endpoints. POST, PUT, PATCH, DELETE
and OPTIONS return 405 there; unknown read routes return 404. Responses disable
caching. There is no configuration-update or trigger-cycle endpoint. The optional
manual action endpoint is described below.

The `/v1/status` object has `api_version: 1` and these fields. Clients should
ignore additional fields, so later additions need not change the URL version.

| Field | Meaning |
| --- | --- |
| `running` | A collection/remediation cycle is in progress |
| `ready` | Latest completed cycle succeeded and is fresh |
| `last_cycle_success` | Outcome of the last completed cycle; false at startup |
| `last_cycle_finished_at` | Unix seconds of last completion, or null |
| `last_success_at` | Unix seconds of last successful completion, or null |
| `age_seconds` | Monotonic age of the last success, or null |
| `last_error_code` | Agent error code for the last failed cycle, otherwise null |
| `cycles_total` / `errors_total` | Completed / failed cycles since process start |
| `actions` | Counters for `executed`, `cooldown`, `pending`, `restart-limit`, `maintenance` and `dry-run` decisions |
| `containers` | Fresh complete snapshot list; empty when not ready |
| `manual_actions` | Capability object, always present: `enabled`; when enabled, `allowed_states` and up to 32 `recent` request results |
| `audit_enabled` | Whether the authenticated journal reader is enabled |

Each container has `id`, `name`, `status`, `pid`, `mem_usage`, `mem_limit`,
`mem_percent`, `cpu_percent`, `io_read`, `io_write`, `net_tx`, and `net_rx`.
It also exposes normalized `health`, filesystem samples, and `restart_attempts`
and `restart_limit` when an automatic restart policy is active.
Newer agents also include `manual_actions_protected`, a boolean derived from the
`monit-docker.protected` Docker label. It restricts only manual actions; it is not
a metric or a rule condition. Older agents omit this field. See
[container protection](ui.md#protect-a-container-from-manual-actions).
Byte fields and percentages are raw numbers, not formatted strings. Unknown or
unrequested fields are null. A successful metadata-only cycle can be ready even
if no numerical metrics were requested. No matching container produces the
existing error 114 and makes the monitor unready.


## Optional manual action API

A protected container stays in `GET /v1/status`, but a new manual request for it
returns HTTP **403** with `{"error": "container_protected"}`. The engine also
rechecks protection from fresh metadata before execution: an accepted request
can finish as `failed` with that same reason. Automatic rules are unaffected.

See [UI and authentication setup](ui.md) before enabling writes.
`POST /v1/actions` accepts only an `application/json` body up to 1024 bytes:

```json
{
  "request_id": "e793021f188b443b880a791be38e277d6e",
  "container_id": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
  "action": "restart"
}
```

Use a fresh cryptographically random 32-character lowercase hexadecimal request
ID and an exact full 64-character container ID. Extra fields, aliases and command
arguments are rejected. This endpoint requires `X-Monit-Action-Token` supplied by
the trusted proxy and an exact configured `Origin`. The token never appears in
status or frontend files. Missing/invalid credentials or origin return 403;
disabled writes return 405, unsupported content type 415, oversized body 413,
invalid input 400 and rejected preconditions 409.

Alongside `start`, `stop` and `restart`, `action: "restart-reset"` explicitly rearms
the selected container's automatic restart counter. It requires a nonzero
counter in fresh status and rechecks persistent state before resetting it. A
missing/already-reset budget is rejected with `no_restart_attempts`. The same
protection, queue, deduplication, cooldown and audit controls apply; no Docker
restart is executed by this operation. Clients must use `allowed_states` to
discover support and confirm that rules may act again on the next cycle.

HTTP 202 returns the request record, including for a retained duplicate ID with
the same payload and actor. Reusing a retained ID for another payload or actor returns 409. Records
contain `request_id`, `container_id`, `action`, `actor`, `container_name`, `status`, `submitted_at`,
`finished_at`, `error` and `error_code`; timestamps are Unix seconds. Status is
`queued`, `running`, `succeeded` or `failed`. Poll `/v1/status` for the result.
Errors expose a short reason and optional numeric agent code, never raw exception
text. Successful submission is not successful execution. A proxy timeout is an
ambiguous result; clients must not blindly repeat a mutation.

Manual requests execute between monitoring cycles and cause a fresh collection
after an attempt. Measurements are invalidated while a manual operation runs.
Read endpoints remain responsive; they never execute queued work. Existing
`actions` counters and their Prometheus metrics describe autonomous rule
decisions only, not manual requests. The manual queue/results are in memory;
only their per-container cooldown reservations are persisted. The UI guide
describes expiry, restart behavior, selection and autonomous-rule interactions.