sakana ¶

1.1 Why watch the watcher ¶

A Prometheus Alertmanager cannot usefully monitor itself. When it stops working, the symptom is silence—and silence is exactly what everything being in order looks like. An operator who relies on Alertmanager to be told about problems has no way to be told that Alertmanager is the problem.

This is the gap sakana fills, and it fills it for a structural reason rather than a clever one: sakana’s reporting path is a pipe to a command of the operator’s choosing, which shares no process, no configuration and no network dependency with Alertmanager’s notification path. When one of the two breaks, the other still speaks.

The same argument applies, less dramatically, to Prometheus: a configuration reload that failed or a rule that cannot be evaluated leaves a server that answers every query, exports every metric and looks entirely healthy, while the alert it was supposed to raise will never fire.

1.2 Design decisions worth knowing ¶

No state on disk

Failure counters live in memory. A restart of sakana starts every counter from zero, which means the first check after a restart reports an existing problem immediately.

One counter per error class

Every error class of every exchange counts its failures separately. A denomination warning that has been pending for weeks can therefore not delay the report of an exchange that has just become unreachable.

Unevaluated classes keep their counters

While an exchange cannot be reached, nothing can be said about its keys. The key classes are then neither escalated nor reset, so a pending warning survives an outage and does not produce a second alert when the exchange comes back.

Expired keys are normal

An exchange keeps publishing keys whose validity has ended. sakana reports expiry only when no usable key remains, which is the situation in which key rotation has actually stopped. Signing keys and denominations have separate warning horizons, since exchanges provision them months and days ahead respectively.

TLS 1.3 only

sakana offers TLS 1.3 and nothing else. A front end that can only speak TLS 1.2 is reported as a failed handshake rather than connected to with the older protocol, so that an outdated HTTPS configuration becomes visible instead of staying silently in use.

No dependencies

sakana is written in Go and uses only the standard library, including for TLS, HTTP and JSON. This is also why the two Prometheus kinds are checked through their JSON APIs and not through /metrics: the exposition format would need a parser, and the official client libraries would end this property.

Meta-monitoring, not alert relay

sakana checks whether the alerting stack works. It deliberately does not read the alerts an Alertmanager holds and does not forward them. Relaying them would make sakana a second notification path, which fits its machinery badly—a firing alert should not wait for TOLERANCE checks, and fifty firing alerts would mean fifty invocations of COMMAND—and would duplicate work Alertmanager already does well. The one thing Alertmanager cannot do is notice its own silence, and that is what sakana is for.

Scrape targets are not checked

On a real Prometheus some scrape target is nearly always down. Reporting that would train the operator to ignore sakana, which costs more than the information is worth.

The reporting command gets a cleared environment

COMMAND does not inherit sakana’s environment; it receives exactly what the configuration gives it. A report must not depend on who started the daemon, because a helper that works by hand and fails under systemd is a wasted evening.

One class, one counter, even for several endpoints

A check of a Prometheus or an Alertmanager makes several requests. All the secondary ones share one error class, whose verdict is the aggregate over the whole check, so that a later endpoint’s success cannot reset the counter an earlier endpoint’s failure raised.

3.1 Kinds of system ¶

KIND selects what a section describes and therefore what is checked beyond reachability. It is exchange, alertmanager or prometheus, and defaults to exchange.

[alertmanager-ops]
KIND = alertmanager
BASE_URL = https://mon.example.com/alertmanager/
AUTH_BEARER = ...
CLUSTER_PEERS = 3

[prometheus-ops]
KIND = prometheus
BASE_URL = https://mon.example.com/prometheus/
AUTH_BASIC = monitor:...

BASE_URL is always the base, never the endpoint: sakana appends keys for an exchange and -/healthy for the other two. For a Prometheus or an Alertmanager behind a reverse proxy this is the prefix the service’s own --web.external-url names, because that is where its /-/healthy lives. A 404 on the primary endpoint is usually a missing prefix, and sakana’s report for that status says so.

An option belonging to another kind is rejected, not ignored. A SIGNKEY_EXPIRATION_HORIZON in a prometheus section stops sakana from starting, with the line number. Silently ignoring it would be the worse behaviour for a monitor: the operator would believe a check is configured that is not.

3.2 Authentication ¶

Prometheus and Alertmanager are often behind a token or behind basic auth. AUTH_BEARER sets a bearer token and AUTH_BASIC takes user:password; the two are mutually exclusive, because a request carries one Authorization header and silently preferring one would hide a mistake. Both are available for every kind—an exchange behind basic auth on a staging host is plausible enough.

Both are credentials. They never appear in --test output, in a log line or in a report. See Credentials in the configuration file, for the file permissions that follow from keeping them here.

3.3 Durations ¶

Durations use the Taler and GNUnet spelling: 5 s, 10m, 1 h, 1 d, 4d 32s. The units are us, ms, s, m, h, d, w and y, with the long forms (seconds, minutes, …) and the abbreviations " and ' accepted as well.

3.4 Frequency and jitter ¶

The delay between two checks of one exchange is FREQUENCY randomized by ±5%. Several monitors watching the same exchange therefore drift apart instead of hitting it in lockstep. The first check of every exchange happens immediately when sakana starts.

3.5 Tuning the horizons ¶

TLS_RENEWAL_HORIZON (five days by default) is the point at which a TLS certificate that has not been renewed becomes a report. Five days is a deliberate compromise: certbot renews at 30 days remaining and retries twice a day, so a certificate with less than five days left means that renewal has been failing for weeks.

The key material has two horizons, because exchanges provision its two kinds on entirely different scales.

SIGNKEY_EXPIRATION_HORIZON (30 days by default) is the point at which expiring online signing keys become a report. Exchanges sign those months ahead—the two public exchanges are currently at two months and ten months respectively—so 30 days of notice is both ample and easy to satisfy.

DENOMINATION_EXPIRATION_HORIZON (14 days by default) is the point at which denominations that can no longer be withdrawn become a report. It is shorter on purpose: denominations are rotated within days, and an exchange publishes them only as far ahead as its LOOKAHEAD_SIGN reaches. The demo exchange at https://exchange.demo.taler.net/ publishes them about 30 days ahead and dips to roughly three weeks between signing runs, so a 30 day horizon would warn about an exchange that rotates exactly as intended. A horizon below the lookahead of the deployment still catches the real failure, because a signing pipeline that has stopped lets the remaining horizon shrink to zero.

Raise the denomination horizon if the exchange you monitor signs further ahead and you want earlier warning; KEY_EXPIRATION_HORIZON sets both at once if you want one knob.

4.1 Reaching the system ¶

Name resolution, the TCP connection and the TLS handshake are performed by sakana itself rather than left to the HTTP client, so that each failure can be classified precisely:

dns-failure

The host name does not resolve. The report distinguishes a name that does not exist from a resolver that did not answer.

host-unreachable

The connection attempt was answered with neither a SYN-ACK nor a RST, or an ICMP host or network unreachable arrived.

port-unreachable

The connection was refused with a RST: something is listening on the host, but not on this port.

4.2 The TLS connection ¶

sakana validates the certificate chain itself, with the system trust store plus every root given with --trust. Because validation happens after the handshake, with the chain in hand, the report can name the certificate that is at fault even when it is rejected:

Only TLS 1.3 is offered. TLS 1.2 and every older version are refused, and a peer that cannot negotiate TLS 1.3 therefore fails the handshake; the detail of the report names the TLS version as the reason. There is no option to re-enable TLS 1.2.

tls-handshake

The handshake failed, or validation rejected the chain. The detail names the reason: a peer that does not support TLS 1.3, a certificate that is not valid at this time, an unknown certification authority, a host name mismatch, a peer that did not speak TLS at all, or the TLS alert the peer sent.

tls-expiring

The handshake succeeded, but the certificate expires within TLS_RENEWAL_HORIZON.

4.3 The HTTP response ¶

http-status

The status was not 200 OK. The report includes the Taler error code and hint if the body carries them, and adds what a 502 or a 503 from /keys usually means.

http-timeout

The connection was established, but no complete response arrived within TIMEOUT.

keys-parse

The body is not a JSON object, or not an exchange /keys object: a missing version, currency, master_public_key, signkeys, denominations, global_fees or wire_fees member, or a malformed timestamp, all land here.

4.4 Key material of an exchange ¶

Both key checks ask whether anything usable is left. Keys whose validity has ended stay in the response and are not by themselves a problem.

signkeys-expired

No entry of signkeys has a stamp_expire in the future. The exchange cannot sign anything and is effectively out of service.

signkeys-expiring

No entry of signkeys is valid beyond SIGNKEY_EXPIRATION_HORIZON.

denoms-expired

No denomination has a stamp_expire_withdraw in the future, so no coin can be withdrawn any more. Denominations marked lost are not counted as usable.

denoms-expiring

No denomination can be withdrawn beyond DENOMINATION_EXPIRATION_HORIZON.

When the expired class fires, the expiring class of the same kind is left unknown instead of being reported as well: the worse report subsumes the milder one, and the milder counter neither escalates nor resets.

4.5 Fees of an exchange ¶

Fees must cover the period from now until the end of the current year, and, during December, until the end of the next year as well — by December an exchange has to have the fees of the coming year signed. sakana sorts the fee periods, walks them and reports the first moment that is not covered. Periods that end exactly when the next one starts are contiguous, which is how consecutive years are configured.

global-fees-missing

global_fees does not cover the required period.

wire-fees-missing

The wire fees of at least one wire method in wire_fees do not cover it. The report names the methods and their configured periods.

4.6 An Alertmanager ¶

After /-/healthy, sakana asks for /-/ready and /api/v2/status.

/-/ready answering 503 is the Alertmanager’s own statement that it is running but not ready to serve, which is am-not-ready. That is briefly normal at start-up and while the gossip cluster settles, which is what TOLERANCE is for; persisting, it means the instance never became ready. A different failure of that endpoint says nothing about readiness, so it becomes am-api-error instead and am-not-ready stays unevaluated.

/api/v2/status yields the rest:

The gossip cluster

A status of settling, an unknown status, or fewer peers than CLUSTER_PEERS requires is am-cluster-degraded. A single Alertmanager with no high availability configured reports its cluster as disabled; that is normal and is not a finding, unless CLUSTER_PEERS says a cluster is expected. A split cluster is worth knowing about because the instances then each notify on their own, duplicating every alert, and because a silence created on one is not in effect on the others.

The start timestamp

An Alertmanager that started less than one FREQUENCY ago was not running at the previous check, so it has restarted: am-restarted, a warning. One of those after an upgrade or a reload is expected; repeated, it means the process is crash-looping, and an Alertmanager that restarts keeps no notification state, so alerts already sent may be sent again and pending ones may be lost.

Because the timestamp is absolute, this is an instantaneous predicate: it needs none of the cross-check state sakana deliberately does not keep.

The loaded configuration

Only its size is used, and only in the ‘service restored’ note. The document carries the whole alertmanager.yml, SMTP passwords and webhook URLs included, and a report is piped to a command that may mail or text it onwards, so its contents must never reach a report. There is a test that asserts exactly that.

A document that cannot be parsed, or that lacks the cluster or uptime field, is am-status-invalid; that usually means BASE_URL does not point at an Alertmanager at all. The cluster and restart classes are then left unevaluated rather than marked in order.

4.7 A Prometheus server ¶

After /-/healthy, sakana asks for /-/ready, /api/v1/status/runtimeinfo and /api/v1/rules.

Every Prometheus API response is wrapped in an envelope carrying a status field, and a 200 OK may still report "status":"error". sakana checks the envelope before believing the data; an API-level error is prom-status-invalid, with the server’s own explanation in the report.

/-/ready answering 503 is prom-not-ready. After a restart this is normal while the write-ahead log is replayed, which on a large database takes minutes—and until it finishes, no rule is evaluated and no alert can fire, which is why it is worth a report rather than a shrug.

From /api/v1/status/runtimeinfo:

reloadConfigSuccess

False means the last configuration reload failed and Prometheus is still running the configuration it had before: prom-config-reload-failed. This is the quietest way for a Prometheus to stop doing what its configuration says. The process keeps running, every query works, every metric is exported, and a rule, target or alerting change made since then is simply not in effect. promtool check config on the server finds the error.

corruptionCount

Anything but zero is prom-tsdb-corruption, usually the result of a full disk or an unclean shutdown. Queries over the affected period, and the rules that depend on them, may return wrong results or none.

startTime

Less than one FREQUENCY ago is prom-restarted, exactly as for an Alertmanager. Every restart costs a write-ahead log replay during which no alert can fire.

From /api/v1/rules, any rule whose health is err is prom-rules-failing, and the report names the rule, its group, its file and its last error. This one deserves its place: an alerting rule that cannot be evaluated will never fire, however bad the situation it was written to catch, and nothing else about the server looks wrong.

Scrape targets are not checked. See Introduction, for why.

6.1 The environment of the command ¶

The command runs with a cleared environment. It does not inherit sakana’s; it receives exactly:

the configured ENV_NAME variables

Each ENV_NAME key sets NAME. A key may appear both in [sakana] and in a system’s section, and the section wins for that one variable, so a shared provider credential can live in [sakana] while each system has its own recipient.

PATH, HOME, LC_ALL=C and TZ=UTC

PATH comes from COMMAND_PATH. The locale is pinned so that a helper’s output does not depend on the machine.

the SAKANA_* variables

SAKANA_SYSTEM, SAKANA_KIND, SAKANA_BASE_URL, SAKANA_ENDPOINT, SAKANA_CLASS, SAKANA_SEVERITY, SAKANA_SUMMARY, SAKANA_VERSION and SAKANA_HOST describe the report. They are applied last, so a configured ENV_* cannot shadow them and make a report lie about itself.

SAKANA_SEVERITY is what makes a single COMMAND practical for something like SMS, where only an outage should wake anybody: the helper can route on it without parsing the report.

This fits the GNU Taler Challenger SMS helpers exactly. They take the message on standard input, the number as an argument and their credential from the environment, which is the whole of sakana’s COMMAND contract:

[sakana]
COMMAND = /usr/bin/challenger-send-sms-gatewayapi +41791234567
ENV_GATEWAYAPI_TOKEN = ...

6.2 The exit status of the command ¶

0 means the report was delivered. The band 200–210 is also a success and means ‘accepted by the provider, not yet confirmed’; the log says so and distinguishes it from a confirmed delivery. Every other status is a failed report, logged as an error.

This is the convention of challenger-send-sms(1), adopted so that those helpers work as COMMAND unmodified. The band deliberately starts at 200 and not at 1: a helper that could not be executed at all is reported as exit 1 by libgnunetutil, and an ‘accepted’ code down there would make a missing helper indistinguishable from a delivered message—the precise failure the convention exists to prevent.

6.3 The report ¶

The first line is a one-line summary suitable as a mail subject:

sakana ALERT https://exchange.example.com/: TLS certificate expiring

A blank line follows, then the headers:

System:       exchange-demo (exchange)
Base URL:     https://exchange.example.com/
Endpoint:     https://exchange.example.com/keys
Host:         exchange.example.com
Error class:  tls-expiring
Severity:     warning
Detected:     2026-11-03T21:30:00Z
Occurrences:  4 consecutive failures of this class; 3 tolerated before this report
Monitor:      sakana 0.2.0 on monitor.example.com, configuration section [exchange-demo]

Then the detail of the class — for a certificate that is expiring, its subject, issuer, names and validity, and what to check about certbot — and finally a footer that says after how many further checks the class would be reported again.

The restoration report follows the same shape, with the summary

sakana RESTORED https://exchange.example.com/: service restored

6.4 The log ¶

For every report sakana writes one line to standard error:

2026-11-03T21:30:00Z ERROR   ERROR https://exchange.example.com/ unexpected  HTTP status [http-status] -- command ok after 7ms, output: "accepted"

It carries the severity, the system’s base URL, the error class in two or three words, the class identifier and the result of the reporting command, including its exit status and its output. Output of the command is collapsed to one line and truncated, so that one event stays one line.

8.1 Under systemd ¶

The shipped unit runs sakana as the unprivileged user sakana with a tight sandbox and logs to the journal:

systemctl enable --now sakana.service
journalctl -u sakana -f

Note that the sandbox applies to the reporting command as well, since it is a child of sakana. The usual case is fine: an SMS or mail helper needs outgoing TCP, which RestrictAddressFamilies allows, and the system trust store, which ProtectSystem=strict leaves readable. But a helper that works when run by hand and fails here is almost always hitting one of those lines— one writing a spool file needs ReadWritePaths=, one talking to a local socket needs that path too. This is why the unit is a plain file in /usr/lib/systemd/system and meant to be overridden with a drop-in: relax deliberately rather than by deleting the block.

8.2 From cron ¶

A one-off check fits a cron job:

*/15 * * * * sakana --once --config /etc/sakana/sakana.conf

Keep in mind that each invocation starts with empty counters, so a tolerance sequence cannot build up across invocations; use TOLERANCE = 0 in this mode, or run sakana as a daemon.

11.1 Transport classes ¶

These apply to every kind and are decided from the response to the primary endpoint.

11.2 Classes of an exchange ¶

11.3 Classes of an Alertmanager ¶

11.4 Classes of a Prometheus server ¶

11.5 Severity ¶

The severity only affects the wording of the report and the level of the log line. Warnings and errors use the same failure counters and both block the restoration report.