Compare commits
No commits in common. "cd1cdbd110d286ca60d9ade40ca90d1a99d17aa4" and "45e06ed524e2534764e33433d35713faa0e26c65" have entirely different histories.
cd1cdbd110
...
45e06ed524
@ -68,11 +68,6 @@ STALWART_S3_BUCKET=stalwart-mail
|
||||
# ----------------------------------------------------------------------------
|
||||
# Most VPS providers block outbound :25. If yours does, relay through a
|
||||
# smarthost (host:port). Leave blank to attempt direct MX delivery.
|
||||
#
|
||||
# IMPORTANT: Use an IPv4 literal or a tailnet IP — never a dual-stack hostname.
|
||||
# The container has no IPv6 and will NOT fall back from AAAA to A; any host
|
||||
# with an AAAA record will fail immediately (os error 101). Relaying over the
|
||||
# tailnet (100.x:587) sidesteps this entirely and also bypasses VPS SMTP blocks.
|
||||
STALWART_SMARTHOST=
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
|
||||
@ -81,12 +81,7 @@ healthcheck, ephemeral OAuth auth). Don't drift it. Tag: `tag:stalwart`.
|
||||
re-init. And never test a password over `127.0.0.1` against these Postgres
|
||||
containers: pg_hba `trust`s loopback and accepts ANY password. Test over the
|
||||
tailnet (scram) or you'll fool yourself.
|
||||
- **Outbound :25 is usually blocked on VPS.** Set `STALWART_SMARTHOST`, and
|
||||
prefer relaying over the tailnet (`100.x:587`) — it bypasses the VPS SMTP-port
|
||||
blocks and, having no AAAA, sidesteps the v6-first trap. The sidecar now has
|
||||
its **own IPv6 egress** (LESSONS.md 9), so dual-stack targets resolve too;
|
||||
before that fix an AAAA-only path would hang (`os error 101`) with no fallback
|
||||
to A. See LESSONS.md 8–9.
|
||||
- **Outbound :25 is usually blocked on VPS.** Set `STALWART_SMARTHOST`.
|
||||
- **Mail forces WAN ports.** `:25` must be world-reachable for inbound
|
||||
federation — this is the one place the tailnet-only model can't hold. Keep
|
||||
submission/IMAP tailnet-only if you want a tighter surface.
|
||||
|
||||
152
LESSONS.md
152
LESSONS.md
@ -94,155 +94,3 @@ duplicate-cert limit, use **certspotter** instead:
|
||||
Also: LE limits are dimensioned — **failed validations** are hourly (5/hr/host,
|
||||
the one a retry storm trips), **issued duplicates** are weekly (5/wk). A renewal
|
||||
task hammering every 10 min trips the hourly one; consolidate to a single task.
|
||||
|
||||
## 8. The Stalwart container has no IPv6 — AAAA targets fail before IPv4 is tried
|
||||
|
||||
**Symptom:** Outbound delivery (and relay-to-smarthost) to any host with an
|
||||
AAAA record fails with `I/O error: Network is unreachable (os error 101)`.
|
||||
Hosts that are IPv4-only deliver fine. Pointing a relay at a *hostname* that
|
||||
has both A and AAAA fails; pointing it at the raw IPv4 works.
|
||||
|
||||
**Cause:** Stalwart shares the `ts-stalwart` sidecar's netns, which has no
|
||||
global IPv6. When it resolves a dual-stack target it tries the AAAA first,
|
||||
gets `ENETUNREACH` immediately, and for a **relay next-hop it does not fall
|
||||
back to the A record** — it just records the v6 failure and backs off. So a
|
||||
single missing address family wedges all mail to dual-stack destinations.
|
||||
|
||||
**Fix:** Either (a) pin the relay/smarthost `address` to an **IPv4 literal**
|
||||
(no AAAA to trip on), or (b) give the container real IPv6. Note that relaying
|
||||
over the **tailnet** sidesteps this entirely — you connect to a tailnet
|
||||
`100.x` address, which has no AAAA, so the v6-first trap never triggers.
|
||||
|
||||
> **RESOLVED (2026-06-11) — option (b) is now done.** The container has real
|
||||
> IPv6 egress; this trap no longer fires. See Lesson 9's fix for how.
|
||||
|
||||
## 9. Configuring IPv6 on the KVM host does NOT give the container IPv6
|
||||
|
||||
**Symptom:** `ip -6 addr` and `ping6 google.com` succeed on the KVM host, but
|
||||
Stalwart still dies with `os error 101` on AAAA targets, and the box is still
|
||||
a broken IPv6 Tailscale exit node.
|
||||
|
||||
**Cause:** The host's `eth0` and the container/sidecar netns are separate
|
||||
network stacks. Adding the provider's `/64` to `eth0` (ifupdown `inet6 static`
|
||||
+ `onlink` default route, since the gateway is in a different /64) fixes the
|
||||
*host*, not the container. Docker doesn't hand IPv6 to containers by default,
|
||||
and the sidecar routes via Tailscale, not eth0.
|
||||
|
||||
**Fix:** Don't assume host IPv6 = container IPv6. Test from *inside* the
|
||||
container's netns. For mail egress, the IPv4-literal relay (Lesson 8) or the
|
||||
tailnet relay avoids needing container IPv6 at all. Enabling true container
|
||||
IPv6 (Docker IPv6 + routing the /64 in) is a separate, larger task.
|
||||
|
||||
**RESOLVED (2026-06-11) — the easy way, no /64 routing or ndppd.** Because the
|
||||
container only needs IPv6 **egress** (inbound arrives via the edge/tailnet,
|
||||
never v6), you don't need a routable prefix or NDP proxy at all — just a **ULA
|
||||
subnet + masquerade**, exactly like Docker does for v4:
|
||||
```yaml
|
||||
# docker-compose.yml
|
||||
networks:
|
||||
default:
|
||||
enable_ipv6: true
|
||||
ipam:
|
||||
config:
|
||||
- subnet: fd00:7a17:600d::/64
|
||||
gateway: fd00:7a17:600d::1
|
||||
```
|
||||
Docker 29 enables `ip6tables` by default and masquerades the ULA out the host's
|
||||
global v6, so the sidecar netns (shared by Stalwart via `network_mode`) gets a
|
||||
working v6 default route with **zero host sysctl/daemon changes** (host
|
||||
`net.ipv6.conf.all.forwarding` was already 1 from the static-v6 setup). Verify
|
||||
from *inside* the netns: `ping6 google.com` + a TCP connect to a v6 literal on
|
||||
:443. Recreating the network (`docker compose down && up`) bounces the stack and
|
||||
the ephemeral sidecar gets a new tailnet IP — MagicDNS covers it (Lesson 6), and
|
||||
the MTA route table rebuilds anyway (Lesson 12). This does **not** give inbound
|
||||
v6; for that you'd still publish AAAA + make the edge listen on v6 (separate).
|
||||
|
||||
## 10. The VPS blocks ALL outbound SMTP ports — relay over the tailnet
|
||||
|
||||
**Symptom:** Direct MX delivery and relay-to-public-host both fail with
|
||||
`Connection timed out (os error 110)`, and the SYN never arrives at the
|
||||
destination. Not just port 25 — `465`, `587`, even alt-port `2525` all time out.
|
||||
|
||||
**Cause:** The KVM provider blocks all outbound SMTP submission ports to prevent
|
||||
spam. Only non-SMTP ports (`443`, etc.) egress. Confirmed with:
|
||||
```bash
|
||||
for p in 25 465 587 2525 443; do
|
||||
timeout 5 bash -c "exec 3<>/dev/tcp/<dst>/$p" && echo "$p OPEN" || echo "$p blocked"
|
||||
done
|
||||
# 443 OPEN, all SMTP ports timeout
|
||||
```
|
||||
|
||||
**Fix:** Relay over the **tailnet**. Tailscale rides WireGuard/DERP (UDP 41641 /
|
||||
443), so it's immune to SMTP port filtering. Point the relay at the smarthost's
|
||||
**tailnet IP** (e.g. `100.x:587`), not its public address. Long-term: ask the
|
||||
provider to unblock outbound 25/587 for verified use.
|
||||
|
||||
## 11. The sidecar can RECEIVE on the tailnet but can't INITIATE without an ACL grant
|
||||
|
||||
**Symptom:** The relay to `<mailbox-tailnet-ip>:587` times out (`os error 110`),
|
||||
yet the **KVM host** (same physical machine) can reach that exact IP:port over
|
||||
the tailnet fine. Looks like a routing or transparent-proxy bug.
|
||||
|
||||
**Cause:** The Stalwart container rides the `ts-stalwart` sidecar — a **separate
|
||||
tailnet node** (`tag:stalwart`) from the KVM host. The `tailwart` ACL block only
|
||||
listed `tag:stalwart` as a **destination** (`"dst": ["tag:stalwart"]`). Tailnet
|
||||
is default-deny, so the sidecar could receive connections but could not
|
||||
*initiate* the relay back to the mailbox → silent drop → timeout. The KVM host
|
||||
worked because it's a different, permitted identity, which masked the real cause.
|
||||
|
||||
**Fix:** Add an ACL rule granting `tag:stalwart` as a **source**:
|
||||
```json
|
||||
{ "src": ["tag:stalwart"], "dst": ["tag:mail"], "ip": ["tcp:587"] }
|
||||
```
|
||||
(mailbox is `tag:mail`). Applies in seconds, no restart. See `acl-snippet.hujson`.
|
||||
|
||||
## 12. Stalwart only rebuilds its MTA route table at container startup
|
||||
|
||||
**Symptom:** You edit an `MtaRoute` (address, etc.) via API/UI, but delivery keeps
|
||||
using the old value. The datastore shows the new value; live delivery ignores it.
|
||||
|
||||
**Cause:** The `routing_strategy` map is built once when the process boots. The
|
||||
`ReloadSettings` action reloads the datastore but does **not** rebuild the SMTP
|
||||
route map. So route/strategy changes are invisible until restart.
|
||||
|
||||
**Fix:** After any `MtaRoute` / `MtaOutboundStrategy` change,
|
||||
`docker restart tailwart-stalwart-1`. (Side effect: the ephemeral sidecar gets a
|
||||
new tailnet IP each restart — anything addressing it by IP must rediscover it;
|
||||
use the MagicDNS name where possible.)
|
||||
|
||||
## 13. "Did Stalwart eat my custom DNS records?" — no; Spaceship is RRSet-upsert
|
||||
|
||||
**Symptom:** A manually-added record (e.g. an `AAAA` for the apex/`mail`) is
|
||||
gone from the zone, and the suspicion is that Stalwart's ACME DNS-01 integration
|
||||
overwrote it on a renewal.
|
||||
|
||||
**Cause:** Almost never Stalwart. Its **only** DNS-provider writes are
|
||||
`_acme-challenge.<name>` TXT (the rotating challenge) and `_validation-persist`
|
||||
TXT (the LE account-pinned persistent-validation record). It does **not** create
|
||||
or modify A/AAAA/MX/SRV — those you add yourself from its "recommended records"
|
||||
page. And the Spaceship API is **RRSet-upsert keyed by (name, type)**, not a
|
||||
whole-zone replace: a `PUT /api/v1/dns/records/{domain}` with
|
||||
`{"force":true,"items":[…]}` only touches the RRSets named in `items`. Proof:
|
||||
25 unrelated records coexist untouched through every rotating `_acme-challenge`
|
||||
write; and adding one apex `AAAA` left the other 25 exactly intact (25→26).
|
||||
|
||||
So a vanished AAAA is far more likely a **provider-side loss/rollback** (e.g.
|
||||
during a data-center DDoS) or a manual edit — not Stalwart.
|
||||
|
||||
**How to inspect / verify (read-only), creds in `.env`:**
|
||||
```bash
|
||||
KEY=$(grep '^SPACESHIP_KEY=' .env | cut -d= -f2)
|
||||
SECRET=$(grep '^SPACESHIP_SECRET=' .env | cut -d= -f2)
|
||||
curl -s "https://spaceship.dev/api/v1/dns/records/<domain>?take=100&skip=0" \
|
||||
-H "X-Api-Key: $KEY" -H "X-Api-Secret: $SECRET" | python3 -m json.tool
|
||||
```
|
||||
To add a record, `PUT` the same endpoint with a single-item `items` array — it
|
||||
won't disturb siblings. **Snapshot the zone (GET) before any write** and diff
|
||||
after; snapshots land in `_backup/` (gitignored). Always re-check at the
|
||||
authoritative NS (`dig +short AAAA <name> @launch1.spaceship.net`), not a cache.
|
||||
|
||||
**Caveat — don't publish `mail` AAAA before the edge listens on v6.** Inbound
|
||||
mail follows `MX → mail.<domain>`; an `AAAA` there with no v6 `:25` listener on
|
||||
the edge makes senders try v6 and some won't fall back → deferred/bounced mail.
|
||||
An **apex** `AAAA` is safe (it doesn't affect MX routing). Do `mail` AAAA + edge
|
||||
v6 listeners together.
|
||||
|
||||
28
README.md
28
README.md
@ -8,20 +8,17 @@ mail server wired into **Postgres + Redis + Garage S3** at once, deployed as a
|
||||
Tailscale sidecar, with a separate `caddy-l4` edge that pipes the raw mail ports
|
||||
over the tailnet. For `infinidim.net`.
|
||||
|
||||
See [CLAUDE.md](./CLAUDE.md) for the architecture and the gotchas, and
|
||||
[LESSONS.md](./LESSONS.md) for the hard-won symptom→cause→fix notes from
|
||||
bringing it up (PG races, ACME DNS-01, PROXY-protocol trust, IPv6 egress, …).
|
||||
See [CLAUDE.md](./CLAUDE.md) for the architecture and the gotchas.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
tailwart/
|
||||
├── docker-compose.yml # the mailbox: ts-stalwart sidecar + stalwart (+ IPv6 egress)
|
||||
├── config/config.json # Stalwart v0.16 bootstrap config — datastore (Postgres) ONLY
|
||||
├── config/config.toml # dead v0.15-era config, kept as historical reference
|
||||
├── docker-compose.yml # the mailbox: ts-stalwart sidecar + stalwart
|
||||
├── config/config.toml # Stalwart config — PG + Redis + S3 wiring (strawman)
|
||||
├── caddy/ # the edge: custom Caddy (caddy-l4) layer-4 mail proxy
|
||||
│ ├── Dockerfile # pulls prebuilt caddy-l4 binary (caddyserver.com, no local build)
|
||||
│ ├── caddy.json # :25/465/587/143/993 mail + :443 SNI fan-out → stalwart over the tailnet
|
||||
│ ├── caddy.json # :25/465/587/143/993 → stalwart over the tailnet
|
||||
│ ├── docker-compose.yml # deploy on any public-IP, tailnet, tag:reverse-proxy host
|
||||
│ └── README.md
|
||||
├── acl-snippet.hujson # tag:stalwart owner + grants to merge into your policy
|
||||
@ -29,11 +26,6 @@ tailwart/
|
||||
└── .gitignore
|
||||
```
|
||||
|
||||
> **v0.16 config model:** `config.json` describes *only* where Postgres lives;
|
||||
> everything else (domains, accounts, listeners, ACME, blob/Redis wiring, proxy
|
||||
> trust, DKIM, spam) lives **in Postgres**, managed via the web UI or JMAP. The
|
||||
> old TOML + `%{env}%` macro model is gone — see CLAUDE.md / LESSONS.md.
|
||||
|
||||
## Quickstart
|
||||
|
||||
```bash
|
||||
@ -53,12 +45,6 @@ configuration in Stalwart's web admin (`mail.infinidim.net`).
|
||||
|
||||
## Status
|
||||
|
||||
Live. Pinned to `stalwartlabs/stalwart:v0.16.7`, booting cleanly against the
|
||||
shared Postgres/Redis/Garage backends, holding a Let's Encrypt wildcard via
|
||||
ACME DNS-01 (Spaceship), and relaying outbound over the tailnet (the VPS blocks
|
||||
all outbound SMTP ports). The container has its own IPv6 egress.
|
||||
|
||||
Not yet exposed for inbound IPv6 mail: `mail.infinidim.net` is intentionally
|
||||
A-only until the edge proxy gains v6 `:25` listeners — publish its `AAAA` and
|
||||
the v6 listeners together, or senders will try v6 and some won't fall back. See
|
||||
LESSONS.md.
|
||||
Scaffold / strawman. The Stalwart `config.toml` keys need verifying against a
|
||||
pinned image version before first real boot — treat it as a starting shape, not
|
||||
a turnkey config.
|
||||
|
||||
@ -21,12 +21,6 @@
|
||||
"ip": ["tcp:25", "tcp:465", "tcp:587", "tcp:143", "tcp:993", "tcp:443", "tcp:8080"],
|
||||
},
|
||||
|
||||
// 4) grant — Stalwart initiates outbound relay to the smarthost (tag:mail).
|
||||
// Without this, the sidecar can RECEIVE but cannot INITIATE over the tailnet
|
||||
// (default-deny; the KVM host is a different identity and won't mask this).
|
||||
// Adjust dst/port to match your smarthost's tag and submission port.
|
||||
{ "src": ["tag:stalwart"], "dst": ["tag:mail"], "ip": ["tcp:587"] },
|
||||
|
||||
// 5) admin console (not this file): assign tag:stalwart to the same OAuth
|
||||
// 4) admin console (not this file): assign tag:stalwart to the same OAuth
|
||||
// client federatedSocial uses, on the Devices/Core + Keys/AuthKeys scopes.
|
||||
// Missing → 403 "calling actor does not have enough permissions" at boot.
|
||||
|
||||
@ -80,19 +80,3 @@ services:
|
||||
|
||||
volumes:
|
||||
stalwart-data:
|
||||
|
||||
# The sidecar's bridge (shared by stalwart via network_mode) gets IPv6 here so
|
||||
# the container can reach AAAA-only / dual-stack hosts. Without it the netns has
|
||||
# no global v6 → Stalwart tries AAAA first, gets ENETUNREACH, and for a relay
|
||||
# next-hop never falls back to A (see LESSONS.md #8). A ULA subnet is fine: we
|
||||
# only need *egress* (inbound arrives via the edge/tailnet, never v6). Docker 29
|
||||
# masquerades it out the host's global v6 via ip6tables — no routable prefix,
|
||||
# NDP proxy, or host sysctl needed. Recreating this network bounces the stack
|
||||
# and the ephemeral sidecar gets a new tailnet IP (MagicDNS handles it).
|
||||
networks:
|
||||
default:
|
||||
enable_ipv6: true
|
||||
ipam:
|
||||
config:
|
||||
- subnet: fd00:7a17:600d::/64
|
||||
gateway: fd00:7a17:600d::1
|
||||
|
||||
Loading…
Reference in New Issue
Block a user