310 lines
9.6 KiB
Markdown
310 lines
9.6 KiB
Markdown
# Panoptikon – Watch the world from NixOS
|
||
A NixOS module for monitoring website content and command output changes.
|
||
|
||
|
||
|
||
## Overview
|
||
|
||
Panoptikon is a flexible, secure, and modular system for monitoring changes to websites, API endpoints, and command outputs. It runs scripts at configurable intervals, detects changes by comparing outputs, and reports differences to various destinations.
|
||
|
||
**Perfect for:**
|
||
- Monitoring service status pages (GitHub, cloud providers)
|
||
- Tracking website content changes (blogs, news)
|
||
- Watching API endpoints (cryptocurrency prices, weather)
|
||
- System metrics monitoring (disk space, load, processes)
|
||
- Security canaries and breach detection
|
||
|
||
## Features
|
||
|
||
- **Flexible Watchers**: Execute any script or command; monitor HTTP endpoints with built-in helpers
|
||
- **Custom Frequencies**: Use systemd timer syntax for any schedule (*/5 minutes, daily, weekly, etc.)
|
||
- **Multiple Reporters**: Notify via IRC, Telegram, Matrix, email, desktop notifications, wall, or custom scripts
|
||
- **Secret Support**: Securely pass credentials using `LoadCredential=` without exposing them in the Nix store
|
||
- **Stateful Tracking**: Automatic diffing; only reports actual changes
|
||
- **Modular Design**: Built-in helpers for HTML, JSON, and plain text; easy to create custom ones
|
||
- **Security Hardened**: Dedicated system user, filesystem isolation, and process protection
|
||
- **systemd Native**: Full integration with systemd for reliable scheduling and logging
|
||
|
||
## Quick Start
|
||
|
||
1. **Enable Panoptikon** in your NixOS configuration:
|
||
|
||
```nix
|
||
{ config, pkgs, ... }:
|
||
|
||
{
|
||
services.panoptikon.enable = true;
|
||
}
|
||
```
|
||
|
||
2. **Add a simple watcher**:
|
||
|
||
```nix
|
||
services.panoptikon.watchers = {
|
||
example = {
|
||
script = pkgs.panoptikonWatchers.plain "https://example.com";
|
||
frequency = "hourly";
|
||
reporters = [ (pkgs.panoptikonReporters.wall { }) ];
|
||
};
|
||
};
|
||
```
|
||
|
||
3. **Deploy and monitor**:
|
||
|
||
```bash
|
||
# Check status
|
||
sudo systemctl status panoptikon-example
|
||
|
||
# View logs
|
||
sudo journalctl -u panoptikon-example -f
|
||
|
||
# Trigger manually
|
||
sudo systemctl start panoptikon-example
|
||
```
|
||
|
||
## How It Works
|
||
|
||
```
|
||
┌─────────────┐
|
||
│ systemd │
|
||
│ timer │─── triggers ───┐
|
||
└─────────────┘ │
|
||
▼
|
||
┌─────────────────────┐
|
||
│ Panoptikon Service │
|
||
│ (oneshot) │
|
||
└─────────────────────┘
|
||
│
|
||
┌─────────┴─────────┐
|
||
▼ │
|
||
┌──────────────────┐ │
|
||
│ Run watcher │ │
|
||
│ script → current │ │
|
||
└──────────────────┘ │
|
||
│ │
|
||
▼ │
|
||
┌──────────────────┐ │
|
||
│ Compare with │ │
|
||
│ .old state │ │
|
||
└──────────────────┘ │
|
||
│ │
|
||
diff? ─┼─── Yes ────────┤
|
||
│ No │
|
||
▼ ▼
|
||
┌──────────────┐ ┌──────────────┐
|
||
│ Exit quietly │ │ Pipe diff to │
|
||
└──────────────┘ │ reporters │
|
||
└──────┬───────┘
|
||
│
|
||
┌────────────┴────────────┐
|
||
▼ ▼
|
||
[reporter 1] [reporter 2]
|
||
│ │
|
||
└──────────┬───────────────┘
|
||
▼
|
||
[Notifications sent]
|
||
│
|
||
▼
|
||
┌──────────────────┐
|
||
│ Rotate state: │
|
||
│ current → old │
|
||
└──────────────────┘
|
||
```
|
||
|
||
**Key Points:**
|
||
- Each watcher runs as its own systemd service with a dedicated timer
|
||
- Output is saved to `/var/lib/panoptikon/<watcher-name>` and `<watcher-name>.old`
|
||
- Only diffs are sent to reporters (full output never transmitted)
|
||
- Reporters run in a clean, empty `/run` directory for isolation
|
||
- State persists across reboots
|
||
|
||
## Configuration Reference
|
||
|
||
### Watcher Options
|
||
|
||
| Option | Type | Description |
|
||
|--------|------|-------------|
|
||
| `script` | path | **Required.** Executable whose stdout will be monitored. |
|
||
| `frequency` | string | systemd.time(7) timer expression. Default: `"daily"` |
|
||
| `reporters` | list of paths | **Required.** Scripts that receive the diff via stdin. |
|
||
| `loadCredential` | list of strings | Credentials to pass from systemd (`LoadCredential=`). |
|
||
|
||
### Timer Syntax
|
||
|
||
Common patterns:
|
||
|
||
| Pattern | Meaning |
|
||
|---------|---------|
|
||
| `*:0/5` | Every 5 minutes |
|
||
| `*:0/15` | Every 15 minutes |
|
||
| `hourly` | At minute 0 of every hour |
|
||
| `daily` | Once per day at midnight |
|
||
| `*-*-1 0:0:0` | First day of month |
|
||
| `Mon *-*-* 0:0:0` | Every Monday |
|
||
| `Sat,Sun *-*-* 0:0:0` | Weekends |
|
||
|
||
See: `man systemd.time`
|
||
|
||
### Environment Variables
|
||
|
||
- `PANOPTIKON_WATCHER` - Name of the watcher (available to both watchers and reporters)
|
||
|
||
## Built-in Helpers
|
||
|
||
The overlay provides convenient watcher and reporter constructors:
|
||
|
||
### Watcher Helpers
|
||
|
||
```nix
|
||
# Fetch raw content
|
||
pkgs.panoptikonWatchers.plain "https://example.com"
|
||
|
||
# Convert HTML to plain text
|
||
pkgs.panoptikonWatchers.html "https://example.com"
|
||
|
||
# Extract specific HTML elements using CSS selector
|
||
pkgs.panoptikonWatchers.htmlSelector "#news h2" "https://example.com"
|
||
|
||
# Process JSON with jq
|
||
pkgs.panoptikonWatchers.json { jqScript = ".data[] | .value" } "https://api.example.com"
|
||
```
|
||
|
||
### Reporter Helpers
|
||
|
||
```nix
|
||
# Send to wall (broadcast to all logged-in users)
|
||
pkgs.panoptikonReporters.wall { }
|
||
|
||
# Send email
|
||
pkgs.panoptikonReporters.mail {
|
||
recipient = "admin@example.org";
|
||
subjectPrefix = "[Alert]";
|
||
}
|
||
|
||
# Telegram Bot API
|
||
pkgs.panoptikonReporters.telegram {
|
||
chatId = "123456";
|
||
tokenPath = "/run/keys/telegram-token"; # Use LoadCredential=
|
||
messagePrefix = "Change detected: ";
|
||
}
|
||
|
||
# Matrix (via REST API)
|
||
pkgs.panoptikonReporters.matrix {
|
||
homeserver = "https://matrix.org";
|
||
roomId = "!roomid:matrix.org";
|
||
tokenPath = "/run/keys/matrix-token";
|
||
}
|
||
|
||
# IRC (simple netcat-based)
|
||
pkgs.panoptikonReporters.irc {
|
||
target = "#channel";
|
||
server = "irc.libera.chat";
|
||
port = "6667";
|
||
nick = "panoptikon-bot";
|
||
}
|
||
|
||
# kpaste + ircsink (for retiolum)
|
||
pkgs.panoptikonReporters.kpaste-irc {
|
||
target = "#nixos";
|
||
server = "irc.r";
|
||
retiolumLink = true; # Generate retiolum link
|
||
}
|
||
```
|
||
|
||
**Important:** For security, always use `LoadCredential=` for tokens instead of embedding them in your Nix store.
|
||
|
||
## Advanced Examples
|
||
|
||
### Monitor Bitcoin Price with Telegram Alert
|
||
|
||
See [examples/bitcoin.nix](./examples/bitcoin.nix)
|
||
|
||
### Website Content Change with Email Notification
|
||
|
||
See [examples/nixos.nix](./examples/nixos.nix)
|
||
|
||
### System Metrics with Console Notifications
|
||
|
||
See [examples/system.nix](./examples/system.nix)
|
||
|
||
### Custom Script with Multiple Reporters
|
||
|
||
See [examples/simple.nix](./examples/simple.nix) for more.
|
||
|
||
## Testing
|
||
|
||
### Run a VM with Example Configurations
|
||
|
||
```bash
|
||
nix run .#panoptikon-vm
|
||
```
|
||
|
||
This builds and boots a NixOS VM with all example watchers pre-configured.
|
||
|
||
### Inside the VM
|
||
|
||
```bash
|
||
# List all panoptikon services
|
||
systemctl list-units "panoptikon-*"
|
||
|
||
# Check Bitcoin watcher status
|
||
systemctl status panoptikon-bitcoin-price
|
||
|
||
# Follow its logs
|
||
journalctl -u panoptikon-bitcoin-price -f
|
||
|
||
# Force a run (useful for testing)
|
||
systemctl start panoptikon-bitcoin-price
|
||
|
||
# Check state directory
|
||
ls -la /var/lib/panoptikon/
|
||
```
|
||
|
||
### Manual Script Testing
|
||
|
||
Run the watcher script as the panoptikon user to see its output:
|
||
|
||
```bash
|
||
sudo -u panoptikon /nix/store/<hash>-watch-bitcoin
|
||
```
|
||
|
||
Check that it produces clean output without errors.
|
||
|
||
## FAQ
|
||
|
||
**Q: Can Panoptikon monitor FTP or SSH?**
|
||
|
||
A: Yes! Write a custom watcher script that uses `curl` (for SFTP), `ssh`, or any CLI tool. Example:
|
||
|
||
```nix
|
||
script = pkgs.writers.writeDash "ssh-check" ''
|
||
${pkgs.openssh}/bin/ssh user@host "uptime"
|
||
'';
|
||
```
|
||
|
||
**Q: What happens if the watcher script fails?**
|
||
|
||
A: The service will be marked as failed and will restart on next timer activation (unless configured otherwise). The error is logged to the systemd journal. Reporters are skipped.
|
||
|
||
**Q: Can I run multiple reporters for the same event?**
|
||
|
||
A: Yes! Reporters are executed sequentially. If one fails, others still run (errors are suppressed with `|| :`).
|
||
|
||
**Q: How do I handle JSON pretty-printing?**
|
||
|
||
A: Use jq:
|
||
|
||
```nix
|
||
script = pkgs.panoptikonWatchers.json { jqScript = "." } "https://api.example.com/data";
|
||
```
|
||
|
||
Or in a custom script:
|
||
|
||
```bash
|
||
curl -s ... | jq -S . # -S sorts keys
|
||
```
|
||
|
||
**Q: Can I send rich formatting (HTML, Markdown) to reporters?**
|
||
|
||
A: Yes! Reporters receive the raw diff. For IRC, use colors sparingly. For Telegram/Matrix, you can send Markdown or HTML by constructing appropriate payloads in custom reporters.
|