panoptikon/README.md

# Panoptikon – Watch the world from NixOS
A NixOS module for monitoring website content and command output changes.

![](./panoptikon.jpg)

## Overview

Panoptikon is a flexible, secure, and modular system for monitoring changes to websites, API endpoints, and command outputs. It runs scripts at configurable intervals, detects changes by comparing outputs, and reports differences to various destinations.

**Perfect for:**
- Monitoring service status pages (GitHub, cloud providers)
- Tracking website content changes (blogs, news)
- Watching API endpoints (cryptocurrency prices, weather)
- System metrics monitoring (disk space, load, processes)
- Security canaries and breach detection

## Features

- **Flexible Watchers**: Execute any script or command; monitor HTTP endpoints with built-in helpers
- **Custom Frequencies**: Use systemd timer syntax for any schedule (*/5 minutes, daily, weekly, etc.)
- **Multiple Reporters**: Notify via IRC, Telegram, Matrix, email, desktop notifications, wall, or custom scripts
- **Secret Support**: Securely pass credentials using `LoadCredential=` without exposing them in the Nix store
- **Stateful Tracking**: Automatic diffing; only reports actual changes
- **Modular Design**: Built-in helpers for HTML, JSON, and plain text; easy to create custom ones
- **Security Hardened**: Dedicated system user, filesystem isolation, and process protection
- **systemd Native**: Full integration with systemd for reliable scheduling and logging

## Quick Start

1. **Enable Panoptikon** in your NixOS configuration:

```nix
{ config, pkgs, ... }:

{
  services.panoptikon.enable = true;
}
```

2. **Add a simple watcher**:

```nix
services.panoptikon.watchers = {
  example = {
    script = pkgs.panoptikonWatchers.plain "https://example.com";
    frequency = "hourly";
    reporters = [ (pkgs.panoptikonReporters.wall { }) ];
  };
};
```

3. **Deploy and monitor**:

```bash
# Check status
sudo systemctl status panoptikon-example

# View logs
sudo journalctl -u panoptikon-example -f

# Trigger manually
sudo systemctl start panoptikon-example
```

## How It Works

```
┌─────────────┐
│ systemd     │
│ timer       │─── triggers ───┐
└─────────────┘                │
                               ▼
                    ┌─────────────────────┐
                    │ Panoptikon Service  │
                    │ (oneshot)           │
                    └─────────────────────┘
                               │
                     ┌─────────┴─────────┐
                     ▼                   │
          ┌──────────────────┐         │
          │ Run watcher      │         │
          │ script → current │         │
          └──────────────────┘         │
                     │                 │
                     ▼                 │
          ┌──────────────────┐        │
          │ Compare with     │        │
          │ .old state       │        │
          └──────────────────┘        │
                     │                 │
              diff? ─┼─── Yes ────────┤
                     │ No             │
                     ▼                ▼
            ┌──────────────┐  ┌──────────────┐
            │ Exit quietly │  │ Pipe diff to │
            └──────────────┘  │ reporters    │
                              └──────┬───────┘
                                     │
                        ┌────────────┴────────────┐
                        ▼                          ▼
                 [reporter 1]            [reporter 2]
                    │                          │
                    └──────────┬───────────────┘
                               ▼
                     [Notifications sent]
                               │
                               ▼
                    ┌──────────────────┐
                    │ Rotate state:    │
                    │ current → old    │
                    └──────────────────┘
```

**Key Points:**
- Each watcher runs as its own systemd service with a dedicated timer
- Output is saved to `/var/lib/panoptikon/<watcher-name>` and `<watcher-name>.old`
- Only diffs are sent to reporters (full output never transmitted)
- Reporters run in a clean, empty `/run` directory for isolation
- State persists across reboots

## Configuration Reference

### Watcher Options

| Option | Type | Description |
|--------|------|-------------|
| `script` | path | **Required.** Executable whose stdout will be monitored. |
| `frequency` | string | systemd.time(7) timer expression. Default: `"daily"` |
| `reporters` | list of paths | **Required.** Scripts that receive the diff via stdin. |
| `loadCredential` | list of strings | Credentials to pass from systemd (`LoadCredential=`). |

### Timer Syntax

Common patterns:

| Pattern | Meaning |
|---------|---------|
| `*:0/5` | Every 5 minutes |
| `*:0/15` | Every 15 minutes |
| `hourly` | At minute 0 of every hour |
| `daily` | Once per day at midnight |
| `*-*-1 0:0:0` | First day of month |
| `Mon *-*-* 0:0:0` | Every Monday |
| `Sat,Sun *-*-* 0:0:0` | Weekends |

See: `man systemd.time`

### Environment Variables

- `PANOPTIKON_WATCHER` - Name of the watcher (available to both watchers and reporters)

## Built-in Helpers

The overlay provides convenient watcher and reporter constructors:

### Watcher Helpers

```nix
# Fetch raw content
pkgs.panoptikonWatchers.plain "https://example.com"

# Convert HTML to plain text
pkgs.panoptikonWatchers.html "https://example.com"

# Extract specific HTML elements using CSS selector
pkgs.panoptikonWatchers.htmlSelector "#news h2" "https://example.com"

# Process JSON with jq
pkgs.panoptikonWatchers.json { jqScript = ".data[] | .value" } "https://api.example.com"
```

### Reporter Helpers

```nix
# Send to wall (broadcast to all logged-in users)
pkgs.panoptikonReporters.wall { }

# Send email
pkgs.panoptikonReporters.mail {
  recipient = "admin@example.org";
  subjectPrefix = "[Alert]";
}

# Telegram Bot API
pkgs.panoptikonReporters.telegram {
  chatId = "123456";
  tokenPath = "/run/keys/telegram-token";  # Use LoadCredential=
  messagePrefix = "Change detected: ";
}

# Matrix (via REST API)
pkgs.panoptikonReporters.matrix {
  homeserver = "https://matrix.org";
  roomId = "!roomid:matrix.org";
  tokenPath = "/run/keys/matrix-token";
}

# IRC (simple netcat-based)
pkgs.panoptikonReporters.irc {
  target = "#channel";
  server = "irc.libera.chat";
  port = "6667";
  nick = "panoptikon-bot";
}

# kpaste + ircsink (for retiolum)
pkgs.panoptikonReporters.kpaste-irc {
  target = "#nixos";
  server = "irc.r";
  retiolumLink = true;  # Generate retiolum link
}
```

**Important:** For security, always use `LoadCredential=` for tokens instead of embedding them in your Nix store.

## Advanced Examples

### Monitor Bitcoin Price with Telegram Alert

See [examples/bitcoin.nix](./examples/bitcoin.nix)

### Website Content Change with Email Notification

See [examples/nixos.nix](./examples/nixos.nix)

### System Metrics with Console Notifications

See [examples/system.nix](./examples/system.nix)

### Custom Script with Multiple Reporters

See [examples/simple.nix](./examples/simple.nix) for more.

## Testing

### Run a VM with Example Configurations

```bash
nix run .#panoptikon-vm
```

This builds and boots a NixOS VM with all example watchers pre-configured.

### Inside the VM

```bash
# List all panoptikon services
systemctl list-units "panoptikon-*"

# Check Bitcoin watcher status
systemctl status panoptikon-bitcoin-price

# Follow its logs
journalctl -u panoptikon-bitcoin-price -f

# Force a run (useful for testing)
systemctl start panoptikon-bitcoin-price

# Check state directory
ls -la /var/lib/panoptikon/
```

### Manual Script Testing

Run the watcher script as the panoptikon user to see its output:

```bash
sudo -u panoptikon /nix/store/<hash>-watch-bitcoin
```

Check that it produces clean output without errors.

## FAQ

**Q: Can Panoptikon monitor FTP or SSH?**

A: Yes! Write a custom watcher script that uses `curl` (for SFTP), `ssh`, or any CLI tool. Example:

```nix
script = pkgs.writers.writeDash "ssh-check" ''
  ${pkgs.openssh}/bin/ssh user@host "uptime"
'';
```

**Q: What happens if the watcher script fails?**

A: The service will be marked as failed and will restart on next timer activation (unless configured otherwise). The error is logged to the systemd journal. Reporters are skipped.

**Q: Can I run multiple reporters for the same event?**

A: Yes! Reporters are executed sequentially. If one fails, others still run (errors are suppressed with `|| :`).

**Q: How do I handle JSON pretty-printing?**

A: Use jq:

```nix
script = pkgs.panoptikonWatchers.json { jqScript = "." } "https://api.example.com/data";
```

Or in a custom script:

```bash
curl -s ... | jq -S .  # -S sorts keys
```

**Q: Can I send rich formatting (HTML, Markdown) to reporters?**

A: Yes! Reporters receive the raw diff. For IRC, use colors sparingly. For Telegram/Matrix, you can send Markdown or HTML by constructing appropriate payloads in custom reporters.