Host 3 — Edge host (RHEL 9)¶

Runs ironshep-edge: the protocol collector for this site. Holds no database. Needs its client certificate from Host 2 before it can connect.

Substitute: <CORE_HOSTNAME_OR_IP> (Host 2, as reached from here), <EDGE_ID> (must match what you passed to gen_certs.sh --edge on Host 2, e.g. lab1), <CORE_HOST_ADMIN> (your ssh user on Host 2).

1. Toolchain¶

sudo dnf install -y gcc openssl-devel git   # openssl-devel is for the opcua crate
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
source "$HOME/.cargo/env"

2. Build¶

git clone https://gitlab.com/ironshep/ironshep-edge.git
cd ironshep-edge
cargo test --locked              # SNMP/BACnet decoder unit tests
cargo build --release --locked

--locked builds the exact dependency versions recorded in the committed Cargo.lock, so this host gets what was tested rather than whatever is newest today. If the lockfile and Cargo.toml ever disagree the build fails loudly instead of silently upgrading. Versions are inventoried in Code/TECHNOLOGY.md.

If the OPC-UA dependency ever gives you trouble: cargo build --release --locked --no-default-features builds everything else.

3. Get this edge's certificate from Host 2¶

Run on Host 2 first if you haven't: bash scripts/gen_certs.sh --edge <EDGE_ID>. Then, from this host:

mkdir -p certs
scp <CORE_HOST_ADMIN>@<CORE_HOSTNAME_OR_IP>:~/ironshep-core/certs/ca.crt certs/
scp <CORE_HOST_ADMIN>@<CORE_HOSTNAME_OR_IP>:~/ironshep-core/certs/edge-<EDGE_ID>.crt certs/
scp <CORE_HOST_ADMIN>@<CORE_HOSTNAME_OR_IP>:~/ironshep-core/certs/edge-<EDGE_ID>.key certs/
chmod 600 certs/edge-<EDGE_ID>.key

(Paths above assume Host 2's repo checkout lives in ~/ironshep-core — adjust to wherever it actually cloned.)

4. Configure¶

mkdir -p /etc/ironshep/certs
sudo cp certs/ca.crt certs/edge-<EDGE_ID>.crt certs/edge-<EDGE_ID>.key /etc/ironshep/certs/
sudo chmod 600 /etc/ironshep/certs/edge-<EDGE_ID>.key
sudo cp config/edge.toml /etc/ironshep/edge.toml

Edit /etc/ironshep/edge.toml:

[core]
edge_id     = "<EDGE_ID>"
endpoint    = "https://<CORE_HOSTNAME_OR_IP>:50051"
ca_cert     = "/etc/ironshep/certs/ca.crt"
client_cert = "/etc/ironshep/certs/edge-<EDGE_ID>.crt"
client_key  = "/etc/ironshep/certs/edge-<EDGE_ID>.key"
# Only needed if <CORE_HOSTNAME_OR_IP> above is an IP that was NOT itself
# passed to gen_certs.sh as a SAN on Host 2 - point this at one that was:
#tls_server_name = "<CORE_HOSTNAME>"

[forward]
batch_size = 500
flush_ms = 500
buffer_max = 50000

Then enable/configure only the collectors this site actually has — [syslog], [snmp] (trap receiver), [snmp_poll] (active polling), [mqtt], [modbus], [opcua], [bacnet] — each is independent. Defaults in the shipped edge.toml are a reasonable starting point; set enabled = false for anything not present at this site.

Polling network devices via SNMP. [snmp_poll] reads agents on an interval (the twin of [snmp], which only receives traps). edge.toml ships two worked examples: a Synology NAS (temperatures, RAID status) and a commented UniFi UDM Pro (CPU, load, memory, per-interface throughput and errors). Set the target's host + community and drop the leading # to enable. Enabling SNMP on the device itself is vendor-specific — e.g. the UDM Pro usually needs SSH into UniFi OS to configure snmpd. Values are stored as raw counters; derive rates (throughput) as a delta over time in your queries. See the edge repo's README (§ SNMP polling) for the OID details.

5. Install as a service¶

sudo useradd --system --home /opt/ironshep-edge --shell /sbin/nologin ironshep
sudo mkdir -p /opt/ironshep-edge
sudo cp target/release/ironshep-edge /opt/ironshep-edge/
sudo chown -R ironshep:ironshep /opt/ironshep-edge /etc/ironshep

sudo cp systemd/ironshep-edge.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now ironshep-edge
sudo systemctl status ironshep-edge --no-pager
journalctl -u ironshep-edge -f    # Ctrl-C to stop watching

Expect one line per enabled listener, then connected to core at https://<CORE_HOSTNAME_OR_IP>:50051.

6. Firewall — only what you enabled¶

# syslog, snmp traps, bacnet respectively - open only the ones enabled above:
sudo firewall-cmd --permanent --add-port=5514/udp --add-port=5514/tcp
sudo firewall-cmd --permanent --add-port=5162/udp
sudo firewall-cmd --permanent --add-port=47808/udp
sudo firewall-cmd --reload

MQTT, Modbus, OPC-UA, and SNMP polling are outbound connections from this host — no inbound ports needed for those; just allow egress to the polled devices (SNMP polling dials their UDP 161). The gRPC link to Host 2 is outbound too (already covered by Host 2's firewall rule for this host's IP).

Want standard ports 514/162 instead of 5514/5162? Two options — redirect (no privileges needed) or bind directly (the systemd unit already grants CAP_NET_BIND_SERVICE):

sudo firewall-cmd --permanent --add-forward-port=port=514:proto=udp:toport=5514
sudo firewall-cmd --permanent --add-forward-port=port=162:proto=udp:toport=5162
sudo firewall-cmd --reload

SELinux stays Enforcing either way.

7. Smoke test¶

sudo dnf install -y net-snmp-utils nc
bash scripts/send_test_traffic.sh

Then, back on Host 1 (database):

sudo -u postgres /usr/pgsql-16/bin/psql -d ironshep -c \
  "SELECT ts, edge_id, protocol, category, left(message,50) msg FROM events ORDER BY ts DESC LIMIT 10;"

Rows tagged edge_id = <EDGE_ID> confirm the full chain worked.

8. Scheduled test jobs (cron)¶

Two scripts are built for unattended runs — both are quiet when things are fine, so cron only mails you when something needs attention.

Script	Job	Suggested schedule
`scripts/synthetic_traffic.sh`	Sends varied, realistic OT syslog/SNMP/BACnet messages so the pipeline is continuously exercised and the console/anomaly work has data	every 5 min
`scripts/edge_healthcheck.sh`	Verifies the service is active, listeners are bound, core is reachable, and the buffer isn't dropping captures	every 15 min

8a. Install the scripts and a log location¶

sudo mkdir -p /opt/ironshep-edge/scripts /opt/ironshep-edge/tools/simulators
sudo cp scripts/synthetic_traffic.sh scripts/edge_healthcheck.sh /opt/ironshep-edge/scripts/
sudo cp tools/simulators/bacnet_iam.py /opt/ironshep-edge/tools/simulators/
sudo chmod +x /opt/ironshep-edge/scripts/*.sh

sudo touch /var/log/ironshep-synthetic.log
sudo chown ironshep:ironshep /var/log/ironshep-synthetic.log

Prereqs for the full message mix (syslog works with bare bash):

sudo dnf install -y net-snmp-utils python3    # SNMP traps + BACnet I-Am

8b. Verify by hand before scheduling¶

Always run each once interactively — a cron entry that fails silently is worse than no cron entry:

/opt/ironshep-edge/scripts/synthetic_traffic.sh --verbose ; echo "exit=$?"
/opt/ironshep-edge/scripts/edge_healthcheck.sh --verbose ; echo "exit=$?"

Expect exit=0 from both. The health check prints each listener it found bound and confirms the core connection.

8c. Install the crontab¶

Run as the ironshep service user (no root needed — these only send UDP/TCP to localhost and read the journal):

sudo -u ironshep crontab -e

# ── IronShep edge — scheduled test jobs ──────────────────────────────
# The edge listens on non-privileged ports by default; override here if
# you changed them in /etc/ironshep/edge.toml.
SYSLOG_PORT=5514
SNMP_PORT=5162
BACNET_PORT=47808
MAILTO=ops@example.com

# Synthetic plant traffic every 5 minutes. Quiet on success; a failed send
# prints to stderr, which cron mails to MAILTO.
*/5 * * * * /opt/ironshep-edge/scripts/synthetic_traffic.sh >> /var/log/ironshep-synthetic.log 2>&1

# Health check every 15 minutes. Silent when healthy; mails you when not.
*/15 * * * * /opt/ironshep-edge/scripts/edge_healthcheck.sh

Notes on the entries above:

The health check deliberately has no redirect — its output is the alert. Redirecting it to a log would silence the thing you installed it for.
The traffic generator appends to a log because on success it's silent anyway; the log gives you a timeline when you're debugging.
synthetic_traffic.sh takes a lock, so a slow run never overlaps the next tick (it exits 2 and says so rather than piling up).
Set MAILTO to a real address, or drop it and rely on the log plus the web console.

8d. Confirm cron is actually running them¶

sudo systemctl enable --now crond            # RHEL 9: crond, not cron
sudo -u ironshep crontab -l                  # entries are installed
journalctl -u crond --since -20min | grep ironshep
tail -f /var/log/ironshep-synthetic.log      # should stay empty on success

Then watch the data arrive — on Host 2's web console the edge card's "Events · 24h" should climb every 5 minutes, or query Host 1 directly:

sudo -u postgres /usr/pgsql-16/bin/psql -d ironshep -c \
  "SELECT date_trunc('minute', ts) AS minute, count(*)
   FROM events WHERE ts > now() - interval '30 minutes'
   GROUP BY 1 ORDER BY 1 DESC;"

8e. Log rotation¶

Left alone the log grows slowly (it's silent on success), but wire it up anyway:

sudo tee /etc/logrotate.d/ironshep-synthetic > /dev/null <<'EOF'
/var/log/ironshep-synthetic.log {
    weekly
    rotate 4
    compress
    missingok
    notifempty
    create 0644 ironshep ironshep
}
EOF

Turning the synthetic traffic off¶

Once real devices are feeding this edge, the generator has done its job:

sudo -u ironshep crontab -e     # comment out the */5 line

Keep the health check running — that one stays useful for the life of the deployment.

9. Continuous device simulation (optional)¶

For a demo or a soak test, the Modbus pump simulator gives the edge a degrading asset to poll (temperature creeping up, vibration following) — much better anomaly-detection material than syslog alone. Run it as a service rather than from cron, since it's a long-lived server:

sudo dnf install -y python3-pip
sudo pip3 install pymodbus

sudo tee /etc/systemd/system/ironshep-modbus-sim.service > /dev/null <<'EOF'
[Unit]
Description=IronShep Modbus pump simulator (POC test data)
After=network-online.target

[Service]
Type=simple
User=ironshep
ExecStart=/usr/bin/python3 /opt/ironshep-edge/tools/simulators/modbus_sim.py
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

sudo cp tools/simulators/modbus_sim.py /opt/ironshep-edge/tools/simulators/
sudo systemctl daemon-reload
sudo systemctl enable --now ironshep-modbus-sim

Then enable Modbus polling in /etc/ironshep/edge.toml and restart the edge:

[modbus]
enabled = true
[[modbus.targets]]
host = "127.0.0.1"
port = 1502
unit_id = 1
poll_interval_secs = 10
registers = [
    { address = 0, name = "pump_temperature_c", scale = 0.1 },
    { address = 1, name = "pump_vibration_mm_s", scale = 0.01 },
    { address = 2, name = "pump_rpm", scale = 1.0 },
]

sudo systemctl restart ironshep-edge

The console's edge view will show the three pump metrics with live sparklines within a minute.

10. Resilience check (optional)¶

Stop ironshep-core on Host 2 for a minute, keep sending traffic here — this host's log shows captures buffering and reconnect attempts backing off (1s → 60s). Restart core; the backlog ships within the next flush interval, with each row's original capture time preserved.

Adding another edge later¶

Repeat this whole doc on a new Host 3, with a new <EDGE_ID> — after Host 2 mints that edge's certificate (02-core-host.md, step 4) and its firewall rule (step 9) for the new host's IP.

Troubleshooting¶

Symptom	Likely cause
`cannot reach core (transport error)`	Host 2's firewall doesn't allow this host's IP on 50051, or `endpoint` is wrong
TLS handshake failure	`endpoint`'s name/IP isn't a SAN in Host 2's server cert → set `tls_server_name`, or regenerate that cert on Host 2 with the right names
`certificate required` on Host 2's side	cert paths here are wrong/unreadable, or this cert wasn't signed by the CA Host 2 trusts
rows never appear, no errors logged	the relevant `[listener]` is `enabled = false`, or the host firewall is silently dropping the UDP
cron jobs never run	`crond` not enabled (`systemctl enable --now crond`), or the crontab went to the wrong user — check `sudo -u ironshep crontab -l`
`synthetic_traffic: previous run still in progress`	a prior run hung; the lock self-clears once that PID exits, or `rm -rf /tmp/ironshep-synthetic.lock`
cron mails "snmptrap send failed"	`net-snmp-utils` missing, or SNMP is disabled in edge.toml while the script still targets its port
`snmp_poll` logs `no response within 5s`	the device's SNMP agent isn't enabled/reachable, wrong IP/port, or a firewall on the device is dropping UDP 161
polled device shows as an asset but few/no metrics	community-string mismatch (the agent silently ignores a wrong community), or those OIDs aren't exposed by that firmware — `snmpwalk -v2c -c <community> <ip> .1.3.6.1` to see what's actually there, then trim the config
polled metrics look like ever-growing numbers	expected — interface/CPU counters are cumulative; derive throughput as a delta over time in your query (see the edge README)
`snmp <addr>: bad OID component` at startup	a typo in an `snmp_poll` OID — the log names the offending target and OID

Turn up logging with RUST_LOG=debug on either host's service (edit the Environment= line in the systemd unit, daemon-reload, restart). At debug level snmp_poll also logs the OIDs it skipped as non-numeric or that returned no value — useful when a device exposes fewer OIDs than configured.