Project Overview
CyberGuard RT-IDS is an autonomous defense pipeline engineered to protect modern hybrid infrastructure against known threat signatures and novel zero-day anomalies. Rather than acting as a passive alerting dashboard that overwhelms security personnel, CyberGuard actively inspects network telemetry, performs dual-engine machine learning scoring, and safely mutates operating system packet filtering rules with verifiable audit rollback guarantees.
Problem and Motivation
Traditional Intrusion Detection Systems (IDS) suffer from a crippling trade-off: signature-based rules (like Snort or Suricata) fail against previously unseen zero-day mutations, while raw machine learning models generate high false-positive rates that make automatic active response dangerous. If a naive automated system blocks a legitimate gateway or core database due to a model error, it inflicts a self-inflicted Denial of Service. CyberGuard was designed to solve this by decoupling statistical anomaly screening from deterministic classification and wrapping every automated containment action in a durable database receipt with strict TTL expirations.
Requirements and Constraints
- ■Sub-5 millisecond end-to-end packet flow classification latency to prevent network bottlenecking.
- ■Zero train-serve feature skew between training parquet logs and live network packet capture buffers.
- ■Deterministic containment rollback: every OS firewall mutation (Windows netsh / Linux iptables) must automatically expire unless refreshed.
- ■Safe multi-worker thread safety: prevent CUDA contention and CPU GIL deadlocks during concurrent stream processing.
- ■Tamper-proof audit trail: all autonomous defensive actions must be recorded with cryptographically signed database receipts.
My Role & Contributions
- ›Designer and primary developer of the backend, ML inference pipeline, containment actuators, and test suites.
- ›Engineered the PyTorch Deep Autoencoder architecture for reconstruction loss anomaly scoring.
- ›Designed the durable TTL containment reaper with SQLite WAL mode and atomic rollback transactions.
- ›Authored 46 comprehensive unit, integration, and attack matrix simulation test files.
System Architecture
The system utilizes a 4-tier decoupled pipeline communicating via asynchronous queues and atomic state transactions.
1. Telemetry Ingestion Layer
Suricata EVE-JSON flow parser and raw socket packet monitor extracting 78 statistical network flow features in real-time.
2. Dual-Engine ML Pipeline
PyTorch Deep Autoencoder computes anomaly reconstruction error; parallel XGBoost multi-class classifier identifies specific threat families (DoS, PortScan, BruteForce, Infiltration).
3. Guarded Containment Controller
Evaluates severity thresholds; issues host-level firewall rules (`netsh` or `iptables`) with cryptographic receipts and automated TTL expiration timers.
4. Async Command Center & API
FastAPI async backend providing SSE/WebSocket live telemetry streaming, RBAC authentication, and operator override controls.
Technical Implementation
Durable Containment Actuator with Zero-Loss Rollback
The active defense actuator executes OS-level packet drops while preventing persistent accidental lockouts. Each rule is registered with an absolute Unix expiration timestamp. A resilient background daemon reaps expired rules even if the main server restarts.
async def apply_containment_action(target_ip: str, rule_type: str, ttl_seconds: int = 300) -> ContainmentReceipt:
receipt_id = str(uuid.uuid4())
expiration = time.time() + ttl_seconds
# Platform-aware command execution
if sys.platform == "win32":
cmd = f"netsh advfirewall firewall add rule name=\"CG_BLOCK_{target_ip}\" dir=in action=block remoteip={target_ip}"
else:
cmd = f"iptables -I INPUT -s {target_ip} -j DROP"
await execute_system_command(cmd)
receipt = ContainmentReceipt(id=receipt_id, target_ip=target_ip, expires_at=expiration, status="ACTIVE")
await db.save_receipt(receipt)
return receiptTrain-Serve Feature Alignment Pipeline
To prevent training-serving skew, flow features are normalized using an immutable serialized vector schema ensuring uniform normalization across offline training and live packet inspection buffers.
Important Algorithms & Decisions
Deep Autoencoder Reconstruction Error for Anomaly Detection
Rationale: Supervised models fail on unknown attack payloads. An autoencoder trained exclusively on clean network baseline flows exhibits high reconstruction error (MSE) when processing zero-day anomalies.
Trade-off: Requires careful tuning of the reconstruction threshold to balance false discovery against sensitivity.
Thread-Safe CPU Model Inference in Fast Server Loop
Rationale: Forced CPU evaluation in XGBoost and PyTorch inference inside the async event loop to prevent CUDA context switching overhead on small batch sizes.
Trade-off: Consumes predictable CPU cores but completely eliminates GPU memory bus latency for micro-batches.
Engineering Challenges
Issue: Train-Serve Skew in Network Flow Feature Extraction
Root Cause: Offline CIC-IDS2017 training datasets computed bidirectional flow ratios over completed TCP sessions, while live streaming packets arrive incrementally.
Resolution: Implemented a sliding-window session accumulator in Cython/Python that caches TCP stream metadata up to a 1.5s time boundary before generating vector inputs.
Issue: Firewall Rule Accumulation and Zombie Lockouts
Root Cause: Server crashes during active attack mitigation left hundreds of blocking rules stuck in the Windows Firewall registry.
Resolution: Built an idempotent startup reconciliation reaper that scans the OS firewall table for `CG_BLOCK_*` tags and cross-references the durable SQLite database.
Testing & Evaluation
46 pytest test files covering unit behavior, mock packet injection, adversarial payload mutation, RBAC token expiration, and synthetic zero-day labs.
Results & Known Limitations
- ✓Successfully defended synthetic network against simulated multi-vector DDoS and port scan attacks.
- ✓Demonstrated zero persistent rule leakage across 200 forced kill-restart cycles.
- ✓Documented in a comprehensive technical audit report with reproducible JSON benchmark logs.
- ⚠Encrypted TLS 1.3 payloads without server-side decryption must be analyzed purely through flow timing and packet size metadata.
- ⚠Ultra-high bandwidth links (>10 Gbps) require DPDK or eBPF kernel offloading instead of user-space raw sockets.
Future Improvements
- →Implement eBPF XDP (eXpress Data Path) kernel driver for sub-microsecond packet drops on Linux.
- →Add graph neural network (GNN) module for multi-host lateral movement correlation.
Repository & Verification
Public open-source repository on GitHub with CI test suite.