🎯 OVERWATCH

An open, hackable take on connected-warfare-style perception — running on a $500 dev kit.
Multi-sensor fusion · Cross-camera tracking · Tactical AR HUD · Edge inference on Jetson Orin Nano

Inspired by Anduril Connected Warfare & the Lattice OS concept —
built as a community reference implementation, not affiliated with Anduril Industries.

Demo · Inspiration · Features · Architecture · Quick Start · Deployment · Testing · API · Troubleshooting

Demo Video

Watch on YouTube — prototype iteration 1.

🛰️ Inspiration & scope

OVERWATCH is a publicly-available reference implementation of the multi-sensor situational-awareness concept popularised by Anduril's Connected Warfare and its Lattice software platform — the idea that a network of low-cost, heterogeneous sensors can be fused at the edge into a single, AI-driven view of the battlespace.

This project takes that idea and runs with it on commodity hardware:

• A $500 NVIDIA Jetson Orin Nano instead of a hardened tactical server
• IP webcams + mobile phone cameras instead of dedicated military-grade sensors
• YOLOv8 + Kalman + homography instead of classified perception stacks
• A FastAPI/React stack instead of proprietary tactical software

The visual language — diamond IFF markers, compass ribbon, threat rings, ghost predictions — is inspired by Anduril's EagleEye HUD aesthetic (one of the publicly-shown UI surfaces of Lattice). It is not a clone, not affiliated with or endorsed by Anduril Industries, and not a substitute for their products. Trademarks belong to their respective owners.

Scope honesty: this is a research/educational project. It demonstrates the principles of connected sensing — sensor fusion, cross-camera re-ID, edge inference, real-time broadcast — at a scale that fits in a backpack. It is not military-grade, not C2-system-grade, and not certified for any operational use.

What's the same idea, what's different

If you're building something in this space — researchers, students, civilian defense-tech tinkerers, public-safety folks — this repo is meant to be a starting point you can fork, hack on, and learn from.

Overview

OVERWATCH is a real-time multi-camera situational awareness platform built for edge deployment on NVIDIA Jetson Orin Nano. It fuses video from IP cameras and mobile phones into a unified world model using YOLOv8 detection, Hungarian-assignment tracking, adaptive Kalman filtering, and cross-camera appearance re-identification — all at TensorRT FP16 speeds.

The system runs a singleton perception pipeline: detection, tracking, and fusion execute once per tick regardless of how many viewers are connected, then broadcast pre-serialized snapshots to all clients over binary WebSocket.

1 camera + 10 viewers = 1 GPU inference, not 10.

🚀 Features

Core perception

Platform

🏗️ Architecture

                          ┌─────────────────────────────────┐
                          │       OVERWATCH  v2.0.0         │
                          └─────────────────────────────────┘

  ╔═══════════════╗       ╔═══════════════════════════════════════════════════╗
  ║  DATA SOURCES ║       ║          JETSON ORIN NANO  (backend :8000)        ║
  ╠═══════════════╣       ╠═══════════════════════════════════════════════════╣
  ║               ║       ║                                                   ║
  ║  📷 IP Camera ─────────►  CameraCapture (OpenCV, MJPEG/RTSP)              ║
  ║               ║       ║       │                                           ║
  ║  📱 Mobile    ─────────►  VirtualCamera (binary JPEG push)                ║
  ║   Phone       ║ws/cam ║       │         + GPS/IMU sensor data             ║
  ║               ║       ║       ▼                                           ║
  ║               ║       ║  ┌──────────────────────────────────────────┐     ║
  ║               ║       ║  │     PerceptionPipeline  (singleton)      │     ║
  ║               ║       ║  │                                          │     ║
  ║               ║       ║  │  1. DETECT   YOLOv8n TensorRT FP16       │     ║
  ║               ║       ║  │              + HSV appearance features   │     ║
  ║               ║       ║  │              │                           │     ║
  ║               ║       ║  │  2. TRACK    Hungarian assignment        │     ║
  ║               ║       ║  │              IoU + cosine appearance     │     ║
  ║               ║       ║  │              │                           │     ║
  ║               ║       ║  │  3. FUSE     Adaptive Kalman 6-state     │     ║
  ║               ║       ║  │              Cross-camera matching       │     ║
  ║               ║       ║  │              Sensor trust scoring        │     ║
  ║               ║       ║  │              │                           │     ║
  ║               ║       ║  │  4. SNAPSHOT Pre-serialized msgpack      │     ║
  ║               ║       ║  └──────────────┬───────────────────────────┘     ║
  ║               ║       ║                 │                                 ║
  ╚═══════════════╝       ║                 ▼  broadcast                      ║
                          ║     WebSocketManager (/ws, msgpack binary)        ║
                          ║         │           │           │                 ║
                          ╚═════════╪═══════════╪═══════════╪═════════════════╝
                                    │           │           │
                          ┌─────────▼──┐  ┌─────▼──┐  ┌─────▼─────┐
                          │  Viewer 1  │  │Viewer 2│  │ Viewer N  │
                          │  React     │  │  React │  │  React    │
                          │  AR Canvas │  │  ...   │  │  ...      │
                          └────────────┘  └────────┘  └───────────┘

Pipeline design

OVERWATCH runs a single shared pipeline rather than per-viewer. The PerceptionPipeline singleton executes detect → track → fuse once per tick, produces a PerceptionSnapshot with pre-serialized msgpack packets, and all connected viewers read from the latest snapshot.

• 1 camera + 10 viewers = 1 GPU inference
• Zero-copy broadcast via pre-serialized binary packets
• Slow viewers gracefully skip intermediate frames (per-client 2 s send timeout)

📁 Project structure

OVERWATCH/
├── backend/                              # FastAPI + perception engine
│   ├── main.py                           # App entry, lifespan, REST + WS endpoints
│   ├── requirements.txt                  # Python deps (CPU/Windows dev)
│   ├── requirements-jetson.txt           # Jetson Orin Nano deps (pinned)
│   ├── .env.example                      # Config template
│   └── app/
│       ├── domain/entities.py            # Detection, Track, WorldObject, ...
│       ├── application/
│       │   ├── ports.py                  # Repository interfaces
│       │   └── services.py               # PerceptionPipelineService
│       └── infrastructure/
│           ├── auth.py                   # Optional JWT verify/issue
│           ├── camera_adapter.py         # OpenCV capture + virtual cameras
│           ├── config_adapter.py         # Pydantic settings
│           ├── container.py              # DI container
│           ├── detection_adapter.py      # YOLO wrapper
│           ├── frame_encoder_adapter.py  # JPEG encode
│           ├── tracking_adapter.py       # Hungarian + DeepSORT
│           ├── websocket_adapter.py      # msgpack broadcast
│           └── world_model_adapter.py    # Kalman fusion + homography
│   └── tests/
│       ├── conftest.py                   # Shared fixtures
│       └── unit/                         # 57 unit tests
│
├── frontend/                             # React 18 admin dashboard
│   ├── package.json
│   └── src/
│       ├── pages/
│       │   ├── AdminDashboard.jsx        # Main camera grid
│       │   └── MobileCamera.jsx          # Phone camera streaming UI
│       ├── components/
│       │   ├── CameraDisplay.jsx         # Canvas AR overlay renderer
│       │   ├── ErrorBoundary.jsx         # Top-level error fallback
│       │   ├── StatsPanel.jsx
│       │   └── ConnectionStatus.jsx
│       ├── application/hooks/            # useCameraData, useWebSocket, useSystemStats
│       └── infrastructure/               # websocketAdapter, cameraStreamAdapter, apiAdapter
│
├── scripts/                              # Deployment & ops
│   ├── _jetson_common.py                 # Shared SSH/SFTP helper (env-driven creds)
│   ├── deploy_jetson.py                  # Atomic deploy with --rollback
│   ├── restart_jetson.py                 # Quick backend restart
│   ├── check_logs.py / check_status.py
│   ├── ws_test.py
│   └── archive/                          # Retired/duplicate scripts (reference only)
│
├── certs/                                # SSL certificates (gitignored)
├── .github/workflows/ci.yml              # GitHub Actions test runner
├── pyproject.toml                        # pytest + project metadata
└── README.md

⚡ Quick start

Prerequisites

• Python 3.10+ with pip
• Node.js 18+ with npm
• NVIDIA Jetson Orin Nano for production, or any machine with CUDA for development

1. Clone

git clone https://github.com/mandarwagh9/overwatch.git
cd overwatch

2. Local development

# Backend
cd backend
pip install -r requirements.txt
python main.py

# Frontend (new terminal)
cd frontend
npm install
npm start

Open https://localhost:3000 — accept the self-signed certificate warning.

3. Single-binary mode (frontend served by backend)

cd frontend && npm install && npm run build && cd ..
cd backend && python main.py

Backend serves both the React app and the API at https://localhost:8000.

🚀 Deployment

Deploy to Jetson

Credentials are read from environment — never hardcoded:

export JETSON_HOST=192.168.1.10        # default if unset
export JETSON_USER=mandar              # default if unset
export JETSON_PASS=...                 # or use JETSON_KEY=/path/to/id_rsa
python scripts/deploy_jetson.py

The script:

1. Connects via SSH/SFTP using paramiko
2. Uploads backend, frontend build, certs to <remote>.new/ (staging)
3. Atomically swaps <remote>.new → <remote>, keeping previous version at <remote>.bak
4. Generates a fresh JWT_SECRET and writes a chmod 600 .env
5. Installs Python dependencies
6. Starts the backend

Rollback

python scripts/deploy_jetson.py --rollback

Swaps the last .bak directory back into place. Use after a bad deploy.

Quick operations

# Restart backend without redeploying
python scripts/restart_jetson.py

# Tail logs
python scripts/check_logs.py

# Quick status
python scripts/check_status.py

Access (replace with your JETSON_HOST)

🧪 Testing

The backend has 57 unit tests covering domain primitives, Kalman filtering, coordinate transforms, tracking, and configuration. CI runs them on every push and PR.

python -m pytest backend/tests/unit -v

Tests are pure-Python and do not require CUDA, ultralytics, or torch. They use pytest.importorskip("cv2") where OpenCV is needed.

📡 API reference

REST endpoints

WebSocket endpoints

When AUTH_ENABLED=true, both endpoints require a ?token=<jwt> query parameter; unauthorized connections close with 1008.

Mobile registration handshake

Client → { "type": "register", "role": "camera_source", "camera_id": null }
Server → { "type": "registered", "camera_id": 0, "target_fps": 15 }
Client → [binary JPEG frames at target FPS]
Client → { "type": "sensor_data", "gps": {...}, "orientation": {...} }

🔧 Configuration

Backend .env

Frontend .env

Deploy environment

🎯 AR overlay system — EagleEye-inspired tactical HUD

The frontend renders a tactical HUD inspired by Anduril's EagleEye UI — diamond IFF markers, compass ribbon, threat rings — implemented entirely in HTML5 Canvas. (Visual style only; rendered from open code, no Anduril assets used.)

Detection overlays show what the model sees right now. Track overlays show persistent identity across frames. Predictions show cross-camera projections — green for homography (most accurate), orange for world-model fallback (rough but always available), red for pixel extrapolation (last resort).

🌍 World model & sensor fusion

Kalman filter

Each fused world object maintains a 6-state Kalman filter [x, y, z, vx, vy, vz] with constant-velocity dynamics. Measurement noise R adapts per-update based on detection confidence, bounding box area, and sensor trust — higher-quality observations tighten the filter, while noisy or untrusted sensors widen it. dt is clamped to ≥ 0 to defend against cross-camera clock skew.

Cross-camera association

Objects from different cameras are matched when:

• Euclidean distance < 2 m
• Same class_id
• Appearance cosine similarity > 0.5 (when feature vectors available)

Sensor trust

Each camera/sensor earns trust through consistency:

• Consistent measurements → trust increases (capped at 1.0)
• Innovation outliers → trust decays (floored at 0.1)

Appearance re-ID

• 64-dimensional HSV histogram descriptors computed per detection (~0.1 ms each)
• L2-normalized for cosine similarity
• Exponential moving average (α = 0.3) for descriptor stability across frames

📐 Cross-camera homography — how it works

The signature feature is ghost prediction: when Camera 0 can't see a person but Camera 1 can, the system renders a ghost overlay on Camera 0's feed showing where that person is.

The problem with naive extrapolation

Sliding a person's last-known pixel position forward in time fails within seconds because:

• Different cameras have completely different pixel coordinate systems
• The mapping between camera views is a projective transformation, not a linear offset
• A person at pixel (400, 300) in Camera 1 might correspond to (800, 500) in Camera 0

The solution: learn the camera-to-camera transform

When both cameras simultaneously observe the same person (matched via appearance re-ID), the system records foot-point correspondence pairs — the bottom-center of the bounding box in each view. These foot points project to the same physical ground-plane location.

With ≥ 4 such pairs, cv2.findHomography() + RANSAC computes a 3×3 homography matrix $H$ that maps any ground-plane point from one camera's pixel space to another's:

$$\begin{pmatrix} x' \ y' \ w \end{pmatrix} = H \cdot \begin{pmatrix} x \ y \ 1 \end{pmatrix}$$

Self-calibrating pipeline

1. Collect: when re-ID matches a person across Camera 0 and Camera 1, record (foot_cam0, foot_cam1) pair
2. Estimate: after 4+ pairs, compute $H{0\to 1}$ and $H{1\to 0}$ via RANSAC (re-estimated every 5 new pairs)
3. Project: when Camera 0 loses a person but Camera 1 still sees them, apply $H_{1\to 0}$ to Camera 1's current foot point → position on Camera 0's feed
4. Validate: monitor reprojection error; if it spikes (camera moved), flush and re-learn

Computational cost

• Homography estimation: < 0.1 ms (called every 5 new pairs, not every frame)
• Per-prediction projection: < 0.001 ms (one 3×3 matrix multiply)
• Total overhead per frame: effectively zero on Jetson Orin Nano

Visual indicators

📱 Mobile camera streaming

Any phone on the same LAN can become a camera source:

• Via React app: https://<frontend-ip>:3000/mobile
• Standalone page: https://<jetson-ip>:8000/mobile

The mobile client:

1. Opens the rear camera via getUserMedia (1280×720)
2. Renders to an offscreen canvas, extracts a JPEG blob
3. Sends binary frames over WebSocket to /ws/camera
4. Captures GPS (watchPosition, high-accuracy) and IMU (DeviceOrientationEvent) at 2 Hz
5. Sends sensor data as JSON for camera calibration fusion

getUserMedia requires HTTPS — this is why SSL certificates are mandatory even on LAN.

⚠️ Edge cases & known limitations

Cross-camera prediction

Tracking & re-ID

Sensor fusion

Network & deployment

🐛 Troubleshooting

python scripts/restart_jetson.py
# Or manually over SSH:
JETSON_HOST=192.168.1.10 JETSON_PASS=... ssh "$JETSON_USER@$JETSON_HOST" \
  'pkill -9 -f "python3 main.py"; sleep 2; cd /home/$USER/overwatch/backend && nohup python3 main.py > /tmp/overwatch.log 2>&1 &'

python scripts/check_logs.py        # Tails the last 50 lines
python scripts/check_status.py      # Backend status snapshot

python scripts/deploy_jetson.py --rollback

🧰 Tech stack

📚 References & sources

The cross-camera homography system is built on established multi-view geometry principles and inspired by several academic works and open-source implementations.

Foundational theory

Research papers

Open-source implementations

Datasets referenced

📄 License & trademarks

OVERWATCH is released under the MIT License — copyright © 2024–2026 Mandar Wagh. You're free to use, copy, modify, merge, publish, distribute, sublicense, and sell copies of the software, subject to the conditions in the license file. If you build something cool with it, a link back is appreciated but not required.

"Anduril," "Lattice," "Connected Warfare," and "EagleEye" are trademarks of Anduril Industries, Inc. This project is an independent community implementation inspired by publicly-shown concepts of those products. It is not affiliated with, endorsed by, or sponsored by Anduril Industries. No proprietary information, code, or assets from Anduril are used.

All other third-party trademarks (NVIDIA, Jetson, TensorRT, React, FastAPI, etc.) belong to their respective owners.

Connected sensing, at hackathon scale. 🎯
Inspired by Anduril Connected Warfare · Built on open tools.