Smart Glasses

Overview

FERAL ingests first-person camera frames over HUP and feeds them to the vision pipeline for scene analysis and conversational context.

What ships today

FERAL does not ship a driver for any glasses vendor. What it ships is the receiving half: any node that can send a glasses_frame over HUP becomes a first-person camera for the brain.

Piece	Status
`glasses_frame` ingest over HUP	Ships — `api/server.py:4471`
Frame ring buffer per device	Ships — `perception/glasses_buffer.py`
Scene analysis + context attach	Ships — `perception/scene.py`, `context_attach.py`
Phone camera as the frame source	Ships — via the companion app node
A driver for any specific glasses product	Does not ship

Earlier versions of this page listed Meta Ray-Ban, Vuzix Blade 2 and a “FERAL Glasses (reference)” device as supported, and documented a hardware.glasses settings block with ar_overlay, stream_resolution and stream_fps. None of that existed. No code in this repository implements any of those vendors or reads any of those settings keys. The table above is what the code actually does.

Connecting glasses

Because the brain accepts frames rather than driving hardware, the integration point is a node, not a FERAL-side driver. Write (or use) a node that owns the Bluetooth or Wi-Fi connection to the glasses and forwards frames as glasses_frame messages. The node SDKs (feral-nodes/python-node-sdk, ts-node-sdk, Swift, Kotlin) all speak the envelope. The practical path today is a phone: the companion app already holds the BLE connection to a wearable and is already a registered node, so it forwards frames without any new transport work.

Connection Setup

There is no hardware.glasses settings block — the brain reads no such keys. A node registers over HUP like any other, declaring a camera capability, and then sends glasses_frame messages. See the node SDK README for the registration handshake.

Pair the glasses

Put your glasses in pairing mode (varies by device), then list paired devices and complete pairing through the WebUI or the device-pair API:

feral devices
curl -X POST http://localhost:9090/api/devices/pair \
  -H "Content-Type: application/json" \
  -d '{ "name": "theora-glasses", "type": "glasses" }'

The WebUI’s Settings → Devices screen renders the same registry with a “Pair” button per advertised device.

Start the video stream

The video stream starts automatically once paired. Verify:

feral status

Hardware    ● 1 device   theora-glasses (streaming 720p @ 15fps)

Video Streaming Protocol

Frames reach the vision pipeline as discrete HUP messages, not as a video stream the brain connects to:

glasses → node (owns the BLE/Wi-Fi link) → HUP glasses_frame
        → GlassesBuffer → scene analysis → LLM context

The brain never opens a stream. It receives whatever the node chooses to send, so resolution, frame rate and codec are the node’s decisions, not FERAL settings.

This section previously documented stream_resolution, stream_fps, codec and vision_interval as configurable settings. The brain reads none of those keys — they exist in no source file. Set the capture rate in the node that owns the camera.

Scene Analysis Pipeline

The brain processes video frames through a multi-stage pipeline:

Frame capture

Frames are sampled at the configured vision_interval (default: every 2 seconds).

Object detection

Local YOLO model identifies objects, people, and text in the scene.

Scene description

The vision LLM generates a natural-language description of the current scene.

Context injection

The scene description is added to the brain’s context window, available for queries and proactive reasoning.

Example scene context:

{
  "type": "scene_analysis",
  "timestamp": "2026-04-12T14:30:00Z",
  "description": "Indoor office. Monitor displaying VS Code with Python file. Coffee mug on left. Two people in background near whiteboard.",
  "objects": ["monitor", "keyboard", "coffee_mug", "person", "whiteboard"],
  "text_detected": ["def process_data(", "TODO: fix auth"]
}

AR Overlay Commands

Send visual overlays to the glasses display:

curl -X POST http://localhost:9090/api/devices/connected \
  -H "Content-Type: application/json" \
  -d '{
    "type": "notification",
    "title": "Meeting in 5 min",
    "body": "Team standup — Conference Room B",
    "duration_ms": 5000,
    "position": "top_right"
  }'

Overlay Type	Description
`notification`	Toast-style message in corner
`hud`	Persistent heads-up data (HR, time, next event)
`highlight`	Outline a detected object in the scene
`navigation`	Turn-by-turn directional arrows
`text`	Arbitrary text at a specified position

Hands-Free Voice

With glasses connected, FERAL’s voice pipeline uses the glasses microphone and speaker:

Wake word detection runs on-device (glasses)
Audio streams to the brain for processing
Responses play through the glasses speaker or bone conduction

"Hey FERAL, what am I looking at?"
→ "You're looking at a whiteboard with a system architecture diagram.
   I can see boxes labeled 'API Gateway', 'Auth Service', and 'Database'."

Privacy Controls

settings.json

{
  "hardware": {
    "glasses": {
      "auto_record": false,
      "blur_faces": true,
      "store_frames": false,
      "vision_only_on_request": false
    }
  }
}

When store_frames is true, video frames are saved locally for memory recall. This uses significant disk space (~2 GB/day at 720p). Frames never leave your machine.

Troubleshooting

No frames reaching the brain

Frames arrive as HUP glasses_frame messages from a registered node, so check the node first: is it connected to /v1/node, did node_register succeed, and is it actually sending? A node that connects but never sends looks identical to glasses that are off.

High latency on scene analysis

Lower the capture rate in the node — the brain has no frame-rate setting. If the vision model is a cloud one, latency also tracks your connection; a local model removes that leg.

AR overlay not displaying

FERAL has no AR overlay feature. Nothing in the brain renders to a glasses display, and there is no ar_overlay setting. Drawing on the lens is the node’s and the vendor SDK’s job.

Overview

Getting Started

Guides

Marketplace

Memory

Hardware

Channels

Connectivity

Security

Native Apps

Operations

SDKs

Reference

Help

Community

Smart Glasses

Overview

What ships today

Connecting glasses

Connection Setup

Video Streaming Protocol

Scene Analysis Pipeline

AR Overlay Commands

Hands-Free Voice

Privacy Controls

Troubleshooting

​Overview

​What ships today

​Connecting glasses

​Connection Setup

​Video Streaming Protocol

​Scene Analysis Pipeline

​AR Overlay Commands

​Hands-Free Voice

​Privacy Controls

​Troubleshooting

Overview

What ships today

Connecting glasses

Connection Setup

Video Streaming Protocol

Scene Analysis Pipeline

AR Overlay Commands

Hands-Free Voice

Privacy Controls

Troubleshooting