Skip to main content

​
Overview

FERAL’s smart glasses integration enables real-time video streaming, scene analysis via the vision pipeline, AR overlay commands, and hands-free voice interaction. The glasses act as a first-person sensor for the brain — providing continuous visual context about your environment.

​
Supported Devices

DeviceConnectionFeatures
FERAL Glasses (reference)BLE + Wi-FiVideo, audio, AR overlay, touch
Meta Ray-BanCompanion app bridgePhoto capture, audio, voice
Vuzix Blade 2Wi-Fi directVideo, AR overlay
Custom ESP32-CAM buildsBLE / Wi-FiVideo stream, basic audio
Third-party glasses support varies. The FERAL reference glasses support the full feature set. Other devices may only support a subset.

​
Connection Setup

1

Enable the glasses adapter

settings.json
{
  "hardware": {
    "glasses": {
      "enabled": true,
      "stream_resolution": "720p",
      "stream_fps": 15,
      "ar_overlay": true
    }
  }
}
2

Pair the glasses

Put your glasses in pairing mode (varies by device) and scan:
feral hardware scan --type glasses

Scanning for smart glasses…
✓ FERAL-Glasses-X1     BLE + Wi-Fi    Battery: 72%
? Pair this device? (Y/n)
✓ Paired. Video stream available.
3

Start the video stream

The video stream starts automatically once paired. Verify:
feral status

Hardware    ● 1 device   FERAL-Glasses-X1 (streaming 720p @ 15fps)

​
Video Streaming Protocol

Glasses stream video to the brain over Wi-Fi using a lightweight protocol: 
Glasses → Wi-Fi (MJPEG/H.264) → FERAL Brain → Vision Pipeline → LLM Context
SettingDefaultOptions
stream_resolution720p480p, 720p, 1080p
stream_fps155 – 30
codecmjpegmjpeg, h264
vision_interval2sHow often frames are sent to the vision model
Lower resolution and FPS extend battery life. Use 480p at 10fps for all-day wear.

​
Scene Analysis Pipeline

The brain processes video frames through a multi-stage pipeline:
1

Frame capture

Frames are sampled at the configured vision_interval (default: every 2 seconds).
2

Object detection

Local YOLO model identifies objects, people, and text in the scene.
3

Scene description

The vision LLM generates a natural-language description of the current scene.
4

Context injection

The scene description is added to the brain’s context window, available for queries and proactive reasoning.
Example scene context: 
{
  "type": "scene_analysis",
  "timestamp": "2026-04-12T14:30:00Z",
  "description": "Indoor office. Monitor displaying VS Code with Python file. Coffee mug on left. Two people in background near whiteboard.",
  "objects": ["monitor", "keyboard", "coffee_mug", "person", "whiteboard"],
  "text_detected": ["def process_data(", "TODO: fix auth"]
}

​
AR Overlay Commands

Send visual overlays to the glasses display: 
curl -X POST http://localhost:9090/api/hardware/glasses/overlay \
  -H "Content-Type: application/json" \
  -d '{
    "type": "notification",
    "title": "Meeting in 5 min",
    "body": "Team standup — Conference Room B",
    "duration_ms": 5000,
    "position": "top_right"
  }'
Overlay TypeDescription
notificationToast-style message in corner
hudPersistent heads-up data (HR, time, next event)
highlightOutline a detected object in the scene
navigationTurn-by-turn directional arrows
textArbitrary text at a specified position

​
Hands-Free Voice

With glasses connected, FERAL’s voice pipeline uses the glasses microphone and speaker:
  • Wake word detection runs on-device (glasses)
  • Audio streams to the brain for processing
  • Responses play through the glasses speaker or bone conduction
"Hey FERAL, what am I looking at?"
→ "You're looking at a whiteboard with a system architecture diagram.
   I can see boxes labeled 'API Gateway', 'Auth Service', and 'Database'."

​
Privacy Controls

settings.json
{
  "hardware": {
    "glasses": {
      "auto_record": false,
      "blur_faces": true,
      "store_frames": false,
      "vision_only_on_request": false
    }
  }
}
When store_frames is true, video frames are saved locally for memory recall. This uses significant disk space (~2 GB/day at 720p). Frames never leave your machine.

​
Troubleshooting

Ensure the glasses and FERAL host are on the same Wi-Fi network. Check firewall rules for the streaming port.
Reduce stream_fps and stream_resolution. If using a cloud vision model, latency depends on your internet connection — switch to a local model for faster results.
Verify ar_overlay is true in settings and that your glasses model supports overlays.