SonicARray

AR-assisted spatial audio visualization for multi-speaker VBAP rendering.

SonicARray lets you place and drag virtual sound sources inside a real speaker array using a Meta Quest 3 headset. Coloured lines connect each source to its active speakers; line width and brightness reflect the live VBAP gain computed by a C++ backend running on a connected PC. Built as a research tool for studying how AR visualization helps people understand spatial audio routing.

---

System Overview

┌─────────────────────────────┐   OSC / UDP :7000 (src :7002)   ┌──────────────────────────────┐
│   Meta Quest 3  (Unity AR)  │ ──────────────────────────────► │   C++ Backend  (PC / Linux)  │
│                             │                                 │                              │
│  • AR passthrough           │ ◄────────────────────────────── │  • VBAP renderer             │
│  • Drag source balls        │  OSC / UDP :7002 (src :7000)    │  • PortAudio / JACK output   │
│  • Visualise speaker lines  │                                 │  • Terminal dashboard        │
└─────────────────────────────┘                                 └──────────────────────────────┘
                                                                              │
                                                                       speakers.yaml
                                                                      (speaker layout)

Data flow:

1. Unity streams each source's 3-D position (relative to the listener) to the backend over OSC/UDP.
2. The C++ backend runs VBAP, computes per-speaker gains, and plays audio through the physical speaker array.
3. Gains are sent back to Unity over TCP; Unity draws gain-weighted lines from each source to its active speakers.

---

Features

• Up to 8 simultaneous spatial sources draggable in AR
• Real-time VBAP with gain smoothing (lock-free, audio-thread safe)
• Live terminal dashboard — uptime, OSC packet rate, per-speaker gain bars
• Flexible speaker layout — edit speakers.yaml; no recompile needed
• Cross-platform backend — PortAudio on Windows/macOS, JACK on Linux
• Hand-tracking & controller support via Meta XR SDK
• Adjustable source ball size — sourceScale on each SpatialSource controls the visual radius of the sphere in world space (metres) and the grab collider radius, letting you tune the ball size to match your physical room scale

---

Repository Structure

SonicARray/
├── cpp/                            # C++ backend
│   ├── src/
│   │   ├── main.cpp                # Entry point; starts audio stream and terminal dashboard
│   │   ├── AudioEngine.{h,cpp}     # PortAudio / JACK stream lifecycle management
│   │   ├── VBAPRenderer.{h,cpp}    # VBAP gain computation; supports up to 8 simultaneous sources
│   │   ├── OSCReceiver.{h,cpp}     # UDP socket bound to :7000; parses /spatial/source_pos
│   │   └── OSCSender.{h,cpp}       # Sends /spatial/speaker_gains back to the Quest on :7002
│   ├── third_party/
│   │   ├── oscpack/                # OSC encoding & decoding (vendored, BSD/MIT-like)
│   │   └── portaudio/              # PortAudio cross-platform audio I/O (vendored, MIT)
│   └── CMakeLists.txt              # Builds SoundARray binary; auto-copies speakers.yaml to output
├── unity/                          # Unity AR frontend (Meta Quest 3)
│   └── Assets/
│       ├── Scenes/
│       │   └── MainScene.unity     # Main AR scene with SpeakerManager and source prefabs
│       └── Scripts/
│           ├── SpatialSource.cs    # Per-source ball: OSC send, gain-weighted line drawing, sourceScale
│           ├── SpeakerManager.cs   # Loads speakers.yaml at runtime; stores per-source gain tables
│           ├── OSCReceiver.cs      # Single UDP socket (:7002) used for both sending and receiving
│           ├── HandGestureController.cs  # Hand-tracking and controller drag input logic
│           ├── OSCClient.cs        # Low-level OSC packet serialisation
│           ├── ARPassthroughSetup.cs     # Enables Meta Quest passthrough layer
│           └── …                   # VRStatusPanel, GazeExitButton, StreamingAssetsHelper, …
├── speakers.yaml                   # Speaker layout (edit for your room; no recompile needed)
└── user_study_draft.md             # Research protocol and user study design

---

Requirements

Backend (C++)

Requirement	Notes
CMake ≥ 3.10
C++17 compiler	MinGW-w64 on Windows; GCC/Clang on Linux/macOS
PortAudio	Windows/macOS — built automatically from third_party/portaudio
JACK	Linux — detected via pkg-config; enable with -DUSE_JACK=ON
oscpack	Vendored in third_party/oscpack

Frontend (Unity)

Requirement	Notes
Unity 2022 LTS or later
Meta XR SDK (Core + Interaction)	Via Unity Package Manager or Meta developer portal
Android Build Support module	Target device: Meta Quest 3

---

Building the C++ Backend

Windows (MinGW-w64)

cd cpp
cmake -B build_mingw -G "MinGW Makefiles" -DCMAKE_BUILD_TYPE=Release
cmake --build build_mingw --config Release

The executable and speakers.yaml are copied automatically to cpp/build_mingw/.

Linux (JACK)

cd cpp
cmake -B build -DUSE_JACK=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build

macOS (PortAudio)

cd cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build

---

Running the Backend

# Default: load speakers.yaml from the same directory, play a sine wave
./SoundARray

# Custom speaker layout
./SoundARray /path/to/my_room.yaml

# WAV file as audio source
./SoundARray speakers.yaml recording.wav

The terminal dashboard refreshes every 500 ms. Press Enter to quit.

---

Speaker Layout (speakers.yaml)

Coordinates follow the same convention as the Unity scene: x = right, y = forward (front), z = up. Edit this file to match your physical array; no recompile is needed.

speakers:
  - id: 1
    x: -0.414
    y:  1.000
    z:  0.000
  - id: 2
    x:  0.414
    y:  1.000
    z:  0.000
  # add more speakers …

The project ships with a 13-speaker layout derived from the Couch 204 IEM AllRADecoder configuration.

---

Unity Frontend Setup

1. Open the unity/ folder as a Unity project.
2. Go to Edit → Project Settings → XR Plug-in Management and enable Meta OpenXR.
3. Open Assets/Scenes/MainScene.unity.
4. Select the SpeakerManager GameObject and set Backend IP to your PC's local IP address.
5. Copy speakers.yaml into Assets/StreamingAssets/ so the app can load the layout at runtime.
6. (Optional) Select a SpatialSource prefab and adjust the Source Scale field (default 0.15 m) to match your room scale. This controls both the rendered sphere radius and the grab collider size.
7. Connect your Quest 3 and click File → Build Settings → Android → Build and Run.

Network note: Both devices must be on the same Wi-Fi network. All communication is UDP — the backend uses a single socket on port 7000 for both receiving positions and replying with gains. On Windows, run open_firewall.bat as administrator to open port 7000.

---

Network Ports

Port	Protocol	Direction	Purpose
7000	UDP	Quest → PC	Source position OSC messages (/spatial/source_pos) — Quest sends from port 7002
7002	UDP	PC → Quest	Per-speaker VBAP gains (/spatial/speaker_gains) — PC replies from port 7000 to Quest's port 7002

Both directions share the same UDP socket on the PC (bound to port 7000). Replies are sent from port 7000 so Android's stateful firewall recognises them as return traffic and allows them through.

On Windows, run open_firewall.bat (included) as administrator to open port 7000.

---

Screenshots

Backend

Frontend

Research Context

SonicARray was developed to support a user study comparing 2D video explanation against live AR interaction for teaching VBAP concepts. See user_study_draft.md for the full protocol.

---

Third-Party Libraries

Library	License	Purpose
oscpack	BSD/MIT-like	OSC encoding & decoding
PortAudio	MIT	Cross-platform audio I/O
JACK Audio	LGPL	Low-latency audio on Linux
Meta XR SDK	Meta Platform SDK License	AR passthrough & hand tracking

---

License

This project is released under the MIT License.