Open3DBroadcast
Lightweight, low-latency live streaming of 3D animation and audio in and out of Unreal Engine
Overview
Open3DBroadcast is an Unreal Engine plugin for real-time streaming of skeletal animation, morph target curves, and audio. It is built on Open3DStream, the open protocol and core library that defines the wire format — the plugin is Open3DBroadcast; Open3DStream is what it speaks. The plugin ships both halves of the pipeline, a sender component and a Live Link receiver source, so Unreal can act as either the source or the destination of animation data. Five transports (Sockets TCP, Sockets UDP, NNG, WebRTC, and MoQ, plus an in-process Loopback for testing) carry the same subject data, and every one of them carries audio as well. Delta-optimized updates, channel quantization, and morph-curve filtering keep bandwidth low, while support for hierarchical transform chains, multiple subjects per stream, and animation curves for facial animation make it a flexible backbone for virtual production and live performance pipelines.
Features
Real-Time Skeletal Streaming
Stream transforms, rotations, and scales for full skeletal hierarchies with delta-optimized updates for efficiency and low latency.
Facial Animation Curves
Animation curves for morph targets let you drive facial animation and blend shapes alongside skeletal motion.
Five Transports
Choose the transport that fits your setup — Sockets TCP, Sockets UDP, NNG, WebRTC, or MoQ (Media over QUIC), plus an in-process Loopback for local testing and validation.
Live Link Integration
Adds an Open3D receiver source to Unreal's Live Link system so incoming subjects can drive characters through the standard Live Link workflow.
Broadcast & Receive
Includes both the sender component and the Live Link receiver source, letting Unreal act as either the source or the destination of animation data.
Audio on Every Transport
Audio is a plugin-level capability, not a per-transport add-on: every transport supports it on both the sender and receiver side, carried as Opus or PCM16 through shared encode/decode code.
Bandwidth Controls
Adaptive channel quantization, opt-in predictor-based residual coding, and morph-curve filtering and sanitization cut the bytes sent per frame.
Resilient Frame Delivery
Frame sequencing with receiver-side reorder, dedup, and stale-frame rejection, clock-offset estimation with latency/jitter/loss reporting, and predictive loss concealment for late or lost frames.
Installation
Open3DBroadcast will be available on the Fab Marketplace soon. In the meantime you can install it from a release package: download the latest release from the project's GitHub Releases, extract it, and copy the appropriate `UE_X.X/Plugins/Open3DBroadcast` folder into your project's `Plugins` directory (or extract the whole `UE_X.X` folder to your project root to merge it in). Restart the Unreal Editor, then enable the plugin from Edit → Plugins by searching for "Open3DBroadcast" and checking the box. Note that the WebRTC, MoQ, and NNG transports ship Win64 binaries only — see Platform Support below.
Usage
Open the Live Link window (Window → Virtual Production → Live Link) and add a new source: + Source → Open3D Receiver Source. Configure the connection by entering a URL such as `tcp://localhost:5555` and selecting the matching transport (TCP Client, UDP Server, WebRTC Client, MoQ, and so on), then click Create. Once connected, incoming subjects appear in Live Link and can drive your characters. To hear audio from a sender, enable audio on both ends and add an `O3DRemoteAudioComponent` to an actor in your level — this works with any transport, not just WebRTC. See the plugin's `USER_GUIDE.md` for step-by-step setup and troubleshooting.
Documentation
Core Concepts
Subject-Based Streaming
Each sender broadcasts data for a subject — a named stream of animation data. Multiple subjects can share the same transport channel, and receivers subscribe to all subjects on a channel and expose them as Live Link subjects.
Sender & Receiver
The plugin ships with both halves of the pipeline: an O3D Sender Component captures animation (and optionally audio) from a skeletal mesh actor, while the Open3D Live Link receiver source receives incoming frames. Unreal can act as either end — capture in one instance and receive in another, or the same one.
Open3DBroadcast & Open3DStream
Open3DBroadcast is the plugin and the product. Open3DStream is the underlying protocol and core library it builds on — the wire format the transports carry. (An earlier standalone Open3DStream plugin has been removed; its functionality lives in Open3DBroadcast.)
Transport Modules
Five transports are available, plus an in-process Loopback for testing. All of them carry the same subject data — and all of them carry audio — so you can switch between them without changing the rest of your pipeline.
- •Loopback — in-process transport for local testing and validation, no network required.
- •Sockets TCP — direct peer-to-peer streaming over a reliable, ordered connection, for local networks.
- •Sockets UDP — direct peer-to-peer streaming over an unreliable connection, for low-latency links where occasional loss is acceptable.
- •NNG — lightweight messaging with Pub/Sub, Pair, and Push/Pull patterns, well suited to microservice-style setups. Win64 only.
- •WebRTC — cloud-ready streaming with NAT traversal and encryption, using libdatachannel (P2P) or LiveKit (SFU) backends. Win64 only.
- •MoQ (Media over QUIC) — a full first-class transport with its own sender, receiver, editor customization, audio pipeline, and automation test suite, registered as `MoQ`. It carries data over QUIC / WebTransport through a relay (Cloudflare relays are supported and tested), so it traverses NAT the way WebRTC does. It is implemented in the `Open3DTransportMoQ` module, backed by a Rust FFI library (`moq_ffi`) built on Cloudflare's `moq-rs`. Win64 only.
Quick Start (Loopback)
This local test needs no network. It streams one character's animation to a second character through the in-process loopback transport.
- •Select a skeletal-mesh actor and add an O3D Sender Component. Set Subject Name (e.g. MyCharacter), Capture Rate to 60 Hz, and Transport Name to `Loopback`.
- •Open Window → Live Link, click + Source, and add an Open3D Receiver Source with the same `Loopback` transport and channel.
- •Press Play and confirm the subject appears in Live Link with a green status.
- •Add a Live Link Component to a second skeletal-mesh actor and point it at the same subject to mirror the animation.
Audio Streaming
Audio is a plugin-level capability rather than a feature of one transport. The base sender and receiver interfaces default `SupportsAudio()` to false, and every transport — Loopback, Sockets TCP, Sockets UDP, NNG, WebRTC, and MoQ — overrides it to true on both the sender and the receiver side. Audio is carried as Opus or PCM16 using shared encode/decode code rather than a per-transport implementation: senders run `ProcessCapturedAudio()` → `SendEncodedAudio()`, and receivers run `SetAudioSink()` → `ProcessAudioPayload()`. Enable it with `bEnableAudio` on the sender component and on the receiver source config.
Capture
An O3DSenderAudioCaptureComponent handles the capture side, with settings for `SampleRate`, `NumChannels`, `BitrateKbps`, `DeviceIndex`, `GameGain`, and `MicGain`.
Receive-Side Playback
Add an O3DRemoteAudioComponent to an actor in your level to hear incoming audio. It lives in the Open3DReceiver module and is transport-agnostic — it works with any of the transports above. It manages an internal audio component and procedural sound wave, and mirrors standard AudioComponent settings — volume and pitch, attenuation, submix sends, source effect chains, concurrency, and spatialization. Client-first connection and reconnect flows are supported.
Bandwidth & Curve Controls
The O3DSenderComponent exposes the capture and compression settings as editor properties.
- •`bEnableQuantization`, `QuantizationByteRange`, `QuantizationHalfRange`, `QuantizationDeltaThreshold` — adaptive/variable-bit channel quantization, cutting the bytes sent per frame on the wire.
- •`bEnableResidualCoding`, `ResidualKeyframeIntervalFrames`, `ResidualDeltaThreshold` — predictor-based residual compression. Opt-in and off by default, and available on reliable/ordered transports only.
- •`bEnableCurveFiltering`, `CurveEpsilon`, `CurveDeltaThreshold`, `bClampMorphCurvesToUnit`, `bDropNaNAndInfinity` — morph-curve filtering and sanitization.
- •`CaptureRateHz`, `bAutoStartCapture`, `bAutoCreateTransport`, `bEnableAudio` — capture basics.
Frame Delivery & Latency
The receiver source config exposes the loss-handling and latency settings.
- •`bEnableConcealment`, `StarvationThresholdMs`, `MaxHorizonMs`, `CorrectionWindowMs` — predictive loss concealment, which synthesizes poses when frames are late or lost and then blends back to the real data.
- •`RenderAheadMs` — latency hiding. Opt-in and off by default.
- •`bEnableAudio` — receive-side audio.
Under the Hood
- •Frame sequencing and a transmit clock, with receiver-side reorder, dedup, and stale-frame rejection.
- •Clock-offset estimation that maps sender time to local engine time, with latency, jitter, and loss reporting.
- •Capture and replay of the wire protocol to a `.o3dscap` file, for deterministic testing.
Platform Support
Transport availability is not uniform across platforms. Check this before you build a pipeline around a particular transport.
- •WebRTC — Win64 only. Its build rules fail if the `livekit_ffi` .lib, header, or DLL is absent, and only Win64 artifacts are shipped.
- •MoQ — Win64 only. Only Win64 `moq_ffi` artifacts ship; on other platforms the transport auto-disables.
- •NNG — Win64: the vendored library is Win64.
- •Loopback and Sockets (TCP/UDP) — not gated this way.
Requirements
- •Unreal Engine with the Open3DBroadcast plugin enabled (it provides both the sender and the Live Link receiver source).
- •For WebRTC: a signaling URL and Room, plus a Token when using the LiveKit backend. Win64 only.
- •For MoQ: a reachable relay (Cloudflare relays are supported and tested). Win64 only.
- •Install from a GitHub release package until the Fab Marketplace listing is available.
Known Issues
- Not yet listed on the Fab Marketplace — install from a GitHub release package for now.
- The WebRTC, MoQ, and NNG transports are Win64 only. On other platforms, use Loopback or Sockets (TCP/UDP).
- WebRTC backends require additional signaling configuration (URL/Room, and a Token for LiveKit) as described in the plugin documentation.
- MoQ requires a reachable relay; Cloudflare relays are supported and tested.
- Residual coding (`bEnableResidualCoding`) and render-ahead (`RenderAheadMs`) are opt-in and off by default.
Changelog
- •Real-time skeletal animation streaming over Loopback, Sockets TCP, Sockets UDP, NNG, WebRTC, and MoQ.
- •Animation curve support for morph target / facial animation, with curve filtering and sanitization.
- •MoQ transport (Open3DTransportMoQ) carrying data over QUIC / WebTransport through a relay, backed by the moq_ffi Rust library built on Cloudflare's moq-rs.
- •WebRTC backends: libdatachannel (P2P) and LiveKit (SFU).
- •Audio on every transport, carried as Opus or PCM16 through shared encode/decode code.
- •Receive-side audio playback via O3DRemoteAudioComponent with reconnect support.
- •Channel quantization, opt-in residual coding, frame sequencing with reorder/dedup/stale-frame rejection, clock-offset estimation, and predictive loss concealment.
- •Capture and replay of the wire protocol to .o3dscap files for deterministic testing.
- •Shared Open3DShared module powering both the receiver and the sender.
Support
Open3DBroadcast is open source. For bug reports, questions, or feature requests, open an issue on the GitHub repository (github.com/lifelike-and-believable/Open3DBroadcast) or join the community Discord. You can also use the contact form on this site and include "Open3DBroadcast" in the subject line, along with your Unreal Engine version and a description of the issue.
Contact Support