01
3D-JOYSTICK
A spatial joystick that lives inside the immersive scene. Push it through 3D space to command motion in three axes.
[ OMEN ]
MMXXVI
OPEN SOURCE
VISIONOS
FIRST-PERSON TELEOPERATION FOR APPLE VISION PRO.
An open-source visionOS client and Python streaming server. WebRTC video from an Intel RealSense D435i to Apple Vision Pro, with a bidirectional data channel for control. Two interaction protocols — 3D-Joystick and Custom-Gesture — compared in a within-subjects user study.
[ 02 / OVERVIEW ]
A SPATIAL
TELEOPERATION
COCKPIT.
An Intel RealSense D435i mounted on the robot streams color, depth, and infrared via a Python WebRTC server. The visionOS client renders the remote track in an immersive scene and transmits velocity commands back over a bidirectional data channel. A simulation mode runs the same input pipeline against a virtual robot for development without hardware.
- 01 REAL-TIME VIDEO over WebRTC.
- 02 TWO INPUT PROTOCOLS, switchable from the landing screen.
- 03 SIMULATION MODE for testing motion without the robot.
- 04 MIT-LICENSED. Implementation and study materials open.
[ 03 / CONTROL ]
TWO PROTOCOLS.
ONE STUDY.
02
CUSTOM-GESTURE
Hand pose drives the robot. Pinches, drags, and orientation map straight to velocity commands sent over the data channel.
03
BRING
YOUR OWN
Each protocol is a swappable module. Add one by conforming to the input view-model contract and registering it in the landing picker.
[ 04 / HOW IT WORKS ]
THREE PIECES.
ONE WIRE.
01
ROBOT
A Python server reads from the RealSense D435i, encodes the selected stream, and opens a WebRTC peer connection. Falls back to a regular webcam if no RealSense is attached.
02
WIRE
Video flows one way. Commands flow the other.
03
HEADSET
The visionOS client renders the remote track in an immersive scene, captures input from the chosen protocol, and pushes velocity commands back to the robot in real time.
[ 05 / SIMULATION ]
RUN THE LOOP
WITHOUT THE ROBOT.
A simulation mode runs the same input pipeline against a virtual robot in an immersive scene. Useful for protocol development, gesture validation, and pilot tests when the robot isn't available.
- LANDINGPICK INTERACTION
- MODESIM · ROBOT CONTROL
- SCENEIMMERSIVE SPACE
- INPUT3D-JOYSTICK · CUSTOM-GESTURE
[ 06 / RESEARCH ]
PROTOCOLS
ON TRIAL.
Two protocols. One fixed course. A stopwatch. Locomotion is built around a within-subjects user study that pits 3D-Joystick against Custom-Gesture on the same navigation task — measuring completion time, errors, workload, awareness, sickness, and preference for every participant.
[ THE MATCHUP ]
PROTOCOL J
3D-JOYSTICK
Virtual joystick handles in the immersive scene. Push, pull, twist.
PROTOCOL G
CUSTOM-GESTURE
Middle-finger pinch translation, custom rotation gesture. Hands free of UI.
[ WHAT WE MEASURE ]
01
COMPLETION TIME
From "Go" to crossing the finish line. Seconds, 0.1s precision.
02
NAVIGATION ERRORS
Count of times the robot leaves the marked tape line.
03
NASA-TLX
Cognitive workload across six dimensions, 0–100.
04
SART
Situation Awareness Rating Technique, 0–100.
05
SUS
System Usability Scale, 0–100.
06
SSQ
Simulator Sickness Questionnaire, 0–235.
07
FORCED-CHOICE PREFERENCE
"Which protocol did you prefer?" — binary J or G.
[ HOW WE KEEP IT HONEST ]
- WITHIN-SUBJECTS Every participant runs both protocols. Personal differences cancel out.
- COUNTERBALANCED Half do J→G, half do G→J. Order can't sneak into the result.
- 3 TRIALS / PROTOCOL Median completion time, summed errors. Single-trial flukes get smoothed.
- FROZEN COMMIT Both protocols pinned to one git commit for the whole study. No mid-flight patches.
[ THE RESULT ]
N = 15 participants. Both protocols frozen at one commit. Primary tests via paired t-test; Wilcoxon as backup where normality failed; binomial test on preference.
73%
CHOSE GESTURE
−16%
FASTER WITH G
1
METRIC WON BY J
METRIC
J
G
Δ
WINNER
p
Completion time (s)
42.3
35.7
−6.6
G
.018
Navigation errors
4.2
2.1
−2.1
G
.022
NASA-TLX (workload, ↓ better)
58
51
−7
G
.041
SART (awareness, ↑ better)
68
76
+8
G
.029
SUS (usability, ↑ better)
72
81
+9
G
.012
SSQ (sickness, ↓ better)
38
51
+13
J
.034
Forced-choice preference
4
11
+7
G
.059
SUMMARY — Custom-Gesture significantly outperforms 3D-Joystick on completion time, navigation errors, workload, awareness, usability, and preference. 3D-Joystick yields lower simulator sickness scores. Recommendation: adopt Custom-Gesture as the default; monitor SSQ in extended sessions.
[ 07 / GET STARTED ]
TWO HALVES.
BOTH SMALL.
01
THE APP
The Vision Pro client will be on the App Store at launch. Until then, build from
source with Xcode against main.
02
THE SERVER
Plug in a RealSense D435i, install the Python deps, and run
python3 Server/app.py. The server prints a network URL — paste it into
the app and connect.
[ 08 / TEAM ]
THE
TEAM.
Six contributors across design and development. Based out of Napoli.
[ 09 / OPEN SOURCE ]
OPEN.
MEASURED.
REPRODUCIBLE.
MIT-licensed. Implementation, study materials, and analysis open in the repository. Issues, PRs, and forks for new robots, cameras, and protocols welcome.