Instead of just rendering a flat desktop mirror onto an in-game block, this setup actually bridges the Wayland protocol into the game engine. You can launch real Linux apps (terminals, Firefox, GIMP, etc.), resize them, move them around in 3D voxel space, and use mouse/keyboard input dynamically mapped to the 3D surface coordinates. It even manages a fully functional Picture-in-Picture mode and runs high-frame-rate apps like Osu! smoothly.
While it’s easy to look at this as just a funny “because we can” flex, it actually got me thinking a lot about the future of 3D window management.
Projects like this feel like an awesome, accidental proof-of-concept for things like Stardust XR (the Linux display server designed for VR/AR environments). We are slowly moving away from the assumption that a desktop environment has to be a 2D grid of pixels bound by physical plastic bezels. Seeing a Linux ecosystem adapt natively to a voxel world shows how flexible Wayland’s architecture can be when you start decoupling buffers from standard monitors.
Curious to hear the group’s thoughts:
Has anyone experimented with Stardust XR or other spatial environments on Linux?
What are your thoughts on input latency and ergonomics when we inevitably move window management into 3D environments?
Is anyone else just purely impressed by the nested input routing required to make this work inside a Java/LWJGL game?;
Hi @VioletLain, this is a pretty cool demo, thanks for sharing!
I haven’t had experience with VR/AR or other spacial environments on Linux. I think input latency and ergonomics will matter a lot based on the specific use case, whether it’s more our presenting info in context in the 3D space versus interacting with it and what the actual impact is (try coding with latency - hard to buffer this, but streaming into the context you need, then that seems very compelling).
I would think there’s a a lot of useful use cases for this capability to allow developers to create reusable VR/AR components. While this demo in the video was for Minecraft (gaming always seems a great place to experiment), I can see this be used in other fields, from education, engineering, architecture, medical, manufacturing, etc.
While I don’t know the full details for what’s involved under the tech stack in this particular case, I’m impressed.
I do not think moving into 3D environments is inevitable, so I have zero thoughts on the matter.
No, LLMs and any user putting effort and time can accomplish anything at this point, so the bar for impressing me is dramatically higher than justifying usage of proprietary software for novel use cases.