Virtual reality: The future of chat in VR

Have you ever wondered what it might truly feel like to roam dungeons and fire arrows in a mythical world? Or perhaps you’re more practically minded and you’d rather understand what goes on behind the scenes while operating heavy – and potentially dangerous – machinery. Or maybe you are looking for a new kind of workout and want to get in on the action of blasting asteroids in outer space!

You can experience all these with Virtual Reality (VR). VR, from its inception through today, has both utopian and practical aspirations. First, the utopic vision is the ability to create an accessible world that allows people to freely engage with knowledge and enjoy new experiences, even fantasy experiences such as fighting mythical creatures in worlds that don’t exist. Indeed, today’s consumer VR headsets focus on gaming: popular games like Beat Saber, Cloudlands VR Minigolf, and The Elder Scrolls V: Skyrim VR bring a new level of immersion into our favorite playable worlds. Meta’s Horizon platform is building an explorable, immersive world where your avatar can attend events like concerts and comedy shows, shop for digital goods, and even own virtual real estate.

Beyond the consumer space, though, virtual reality is changing commercial industries. High-skill jobs are turning to VR to make it easier, for example, to train for highly specialized medical procedures, flying a plane for a trainee pilot, or to experience new architectural designs in virtual environments before committing to building. New marketing applications in industries like tourism and real estate are allowing vendors to let consumers try before they buy in a whole new immersive way.

In this guide, we’ll introduce you to virtual reality—where it has been and where it is today—and share our vision for how 2D chat interfaces will become an integral part of an interactive 3D world. We’ll also discover how you can build such a 2D chat interface and what it looks like in VR.

Building virtual reality into the future we’ve always imagined

According to TechTarget, virtual reality is “a simulated 3D environment that enables users to explore and interact with a virtual surrounding in a way that approximates reality, as it is perceived through the users’ senses.” Virtual reality seems to be everywhere right now, with Meta (formerly Facebook) going all-in on VR with the metaverse, a sprawling virtual reality platform designed to bring gaming, shopping, social media, and more to a simulated, interactive 3D environment.

There are varying degrees of immersion for VR experiences, including:

• Non-immersive: A 3D simulated environment accessible through a screen. It may have sound, but further sense-based interaction is limited. Input is controlled through mouse and keyboard or touchscreens. Examples include room design simulators.
• Semi-immersive: A 3D simulated environment accessed through glasses or a headset, much like the Meta Quest. Though it is immersive visually, it doesn’t include other senses or movement.
• Fully immersive: This is the holy grail of virtual reality. Users are submerged into a 3D world with sight, sound, touch, and sometimes smell—providing input on what’s happening in that virtual environment. You’re more likely to come across this in fictional representations like Ready Player One or The Matrix. Gaming, healthcare, and other industries are working to make this technology a reality.

A brief history of virtual reality

For decades, artists and scientists have been working—sometimes together—on ways to immerse viewers into fabricated worlds. Most iterations have been limited to non- or semi-immersive VR. Let’s take a look at a few notable attempts at virtual reality for the masses before briefly reviewing the state of the technology today.

Panoramic paintings

Before cameras, painters were recreating the world in exquisite detail. 18th and 19th-century panoramic painters created enormous, 360-degree vantages of cities and nature to immerse viewers in the world. Often, viewers were asked to stand at a place and angle calculated to have the maximum immersive effect of the circular panorama.

Stereoscopes

Those of us who grew up with View-Masters are familiar with stereoscopes. First invented in the early 1800s, stereoscopes place two 2D images in a pair of what look like binoculars to trick the brain into creating a 3D view of the image.

“Pygmalion’s Spectacles” and other sci-fi stories predict the future

Stanley G. Weinbaum’s 1930s story, “Pygmalion’s Spectacles” portrayed a vision of goggles that gave wearers smell, taste, touch and sight into a different world, closely predicting today’s VR headsets.

VR head-mounted displays

In 1960, the first head-mounted VR device, or HMD (head-mounted display), gave users a hands-free, immersive visual and aural experience.

Sword of Damocles

In 1968, the first virtual reality/artificial reality head-mounted display, called the Sword of Damocles, was connected to a computer to project wireframe images of an interactive environment into the headset.

Artificial reality is born

With the advent of computers helping to create new visual, virtual environments, the term “artificial reality” was coined in 1969 by Myron Krueger.

Krueger’s VIDEOPLACE

Krueger’s work in artificial reality led directly to what is considered the first interactive VR system, the VIDEOPLACE, in 1969.

The McDonnell-Douglas VITAL helmet

Building on decades of work to create flight training simulators, McDonnell-Douglas’ VITAL 1979 helmet provided the first VR headset for pilots for use outside the lab.

Virtual reality is born

Until 1987, virtual reality wasn’t a term people used in the way we do today. Jaron Lanier, founder of the visual programming lab (VPL), is known for coining the term for the devices and area of study we now know as VR.

SEGA VR glasses (conceptual)

In 1993, Sega announced a VR headset for the SEGA Genesis, marking the first prototype of an at-home, consumer VR product. The device never made it to production, though.

The Matrix

The Matrix trilogy of movies, which began in 1999, brought the concept of fully immersive virtual reality into the mainstream and set the stage for what people expect from the technology.

Google Street View

2D maps got a huge upgrade in 2007 when Google created a 360-degree, immersive rendering of the world that we could stroll through from our computers and phones.

Oculus Kickstarter

Oculus, a VR company, began a Kickstarter campaign in 2012 to bring affordable VR headsets to the masses. The technology was bought in 2014 by Facebook, setting the stage for the emergence of Meta and the Metaverse less than a decade later.

Chat in VR today: Gaming, business, and more

As virtual reality finds more applications and becomes part of our everyday working or leisure lives, the reality is that text-based communication will become more necessary. For example, text-based chat might be a key way for any VR headset company to help users acclimate to the virtual world. New users just being born into the new world will want to chat with other users to ask for help. In gaming, chat can be an important way for gamers to collaborate, build a strategy, and execute on opportunities. During or before virtual concerts, sports, classes, or other virtual events, users can chat with fellow attendees to share reactions or thoughts. When sharing a physical space with people you don’t know well, chat becomes even more necessary to avoid voice commands and maintain privacy. Even in a virtual business use case, chat can play a critical role in networking and making new connections. That’s why the lack of text-based chat in VR is such a missed opportunity.

To make chat part of the VR experience, one has to reimagine the elements that make up today’s 2D chat interfaces for the interactive 3D world.

These elements include:

• The keyboard – Users need a way to type messages. This may be through a 3D rendering of a keyboard that floats in front of their field of vision, or it may be from their own physical keyboard rendered into the 3D world in real-time, allowing people to see what they’re typing on a 3D representation of their actual keyboard.
• The user interface – VR chat interfaces need to display messages in 3D without being obtrusive. Users also need to be able to hide the interfaces for a richer experience in the context of the use case.
• Group chats – Virtual reality is poised to bring huge communities together, and group chats will be vital ways to connect people. Feature-rich chat functionality will therefore need to scale, without breaking, to support thousands of users.
• Chat moderation – Keeping users safe is just as important as giving them a platform to connect with others. Any VR chat needs the ability to moderate conversations and quell toxic discussions.
• Chatbots – With so much to learn and do in VR, chatbots can support users without bogging down human resources. This can be especially useful in customer support use cases or areas in which human intervention is not required.

Translating these 2D elements into the 3D world will take some adaptation on the part of users as well as developers. For users, how they interact with chat interfaces will depend on the hardware they’re using and their own capabilities. Creating a 3D chat-based interface in virtual reality will require users to interact with their controllers to point at a floating 3D keyboard. That of course requires a level of dexterity that not everyone will have.

Developers will need to build their chat interfaces for a variety of headsets and inputs. The difference between using a wobbly controller to point at a floating keyboard and a physical keyboard rendered in 3D in real time is massive – and developers may need to build for both of these. Whichever interface developers create, its accessibility and ease of use will be critical to help people to get over the learning curve. In addition, it will be important for developers to make text-based chat in VR capable of handling legacy apps—like Slack and email—while also opening up new uses unique to the virtual world.

Finally, think back to when we first were introduced to the original iPhone and how strange typing on that little virtual keyboard felt compared to the tactile input of a BlackBerry. Developers will need to create assistive and predictive aids to help speed adoption and improve chat experiences for everyone—but so much of that depends on the hardware people are using.

Which VR hardware is available today?

While we’re not all going to work, shopping and role-playing as our favorite heroes in fully immersive VR headsets right now, the VR market is growing. Users have multiple hardware options; Meta and its Oculus headsets get a lot of attention, but other companies, like Sony, Apple, Google and Microsoft, are all working on variations of VR glasses and headsets. Advances in input and feedback technology, like HTC’s wrist tracker, make experiencing the virtual world through our other senses even easier and more engaging.

Today’s VR hardware can be broken into three different categories: standalone mobile headsets, console-based headsets, and PC-based headsets. The requirement of consoles and PCs can drive the price, but these provide much better graphics and peripheral support. The addition of peripherals like controllers and speakers help facilitate further immersion.

Standalone VR Headsets

1. Oculus Quest 2 – The lightweight entry point for many people; it requires a Facebook account.

Console VR Headsets

1. PlayStation VR – An addition to unlock virtual reality for the PS4 or PS5 if you already own those consoles.

PC VR Headsets

1. HTC Vive Pro 2 – A high-resolution, beautiful gaming VR rig with a price tag nearly 5 times higher than that of the Oculus Quest 2—and that’s before including the high-end PC.
2. Oculus Rift S – An older, but more affordable, PC-connected VR experience.
3. Valve Index – A high-end gaming VR rig at a more affordable price than some similar headsets with great controllers for tracking.

High image resolution and smooth controller tracking, if available, will be important things for developers to consider when building out a 3D chat interface for VR. Greater resolution and input accuracy make virtual keyboards much more palatable experiences.

When we’re all set up with the optimal hardware, it’s time to think about how to build chat for VR.

How to build a 2D chat interface for a virtual environment

We’ve seen how chat interfaces can be useful for facilitating communication in VR. So how can developers build chat for virtual reality? Let’s take an example of a 3D keyboard for a tech conference in VR. In this conference, attendees have a fully featured 3D chat through which they can connect, converse, and congregate. The chat exists within the virtual world, with no need to use 3rd-party platforms such as Slack or WhatsApp. Each conference topic e.g. frontend, backend, UX design, etc. has its own group in which like-minded individuals may chat. Participants can even teleport to the location of other people straight from a chat conversation.

So how can you get started building this? The VR tech conference is built with an extended version of Mozilla Hubs. Mozilla Hubs is a great foundation for building easily distributed VR experiences. Hubs, relatively easy to customize and extend, runs in a browser and is built using WebVR and the A-frame library, so we recommend starting your journey with the excellent A-Frame documentation.

Please note that Hubs is in the process of slimming down its entity component system and moving away from a-frame. A-frame remains a great open source project to learn and work with, but note it will not no longer be a dependency for Hubs in the future.

The custom Hubs experience has the following additions:

1. 3D compatible keyboard: The 3D compatible keyboard is how we input text. There is some prior art for adding 3D JavaScript keyboards to VR, so we can build on that. The keyboard is a Hubs port of this project, which renders a functional 3D keyboard that works in VR. If Hubs requires a generic 3D keyboard component in the future, this code is a great starting point. This is what the rendered keyboard looks like:
2. 3D UI for displaying messages: We need to seamlessly render messages in 3D. Crucially, we need a way to hide messages when we are not chatting.
3. Group chat: Hubs currently comes with a 2D chat that allows all users to message via an input in a web browser. For this demo, we need fine-grained chat channels, one for each track at the conference. Sendbird, a chat API and SDK, is going to be our tool of choice. With Sendbird, we don’t need to build the nuts and bolts of a chat system. Extending the Hubs backend to support additional chat features could be another way to achieve this.
4. Chat moderation: With many user chats occurring simultaneously, we will need a way to moderate conversations and prevent toxicity. Sendbird has an admin dashboard with built-in functionality that allows you to block a user, so we get this for free.
5. Chatbot: A chat bot that allows teleportation through a simple text command.

The images below depict chat for the tech conference in VR. To experience it in action, we highly recommend exploring the VR demo. For the best experience in the demo, use a VR headset like Oculus.

Note that when you are building a new 3D component, it may be easier to take a look at a similar existing component. For example, when trying to build a key for a 3D keyboard, you might realize that an existing mute mic button has the same interaction functionality needed.

One final thing to consider is that you may expect a reasonably steep learning curve; coming from building traditional 2D web apps, building for the 3D world is hard. Expect to spend a lot of time calculating where to position elements from scratch. Hubs is built with JavaScript so many devs will know the language, but still expect to invest time learning new skills.

Finding a group chat

Seeing what others are chatting about

Typing a message

Chatbot teleportation

For those excited about bringing chat to virtual reality, open-source VR projects like Mozilla Hubs, as mentioned earlier, and the developer communities on Reddit and Discord are great places to start. Check out the code for the entire live demo and don’t forget to explore it in VR!

The future of VR, one message at a time

As more people plug into the Metaverse, the ability to communicate via private or group text-based chats without leaving the virtual world will make it easier for people to adopt the technology as part of their play and work. In the next few years, advances like physical keyboards tied into 3D spaces within VR setups will make the technology more feasible as an everyday working environment.

Virtual reality is a growing industry that will likely explode in its use and applications over the next few years as people and businesses look for more cost-efficient, sustainable – and fun! – ways to bring people together. While text-based chat hasn’t been the focus of VR headset manufacturers or developers, the need for effective chat interfaces will be a prerequisite to unlock general adoption and business-friendly use cases.

At Sendbird, we’ve helped companies from all industries to build in-app chat with easy-to-use APIs and SDKs, and we’ve seen how powerful a great chat experience can be to help build a community. We’ve already worked with gaming companies like Krafton, a game developer behind hits like PlayerUnknown Battlegrounds (PUBG), to connect thousands of players across the globe through Sendbird Chat. From a business growth perspective, push notifications encouraging people to take a desired action help to deliver relevant offers to your target audience.

We’re working hard at Sendbird to help shape the reality of chat for a variety of applications. Stay connected with us on social media and keep an eye on our blog for more on this subject!

In the meantime, sign up for a free trial of Sendbird or for our Developer plan and get hands-on experience with how easy it is to bring engaging conversations to your app with chat, voice, video, and live streaming solutions.