What is Mixed Reality?

I’ve not created a new category for my website for years – but there’s something that I have spending a lot of time working and I need to start writing about: mixed reality.

Mixed reality is a broad umbrella of technologies which enable a real and virtual worlds to be blended together to create a single environment where physical and digital objects interact with each other.

I remember when I was in school I would imagine what it would be like to have a “heads up display” all the time. I’d imagine information that I’d see about people and surroundings, interactive maps giving me directions, reminders about errands I need to run, all sorts of things.

Thanks to science fiction there was no doubt in my mind that it was an inevitability, and it would happen in my life time – but I didn’t really know how or when it would happen.

Fortaleza Leak

Back in 2012 a very interesting image appeared on the internet. It was from an Xbox presentation which mentioned something called Fortaleza Glasses. Rumours of Project Fortaleza had been going for a while, and I followed the subject with great interest.

Linked to Alex Kipman, the creator of the Kinect, it was thought that these glasses would be Microsoft’s move to leap ahead of virtual reality and into some new world of augmented reality. I hope so much that it was true…

HoloLens

..and I remember how amazed I was when I found out it was true.

HoloLens at Home

Well, over the last six months I have not only tried the HoloLens, but spend many hours using it in both professionally and in my home. I’ve been learning how it works, how to develop software for it, and most importantly; I’ve been learning about what it means for the future of software.

This is a whole new kind of user experience, and I believe that mixed reality is the future of how humans will interact with software.

Actiongram

But mixed reality is about experience, not technology. When I say that I believe it is the future of how humans will interact with software – I am not saying that everyone will wear headsets. It won’t be long until there are other ways to project digital items into the physical world.

It breaks down the walls between physical and virtual reality – today this means vision and sound, but the concept of mixed reality goes beyond those senses. As a software architect it excites me to think about the user experiences which everyone in the augmented/virtual/mixed reality community is going to help shape, and I’m to write about what I learn as I go.

Now that I have experienced mixed reality, there’s no going back.

It’s real, it’s getting better all the time, and it’s here to stay.

Microsoft Future Vision 2015 – Redux

Back in April I wrote about the latest ‘Future Vision’ video from Microsoft and I was very pleased to see this video come back onto my radar.

Dave Jones and Anton Andrews

Larry Larsen at Channel 9 posted an interview with Dave Jones and Anton Andrews – a couple of the guys who worked on creating this fantastic (yet realistic) vision of the future.

Dave and Anton give us some context on the decisions and thoughts behind the various ideas, including a few extra details about my favourite concepts – the flexible digital notebook and the wrist device.

Future Vision

It’s well worth watching if you are interested in these forward looking concepts, but make sure you watch the Future Vision video first!

Something I had missed when watching the original video was the idea that the system itself had noticed Kat had gone into a flow state. The suggestion here is that the various devices would work together, taking sensor information like heart rate and galvanic skin response, to automatically switch into this mode.

Automatic Mode Switching

The system would then automatically block out any unwanted distractions like notifications and set her communication status to do not disturb.

A nice touch is that the earpiece also switches to red to show other human beings. Very cool.

In The Flow State

As a software architect and technology enthusiast I find myself bombarded with huge amounts of information – communication requests, push notifications, reminders, and much more.

Getting myself into the flow state is hard enough (music helps) but keeping it can be even more difficult. The idea of having the system automatically sense this and move things into a ‘do not disturb’ state is very attractive.

An important part of these Future Vision videos is that they are realistic, and all of this could be done today:

  • Notice high amounts of use in productivity apps (Office, Visual Studio)
  • Sense physical changes in the user (Band)
  • Set ‘Quiet Hours’ for notifications (Windows)
  • Change status to ‘Do Not Disturb’ (Skype)
  • Handle exceptions that can break through (Cortana)

Microsoft controls each one of those components, but the fact is that the most futuristic part of these videos is not the hardware or the software – it’s the integration.

Considering Microsoft reaffirmation as the productivity company, it’s probably something they should try to integrate in order to achieve their goal.

Microsoft Future Vision 2015

I’ve always been inspired by the Microsoft ‘Future Vision’ videos which depict a not-too-distant vision of productivity. This year’s entry has not been a disappointment, with a number of interesting UI concepts explored.

The best thing to do is watch the video above to see them all, but I’ve picked four of my favourites below.

Augmented Reality + Tactile Controls

I’m not really what to call this, so I’m just going to call it a ‘holographic puck’. In this instance, a round hardware device can be rotated to make selections on a holographic UI which has been augmented over the top of Kat’s vision.

By mixing the feel of tactile controls with the holographic interfaces you can avoid the strange experience of ‘tapping thin air’ while still providing the users with the infinite possibilities of augmented reality.

Holographic puck

I really like this concept, and it’s not too unrealistic considering the holographic technology coming in Windows 10. Later in the video you see the same hardware device used to transfer the data collected in the first scene.

Flexible Digital Notebooks

Opening the flexible computer

My favourite concept from the whole video is shown when our hero attends a café. The tea selection is shown on this flexible display, and when Kat opens it all of her personal stuff is automatically available to her.

While I think the folding doesn’t look as amazing as it could be (give me a proper notebook style folding, please) – it is a great example of the kind of computers we will be using in the future, and something I really want.

Flexible computer

Being a massive notebook and stationery nerd, I really love the idea of having a flexible notebook computer like this. I hope it happens in the not-too-distant future. The Surface line of computers already has rich inking capability, so it’s only going to get better over time.

Wearable Computing Devices

Throughout the video only one computer looks like it belongs exclusively Kat. The screen on her wrist is probably the equivalent of the smartphone today, being a general purpose communication and computing device.

Wrist computer

This is quite a way off the current Microsoft Band, but the technology sector is certainly going this direction. My Band has already helped give me the motivation to be fitter and healthier, and while we don’t really see much in the way of health statistics in this video, it can certainly be inferred from the way things are going.

Large Table-like Displays

Large table

I also love this large table computer concept. When Kat needs to get some real work done, she just uses her wearable computer to hand off to a bigger computer in a shared workspace.

I’m sure this kind of keyboard-free interaction will be best suited to a world where voice interaction has been perfected. Though I’m sure a software keyboard could be provided. You can even see Kat use a Bluetooth headset (Bragi Dash?) to work with Cortana-like assistant in the top left of the UI.