What is Mixed Reality?

I’ve not created a new category for my website for years – but there’s something that I have spending a lot of time working and I need to start writing about: mixed reality.

Mixed reality is a broad umbrella of technologies which enable a real and virtual worlds to be blended together to create a single environment where physical and digital objects interact with each other.

I remember when I was in school I would imagine what it would be like to have a “heads up display” all the time. I’d imagine information that I’d see about people and surroundings, interactive maps giving me directions, reminders about errands I need to run, all sorts of things.

Thanks to science fiction there was no doubt in my mind that it was an inevitability, and it would happen in my life time – but I didn’t really know how or when it would happen.

Fortaleza Leak

Back in 2012 a very interesting image appeared on the internet. It was from an Xbox presentation which mentioned something called Fortaleza Glasses. Rumours of Project Fortaleza had been going for a while, and I followed the subject with great interest.

Linked to Alex Kipman, the creator of the Kinect, it was thought that these glasses would be Microsoft’s move to leap ahead of virtual reality and into some new world of augmented reality. I hope so much that it was true…

HoloLens

..and I remember how amazed I was when I found out it was true.

HoloLens at Home

Well, over the last six months I have not only tried the HoloLens, but spend many hours using it in both professionally and in my home. I’ve been learning how it works, how to develop software for it, and most importantly; I’ve been learning about what it means for the future of software.

This is a whole new kind of user experience, and I believe that mixed reality is the future of how humans will interact with software.

Actiongram

But mixed reality is about experience, not technology. When I say that I believe it is the future of how humans will interact with software – I am not saying that everyone will wear headsets. It won’t be long until there are other ways to project digital items into the physical world.

It breaks down the walls between physical and virtual reality – today this means vision and sound, but the concept of mixed reality goes beyond those senses. As a software architect it excites me to think about the user experiences which everyone in the augmented/virtual/mixed reality community is going to help shape, and I’m to write about what I learn as I go.

Now that I have experienced mixed reality, there’s no going back.

It’s real, it’s getting better all the time, and it’s here to stay.

Microsoft Future Vision 2015 – Redux

Back in April I wrote about the latest ‘Future Vision’ video from Microsoft and I was very pleased to see this video come back onto my radar.

Dave Jones and Anton Andrews

Larry Larsen at Channel 9 posted an interview with Dave Jones and Anton Andrews – a couple of the guys who worked on creating this fantastic (yet realistic) vision of the future.

Dave and Anton give us some context on the decisions and thoughts behind the various ideas, including a few extra details about my favourite concepts – the flexible digital notebook and the wrist device.

Future Vision

It’s well worth watching if you are interested in these forward looking concepts, but make sure you watch the Future Vision video first!

Something I had missed when watching the original video was the idea that the system itself had noticed Kat had gone into a flow state. The suggestion here is that the various devices would work together, taking sensor information like heart rate and galvanic skin response, to automatically switch into this mode.

Automatic Mode Switching

The system would then automatically block out any unwanted distractions like notifications and set her communication status to do not disturb.

A nice touch is that the earpiece also switches to red to show other human beings. Very cool.

In The Flow State

As a software architect and technology enthusiast I find myself bombarded with huge amounts of information – communication requests, push notifications, reminders, and much more.

Getting myself into the flow state is hard enough (music helps) but keeping it can be even more difficult. The idea of having the system automatically sense this and move things into a ‘do not disturb’ state is very attractive.

An important part of these Future Vision videos is that they are realistic, and all of this could be done today:

  • Notice high amounts of use in productivity apps (Office, Visual Studio)
  • Sense physical changes in the user (Band)
  • Set ‘Quiet Hours’ for notifications (Windows)
  • Change status to ‘Do Not Disturb’ (Skype)
  • Handle exceptions that can break through (Cortana)

Microsoft controls each one of those components, but the fact is that the most futuristic part of these videos is not the hardware or the software – it’s the integration.

Considering Microsoft reaffirmation as the productivity company, it’s probably something they should try to integrate in order to achieve their goal.

Microsoft Future Vision 2015

I’ve always been inspired by the Microsoft ‘Future Vision’ videos which depict a not-too-distant vision of productivity. This year’s entry has not been a disappointment, with a number of interesting UI concepts explored.

The best thing to do is watch the video above to see them all, but I’ve picked four of my favourites below.

Augmented Reality + Tactile Controls

I’m not really what to call this, so I’m just going to call it a ‘holographic puck’. In this instance, a round hardware device can be rotated to make selections on a holographic UI which has been augmented over the top of Kat’s vision.

By mixing the feel of tactile controls with the holographic interfaces you can avoid the strange experience of ‘tapping thin air’ while still providing the users with the infinite possibilities of augmented reality.

Holographic puck

I really like this concept, and it’s not too unrealistic considering the holographic technology coming in Windows 10. Later in the video you see the same hardware device used to transfer the data collected in the first scene.

Flexible Digital Notebooks

Opening the flexible computer

My favourite concept from the whole video is shown when our hero attends a café. The tea selection is shown on this flexible display, and when Kat opens it all of her personal stuff is automatically available to her.

While I think the folding doesn’t look as amazing as it could be (give me a proper notebook style folding, please) – it is a great example of the kind of computers we will be using in the future, and something I really want.

Flexible computer

Being a massive notebook and stationery nerd, I really love the idea of having a flexible notebook computer like this. I hope it happens in the not-too-distant future. The Surface line of computers already has rich inking capability, so it’s only going to get better over time.

Wearable Computing Devices

Throughout the video only one computer looks like it belongs exclusively Kat. The screen on her wrist is probably the equivalent of the smartphone today, being a general purpose communication and computing device.

Wrist computer

This is quite a way off the current Microsoft Band, but the technology sector is certainly going this direction. My Band has already helped give me the motivation to be fitter and healthier, and while we don’t really see much in the way of health statistics in this video, it can certainly be inferred from the way things are going.

Large Table-like Displays

Large table

I also love this large table computer concept. When Kat needs to get some real work done, she just uses her wearable computer to hand off to a bigger computer in a shared workspace.

I’m sure this kind of keyboard-free interaction will be best suited to a world where voice interaction has been perfected. Though I’m sure a software keyboard could be provided. You can even see Kat use a Bluetooth headset (Bragi Dash?) to work with Cortana-like assistant in the top left of the UI.

Captivated by Her

When reading an article about how design is shaping the new Microsoft, I stumbled upon a video by Kat Holmes from last year.

In this video, Kat talks about how conversations power innovation and new experiences. She goes on to discuss how the movie Her both validated and shaped Microsoft’s thinking around their Cortana digital assistant.

Like Kat I have found the movie Her to have a fascinating look at the future of human and computer interaction, primarily through verbal communication and relationship building. While the movie certainly takes the relationship part of the equation to the extreme, I firmly believe that this kind of trust between digital agents and ourselves will eventually enable a new kind of human excellence.

her

Microsoft also collaborated with Vice’s Motherboard to produce a couple of short documentary videos titled Captivated by Her. They’re well worth watching if you’re interested in how human emotion inspires and shapes technology.

Windows 8.1 – Ultrawide Multitasking

When I recently decided to set up a new workstation at home, I had a look at the available monitors. Without really thinking about it I assumed I’d just get a standard 16:9 monitor, but then I stumbled upon an article about an LG all in one PC with a crazy wide 21:9 screen. This really sparked my interest. Not in the PC itself – but how these ultra wide screens work with Windows 8.1.

Being a fan of Dell monitors, I decided to invest in a 29 inch Dell Ultrawide – so far it has done everything I’ve wanted, and I’ve been very happy with it.

Dell Ultrawide

Windows 8.1’s snap feature allows you to use up to four different applications at the same time with one of these Ultrawide screens. This extra horizontal space has drastically changed how I use Windows at home.

All of these screenshots are real examples of how I use Windows, and were taken over a few weeks of actual use. Basic tasks like email and note taking aren’t included, as I didn’t want to have to censor the content.


Writing a Blog Post

Writing a Blog Post

  1. Xbox Music
  2. Internet Explorer
  3. Internet Explorer

Arranging Tasks & Calendar Appointments

Arranging Tasks & Calendar Appointments

  1. Mail
  2. Calendar
  3. OneNote
  4. Xbox Music

Finding New Music

Finding New Music

  1. Internet Explorer
  2. Xbox Music

Looking for a Computer New Bag

Looking for Computer Bag Ideas

  1. Flipboard
  2. OneNote

Watching Windows Weekly

Watching Windows Weekly

  1. Twitter
  2. Twit.tv

ultrawide-10

Exploring the World

  1. Bing Maps
  2. Star Chart