Introducing Motion Controllers for Windows Mixed Reality

Microsoft have been using gestures like the air tap and bloom when interacting with the HoloLens, but when you’re in a fully occluded VR world, you need to be able to interact with your world without seeing your hands. This is done by using motion controllers, as seen by companies like HTC and Oculus.

Now it’s Microsoft’s turn to show how users are going to interact with the experience provided by Windows Mixed Reality. They are coming later this year and I can’t wait to get my hands on them.

Microsoft have a long history of creating new types of hardware to provide a consistent experience across third party devices. The original Microsoft Mouse was released alongside Windows Word in order to provide a way for users to move their cursor. Like the mouse it is in Microsoft’s interest to allow third parties to create their own motion controllers, but I expect that they’ll all be compatible and have the same technology inside.

Interestingly it seems that Microsoft has decided that they do not need to make these controllers work with the HoloLens – at least not with the existing version. The HoloLens hasn’t seen much in the way of software updates recently, so I won’t rule out the chances of them adding it in the future, but I get the feeling that Microsoft want people to use their hands for that device.

In my opinion we need these controllers for the HoloLens too – the Clicker is not enough… but that’s a story for another time. How do these new controllers work?

Optical Tracking

All of the Windows Mixed Reality headsets provide inside-out tracking – this means that each device will have the sensors required to track the world in the head-mounted display without any additional sensors or tracking devices around the room.

The new Motion Controllers take advantage of this technology to provide six degrees of freedom without any extra wires or mess. This also reduces the complexity of the controller, allowing for a complete tracking solution without being too expensive.

It’s worth mentioning that I assume there is some additional motion tracking in the controllers (so you can put it behind your back, for example), but the truly accurate measurements will be done optically when the headset have a line of sight to the lights around the controller.

Buttons

So what kinds of controls can we expect to have?

  • Windows buttons
  • Menu buttons
  • Trigger buttons
  • Grab buttons
  • Analogue thumbsticks
  • Trackpad surface

Currently, it’s unclear if the trigger and grab buttons are analogue or digital. Analogue buttons would enable the user to gently grab items as well as provide a wide range of trigger actions, much like accelerating in Forza when using an Xbox One controller.

I’m also super interested to know how well the trackpad surface can be used. It seems to have the ability to click, so it can be used much like a primary button too.

Watch the introduction video and see for yourself!

What is Mixed Reality?

I’ve not created a new category for my website for years – but there’s something that I have spending a lot of time working and I need to start writing about: mixed reality.

Mixed reality is a broad umbrella of technologies which enable a real and virtual worlds to be blended together to create a single environment where physical and digital objects interact with each other.

I remember when I was in school I would imagine what it would be like to have a “heads up display” all the time. I’d imagine information that I’d see about people and surroundings, interactive maps giving me directions, reminders about errands I need to run, all sorts of things.

Thanks to science fiction there was no doubt in my mind that it was an inevitability, and it would happen in my life time – but I didn’t really know how or when it would happen.

Fortaleza Leak

Back in 2012 a very interesting image appeared on the internet. It was from an Xbox presentation which mentioned something called Fortaleza Glasses. Rumours of Project Fortaleza had been going for a while, and I followed the subject with great interest.

Linked to Alex Kipman, the creator of the Kinect, it was thought that these glasses would be Microsoft’s move to leap ahead of virtual reality and into some new world of augmented reality. I hope so much that it was true…

HoloLens

..and I remember how amazed I was when I found out it was true.

HoloLens at Home

Well, over the last six months I have not only tried the HoloLens, but spend many hours using it in both professionally and in my home. I’ve been learning how it works, how to develop software for it, and most importantly; I’ve been learning about what it means for the future of software.

This is a whole new kind of user experience, and I believe that mixed reality is the future of how humans will interact with software.

Actiongram

But mixed reality is about experience, not technology. When I say that I believe it is the future of how humans will interact with software – I am not saying that everyone will wear headsets. It won’t be long until there are other ways to project digital items into the physical world.

It breaks down the walls between physical and virtual reality – today this means vision and sound, but the concept of mixed reality goes beyond those senses. As a software architect it excites me to think about the user experiences which everyone in the augmented/virtual/mixed reality community is going to help shape, and I’m to write about what I learn as I go.

Now that I have experienced mixed reality, there’s no going back.

It’s real, it’s getting better all the time, and it’s here to stay.

Upgrading to the Xbox One S

The original Xbox One was announced over two and a half years ago, but I skipped that version and continued to use an Xbox 360 for my living room entertainment.

The new Xbox One S is much smaller and slightly faster than the original, but mostly it is the same. For me this upgrade is perfect timing, as I have been waiting for a hardware refresh before making the switch from the now-discontinued Xbox 360.

Watch the Xbox One S announcement

For people who already have an Xbox One: it may be better to wait for “Project Scorpio” – which will be a significant upgrade to the Xbox coming late 2017. Microsoft have stated that all three iterations of the Xbox One will be compatible with each other – so I am not concerned about being left behind, though “Scorpio” will almost certainly support VR as well.

Watch the Project Scorpio announcement

Hardware

The industrial design of the Xbox One S is absolutely fantastic. Anyone who wasn’t keen on the “VCR” looks of the original Xbox One would agree that Microsoft has gone in a different direction this time.

Xbox One S Design

At the moment the device only comes in white, though I know plenty of people would have liked to see it in black. To me the white polycarbonate really shows off the features of design itself – it would be a lot harder to make out some of the nice touches if the device was just black and in the shadows.

Internally the biggest noticeable improvement is around storage: the disk in the launch edition is 2 TB – that’s four times larger than the disk that shipped with the original Xbox One. You need it too, as games are regularly over 80 GB.

No Kinect in the box this time

In contrast to the original Xbox One launch bundle, the Kinect sensor is not included this time. In fact, it is prohibitive to get the Kinect for the Xbox One S: you’d have to buy both the £80 sensor and a £30 adapter to make it work over USB. The adapter is free if you are upgrading from an original Xbox One – but I am not, and so I will skip this expense for now. However, skipping the Kinect does come at a cost…

At a high level the Kinect provides three main features to the Xbox One:

  • A camera array which supports video and motion capture
  • A microphone array for voice commands as well as chat functionality
  • An infrared transmitter for controlling audio and video equipment

The infrared transmitter is now built in to the front of the Xbox One S. This means I can control my TV without the need for the expensive Kinect add-on.

But the other two features are not included.

I can understand the fact that the camera and motion detection technology is not built in, as most games don’t use it, but I don’t understand why the Xbox One S didn’t include a microphone array by default. Especially as being able to turn on the Xbox with your voice was one of the most impressive features of the Kinect, and the new Cortana assistant requires a microphone to work.

If they can include an infrared transmitter, why didn’t they include a microphone array too? I don’t get it.

Software

The Xbox One S comes pre-loaded with the “Redstone” version of the operating system which matches the Windows 10 Anniversary Update. It includes features like Xbox 360 backwards compatibly, background music, Windows 10’s Universal apps, Cortana, and loads of other tweaks. This update is available for the previous model too, but for me this is a massive leap from the experience provided by the Xbox 360.

Switching between apps is fast

On the Xbox 360 there was no way to pause a game and then watch Netflix for a while, the Xbox One’s software allows this with ease, allowing seamless switching between games, live television, and apps without long loading times.

Being able to use two apps at the same time is supported, but it isn’t as great as I’d like to be. Some applications don’t support snapping at all, and the interface for snapping unintuitive when compared Windows 10.

Xbox One Dashboard

As a Groove Music user the new background audio features work really well – all of the music from my Music Pass and OneDrive collections are available to stream from the same Universal app that I use on my Surface and my Lumia.

While I could access my music through a subpar app on my Xbox 360, I couldn’t multitask at the same time – there was no way I could listen to music on Groove and then go looking for something to watch on Netflix or iPlayer. As an entertainment device this makes a lot of sense, and I’m really pleased they’ve added it to the Xbox One.

Background Audio

Television

In the UK you can use a USB adapter for the Xbox One to gain access to Freeview. In my area of England this includes over a hundred television channels – including 17 in HD. This works without having to do IR blasting to a cable box, and integrates well with Microsoft’s OneGuide software.

I’m not a big television watcher, but now you have to have a TV Licence to watch iPlayer it makes sense to have Freeview as well.

At least this way everything is handled through the Xbox One itself.

Streaming

The Xbox app for Windows includes access to all of the Xbox Live features you’d expect. Including friends, messages, achievements, and (most importantly) game streaming.

Xbox App for Windows

I have been impressed by how well the streaming works – at the medium quality settings there is no noticeable lag, and the compression doesn’t look bad at all.

Personally I have no interest in installing 80 GB games directly on my Surface Book (which I use for software development) but having the option of streaming directly from the Xbox One S means that playing in another room is now an option.

Streaming TV and Games to Surface…
…and HoloLens

And yes, streaming even works with the HoloLens. In the photo below you can see me streaming Halo 5: Guardians onto the wall in my office.

Halo on the HoloLens

More to come

The Xbox One is still early in its life cycle – especially when you consider how much the Xbox 360 changed in the 10 years it was in the market. I still have more to write around the various options for controlling and using the device, and my thoughts on how the software needs to be improved… but that will come another day.

For now I’m pleased to have finally made the jump to the Xbox One platform, and I’m glad I waited for this beautifully designed Xbox One S.

Highlights from Build 2016

Satya

Even though I have never attended a Microsoft Build conference in person I always learn so much from them.

Every year there are new platforms to try, lots of documentation to read, and many presentations and recoded sessions to watch.

I still have a lot of videos to watch, but here are some of top announcements from Build 2016 which matter to me the most as a developer.

Windows 10 as the best OS for Developers

A number of new features coming to Windows 10 in the “Anniversary” update were shown in the day-one keynote, and then even more features where shown at sessions throughout the conference. Solid improvements to the inking, biometrics, and the Action Center were all well received.

Windows Ink

Many of the features shown help fix minor annoyances in the system. For example, pressing on a live tile showing a preview of a news article can now take you directly to it, and notifications dismissed on the PC or tablet will automatically get dismissed on the phone too.

One of the most exciting new features was the addition of Bash (on Ubuntu) on Windows which is both technically very interesting and extremely useful for many development workflows. The new Ubuntu subsystem will allow any (command-line) Linux application to run natively on Windows. This instantly unlocks a massive amount of tools and utilities for developers, making common scenarios significantly easier from Windows.

Bash on Windows

As a huge fan of command line interfaces I’m going to go into this in more detail in a future article – but essentially Microsoft are positioning Windows to be the ultimate developer platform, no matter what operating systems you use for your solutions.

Azure is growing up with more data centres and services

Microsoft would prefer you use Azure when you deploy your applications though, and the day-two keynote showed that is still serious about the cloud.

Improvements which interested me the most included Azure Functions, Service Fabric, Containers, DocumentDB, and much, much more.

Azure

Azure is the future of Microsoft, and by the numbers they going strong. They’re expanding their datacentres and really betting big on the cloud. This is no surprise to Microsoft watchers, but it’s good to see steady improvements here. Many of which I will use.

Visual Studio keeps getting better

I spend absolutely huge amounts of time in Visual Studio so any improvements here have a very positive effect on my productivity.

Visual Studio 2015 Update 2 was released (with lots of improvements) and an early preview version of Visual Studio vNext was also shown. I’ve tried both and they’re definitely going in the right direction for me.

Visual Studio

I’m especially looking forward to some of the improvements coming in the Visual Studio installation experience moving forward. This should make setting up new development environments much faster, and the side-by-side installations means there’s much less risk when installing previews.

App development for Windows, iOS, Android

The mobile app development story from Microsoft is stronger than it ever has been. This year brings a number of improvements to the Universal Windows Platform (UWP) itself, and a more integrated store experience which now includes the apps on the Xbox One and HoloLens.

The Desktop App Converter lets you wrap up existing Win32 and .NET apps into UWP packages, allowing access to new features like UWP APIs – including Live Tiles. Even though I don’t currently develop any Win32 or .NET applications that I want to put in the store, this is an important step and I’m looking forward to the benefits of this as an app user.

Xamarin

For targeting non-Windows devices, the Xamarin platform is now the obvious choice. After recently purchasing Xamarin (and their amazing talent) they’ve decided to make Xamarin available for no extra charge with Visual Studio. And that includes shipping it with the free Community version. Very cool.

The combination of UWP and Xamarin means I can directly apply my C# and .NET skills to making applications for a wide range of platforms, sharing many code components. It’s really coming together nicely.

.NET and the continued move into Open Source

As well as making Xamarin’s development tools free to Visual Studio users, the folks over at Microsoft also announced their intention to open source the Xamarin SDK (including the runtime, the libraries, and command line tools), and give the governance of it over to the .NET Foundation.

Mono, the cross platform and open source sibling of the full .NET Framework has also been re-licenced to be even more permissive, and given to the .NET Foundation. (To be honest I actually thought this was already the case!)

.NET Core, the future replacement of both the .NET Framework and Mono, also saw steady improvements – my favourite of which was official F# language support:

$ cd hellofs/
$ ls
$ dotnet new --lang f#
Created new F# project in /home/julian/hellofs.
$ # I can now dotnet restore and run this F# app using .NET Core!

The Future of Cortana and Conversation as a Platform

So far everything I have mentioned has been mostly around solid updates to existing platforms, but this year’s Build included a slightly different way of thinking about productivity with the idea of Conversation as a Platform.

Conversation as a Platform

The Microsoft Bot Framework provides templates for creating bots with C# and JavaScript, as well as connectors to simplify their interaction with services like Slack and Skype. When linked with the new Cognitive Services, these bots can understand natural language and perform tasks for the user.

build-cortana

The demonstration of talking to Cortana through Skype was very interesting – where essentially Cortana can act as a broker between the user and other bots on the Internet which can act as experts in their field. I found this very compelling, and something I can see myself using.

As this is as subject that interests me greatly, I’ll be writing more about this over the next week or so.

And everything else…

Of course, there’s no way I could summarise everything I looked at so I have skipped a number of cool announcements ranging from Microsoft Graph to HoloLens.

The hard-working folk over at Channel 9 have videos for many of the events and topics, so be sure to check them out if you’re interested. I’m very thankful that these videos are all made available for everyone to watch, I really enjoy watching them.

Highlights from Build 2015

The Build 2015 conference has just taken place in San Francisco.

Like last year, this has been another huge event for Microsoft, and a big deal for the people who build solutions using their technologies.

There have been way more interesting things happening than I can possibly cover in one article, but I have decided to cover the three most important to me:

.NET, Windows and Azure.

An exciting future for .NET and Open Source

The future of .NET is the continued push to an open source .NET Core, which is at the centre of both the latest ASP.NET runtime and the Universal Windows app platform. In the future, this will expand and include other application types. In my opinion, they’ve picked the right place to start.

Applications running on the CoreCLR can be developed and deployed on cloud and server-based infrastructures running different operating systems including Windows, Linux and OS X. I have been watching the development efforts on GitHub for a while now, and I’ve set it up on my own machines running both Windows and Linux. It sure is a sight to see.

As well as the core runtime itself going open source, other technologies like Roslyn have enabled products that many wouldn’t have guessed would see the light of day. Having an open source compiler platform has enabled Visual Studio Code – a new cross platform text editor with Intellisense – to be built.

I was lucky enough to see Visual Studio Code before it was announced, and it changed the way I thought about collaboration with Mac users instantly. I’ll have more on this new text editor soon.

Visual Studio Code

With the RC of Visual Studio 2015 there have been some big improvements in the languages supported including both the more traditional C# and Visual Basic, and (my personal favourite) F#.

The Visual F# improvements in ‘every day’ activities are dramatic for anyone who has been using the language. This is all thanks to the new open source attitude, and the amazing community around F# who have helped to develop the Visual F# tools on GitHub.

This new world of cross-platform and open source .NET technology is going to enable some amazing scenarios for .NET developers like myself.

Windows 10’s application platform takes shape

The aforementioned Universal Windows app platform is really taking shape now. Gone are the days of very prescriptive (and maybe too forward-looking) design patterns of Windows 8, and in is the ‘do what’s right for your applications‘ model that has been working well for some for a while.

Universal Windows apps scale from the smallest phones and Internet of Things devices up to the large screens of the Xbox One and the Surface Hub. The most ‘universal’ of these apps are built with just one binary which includes a scalable UI. This allows you to even have the ‘desktop’ app experience when used on a landscape 5.7 inch phone, or when plugged into an external screen using an amazing new Continuum for Phones feature.

For app developers there are some interesting (and controversial) new ways for software venders to build for Windows. The biggest of which are the bridges from Android and iOS. These two are extremely important for the phone and work especially well for iOS games which don’t rely too heavily on the operating specific UI elements. Combined with the bridges for ‘classic windows’ apps and websites using Microsoft Edge, the Store should get a lot more apps on this Windows 10 wave of releases.

From a user’s view, Windows 10 has really rounded out, with the latest Insider Preview feeling a lot more polished than any of the previous builds. Seeing HoloLens run standard Windows Universal apps was a big deal too.

I’ll have more thoughts on these in the future as the Insider Preview continues, and more information for HoloLoens is released at E3.

<

h3>Microsoft <3 Docker and other Azure improvements

Azure, and the Microsoft Cloud in general, continue to amaze me. Microsoft has managed to embrace this new way of building (and selling) software in at breakneck speed. Additional services were added throughout the platform all the way from storage and networking, to analytics and machine learning. Way too many for this article.

Two of the biggest highlights were the ability to run the complete Azure Stack locally, and Azure’s new Data Lake features too, something which Amazon has had a lot of success with.

Microsoft <3 Docker

For me though, the most interesting changes were around Docker support across Windows and Azure. Docker has been on my radar for a while, but I have yet to use it in production. I have plans to do so in the not too distant future.