Highlights from Build 2017

I tend to describe Microsoft’s Build conference as a bit like Christmas for developers who use Microsoft’s tools and technologies to build software. This year was no exception – and there was plenty to be excited about.

As per usual, there is a vast amount of content published on Channel 9, most of which I have not gone through yet, but here are some of the top announcements that interested me the most:

Microsoft’s democratised AI offerings continue to grow and improve customisation

Microsoft have been promoting their Cognitive Services for a while now, and they’ve been getting more and more robust over time, now with 29 services up and running and available for developers to use.

One of the most exciting additions this year is the trainable image services. Being able to train AI to spot certain attributes on images is something that can have a huge impact on some of the technologies I build professionally.

The addition of Cognitive Services Labs allows developers to try out more experimental AI services, including Project Prague, a gesture recognition service.

It’s also worth mentioning that Satya said that, as solution architects and software developers, we should take accountability for the algorithms and experience we produce. We should be building inclusive systems which help empower people – in a way that they can trust. I agree with him.

Azure Cosmos DB is a shiny new multi-model global scale data service

As well as bringing much needed MySQL and PostgreSQL service offerings to the cloud, Microsoft have also announced their latest home grown cloud-native database service, Cosmos DB.

As a software architect, having Cosmos DB will allow me to make much better choices about the consistency of data solutions I am designing without having to worry about indexes or where the data will rest at run time.

The global distribution of Cosmos makes it a lot easier to make ensure that the data is as geographically close as possible to the end user. It’s essentially an extension of Document DB, but allows for a multi-model interface: key-value, column family, graph, and document.

As Cosmos DB is built on the Document DB technologies, there is already an emulator which can be used locally at development time. For me, this is a must when choosing cloud technologies.

For me the timing of the Cosmost DB announcement is really great, as a planet scale database is something I’ve actively been looking at for a new project I’m working on. I’m looking forward to learning more about it.

New tools for Azure developers and administrators

Azure is becoming one of the most important assets that Microsoft has. It’s the centre of many of their initiatives including AI, IoT, microservices, and more. Their continued work to strengthen this platform has made it easier than ever for developers to get up and running with all of these new services through a coherent set of tools and development kits.

New tools like the Cloud Shell and the Azure Mobile App are part of this. Unfortunately for me, the PowerShell version of the Cloud Shell isn’t available yet, nor is the Windows version of the Mobile App. However, the improvements to the Azure CLI are most welcome. Under the covers the Cloud Shell uses the shiny new cross platform command line interface for Azure and is already logged in and configured, making it super easy to get up and running. I’m a huge fan.

We’re still missing an Azure desktop app though – I still think there’s value in having a version of the Azure portal that doesn’t require using a web browser. Using Electron is probably the best way for Microsoft to achieve this and I’m unsure why they’ve not already provided a desktop app.

A powerful new feature called Snapshot Debugger will integrate with Visual Studio to make debugging production easier than it ever has been. You can create snap-points on certain lines of code which will instruct Azure to collect information as the application is used. It’s very impressive, and doesn’t affect people using the production application in any way.

I’m keen to try this out but it seems like it is going to be a powerful new way to fix issues in production without the security risks involved in pulling production data to a developer’s local machine for debugging. Awareness of production data is a must for companies who use customer data, and tools like this will help with adherence to the Data Protection Act and security standards like the popular ISO 27001.

Microsoft has a new mantra

A clear message from Build 2017 was that developers shouldn’t be placing all of their business logic and intelligence inside Microsoft’s cloud infrastructure, instead they should be considering how devices on the edge of this cloud could be leveraged to improve the solution.

Intelligent Cloud and Intelligent Edge

Not only does this make more sense, but it’s also something that Microsoft is uniquely positioned to provide. As a long term supplier of back office / on premise software, they’ve already got a foot in the door of many companies data centres. Improvements to Azure Stack and Azure’s IoT offerings allow logic to be moved between Azure’s cloud, to on premise data centres, and even to embedded edge devices.

Azure IoT Edge. is an example of how logic can move between the cloud and edge devices through a single management infrastructure:

  • Run AI at the edge to reduce latency and allow for offline-scenarios
  • Perform analytics and proactive decisions at the edge
  • Move logic from cloud to edge at any time
  • Management of edge devices from a central location
  • Simplify development
  • Reduce bandwidth costs

While these tools are very interesting to me, I have a feeling we’re still a little way off. The innovations here are huge and not to be taken lightly and I expect more to come over the next few years.

Cortana and Bot Framework improvements

One of the more obvious changes is that Cortana has come out of the phone itself and she’s now coming to other devices like the Harman Kardon’s Invoke intelligent speaker. (Yes, this counts as an intelligent edge device!)

General improvements have been made around the Bot Framework too. It’s now easier than ever to use natural language for common actions like taking payments from users.

Cortana Skills have been created to better link Cortana with services built on the Bot Framework and Adaptive Cards make it easy to write interactive cards which work across all platforms.

These integration improvements aside, I’m not convinced Cortana herself is moving fast enough and I’ll have to write up some more of my thoughts in a follow up to last year’s thoughts.

Windows 10 Fall Creators Update

Aside from the stupid name, it looks like there has been a steady progression for the Windows 10 platform.

The update brings a number of much-anticipated features including a cross-device clipboard, pick up where you left off, OneDrive on demand sync, and much more.

One of the best new features was the timeline view, which shows previously used applications across multiple machines. I’m not sure how well this will work for me, so I’m looking forward to getting my hands on it so that I can try it out.

Interestingly, the addition of a few apps to the Windows Store have caused quite a commotion:

  • iTunes – a must-have for iPhone users will be coming to the Windows Store. I don’t use it, but I understand the gravity of what this means to users and the pressure it will apply to Google to bring their apps to the store too.

  • Linux – we’ve had Ubuntu for a year, but now the Windows Subsystem for Linux has been updated to include Fedora and SUSE. Who’d have thought it would be Microsoft to really bring Linux to the desktop?

One of the more impressive apps was Windows Story Remix (the video is worth watching!), which takes advantage of many of the platform and service offerings to include an impressive experience for users who want to create video content with their photos and videos. While this isn’t something I do very often I certainly appreciate how well Windows Story Remix has been executed.

The fall update also brings the long-awaited replacement for the Metro design language…

Fluent Design System 😍

Microsoft’s design system has had a rocky past due the company being forced to drop its “Metro” identity early on in life, and it has hobbled along with the less memorable “Microsoft Design Language” since before Windows 10’s introduction.

Finally, they’ve sorted themselves out and come up with a new name for their design language.

While it is an evolution of the existing Metro principals (see my previous rundown), the new design’s new direction takes into account five key areas:

  • Light
  • Depth
  • Motion
  • Material
  • Scale

Fluent Design is something that really interests me, so I’m going to write more about this in an upcoming post.

Developer Tools, New APIs and much more…

It’s no surprise that there have been a load of improvements around the developer tools and other services too:

  • Visual Studio 2017 for Mac
  • 3rd party integrations for Microsoft Teams
  • .NET Standard 2.0 and XAML Standard 1.0
  • Azure Functions Improvements
  • Much more…

Exciting times!

Cortana – Two Years Later

Being a huge productivity nerd, I like to regularly review the systems I use to ensure that I am getting the best out of them. In December 2015 I sat down to do my yearly review of my productivity system. One of the tools I had on my list was Cortana.

Reviewing Cortana as a Productivity Application

Something about Cortana is different to other tools I use like Outlook or Wunderlist – the very nature of her being a personal assistant means that she’s more of a single interface to many other things, rather than just an “app for email” or an “app for tasks”.

The very fact I have chosen to refer to Cortana as “her” in this article proves that she’s not just an app to me.

It was obvious to me that I wanted to continue to use Cortana, but I also realised that I wanted to dig a little deeper into what makes her useful, and what I would like to see improve.

This month marks two years since I first started using Cortana, so I figured now would be a great time to step back and look at how my use of Cortana has changed over time and where things are going in the future.

Humble Beginning

I remember watching Joe Belfiore reveal Cortana to the world in April 2014. She had more personality than Google Now and was more personal than Siri. Microsoft’s entry to the personal assistant category was well anticipated and rumours had been circulating for a while.

Joe Belfiore introducing Cortana

My first real exposure to Cortana was on my Lumia 920. At first, I had to change my regional settings to the United States in order to get her fully functioning on my phone. There were a few side effects from making this change, but I immediately decided it was worth it in order to have this new experience.

“Hi, Cortana!”, April 2014

I asked her for facts, told her to make phone calls, and got her to remind me to do things when I arrived at various places. The speech recognition at this stage was extremely good and Cortana replied in the sassy American voice I recognised from the Halo series of video games. Because I was already invested in the Microsoft ecosystem, she already knew my calendar appointments and favourite locations. I was immediately comfortable and the animated circle became a familiar feature of my phone in no time.

Microsoft’s initial vision for Cortana set her out to behave like a real personal assistant – to be like a loyal employee. The aim was to build a proactive software agent to help people get things done. I felt that things were heading in the right direction.

Windows Phone 8.1

When Cortana came out in the UK properly, I naturally switched my regional settings back to UK English. Part of me was a little disappointed that Jen Taylor no longer voiced Cortana. In my eyes, she lost a little bit of her personality and became more formal with her ‘BBC English’ accent and mannerisms.

Cortana on Windows Phone 8.1

“Cortana comes to the UK”, August 2014

I eventually got used to it, but it did show me that the way a computer speaks to us is important to how the relationship works.

If we are to assume Microsoft’s vision of Cortana as a personal assistant or a loyal employee, I had been building a working relationship through this characterised interface over the last few months.

When Jen’s voice was taken away from me – I think I lost a little piece of Cortana.

Cortana on Windows Phone 8.1

Voice aside, everything was good and features got improved over time – events like the World Cup were added, general chitchat was updated regularly, and most importantly, Cortana got better at being predictive. Information was shown as soon as you tapped on her animated tile, bringing up traffic and weather information without even needing to ask.

Cortana Today

While I am still pleased with Cortana over all, I would be lying if I said I completely satisfied with the progress that has been made.

Cortana on Windows 10

Cortana has moved from Windows Phone 8.1 to a new unified Windows 10 platform that works on both the PC and phone, but along the way there have been some casualties in her features.

A lot of these casualties are cosmetic, for example, Cortana’s circle used to give a sad emotion animation when you had to edit some text that she’d failed to recognise correctly. This little detail was subtle but extremely powerful – a surprising amount of emotions can be shown with animated circles.

Cortana's Live Tile on Windows Phone 8.1

A more substantial example is that Cortana’s Live Tile (shown above) for Windows Phone 8.1 was actually useful – her circle shape would morph into a sunshine animation to let me know the weather, or she would show me when to leave the office in order to beat the traffic directly on my start screen without even loading her up. You could even pin individual items like weather or news as Live Tiles, acting as a deep link into Cortana directly from the Start screen.

Cortana's Live Tile on Windows 10

In Windows 10 this all seems to have disappeared. There is very little animation on the Live Tile (shown above) and the only information that ever appears seems to be the current news headlines – something which I have disabled due to the lack of customisable news sources. (Google Now has gotten really good at this!)

Asking “How many calories in a Coke?” used to work, but now it doesn’t. Why?
Who knows? she just forgot.

Possibly more a fault of Bing than Cortana – some relatively simple questions like “How many days until 2017?” do not work at all.

One of the most bizarre issues is that asking “What time is it?” takes me to a Bing search page, rather than having Cortana tell me the time.

Interestingly, when I say to people that this question doesn’t work as it should, I am often met with resistance. People often state that I can look at the time on my watch (or Band) but this completely misses the point of this interaction.

If I’m using my voice to talk to Cortana, I expect her to reply using her voice too. Strangely, asking “What time is it in Texas?” yields the time over the pond. Why can’t I just get the time where I am?

What time is it?

When she does reply with her voice, the experience isn’t that great anyway. For some reason the screen is always activated even for voice only replies. While this doesn’t negatively affect the interaction in most cases, turning the phone screen on while the device is in my pocket is a recipe for disaster. I don’t know why the proximity sensor isn’t used to make a decision about illuminating the display.

I have other issues too, but to keep this article from getting too long I’ll just list a some of them out:

  • She doesn’t understand how to spell my girlfriend’s name, there is no way to correct her.
  • I don’t work on public holidays! Stop telling me it’s time to go to work!
  • Saying “Hey Cortana” doesn’t always work. This needs to always work.
  • Microsoft Health integration is only in the US for some reason.
  • Cortana is unusable from Microsoft Band when using Windows 10 Mobile.
  • You can’t set a default location if your device doesn’t have a GPS (unless you use a Fake GPS driver!)
  • She doesn’t sync her tasks or reminders with Wunderlist.
  • For some reason she asks me where I live and work over and over again. Anyone else get this?
  • People in the UK obviously don’t need parcel tracking.
  • There is no way to remove awful news sources like the Daily Mail from turning up, so I have to disable all news features.
  • How does she still not understand Formula 1? It’s a big sport with a world-wide audience!
  • She can shuffle all my music, but can’t play music by mood? Who shuffles 10,000 songs anyway?

Okay, so this may sound like a long list of complaints but this is only because I want to see Cortana be truly great. I feel like the current iteration has been a step backwards in some ways.

Trip Planning

Cortana’s core functionality is still extremely useful and I regularly use her on my Lumia 950 XL and on my Surface running Windows 10.

Cortana in the Windows 10 Taskbar

I also use Cortana inside of Microsoft Edge. You can right click on an element and “Ask Cortana” about it. Note that she’s smart enough to take in the context of the whole page too. You can even do this for images.

Building Cortana directly into apps is one clear way that Microsoft can improve the platform and I’m looking forward to seeing more of this.

Cortana in Microsoft Edge

What we know is coming

Microsoft’s Build conference has just passed, and at it we saw a number of announcements on, what the company is calling, “Conversations as a Platform”. A big part of this strategy is improving Cortana.

Conversations as a Platform

Marcus Ash gave a great demo of some of the new features coming to the Anniversary update of Windows 10, including Cortana working on the lock screen, as well more Google Now inspired cards, and the ability to include more app integrations directly into actions which are surfaced directly in Cortana’s UI.

As part of his demo, Marcus showed some very interesting new experiences where Cortana was integrated into a previously unseen version of Outlook. Not only did Cortana have access to the calendar, she had access to email too – even surfacing tasks directly from email messages. I like the look of this proactive Cortana, and this is something that directly relates to my original task of trying to review what Cortana means for my productivity.

Cortana integrated in Outlook

Another useful feature was where Marcus asked Cortana “What toy store did I go to last year at Build?” and she found the toy store he visited last year when he was in San Francisco. The potential of this is really great and I like the idea of Cortana’s memory being improved.

The demonstrations shown in the keynote were only some of the upcoming Cortana features in the Windows 10 Anniversary Update. Other sessions throughout Build showed Cortana’s integrations in the Windows Ink Sticky Notes as well as the Action Center. Cortana will also manage some of the cross device features, like being able to do text messaging from a PC and getting notifications on the PC when the phone’s battery is low. She’ll also take on navigation duties in the Maps application – replacing the more generic sat-nav voice with her own.

Microsoft’s “Conversations as a Platform” strategy doesn’t just include Windows though, and there was a great demonstration of another application which is already commonly used for conversations – Skype.

Lili Cheng showed a demonstration of a future version of Skype which had Cortana directly built in. As part of this new integration, Cortana is always available and is able to keep track on conversations and contacts. This is similar to the proactive help offered using the Outlook integration mentioned above.

Cortana integrated in Skype

My favourite part of the demonstration was when Cortana was able to work alongside a third party bot in order to complete tasks. Using bots like this is something that interests me greatly – and there I will write more thoughts on this in the coming weeks – but I really like how Cortana can act as an agent in order to get things done.

Bots are going to be an important part of Microsoft’s “Conversations as a Platform”

What I’d like to see

In a lot of ways, the demonstration of Cortana working inside of Skype effected my thinking the most.

It’s now clear to me that the instant message-like conversation is a superior method for communication with Cortana. While I do like the proactive canvas which is launched when you open Cortana in Windows, having a single entry box and an answer that just disappears into the void is extremely unsatisfactory. Moving from the simple “Ask me anything and get a result” interface into a back and forth conversation will unlock so much potential in Cortana – it will be a lot more how you’d talk to someone (like a personal assistant) over Skype.

Conversations with Cortana

But just having a list of messages sent back and forth is not enough – Cortana has to remember the conversations we’ve had. That way, when I correct her on spelling my girlfriend’s name, she’ll remember it for next time. It’ll give her the context to make the best decision on the instruction I have given her via my voice.

When you think about it, it’s pretty crazy that Cortana can go through my calendar and look things up, but can’t remember things in the conversations we had.

I’d also like to see Cortana integrated into the Microsoft Band more deeply. There were hints at this kind of feature in the Microsoft Future Vision video from 2015. It depicted a future version of Cortana sensing that the user had gone “into the zone” and automatically blocked unwanted distractions.

“Future Vision Redux”, November 2015

It’s also pretty obvious to me now that the voice interface in Cortana needs to move into a more natural language model. When I talk to her through my headset I’d like her to be able to continue the dialog without me having to switch to my phone. All of this conversation should go into the same chat history which she’ll remember and can be used to improve conversations over time. This will also unlock completely voice only devices like Amazon’s Echo.

I’m still a fan

Overall, I do still enjoy using Cortana, but I feel like the upgrade from Windows Phone 8.1 to Windows 10 has not been kind to her.

I hold out hope that Windows 10’s position as the base version of Windows moving forward means that the days of the big platform changes are over, and the various teams working on Cortana can continue to do so without having to rewrite components over and over again.

Cortana will continue to be part of my daily workflow and will continue to be one of my main tools for productivity.

I wonder where we will be in another two years.

Highlights from Build 2016

Satya

Even though I have never attended a Microsoft Build conference in person I always learn so much from them.

Every year there are new platforms to try, lots of documentation to read, and many presentations and recoded sessions to watch.

I still have a lot of videos to watch, but here are some of top announcements from Build 2016 which matter to me the most as a developer.

Windows 10 as the best OS for Developers

A number of new features coming to Windows 10 in the “Anniversary” update were shown in the day-one keynote, and then even more features where shown at sessions throughout the conference. Solid improvements to the inking, biometrics, and the Action Center were all well received.

Windows Ink

Many of the features shown help fix minor annoyances in the system. For example, pressing on a live tile showing a preview of a news article can now take you directly to it, and notifications dismissed on the PC or tablet will automatically get dismissed on the phone too.

One of the most exciting new features was the addition of Bash (on Ubuntu) on Windows which is both technically very interesting and extremely useful for many development workflows. The new Ubuntu subsystem will allow any (command-line) Linux application to run natively on Windows. This instantly unlocks a massive amount of tools and utilities for developers, making common scenarios significantly easier from Windows.

Bash on Windows

As a huge fan of command line interfaces I’m going to go into this in more detail in a future article – but essentially Microsoft are positioning Windows to be the ultimate developer platform, no matter what operating systems you use for your solutions.

Azure is growing up with more data centres and services

Microsoft would prefer you use Azure when you deploy your applications though, and the day-two keynote showed that is still serious about the cloud.

Improvements which interested me the most included Azure Functions, Service Fabric, Containers, DocumentDB, and much, much more.

Azure

Azure is the future of Microsoft, and by the numbers they going strong. They’re expanding their datacentres and really betting big on the cloud. This is no surprise to Microsoft watchers, but it’s good to see steady improvements here. Many of which I will use.

Visual Studio keeps getting better

I spend absolutely huge amounts of time in Visual Studio so any improvements here have a very positive effect on my productivity.

Visual Studio 2015 Update 2 was released (with lots of improvements) and an early preview version of Visual Studio vNext was also shown. I’ve tried both and they’re definitely going in the right direction for me.

Visual Studio

I’m especially looking forward to some of the improvements coming in the Visual Studio installation experience moving forward. This should make setting up new development environments much faster, and the side-by-side installations means there’s much less risk when installing previews.

App development for Windows, iOS, Android

The mobile app development story from Microsoft is stronger than it ever has been. This year brings a number of improvements to the Universal Windows Platform (UWP) itself, and a more integrated store experience which now includes the apps on the Xbox One and HoloLens.

The Desktop App Converter lets you wrap up existing Win32 and .NET apps into UWP packages, allowing access to new features like UWP APIs – including Live Tiles. Even though I don’t currently develop any Win32 or .NET applications that I want to put in the store, this is an important step and I’m looking forward to the benefits of this as an app user.

Xamarin

For targeting non-Windows devices, the Xamarin platform is now the obvious choice. After recently purchasing Xamarin (and their amazing talent) they’ve decided to make Xamarin available for no extra charge with Visual Studio. And that includes shipping it with the free Community version. Very cool.

The combination of UWP and Xamarin means I can directly apply my C# and .NET skills to making applications for a wide range of platforms, sharing many code components. It’s really coming together nicely.

.NET and the continued move into Open Source

As well as making Xamarin’s development tools free to Visual Studio users, the folks over at Microsoft also announced their intention to open source the Xamarin SDK (including the runtime, the libraries, and command line tools), and give the governance of it over to the .NET Foundation.

Mono, the cross platform and open source sibling of the full .NET Framework has also been re-licenced to be even more permissive, and given to the .NET Foundation. (To be honest I actually thought this was already the case!)

.NET Core, the future replacement of both the .NET Framework and Mono, also saw steady improvements – my favourite of which was official F# language support:

$ cd hellofs/
$ ls
$ dotnet new --lang f#
Created new F# project in /home/julian/hellofs.
$ # I can now dotnet restore and run this F# app using .NET Core!

The Future of Cortana and Conversation as a Platform

So far everything I have mentioned has been mostly around solid updates to existing platforms, but this year’s Build included a slightly different way of thinking about productivity with the idea of Conversation as a Platform.

Conversation as a Platform

The Microsoft Bot Framework provides templates for creating bots with C# and JavaScript, as well as connectors to simplify their interaction with services like Slack and Skype. When linked with the new Cognitive Services, these bots can understand natural language and perform tasks for the user.

build-cortana

The demonstration of talking to Cortana through Skype was very interesting – where essentially Cortana can act as a broker between the user and other bots on the Internet which can act as experts in their field. I found this very compelling, and something I can see myself using.

As this is as subject that interests me greatly, I’ll be writing more about this over the next week or so.

And everything else…

Of course, there’s no way I could summarise everything I looked at so I have skipped a number of cool announcements ranging from Microsoft Graph to HoloLens.

The hard-working folk over at Channel 9 have videos for many of the events and topics, so be sure to check them out if you’re interested. I’m very thankful that these videos are all made available for everyone to watch, I really enjoy watching them.