Connecting Google Glass and Windows Azure Mobile Services

I haven’t posted about it on my blog yet (yeah I’ve meant to and still plan to) but I am a Google Glass Explorer.  I’ve had my glass for just over three months now and it really is incredible.  I don’t want to focus too much on why I think it’s a great device or what I commonly use it for (I’ll try to post about that soon) but instead want to focus on something I’ve spent the last few nights working on.  As you can probably tell if you look at my recent entries on this site, I do a lot with Windows Azure.  More specifically, I do a lot with Windows Azure Mobile Services. 

Windows Azure Mobile Services

Mobile Services is a backend-as-a-service which allows app developers to spend their time where they should: delivering the best client experience they can.  It does this by providing a ton of really great features that most mobile applications need and in such a way that requires very little effort on the part of the devs.  One of the great features of Mobile Services is the ability to implement your own server side logic via scripts.  You can do this in two ways which are then exposed externally (via a client application or a browser): table endpoints and custom API.  Table endpoints are generated whenever you create a table in your Mobile Service.  When you create a table, a REST API is generated that exposes the functionality for reads, inserts, updates, and deletes (via HTTP gets, posts, patches, and deletes respectively).  Each of these endpoints is connected to a script which runs on the server.  These scripts by default just pass the request through to the SQL Database that backs Mobile Services.  However, you can do whatever you want in these scripts.  This works out very well, however, for talking to Glass, custom API is what we’re going to use.  Custom APIs allow you to expose an endpoint that doesn’t map directly to a SQL Database (like the table REST APIs do) and exposes whatever functionality you want for GET, POST, PUT, PATCH, and DELETE requests.  One of the great, though unusual, capabilities of these Custom API endpoints is that they can return HTML (and javascript).  That means we can make a request to a custom API and end up rendering HTML and javascript.  Even more so, we can conditionally do that.  You’ll see what I mean as I demonstrate the authentication portion of getting ready to post to Glass.

You can sign up for a free Windows Azure trial here.  You can read more about developing with Mobile Services for free (even after the trial) here.

The Mirror API

If you haven’t looked at what Glass offers as far as development options go, there are two things that have been announced: the Google Mirror API and the Glass Development Kit (GDK).  The GDK is a forth-coming kit that will enable developers to build applications that run on the actual device itself.  We won’t be touching on the GDK today.  The Mirror API is Google’s way of allowing you to deliver information to Glass.  The way it works it that you get a token when identifies your application and the Glass device you want to interact with.  With that token you can ask the Mirror API to deliver information to that Glass device.  This isn’t all together different from how Push Notifications work (with GCM or any other major push provider).  However, there are many more additional options when it comes to delivering information to Glass.  Today we’re only going to deal with the basics of delivering some text in addition to a few options that we can pass along.

How do I test this myself?

While I can walk everyone through the steps I used to talk to my Google Glass, as of today, not everyone will be able to do the same.  In order to test out development with the Mirror API, you have to request access.  As far as I’m aware, in order to request access, you need to have Glass (or someone that has Glass and access can grant you the ability to use their Google API Project that has access to the Mirror API).  In short, very few people will probably be able to make use of this today.  However, that number will grow and it’s not a bad idea to understand how these things work so down the road you can easily consume them.

Setting up your Google Mirror API Access

You may want to create your Mobile Service first as you will need to know the name of it when you set up your Google Auth Access.  Once you’ve done that, the next step is to go to the Google API Console and create a new project.  Once you’ve created a project, go to the Services (link in the top left of the console).  From there, you’ll need to activate the Google Mirror API:

You’ll need to have been granted access before you’ll see this of course.  Once that is activated, you can go to the API Access link in the top left of the console.  Here you’ll need to click on Create an OAuth 2.0 client ID...:

In the next screen, use whatever name you want and then click Next.  Leave the Application Type as Web Application and for the site or hostname, you’ll enter something like this:  http://glasstest.azure-mobile.net/api/glassauth.  In this case, glasstest is the name of my Mobile Service (so change that to match your Mobile Service).  glassauth is the name of the custom API I’m going to use to authenticate.  After entering that, you can click Create client ID.  In the screen that follows, you’ll need to copy the Client ID and Client secret.  With that step done, we can move on to our Mobile Service.

Setting up the Custom API to Redirect

If you haven’t looked at Custom APIs before, take a look at this blog post.  It should explain some of the details on Custom APIs.  Make sure the name of your first custom API matches the name in the URL you entered for your OAuth 2.0 client (so in the case above, glassauth).  We’re only going to use the GET request for this API, so set the permissions for the GET to Everyone.  That way, anything can access the method without needing any other information.  Once this API is created, we can start by handling the first operation, redirecting users to the Google Auth page.  This is done easily enough with the following script:

The first thing we do is generate a redirect URL.  This is the URL we will push the user to after they hit the API.  The URL we’re going to use is https://accounts.google.com/o/oauth2/auth?response_type=code and as parameters, we pass:

• the redirect URI (which is the URI of our custom API since we want to return to it after authenticating)
• the client ID (this is the client ID from the Google API Console we just set up)
• the scopes (these are the “permissions” we are requesting access to) which include:
  • glass.timeline
  • glass.location
  • userinfo.profile
• the access type (offline)
• approval prompt (forced)

Next we respond to the request to our custom API (using response.send) and specify that the response code is a 200 (everything is OK) and that the response is some HTML with a little javascript mixed in.  specifically we’re just redirecting the browser to the redirect URL we just generated.  Doing so means that when the user goes to our custom API in the browser, they’ll get this:

After the user clicks Accept then the user is redirected back to our custom API (due to the redirect_uri property we specified). 

Handling the Auth Response

When the auth page posts back to our custom API, it includes a query string parameter named code.  We’re already using the custom API to handle the redirect, so now we need to check if that code parameter is there and if it is, we shouldn’t do the redirect again:

If the code parameter is there, we’ll do something else, otherwise we’ll do the redirect.  Once we do have the code, we can pull that out and then post it to another Google URL (https://accounts.google.com/o/oauth2/token) and request an access_token.  We can do this in our script like this:

First we get access to the request module and then we post to that URL.  The data we post is the code, the URL of our custom API to redirect back to, our client ID, client Secret, and what sort of grant type we want (in this case we’re using authorization_code).  The response from that request contains several things including:

• access_token - we use this to tell the Mirror API what device to deliver to
• refresh_token - this token can be used to get new access_tokens (when they expire) without the involvement of the user (only granted because we used the access_type of offline in our initial auth / permission request.
• an expiration (when the access_token expires)
• id_token - this is a JSON Web Token that we’ll use to get the user’s ID

We could stop here and just save the user’s access_token and refresh_token.  We only need to push to the Mirror API with the access_token and get a new token with the refresh_token whenever the access_token expires.  However, as you might imagine, it would be nice to be able to tie the tokens we have back to a specific user.  In order to do that, we need to get a user ID.

Fetching a User ID

We already have an id_token which contains the user ID, we just need some way to extract that information.  Thankfully, Google provides another URL we can do a request to to retrieve it: https://www.googleapis.com/oauth2/v1/tokeninfo?id_token=my_id_token.  We do a HTTP GET request against this URL and the response back will contain, among other things, the user_id.  Once we have this, we can save this information into our Mobile Service’s database (in this case I’ve created a table named Accounts) for this and then let the user they are good to go (by returning more HTML).  You can see the full script here:

Posting to Glass

Once we’ve done all of the auth stuff, posting to Glass becomes incredibly easy.  We just need to pull the information out of our Accounts table and then use the access_token as part of the post.  Here’s a custom API script that will post the text “Check this stuff out!” whenever the API is hit with a GET request:

The first thing we’ll point out is that we’re posting to https://www.googleapis.com/mirror/v1/timeline?access_token=my_token.  If we didn’t want to include the token as a query string parameter, we could also put it in an Authorization header (read more about authorization options when calling APIs here).  Next we specify the content type and say that we’ll be sending over JSON.  Finally we specify the body.  Here we’re sending specific text over and then specifying a notification and menuItems.  Both notification and menuItems are optional and the post will go through just fine without them.  However, if we don’t include the notification, then the device won’t let the user know (via a vibration and a chirp in their ear) that they’ve received a new card.  If we don’t include the  DELETE menu item, then the user won’t be able to delete the card from their timeline.  Both the DEFAULT notification level and the DELETE menu item are built in options that Glass knows what to do with, we just need to specify them.  When we do send this over then we end up seeing this delivered to the device:

Sending text over and specifying a notification and the ability to delete really just scratches the surface of what you can send to Glass.  There are tons of different options for what you can send over and how Glass can then interact with your card.  You can read more about the different options for inserting to the Glass timeline here.  One of the great things to note is that, with the exception of sending actual media data over (bytes), everything is just JSON.  This means it’s super simple to send this stuff from a Mobile Service script.  Even the media can be sent over, it’s just a little more work than I’m demonstrating here.

Summary

The Mirror API is the only official way you can currently send information to a Glass device.  Thankfully, interacting with it is pretty straightforward and is built off of very common standards: JSON, OAuth, HTTP, REST.  While there are quick starts for interacting with Glass in several languages (currently Go, Java, .NET, PHP, Python, and Ruby), you should be able to talk to Glass from many other languages and platforms if you want.  Mobile Services is a backend-as-a-service that exposes tons of great functionality regardless of if you’re building a mobile app for Android, iOS, Windows Phone, or the Windows Store or if you’re building a web site, console app, or anything else.  Being able to talk to Glass from Mobile Services is pretty simple as well.  Maybe one day, down the road, pushing to Glass will be built into Mobile Services just like push notifications are for the other platforms (this is actually quite realistic since you just need to do Google Auth (which is built into Mobile Services) and maintain a token).  I hope this helps you think about new ways of delivering information to different devices and expands what you think Mobile Services is capable of.