Mobile 3D Remote Control
Following our articles on mobile device orientation and using it to mimic the dynamic perspective feature on the Amazon Fire Phone, we're combining device orientation with remote control. And there's a demo with a Bugatti Veyron in it!
Device Orientation Recap
In previous articles, we explained how mobile devices track their orientation in three dimensions, and how this is available to use in web apps via events that are triggered when the device tilts and moves.
We also showed how this could be used to look around three dimensional objects on screen, just like the dynamic perspective feature of Amazon's Fire Phone.
In this article we're going to look at sending this information from the mobile browser to be used for remote control.
Remote Control with Node.js and Websockets
What do we mean by remote control? Imagine a full screen web browser running on your TV, and replace your TV remote with your mobile phone. This is the second screen experience of apps like TVTag (formerly GetGlue) and Beamly (formerly ZeeBox) with an added layer of direct interactivity and control.
There a number of ways to achieve this - Samsung, for example, are working on consumer devices that communicate directly on local networks. However we're aiming for something that you can interact with using your internet-connected device and its web browser, with no special apps or setup required.
This is where node.js and websockets come in. Put simply, a websocket is a bi-directional channel between a browser page and remote server - it's lightweight and low latency, you simply send messages over it without the need for handshakes, protocols, HTTP requests or responses. They haven't been widely supported by browsers until relatively recently.
Node.js has excellent websockets support via the socket.io library, and handles the server side of the connection. In our case, this just involves managing the pairing and interactions of a controlling browser with a viewer (that will be controlled).
An Application
It occurred to us that manipulation of 3D models using two dimensional controllers like touch screens and keyboards was tricky and unnatural, and perhaps - by virtue of its orientation in three dimensions - a mobile device could act as a better, three dimensional controller for a 3D model on screen.
We'll talk more about potential applications in a future post, but with increasingly sophisticated technology making its way into advertising and in-store displays, one can imagine interacting with a digital ad to explore a product and its customisation options. This is much more powerful than (say) a video loop, and has the side effect of generating potentially valuable marketing data and interactions (more on that later).
How it Works
There is a passive viewer page, accessed via a generic URL. On the server side, each new request for the viewer page generates a unique ID, and the viewer page displays a unique controller URL incorporating the ID. A QR code is generated for the URL for convenience, since it's expected to be accessed on a mobile device.
When a request is received for the controller URL, the server establishes a connection with the associated viewer page. Thereafter, control messages are simply relayed from the controller to the viewer, with the interpretation of those messages handled in Javascript by the viewer.
The Prototype
We decided on a car showroom prototype, since cars generally have a wide range of options for customisation, and these will mostly not be available for physical viewing in a showroom.
We chose three.js to handle the display and manipulation of a 3D model (note this requires WebGL), and tracked down a Bugatti Veyron model to work with. This lets us demonstrate the following.
- Manipulation of the model in three dimensions using device orientation.
- Zooming the model view with pinch gestures.
- Customisation of the model (in this case, choosing a colour).
- Generating other interactions (such as social sharing, enquiries and so on).
A fallback touch interface was implemented in case device orientation events are not available.
Check out the viewer interface and start exploring the prototype.
Caveats
As mentioned in our previous post, device orientation events are not always exposed reliably by the browser, and the device sensors can behave erratically at certain boundaries. For this reason, left-to-right and forward-backward rotation in the prototype are limited close to -90 and 90°. If the model seems to behave oddly, use our utility to check what values your device/browser is reporting.
The rotation of the device means the web browser will tend to flip between portrait and landscape mode. This necessitated a responsive design for the controller.
Taking it Further
Clearly, both controller and viewer could be more sophisticated but the real opportunity could be in the marketing data generated as a side effect of interactions.
For example...
- Allowing users to save and share particular configurations, building a database of popular options and combinations.
- Connect with social accounts on-device to add demographic information.
- Follow up with personalised communication based on users' activities.
- Measurability compared to non-interactive content.
So we can see that, as well as providing a novel means of control and interaction, applications of this type provide companies and marketers a brief presence on a consumer's device, that can be built upon to create a longer-term conversation. After all, the controller is likely to be the page the user sees when they next open their browser - the content and message could be changed after an extended period of inactivity.
We'll explore other applications in the next post, in the meantime - let us know what you think!
