Mobile 3D Remote Control
Following our articles on mobile device orientation and using it to mimic the dynamic perspective feature on the Amazon Fire Phone, we're combining device orientation with remote control. And there's a demo with a Bugatti Veyron in it!
Device Orientation Recap
In previous articles, we explained how mobile devices track their orientation in three dimensions, and how this is available to use in web apps via events that are triggered when the device tilts and moves.
We also showed how this could be used to look around three dimensional objects on screen, just like the dynamic perspective feature of Amazon's Fire Phone.
In this article we're going to look at sending this information from the mobile browser to be used for remote control.
Remote Control with Node.js and Websockets
What do we mean by remote control? Imagine a full screen web browser running on your TV, and replace your TV remote with your mobile phone. This is the second screen experience of apps like TVTag (formerly GetGlue) and Beamly (formerly ZeeBox) with an added layer of direct interactivity and control.
There a number of ways to achieve this - Samsung, for example, are working on consumer devices that communicate directly on local networks. However we're aiming for something that you can interact with using your internet-connected device and its web browser, with no special apps or setup required.
This is where node.js and websockets come in. Put simply, a websocket is a bi-directional channel between a browser page and remote server - it's lightweight and low latency, you simply send messages over it without the need for handshakes, protocols, HTTP requests or responses. They haven't been widely supported by browsers until relatively recently.
Node.js has excellent websockets support via the socket.io library, and handles the server side of the connection. In our case, this just involves managing the pairing and interactions of a controlling browser with a viewer (that will be controlled).
An Application
It occurred to us that manipulation of 3D models using two dimensional controllers like touch screens and keyboards was tricky and unnatural, and perhaps - by virtue of its orientation in three dimensions - a mobile device could act as a better, three dimensional controller for a 3D model on screen.
We'll talk more about potential applications in a future post, but with increasingly sophisticated technology making its way into advertising and in-store displays, one can imagine interacting with a digital ad to explore a product and its customisation options. This is much more powerful than (say) a video loop, and has the side effect of generating potentially valuable marketing data and interactions (more on that later).
How it Works
There is a passive viewer page, accessed via a generic URL. On the server side, each new request for the viewer page generates a unique ID, and the viewer page displays a unique controller URL incorporating the ID. A QR code is generated for the URL for convenience, since it's expected to be accessed on a mobile device.
When a request is received for the controller URL, the server establishes a connection with the associated viewer page. Thereafter, control messages are simply relayed from the controller to the viewer, with the interpretation of those messages handled in Javascript by the viewer.
The Prototype
We decided on a car showroom prototype, since cars generally have a wide range of options for customisation, and these will mostly not be available for physical viewing in a showroom.
We chose three.js to handle the display and manipulation of a 3D model (note this requires WebGL), and tracked down a Bugatti Veyron model to work with. This lets us demonstrate the following.
- Manipulation of the model in three dimensions using device orientation.
- Zooming the model view with pinch gestures.
- Customisation of the model (in this case, choosing a colour).
- Generating other interactions (such as social sharing, enquiries and so on).
A fallback touch interface was implemented in case device orientation events are not available.
Check out the viewer interface and start exploring the prototype.
Caveats
As mentioned in our previous post, device orientation events are not always exposed reliably by the browser, and the device sensors can behave erratically at certain boundaries. For this reason, left-to-right and forward-backward rotation in the prototype are limited close to -90 and 90°. If the model seems to behave oddly, use our utility to check what values your device/browser is reporting.
The rotation of the device means the web browser will tend to flip between portrait and landscape mode. This necessitated a responsive design for the controller.
Taking it Further
Clearly, both controller and viewer could be more sophisticated but the real opportunity could be in the marketing data generated as a side effect of interactions.
For example...
- Allowing users to save and share particular configurations, building a database of popular options and combinations.
- Connect with social accounts on-device to add demographic information.
- Follow up with personalised communication based on users' activities.
- Measurability compared to non-interactive content.
So we can see that, as well as providing a novel means of control and interaction, applications of this type provide companies and marketers a brief presence on a consumer's device, that can be built upon to create a longer-term conversation. After all, the controller is likely to be the page the user sees when they next open their browser - the content and message could be changed after an extended period of inactivity.
We'll explore other applications in the next post, in the meantime - let us know what you think!
Dynamic Perspective
Amazon launched their first foray into the Android handset market, the Fire Phone in June 2014. It incorporates unique head tracking that lets users look around 3D objects on screen. But how close can we get with regular device orientation?
Amazon's Fire Phone
The Fire Phone uses 4 additional cameras to achieve its innovative head tracking feature, however we noticed that in all the demos we saw, users moved the device itself, not their head.
Whilst this feature could be a precursor of something more useful on larger format devices (that you might walk around), it got us thinking how well we could mimic it for mobile in HTML5 using the deviceorientation event.
Device Orientation
Modern smartphones incorporate a gyroscope, accelerometer and compass that report the device's orientation, movement and direction. You may have seen games controlled by tilting your phone, but these features are not widely used outisde of navigation and rotating the display when the phone is turned on its side.
This surprising given that device orientation and direction are available to HTML5 web apps in the browser.
Technical details are explored in another post, we will focus on a few prototype applications here.
Applications
The Fire Phone demonstrations showed users looking around 3D objects, and tilting to scroll. The latter has been available on other Android phones for some time, and is arguably tricky to use in practice.
The prototypes make use of the left-right and forward-back tilt of the phone. You can use this handy page to check that your phone reports this information, and that your web browser makes it available. If you don't see values updating on that page as you move your phone, device orientation is not being reported to the browser and the demos won't work... If that's the case, you might want to try a different browser on your phone.
Looking Around 3D Objects
Whilst the Fire Phone detects the viewer's head position to change the rotation of objects on screen, we will achieve a similar result using the orientation of the device. The idea is to rotate the object on screen in the same direction, such that tilting your phone lets you see more of the object in a natural way.
Check it out in action on this page (a QR code will be shown on non-mobile browsers for convenience).
The box was created and rotated using CSS3 3D transforms.
We also created another demo using these fluffy little clouds, also created with CSS3. You can tilt your device to look around the clouds.
Orientation Correction
When creating the dynamic perspective prototypes, we wondered what would happen if you rotated the object in the opposite direction, such that it appeared to maintain it's original position even when you tilted your device.
We've applied this technique to a small example web page for the purposes of illustration. This also shows the power of 3D transforms in CSS3 - rotation is applied to an element (here, the) and inherited by everything contained within as you'd expect.
Conclusion
Device orientation and motion (via the deviceorientation and devicemotion events) are little-used but increasingly relevant in web development for a mobile-first world. There are relatively few real-world applications, but great potential for gestural input that doesn't require your free hand.
We've shown simple examples inspired by the Amazon Fire Phone demos that used multiple cameras to achieve similar results. Our next demo will show a practical application - remote mobile control of a more sophisticated model, rendered with 3D Javascript library three.js.