Mobile 3D Remote Control
Following our articles on mobile device orientation and using it to mimic the dynamic perspective feature on the Amazon Fire Phone, we're combining device orientation with remote control. And there's a demo with a Bugatti Veyron in it!
Device Orientation Recap
In previous articles, we explained how mobile devices track their orientation in three dimensions, and how this is available to use in web apps via events that are triggered when the device tilts and moves.
We also showed how this could be used to look around three dimensional objects on screen, just like the dynamic perspective feature of Amazon's Fire Phone.
In this article we're going to look at sending this information from the mobile browser to be used for remote control.
Remote Control with Node.js and Websockets
What do we mean by remote control? Imagine a full screen web browser running on your TV, and replace your TV remote with your mobile phone. This is the second screen experience of apps like TVTag (formerly GetGlue) and Beamly (formerly ZeeBox) with an added layer of direct interactivity and control.
There a number of ways to achieve this - Samsung, for example, are working on consumer devices that communicate directly on local networks. However we're aiming for something that you can interact with using your internet-connected device and its web browser, with no special apps or setup required.
This is where node.js and websockets come in. Put simply, a websocket is a bi-directional channel between a browser page and remote server - it's lightweight and low latency, you simply send messages over it without the need for handshakes, protocols, HTTP requests or responses. They haven't been widely supported by browsers until relatively recently.
Node.js has excellent websockets support via the socket.io library, and handles the server side of the connection. In our case, this just involves managing the pairing and interactions of a controlling browser with a viewer (that will be controlled).
An Application
It occurred to us that manipulation of 3D models using two dimensional controllers like touch screens and keyboards was tricky and unnatural, and perhaps - by virtue of its orientation in three dimensions - a mobile device could act as a better, three dimensional controller for a 3D model on screen.
We'll talk more about potential applications in a future post, but with increasingly sophisticated technology making its way into advertising and in-store displays, one can imagine interacting with a digital ad to explore a product and its customisation options. This is much more powerful than (say) a video loop, and has the side effect of generating potentially valuable marketing data and interactions (more on that later).
How it Works
There is a passive viewer page, accessed via a generic URL. On the server side, each new request for the viewer page generates a unique ID, and the viewer page displays a unique controller URL incorporating the ID. A QR code is generated for the URL for convenience, since it's expected to be accessed on a mobile device.
When a request is received for the controller URL, the server establishes a connection with the associated viewer page. Thereafter, control messages are simply relayed from the controller to the viewer, with the interpretation of those messages handled in Javascript by the viewer.
The Prototype
We decided on a car showroom prototype, since cars generally have a wide range of options for customisation, and these will mostly not be available for physical viewing in a showroom.
We chose three.js to handle the display and manipulation of a 3D model (note this requires WebGL), and tracked down a Bugatti Veyron model to work with. This lets us demonstrate the following.
- Manipulation of the model in three dimensions using device orientation.
- Zooming the model view with pinch gestures.
- Customisation of the model (in this case, choosing a colour).
- Generating other interactions (such as social sharing, enquiries and so on).
A fallback touch interface was implemented in case device orientation events are not available.
Check out the viewer interface and start exploring the prototype.
Caveats
As mentioned in our previous post, device orientation events are not always exposed reliably by the browser, and the device sensors can behave erratically at certain boundaries. For this reason, left-to-right and forward-backward rotation in the prototype are limited close to -90 and 90°. If the model seems to behave oddly, use our utility to check what values your device/browser is reporting.
The rotation of the device means the web browser will tend to flip between portrait and landscape mode. This necessitated a responsive design for the controller.
Taking it Further
Clearly, both controller and viewer could be more sophisticated but the real opportunity could be in the marketing data generated as a side effect of interactions.
For example...
- Allowing users to save and share particular configurations, building a database of popular options and combinations.
- Connect with social accounts on-device to add demographic information.
- Follow up with personalised communication based on users' activities.
- Measurability compared to non-interactive content.
So we can see that, as well as providing a novel means of control and interaction, applications of this type provide companies and marketers a brief presence on a consumer's device, that can be built upon to create a longer-term conversation. After all, the controller is likely to be the page the user sees when they next open their browser - the content and message could be changed after an extended period of inactivity.
We'll explore other applications in the next post, in the meantime - let us know what you think!
Mobile Device Orientation
The deviceorientation and devicemotion events have been supported by the default iOS and Android browser for some time, and are now supported on the other major mobile browsers too. This creates opportunities for new types of interaction and this post acts as an introduction to those getting started with mobile orientation events, including some practical experience.
What's Available
Most mobile devices incorporate a gyroscope, accelerometer and compass that, between them, can report the orientation and movement of the device in three dimensions.
The deviceorientation event returns an object with three properties: alpha, beta and gamma, corresponding to the device's rotation in degrees around the Z, X and Y axes respectively.
alpha is reported by the compass, for example 90° = west.
A device lying flat should report beta and gamma values of zero.
From this position, tilting the device's screen towards the right increases gamma towards 90° when it is fully pointing right. Tilting left decreases gamma towards to -90° when it is fully pointing left.
Similarly, tilting the device forward towards you increases beta towards 90° (when the screen is vertical and facing you), tilting it away decreases beta towards -90° (when it is upside down and facing away).
The devicemotion event returns information about the device's acceleration and rotation rate in x, y and z.
Device and Browser support
Gyroscopes, compasses and accelerometers are standard in modern mobile devices and, in principal, the deviceorientation event is well supported by current mobile browsers.
In practice though, we have found wild variation in the values returned to the browser by different device-browser combinations.
What's Really Available
The w3c documentation on these events is fairly prescriptive, but leaves a few loopholes for browser implementations. For example:
The event should fire whenever a significant change in orientation occurs. The definition of a significant change in this context is left to the implementation.
Although a threshhold of 1 degree is recommended, tests in Chrome on an HTC One M8 running Android 4.4.2 currently only seem to return alpha and gamma to a granularity of 90 degrees, and return NaN (Not a Number) for beta! Testing in the default Android browser on the same device gave expected values, however they appear to update only once per second.
Our initial testing was with a range of Samsung Galaxy devices using Chrome - these gave values as expected and updated values very frequently.
The exception to all of this is the alpha value reported by the compass - on most devices we tested it seems to be unreliable and erratic, though we found it reported reliably by a Galaxy S3 mini.
You can test your device-browser combo using our handy checker (a QR code link is shown for convenience if you view on a non-mobile browser).
Gotchas
You might expect that rotating your device fully around a single axis would produce values increasing smoothly from 0 to 360° then wrapping round. However, this is not the case:
- beta has a range of -180° to 180°
- gamma has a range of only -90° to 90°
Additionally, for example, tilting your device towards upright vertical increases beta, but as it approaches 90°, gamma can vary wildly across its whole range. This is partly due to ambiguity in the coordinate system, since certain orientations can be achieved by different sets of rotations, but the device and browser can introduce their own issues.
There is a bug logged for Chromium that highlights all of this quite well.
This is related to the phenomenon of Gimbal lock and the w3c documentation explains mapping the values to avoid such issues (for example using Quaternions).
Conclusions
Device orientation can supplement other means of input in an increasingly mobile-first world, and is not just a gimmick - consider that it offers various degrees of control without the use of your free hand or other external input.
However, variability in implementation on specific devices and browsers means that it can't be relied on as the sole way of controlling some interaction. For example, we're working on a prototype for mobile remote control where arrow buttons are displayed for control when device orientation is not being reported.
A post on that application is imminent, for now you can also read about mimicking the Amazon Fire Phone's dynamic perspective feature using device orientation.