Mobile 3D Remote Control
Following our articles on mobile device orientation and using it to mimic the dynamic perspective feature on the Amazon Fire Phone, we're combining device orientation with remote control. And there's a demo with a Bugatti Veyron in it!
Device Orientation Recap
In previous articles, we explained how mobile devices track their orientation in three dimensions, and how this is available to use in web apps via events that are triggered when the device tilts and moves.
We also showed how this could be used to look around three dimensional objects on screen, just like the dynamic perspective feature of Amazon's Fire Phone.
In this article we're going to look at sending this information from the mobile browser to be used for remote control.
Remote Control with Node.js and Websockets
What do we mean by remote control? Imagine a full screen web browser running on your TV, and replace your TV remote with your mobile phone. This is the second screen experience of apps like TVTag (formerly GetGlue) and Beamly (formerly ZeeBox) with an added layer of direct interactivity and control.
There a number of ways to achieve this - Samsung, for example, are working on consumer devices that communicate directly on local networks. However we're aiming for something that you can interact with using your internet-connected device and its web browser, with no special apps or setup required.
This is where node.js and websockets come in. Put simply, a websocket is a bi-directional channel between a browser page and remote server - it's lightweight and low latency, you simply send messages over it without the need for handshakes, protocols, HTTP requests or responses. They haven't been widely supported by browsers until relatively recently.
Node.js has excellent websockets support via the socket.io library, and handles the server side of the connection. In our case, this just involves managing the pairing and interactions of a controlling browser with a viewer (that will be controlled).
An Application
It occurred to us that manipulation of 3D models using two dimensional controllers like touch screens and keyboards was tricky and unnatural, and perhaps - by virtue of its orientation in three dimensions - a mobile device could act as a better, three dimensional controller for a 3D model on screen.
We'll talk more about potential applications in a future post, but with increasingly sophisticated technology making its way into advertising and in-store displays, one can imagine interacting with a digital ad to explore a product and its customisation options. This is much more powerful than (say) a video loop, and has the side effect of generating potentially valuable marketing data and interactions (more on that later).
How it Works
There is a passive viewer page, accessed via a generic URL. On the server side, each new request for the viewer page generates a unique ID, and the viewer page displays a unique controller URL incorporating the ID. A QR code is generated for the URL for convenience, since it's expected to be accessed on a mobile device.
When a request is received for the controller URL, the server establishes a connection with the associated viewer page. Thereafter, control messages are simply relayed from the controller to the viewer, with the interpretation of those messages handled in Javascript by the viewer.
The Prototype
We decided on a car showroom prototype, since cars generally have a wide range of options for customisation, and these will mostly not be available for physical viewing in a showroom.
We chose three.js to handle the display and manipulation of a 3D model (note this requires WebGL), and tracked down a Bugatti Veyron model to work with. This lets us demonstrate the following.
- Manipulation of the model in three dimensions using device orientation.
- Zooming the model view with pinch gestures.
- Customisation of the model (in this case, choosing a colour).
- Generating other interactions (such as social sharing, enquiries and so on).
A fallback touch interface was implemented in case device orientation events are not available.
Check out the viewer interface and start exploring the prototype.
Caveats
As mentioned in our previous post, device orientation events are not always exposed reliably by the browser, and the device sensors can behave erratically at certain boundaries. For this reason, left-to-right and forward-backward rotation in the prototype are limited close to -90 and 90°. If the model seems to behave oddly, use our utility to check what values your device/browser is reporting.
The rotation of the device means the web browser will tend to flip between portrait and landscape mode. This necessitated a responsive design for the controller.
Taking it Further
Clearly, both controller and viewer could be more sophisticated but the real opportunity could be in the marketing data generated as a side effect of interactions.
For example...
- Allowing users to save and share particular configurations, building a database of popular options and combinations.
- Connect with social accounts on-device to add demographic information.
- Follow up with personalised communication based on users' activities.
- Measurability compared to non-interactive content.
So we can see that, as well as providing a novel means of control and interaction, applications of this type provide companies and marketers a brief presence on a consumer's device, that can be built upon to create a longer-term conversation. After all, the controller is likely to be the page the user sees when they next open their browser - the content and message could be changed after an extended period of inactivity.
We'll explore other applications in the next post, in the meantime - let us know what you think!
The ZX404
For those not old enough to remember the Sinclair ZX Spectrum and it’s ubiquitous loading screen, we explain the inspiration for our 404 page, and how we went about building it in Javascript.
The ZX Spectrum
In the early 1980s, games and applications were loaded into personal computers like the Sinclair ZX Spectrum from audio tapes. This took some time so developers tried to entertain you during the process by displaying basic animations, whilst the audio created coloured stripe effects in the background on the TV screen.
The ZX404
I had the idea to create something that looked like a ZX Spectrum loading screen in Javascript (without the horrible noises!) as our 404 page. This was partly for fun, because I couldn’t find anything similar done in Javascript (why would I?), but also because in many ways this shows where New Now comes from – these were the machines on which I first learned to program (my first computer was a ZX81).
I first created a small bitmap version of our logo, and scaled it up in the page so it becomes suitably blocky. The browser scaling makes it a little blurry which is actually not quite right, but kind of adds to the whole CRT TV effect.
Javascript
Firstly, the page is filled with stripes of random height between 1 and 4%, and each assigned a random colour from our brand palette (+ black).
The black “screen” is overlaid in the middle, then the scaled-up logo, then that is covered by a series of horizontal “blinds”, each 2% high.
We’re now ready to kick things off. A timer is set to update all the stripes to a random palette colour every 10ms, then another is set to animate the next blind off to the right of the screen every 0.2 seconds or so, gradually revealing the logo underneath.
When all blinds have been pushed off, the logo image is switched to a page not found message and link.
This is also a nod to the fact that games would often fail to load – anyone familiar with these machines will instantly recognise the feeling of watching something like this for some minutes before ultimately…. Nothing 🙂
Dynamic Perspective
Amazon launched their first foray into the Android handset market, the Fire Phone in June 2014. It incorporates unique head tracking that lets users look around 3D objects on screen. But how close can we get with regular device orientation?
Amazon's Fire Phone
The Fire Phone uses 4 additional cameras to achieve its innovative head tracking feature, however we noticed that in all the demos we saw, users moved the device itself, not their head.
Whilst this feature could be a precursor of something more useful on larger format devices (that you might walk around), it got us thinking how well we could mimic it for mobile in HTML5 using the deviceorientation event.
Device Orientation
Modern smartphones incorporate a gyroscope, accelerometer and compass that report the device's orientation, movement and direction. You may have seen games controlled by tilting your phone, but these features are not widely used outisde of navigation and rotating the display when the phone is turned on its side.
This surprising given that device orientation and direction are available to HTML5 web apps in the browser.
Technical details are explored in another post, we will focus on a few prototype applications here.
Applications
The Fire Phone demonstrations showed users looking around 3D objects, and tilting to scroll. The latter has been available on other Android phones for some time, and is arguably tricky to use in practice.
The prototypes make use of the left-right and forward-back tilt of the phone. You can use this handy page to check that your phone reports this information, and that your web browser makes it available. If you don't see values updating on that page as you move your phone, device orientation is not being reported to the browser and the demos won't work... If that's the case, you might want to try a different browser on your phone.
Looking Around 3D Objects
Whilst the Fire Phone detects the viewer's head position to change the rotation of objects on screen, we will achieve a similar result using the orientation of the device. The idea is to rotate the object on screen in the same direction, such that tilting your phone lets you see more of the object in a natural way.
Check it out in action on this page (a QR code will be shown on non-mobile browsers for convenience).
The box was created and rotated using CSS3 3D transforms.
We also created another demo using these fluffy little clouds, also created with CSS3. You can tilt your device to look around the clouds.
Orientation Correction
When creating the dynamic perspective prototypes, we wondered what would happen if you rotated the object in the opposite direction, such that it appeared to maintain it's original position even when you tilted your device.
We've applied this technique to a small example web page for the purposes of illustration. This also shows the power of 3D transforms in CSS3 - rotation is applied to an element (here, the) and inherited by everything contained within as you'd expect.
Conclusion
Device orientation and motion (via the deviceorientation and devicemotion events) are little-used but increasingly relevant in web development for a mobile-first world. There are relatively few real-world applications, but great potential for gestural input that doesn't require your free hand.
We've shown simple examples inspired by the Amazon Fire Phone demos that used multiple cameras to achieve similar results. Our next demo will show a practical application - remote mobile control of a more sophisticated model, rendered with 3D Javascript library three.js.
Mobile Device Orientation
The deviceorientation and devicemotion events have been supported by the default iOS and Android browser for some time, and are now supported on the other major mobile browsers too. This creates opportunities for new types of interaction and this post acts as an introduction to those getting started with mobile orientation events, including some practical experience.
What's Available
Most mobile devices incorporate a gyroscope, accelerometer and compass that, between them, can report the orientation and movement of the device in three dimensions.
The deviceorientation event returns an object with three properties: alpha, beta and gamma, corresponding to the device's rotation in degrees around the Z, X and Y axes respectively.
alpha is reported by the compass, for example 90° = west.
A device lying flat should report beta and gamma values of zero.
From this position, tilting the device's screen towards the right increases gamma towards 90° when it is fully pointing right. Tilting left decreases gamma towards to -90° when it is fully pointing left.
Similarly, tilting the device forward towards you increases beta towards 90° (when the screen is vertical and facing you), tilting it away decreases beta towards -90° (when it is upside down and facing away).
The devicemotion event returns information about the device's acceleration and rotation rate in x, y and z.
Device and Browser support
Gyroscopes, compasses and accelerometers are standard in modern mobile devices and, in principal, the deviceorientation event is well supported by current mobile browsers.
In practice though, we have found wild variation in the values returned to the browser by different device-browser combinations.
What's Really Available
The w3c documentation on these events is fairly prescriptive, but leaves a few loopholes for browser implementations. For example:
The event should fire whenever a significant change in orientation occurs. The definition of a significant change in this context is left to the implementation.
Although a threshhold of 1 degree is recommended, tests in Chrome on an HTC One M8 running Android 4.4.2 currently only seem to return alpha and gamma to a granularity of 90 degrees, and return NaN (Not a Number) for beta! Testing in the default Android browser on the same device gave expected values, however they appear to update only once per second.
Our initial testing was with a range of Samsung Galaxy devices using Chrome - these gave values as expected and updated values very frequently.
The exception to all of this is the alpha value reported by the compass - on most devices we tested it seems to be unreliable and erratic, though we found it reported reliably by a Galaxy S3 mini.
You can test your device-browser combo using our handy checker (a QR code link is shown for convenience if you view on a non-mobile browser).
Gotchas
You might expect that rotating your device fully around a single axis would produce values increasing smoothly from 0 to 360° then wrapping round. However, this is not the case:
- beta has a range of -180° to 180°
- gamma has a range of only -90° to 90°
Additionally, for example, tilting your device towards upright vertical increases beta, but as it approaches 90°, gamma can vary wildly across its whole range. This is partly due to ambiguity in the coordinate system, since certain orientations can be achieved by different sets of rotations, but the device and browser can introduce their own issues.
There is a bug logged for Chromium that highlights all of this quite well.
This is related to the phenomenon of Gimbal lock and the w3c documentation explains mapping the values to avoid such issues (for example using Quaternions).
Conclusions
Device orientation can supplement other means of input in an increasingly mobile-first world, and is not just a gimmick - consider that it offers various degrees of control without the use of your free hand or other external input.
However, variability in implementation on specific devices and browsers means that it can't be relied on as the sole way of controlling some interaction. For example, we're working on a prototype for mobile remote control where arrow buttons are displayed for control when device orientation is not being reported.
A post on that application is imminent, for now you can also read about mimicking the Amazon Fire Phone's dynamic perspective feature using device orientation.