Telepresence robot interface
“Excuse me, Mr. Wozniak?”
“Oh, hey. Nice virtual presence device.”
“Thank you. I just wanna say I'm a big fan. One of my proudest possessions is a vintage 1977 Apple II. Despite the file system limitations of Apple DOS 3.3... it was a pretty nifty little achievement.”
“Thanks, we were shooting for nifty.”
“You know, if you had it here, I'd autograph it for you.”
“Don't move for 15 to 30 minutes... depending on how the busses are running.”
“Nerds.”
(Big Bang Theory, S04EP02)
***
The internet solved the problem of communication long ago, and now it is time to tackle remote presence.
A telepresence robot is a little screen on wheels that drives around the office and transmits the image and voice of, say, a CEO who has a good reason not to attend the all-hands meeting in person.
Today it is a rather extravagant gadget. The robots are far from mass-produced, and prices range from a thousand to several tens of thousands of dollars. They come in all kinds of shapes, from a clumsy box to an elegant stalk on a highly mobile base.
Telepresence robots: Giraffe, RP-7i, VGO, TiLR, Texai, Anybots QA, Double Robotics, Anybots QB, Sparky Jr.
This article, however, is not so much about how the robots look, which is the job of industrial designers, as about the interface people use to drive these rovers.
It came as a revelation to the author that the interface of a toy costing several thousand dollars could be such a sorry sight. The interfaces of the Gostai Jazz and Botiful robots:
So when a Russian company decided to give its robot a good interface, it was a great opportunity to make the product stand out.
Control analysis
The robot’s design lets it move in any direction: left, right, forward, backward and along both diagonals. It can also raise and lower its head.
Not all movements are equally useful. There are basic moves without which control would be incomplete and from which all other actions can be composed. There are four of them:
- Forward
- Backward
- Turn left
- Turn right
The remaining moves are:
- Move to the left
- Move to the right
- Turnaround
They can be performed as combinations of the basic moves. There are also two special actions that deserve a category of their own:
- Raising the head
- Lowering the head
Obviously, without the four basic actions, the robot can’t be controlled properly, so they deserve the most attention in the interface. Without the two head actions, the robot can still be driven, but it is not much fun, since what you want to look at is often above or below eye level. These commands also deserve a special place.
As for the three additional commands, they add little: experience shows that sideways movement, or strafing, is convenient only in Quake 3.
Basic control
The robot is controlled from a web browser on a desktop or laptop. This means we have a keyboard and a pointing device: a mouse, a laptop touchpad or a graphics tablet.
The first obvious association appears right away: basic actions 1–4 map naturally to the arrow keys. This association is probably so good that even a beginner will figure out how to drive the robot on the first try.
Head control is a bit trickier. Raising and lowering the head is also naturally associated with the up ↑ and down ↓ keys, but they are already taken, and there are no other convenient, self-explanatory keys nearby.
After the keyboard, our eyes turn to the mouse. Using the mouse to move the head is not a bad idea; moreover, the mouse can rotate the camera in any direction, not just up and down.
But let’s not rush. If we simply hand head control over to the mouse, we have to take turning off the left ← and right → arrows, or else mouse movements may conflict with key presses. With mouse control, those keys should not turn the robot but move it sideways. However, we agreed that these actions are not basic and should not get attention until the main operations are in place.
Keyboard-and-mouse control is certainly possible and is used successfully in computer games, but with the WASD keys rather than the arrows. The arrows sit too close to the right edge of the keyboard, so you would have to shove the keyboard across the desk to avoid stretching your left arm.
Next, the mouse wheel comes to mind. What if scrolling up raises the head and scrolling down lowers it? Not the best idea either: scrolling speed varies a lot between systems, and some mice (the Apple Magic Mouse) are too sensitive and often scroll by accident.
In fact, the only more or less acceptable mouse solution is an on-screen slider that can be dragged up and down with the left mouse button, like a scroll bar.
But this approach has plenty of drawbacks too. The user will often have to move their hand from the keyboard to the mouse to adjust the tilt, which slows interaction considerably. So here is the plan: we keep the slider as the most obvious method for beginners and add an advanced keyboard method on top.
Our goal is to drive the robot with one hand, so we won’t involve the other hand or assign two more keys. Instead, we will use what we rejected at the start, the already taken up ↑ and down ↓ keys, and add the Ctrl modifier.
With Ctrl held down, the up ↑ and down ↓ keys tilt the head; without Ctrl, they still drive the robot. It is simple: one-handed operation, the fastest interaction, and the modifier closest to the arrows on all popular keyboards (Cmd and Control on the small and large Apple keyboards). True, you can’t drive forward and move the head at the same time, but that is actually a good thing.
So we have fitted six main actions onto four keys plus one modifier. Now let’s deal with the additional operations.
Additional control
As mentioned above, sideways movement is usually mapped to A and D. The whole set of additional actions therefore fits neatly onto WASD: W moves forward, S backward, A left and D right. With this set, you can also move diagonally: forward-right, back-right, back-left and forward-left.
This approach requires full camera control, not just raising and lowering the head. The mouse, which we rejected for the basic actions, is the most sensible way to provide it. For advanced control, the mouse is a plus: at the first stage the goal was to make the interface as simple and obvious as possible, and now the goal is advanced control.
WASD plus mouse is the standard control scheme in 3D games, with one difference: in a game, controls respond instantly, with almost no lag between the user’s action and the game’s response. A robot is controlled over the internet, and on a poor connection the commands alone can take several seconds to arrive. The second factor is the robot’s mechanics, which have their own lag. The robot doesn’t react to the forward command instantly: it has to start its wheels and get out of the resting state.
The lag itself is not a problem and won’t break the robot, but a lack of activity on the screen may confuse the user: has the system frozen? That is why feedback is needed, including for mouse control.
It is also worth adding robot status indicators, such as Wi-Fi signal strength, battery level and time left in the control session.
The controls should also be shown on the screen, with a mouse alternative to the keyboard, so that the user can confidently start driving right away without reading a manual.
And this is where our analysis turns into actual design.
Designing the controls
To show the main association of the interface, the link between the four basic actions and the arrow keys, we display the keys on the screen as an inverted T in the lower right corner.
The look of the keys matters right away. The controls must be clearly visible on any background, since in real life the background could be a white wall or someone’s black jacket. At the same time, they should be light and unobtrusive. So we use an old trick: the inside of the elements is white, the outlines are black, and the arrows get a subtle white shadow.
Adding head tilt with the Ctrl or Cmd key.
This doesn’t look obvious. The user will surely press Ctrl while exploring the interface, but they need to see that something has changed and that the key has to be held down. So we add feedback: while Ctrl is held, the up ↑ and down ↓ arrows switch to camera control mode.
Unfortunately, there is no universally understood camera control icon that could go on the keys. An eye icon turns the keys into a face. Curved arrows look too odd. After much agonizing, we settle on the simplest symbol: a dot.
Now, when the user presses Ctrl, the dots appear and the up key ↑ lights up for a second. This should read as an invitation to press Ctrl and ↑ together, which raises the camera. This way, users learn the interface on their own.
Complete beginners are not forgotten either: they get a camera tilt slider that moves together with Ctrl + ↑↓.
By the way, note the lovely head control icons made especially for this project.
The main controls are done. WASD control won’t be shown on the screen at all, or we would end up with a pile of buttons. For feedback, the same main keys can be highlighted.
Mouse feedback can take the form of a wheel (or joystick) that appears when the mouse button is pressed and disappears when it is released.
The cursor, shaped like a round pin, moves freely inside the wheel but can’t leave it. The closer the cursor gets to the edge, the faster the camera turns. Moving up, right, down and left respectively raises the head, turns the robot right, lowers the head and turns the robot left. This allows a full 360-degree view. In essence, the wheel is a substitute for the analog stick of a game controller.
And the turnaround, obviously, goes on the space bar.
Designing other elements
The robot’s interface is, of course, not limited to driving. There are auxiliary elements as well: volume control, pause, help, screen size and indicators. The most important indicators are:
- Signal strength. The robot is controlled over Wi-Fi or GSM, and wireless signals tend to drop sharply even within a single room. So the signal level should be monitored and the user informed in time. The signal icon should be universally recognizable.
- Battery charge. The robot is powered by batteries, and if it is not recharged in time, it will shut down. Therefore, when the battery is low, the user should be notified.
- Session time. Control of the robot is allocated in time slots, so sooner or later the user’s session ends, and they should be notified about it.
The standard place for indicators is the top center of the screen. Since there is plenty of room, we will use icons with labels.
All of these indicators are event-driven: they should be shown only when needed. The signal icon is really a weak-signal icon, the battery level matters only when the battery is running low, and the same goes for the time left in the session.
An icon should appear with an animation, such as three blinks. In special cases, color coding can be used: for example, at 10% battery, the icon turns yellow.
As for auxiliary actions, the following come to mind:
- Pause, which freezes video and audio in both directions as well as all control. Hard to replace.
- End session. Practically useless, since closing the browser does the same.
- Turn off the camera. This seems useful, but it contradicts the very idea of the robot: with the camera off, there is no telepresence, so the button is not needed.
- Mute the microphone. Unlike the camera, muting the microphone doesn’t contradict the idea of the robot. You don’t have to speak at a meeting; you can just listen, and you certainly shouldn’t bring household noise (a crying child, a phone conversation, a drill) into the office.
- Turn off incoming video and sound from the robot. Useless. We suggest using your hands to cover your eyes and plug your ears.
The upper left corner is the perfect place for the pause and mute buttons.
By the way, here is an age-old question: should the mute button be plain or crossed out? On the one hand, a crossed-out microphone indicates a state: the microphone is off. On the other hand, it indicates an action: turn the microphone off.
The lower left corner can hold the video zoom and help buttons, and the upper right corner a self-view showing the operator’s face.
The big picture
So it is time to finish with the details and look at the whole picture. Here is the result.
A few words about a competitor
After this interface was finished, the author learned about Double Robotics. This elegant robot stands out from the rest both in appearance:
And in its control interface.
The interface is strikingly simple. Its authors seem to have managed to throw out all the extra functions.
In fact, that is not quite true. The Double Robotics robot is currently controlled only from an iPad, which means that the mute buttons, pause and volume control are on the device itself and don’t need to be in the interface.
Camera control is simpler too: all the controls are on the screen, and the robot is driven with both hands at once, so questions of interaction speed and input devices simply don’t arise.
An excellent, if obvious, solution is the continuous control pad, which lets you slide your finger across the screen without lifting it when you need to change direction.
Unfortunately, none of these techniques are available on an ordinary desktop computer.