Implementing an interactive wall

Creativity is challenging, especially when it comes to actually implementing it. At Lama, we knew we can achieve great results only by pushing our limits. We’ve let our creativity free in order to get the most of our capabilities, and a little more.

The Concept

The idea was to create an interactive wall, with a fluid or particles flowing on a wall(screen/project) passing through a vertically centered smartphone. The flowing speed would be controlled by swiping up and down on the smartphone. As a result, it should create an impressive and pleasant result of controlling the flow of particles.

Here is a quick sketch of how we imagined the interactive wall:

Breaking down the concept

We’ll have a 3D generated video for the background, with a fluid flowing up/down. Then, we would have an abstract infinite list an Android smartphone. A laptop on same local network would create a socket server and allow the client(smartphone) to connect. The client would send the speed and direction for the video. The background video’s playback speed would range from 1x to 30x, forward or backward.

Defining the risks

The Android App

Creating an infinitely scrolling list should not be a problem. Then, connecting to a socket server and passing in the list data should be straightforward as well.

The only unknown was: how do we define what is a 1x to 30x scrolling on the phone?

We can of course set the minimum scrolling speed to 1x and the maximum to 30x. But would the linear mapping of values produce a pleasant effect? We can have it linear to start with, then we can calibrate when the wall’s video is moving.

The Desktop App

This would be the application which plays the video backward and forward at 1x to 30x speed. Based on requirements, the app should have two independent videos. Each video would take half of the screen. So we should support doing the work smoothly on two videos simultaneously.

I’ve researched possible implementations considering the given time and cost. My main concerns were:

• how does an optimized fast-forwarding work?
• how does backwards playback work?
• what would be the optimal video encoding to support both forwards and backwards playback?

Let’s take it one by one and then we’ll see where it gets us.

How does fast-forwarding work?

At first, you would think it simply means to play the video faster. In reality, that would be a bit too much work if done so. At 1x playback video plays 25 frames per second (fps). At 2x, you’d expect to play 50 fps right? Then what about 3x? 75 fps? Kind of. But processing all those frames within a second, would most probably lag the playback.

Do we really need to process every frame? What if we keep playing at 25fps, but we only display/process every 2nd frame? That would result in 2x playback. Same would apply for 10x – display only every 10th frame.

How does backwards playback work?

Playing videos backwards is like reading text through a mirror. It’s there, but takes some effort to read it properly. The main reason it’s harder to playback backwards is because videos are meant to be played forward by default. I will need the video to be put in a way that it can be easily played backwards.

What would be the optimal video encoding to support both forwards and backwards playback?

Video encodings optimize frames using interframe compression. Instead of storing the whole image,

• research
• try concepts on multiple platforms
• choose the right technology to get the optimal time/cost
• distribute the work
• plan on how to actually implement the solution.

Initial Solutions:

• Play 2 videos at same time, forward and reversed videos, then use JavaFX to synchronise them
• Same as 1, but with vlcj
• Process video frame by frame (only mjpeg video encoding)
• What on earth is QTJ(QuickTimeJava)?

The AHA moment

I received an email with a draft of the background video. While previewing the video in the browser, I grabbed the seek bar and saw it actually was previewing very smoothly all of the frames back and forth. Then I tried playing the video forward and backwards with 2x to 10x the speed and it did it so well.

It was the QuickTime from Apple within the browser. Then I found out it was based on AVPlayer. I got Xcode installed and started working in Objective-C. It’s kind of readable.

I got one of my friends to help me. Until 4 AM we got one video working, but when we tried to play the video faster than 2x it was lagging. Took me until afternoon to find out QuickTime doesn’t use the built-in playback speed functionality. It simply uses a feature which skips frames.

The solution: we had to use the right player functionality which would allow us to play at 10x at 25fps by displaying only every 10th frame.

I’ve put two vertical videos together and opened two socket connections. Got the Android App which had the infinite list ready, connected it to the sockets and voila! The scrolling works!

Later on it was used for another event and I’d say it looked pretty damn great.