For Kala Petronijevic, an 11-year-old blind tennis player and super fan in Australia, watching tennis was never easy. "My Dad usually has to commentate for me, and he finds it tiring sometimes because I can't see who is serving or the ball or the points".
This is a problem facing 43 million people living with blindness and 295 million people living with moderate-to-severe visual impairment. So, Tim Devine, executive director at AKQA, alongside Monash University and Tennis Australia, invented what's now known as Action Audio.
Action Audio is now part of the Australian Open Radio, available online, for games being played on centre court. Action Audio allows listeners and fans to experience tennis like never before, using bells and blips to give those with a visual impairment an idea of where the ball goes and how close it is to being out.
Using spatial audio developed by Dolby Atmos with Apple, fans can experience bells and blips from the right ear to the left ear individually. The closer the ball is to the court perimeter, the more blips you will hear. As the ball is hit and bouncing around the court, you will hear a series of bells. A high pitched sound indicates a forehand, a low pitch sound indicates a backhand. Commentary is also available to relay every bit of information alongside the blips and bells.
Tim Devine explained his reasons for developing the innovative piece of technology. “When a low vision person is experiencing tennis, it was through squeaks when the tennis ball hits the ground and on the racket, and potentially the net, and a few grunts, and that’s about it really, there was a bit of information in this radio, which is good, but it takes the agency away from the people to make their own decisions and appraisals about what’s happening”.
Hawkeye
Mr Devine admits that tennis is one of the better sports to test the ActionAudio audio beds on, as tennis already relied on technology like Hawkeye, measuring how fast a serve is and various other stats taken throughout a game. Add to that the hush of the crowd when there’s action followed by the roar after a dramatic ending.
The actual tech specs to create a whole system, according to Devine, was more straightforward than you’d imagine, “because it’s quite low dimensions and by that I mean, there’s only a couple of people and one ball and you know, some lines, and a net, and that’s about it. And so, it’s also tracked really well. So for years now, probably decades, I’d say there’s been a technology invention, and the one provider which is Hawkeye, and they’re able to track where the ball is, the players are in high-resolution 3D.
“Then we took that information and turned it into sounds, abstract sounds that kind of augments the existing audio experience. So we try and make it sound as if you would in an augmented reality. And in a visual sense, this is augmented reality in an audio set. So, it’s kind of trying to put it into the sound space in a convincing way. So it just feels normal and feels part of it. So there’s a lot of ways that we did that in terms of using different reverbs and sounds to kind of make the sounds feel like there happened in that space to make it really feel natural”.
What helps is how accessible the technology is these days. Devine explained how the datafication of sport alongside technology advancements in AirPods and Apple Music can be beneficial in making sport more accessible: “There’s the spatialised audio aspect of things that are being driven by VR (virtual reality) and AR (augmented reality) but also increasingly in music with Apple.
“They have spatialised audio and that Dolby Atmos sort of experience with the AirPods. That’s increasingly becoming a thing, which is awesome because it really gives you more perspective. In that sense, you’re able to really position things.
“We’re really excited about this project because it became something that can kind of ride the coattails of quantification of sports, or the kind of Datafication of sports and the acceleration of the kind of audio 3D experiences. So in that sense, we’re really excited, because when the job is to work with blind and low vision to ask them, what’s interesting in sport and how do we get the experience”.
Devine admits with all his experience in technology and all his innovations, which included a project with Microsoft called sense kit - a way of exploring the concept of sensory substitution, which is feeling vibrations on the skin, and then eventually learning how to understand what they actually represent - this is an experience he’ll remember for a long time.
Profound
He said: “It was one of the most profound projects that I’ve worked on because the feedback was so emotional, and there were some really fun things like, I was asked why are you doing this, and it’s because we actively did this, we actively invested in doing this proactively, because we really thought there was some value in this.
“We came up with lots of design principles, and one of those was that sport is social. It’s hard to be social when you’re not able to make your own appraisal of the game. It’s hard to be social when, if you are low vision, you need to sit really close to the screen away from everybody else who’s on the couch, and you’re kind of blocking the view, and the conversation is not as easy.
“That way, people relied on others to explain what was going on. So in that sense, it kind of levels the experience for everyone. And so there’s been an awesome response from that perspective, from the social and inclusivity perspective”.
Tim has constantly been innovating to help society be better and do better, and while Tennis Australia has been an incredible partner, he hopes other sporting organisations will come on board. Not only are sporting organisations important, but broadcasters and technology companies can use this really minor tweak to make a huge impact.
For visually impaired fans like Kala, the impact is enormous. “With Action Audio, I don’t know why, but I can suddenly see the ball. I can actually hear everything, and I can actually see the ball. I don’t have to ask my Dad, ‘hey can you commentate I can’t see where the ball is going.’”
