Problem Solved: God Is Watching

Leo Burnett and Digitas Taiwan developed the “GOD-IS-WATCHING Behavior Recognition Detector”, an AI-powered device that detects rubbish dumpers and projects a holographic God, reminding people to dispose of their waste correctly. Within three months, it garnered so much attention that illegal waste dropped by 73%. BaoXing Council is now looking to roll out this technology to more neighborhoods in New Taipei City in the future.

 

  

This is a Leo Burnett Taiwan-initiated CSR project that explores how to make our community better, supported by Digitas to design and implement the technology. When we first approached BaoXing Council, they were a little sceptical as they’ve never implemented anything like this before. But proper waste disposal is a challenge, not just in Taiwan but in many countries. They were happy to give it a try.

  

When solving problems, Taiwanese people like to use humour and playfulness rather than being too serious or heavy-handed. We had the idea of using motion sensors and AI deep learning to detect garbage dumpers: AI as the brain, and surveillance camera as the eyes. As soon as random dumping behaviour is identified, this would activate our holographic God, reminding people to dispose of their waste correctly.

 

To build a model which recognises different garbage dumping behaviours, we went on YouTube and used news clips and CCTV footage to help us to train the model. Sadly, it was very easy to find footage of people dumping their garbage. We started teaching our AI model with 5,000 video clips training sets containing annotations of thousands of objects and visual relationships, then fine-tuned the model only to detect "throwing garbage" actions.

Here's one example.

  

There are hundreds of Gods in Taoism, but the God that’s manifested by the GOD-IS-WATCHING Behavior Recognition Detector is modelled on Tudigong, God of Land. Tudigong is respected like a family elder in Taiwan, making him the perfect God, stern yet friendly, to remind people not to dump their garbage illegally.

 

A hologram is basically a high-speed rotating LED fan. When nothing is shown, you see straight through, but if something is played, then you see an image appear from nowhere.

 

This video, captured in our office, shows how it works – it is the first version of Tudigong. The camera’s capture rate is 60 frames per second, that’s why you see the black lines, but in an actual environment to our naked eyes, you will see the perfect image.

 

We had to analyse videos captured from our camera in real time to record human movements and/or objects through computer vision. Then we analysed this footage for spatial-temporal changes in space and time, using AI to interpret each action as text, such as "a man is throwing garbage". For example, a series of stills might reveal someone holding something in their hand (“object”), before bending their knees (“lowering body position”), and the next second, the object has disappeared – this is the first trigger. We can then roll back to analyse if there was an object on the ground two seconds ago.

 

It is almost like a before and after comparison, or “spot the difference” game that we play, but everything has to be done in real-time, within seconds of occurring. The speed of processing and analysing was critical to our success and also the most challenging part. Thanks to 5G (high-speed transmission) and cloud computing (ultra-fast processing), this has become possible and affordable nowadays.

  

Obviously, AI is the hot topic right now. Still, this project is actually more like the evolution of our own technical maturity, as back in 2020, when Covid-19 first hit us, we worked on a Face Mask detection machine that uses computer vision to detect if a person is wearing their mask correctly. Holograms weren’t new to us either, so by knowing how different technologies operate, we’ve become much more comfortable with piecing multiple technologies together to solve a problem.

 

Hence, we simply started by looking at what we did before and how can we do it this time, and we found a good combination of computer vision and holograms.

 

  

Training the model to recognise rubbish dumping behaviour was interesting. We thought we’d have to go to different areas, video record ourselves and pretend to be dumping rubbish in order to get the footage we needed to start the learning process, but one day, one of our team members suddenly saw a video of rubbish dumping on YouTube, and that’s when we realised how big of an issue this is, and not only in Taiwan but a global issue! There are literally thousands and thousands of CCTV videos, news reports or simply people recording on their mobile. So, while this is a good thing, as we had tons of videos to train our model with, it is also quite a sad thing that shows this issue is much more serious than we first thought.

 

What’s really exciting about the GOD-IS-WATCHING Behaviour Recognition Detector is the potential for creativity to improve local neighbourhoods – that’s something we’d love to see agencies do more of in future.

  

We have a team of creative hybrids at Leo Burnett Taiwan who experiment with AR, machine learning, artificial intelligence and all kinds of emerging tools, and they work closely with our digital transformation specialists at Digitas.

  

The recognition model was custom-built, and despite various tests in the office, we wouldn’t know if it would accurately detect rubbish dumping in an uncontrolled environment until we installed it. The most critical part is the lead time between capturing the video and processing through the model, then returning the result and deciding whether to play the Tudigong video or not. Bearing in mind that people are unlikely to stick around after dumping their rubbish, this whole process needs to occur within a second. We solved this by applying a predictive model behind the recognition system. Instead of waiting for the rubbish to be dumped, we calculate the probability that such behaviour is likely to be rubbish dumping behaviour. This will create errors causing confusion. Hence the creative needs to be carefully designed, but over time the model will get better and better at predicting.

 

  

After two months of planning and testing the GOD-IS-WATCHING Behavior Recognition Detector, Leo Burnett and Digitas Taiwan launched a pilot program. It was a big success, and we hope that our GOD-IS-WATCHING Behavior Recognition Detector will return to Taiwanese neighbourhoods later this year – please keep an eye out for Tudigong.

  

As we were installing the device in the neighbourhood, there were things we couldn’t test in the office; for example, car traffic!

Although we have all visited the site before, a number of cars driving by in a narrow street still surprised us; the signboard of a convenience store opposite the location was so bright at night that it affected our camera, so we had to adjust the exposure rate of our camera, the angle, and the height. Any “major” adjustments also meant we needed to make changes to our model etc, and this put a lot of pressure on our engineers.

  

The cool thing about machine learning is that there is no finish line, and you can always improve the accuracy of the model. So, aside from the original training materials (from YouTube), we now also have real videos from the street that helps us to refine the model.

  

When we shared this project with our friends and colleagues, many would say this is technology overkill, using AI, holograms, machine learning and computer vision for a “minor thing” like rubbish dumping. Still, if we apply this to a brand’s marketing campaign, then everything suddenly feels reasonable and interesting. I think there is this gap between the marketing world and the real world, but the truth is, there shouldn’t be, and we should try to solve real-world issues with the same level of creativity, innovation and passion.

 

When residents walked past and saw this, they pretty much all said, “This is such a good idea, and Tudigong is cute!” Many people felt it was an innovative way to solve an old issue and a humorous and unaggressive way to tell those people off.

 

As for concerns, a few people told us that if this activated in the middle of the night, it would really scare that person, but that’s good! He will learn the lesson!