When doorbells go rogue!

Throughout 2020, the researchers at the Spamhaus Project observed swathes of residential and small enterprise IPs emitting an avalanche of spam, listing them on the eXploits Blocklist (XBL). The only similarity was a specific port 25 spam connection. But what was the source? Alex Grosjean, one of the Project’s researchers, shares an incredible story of how one retired man and his doorbell, along with a heap of patience, helped uncover the mystery behind these listings.

Ding Dong! Spam Delivery!

It was spring in 2020. After a long two and a half weeks, and an intense flurry of daily emails, I finally got the email I was waiting for. “Alex!! I found it, I found the source of the spam! It was MY DOORBELL!” He could not believe it, and I was dumbfounded. A doorbell?! Yes, a doorbell, and then as though a dam had broken, the information began flooding in.

The spamming sources were a doorbell, a firestick, a CCTV system, a phone, a tablet, a browser toolbar, a laptop. The causes were a ringtone, a channel unlocker, free streaming, a shopping app, a game. The operating system was Android, whose liberal permissions policy allows third-party apps to be easily installed. The vector was a gigantic and exponentially expanding “residential proxy network,” and altogether, this was a perfect ecosystem for spammers.

Looking for a needle in a haystack

Around the end of 2019, we saw a startling uptick in the number of IP addresses being listed due to spamming with a specific type of port 25 spam connection. As a result, we were suddenly adding a ton of residential, small business, and Carrier Grade NAT (CGNAT) IPs to our blocklists that we had not previously seen. We saw close to a million of these connections every day! Many people were opening tickets to request additional information about why they could not send an email. Most of them were retirees, small business owners, or people newly working from home, thanks to Covid-19. All of them were frustrated, some of them were angry, and some were also very determined.

At the time, though, we had little specific information regarding the cause of the problem, so all we could do was tell people to limit their outbound port 25 to mail servers and to run malware/virus scans. Surprisingly, although limiting port 25 was effective, a minute number of people found any malware source that could account for the mysterious spamming behavior we saw – until I got that revelatory email.

Amateur detective “Mr Shore” investigates

Mr. Shore – the gentleman who sent me that email – lives in the UK, is 75, told me in no uncertain terms he was “non-technical and not up on all this new stuff!” and that he urgently needed his email to run his local Rotary Club. He was initially really hostile – and who can blame him? All of a sudden, he couldn’t send an email, and the only explanation he got was the rejection code in the bounces. His ISP could not help. Those codes referred him to “Spamhaus,” and we told him a story that sounded like science fiction to him.

However, after some discussion, he decided that he believed me and became utterly determined that he would find the source of the problem, his lack of technical skills be damned. The usual malware scans turned up nothing, so he started by unplugging every single thing in the house that was able to connect to the internet, and over the next two weeks, he’d reconnect one item, email me to remove his IP, and then we’d wait to see if his public IP got listed again. This very slow process of elimination finally resulted in the “ITS MY DOORBELL!” email – he had an Android-based smart doorbell that his son had installed a month before. After he disconnected it, the spam detections stopped for good. Of course, then he wanted answers. How could this happen, and why was it allowed?

Truth is often stranger than fiction

This pattern began to repeat itself, revealing more devices and more apps. Finally, the picture became clear: the spam was primarily issuing from Android devices that had third-party apps that included a specific software development kit (SDK). We found this SDK in many different apps, including streaming media, games, shopping apps, a custom ring tone for a Samsung phone, and free VPNs. One such app that could be installed on an Android digital media player was an extremely popular pirate streaming app called Mobdro (which is now apparently permanently offline). Mobdro was the origin of many Spamhaus detections, and upon closer inspection of its license agreements, we found the final clue.

SDKs are often marketed as a way to monetize apps. It turned out that an unscrupulous company had the idea of creating an SDK that included a proxy and burying the relevant part of the End User License Agreement (EULA) two or three EULAs deep, so no one would notice that they were actively consenting to “share their idle resources” in return for “free” features such as no ads. They were correct, of course: no one reads EULAs, especially if they are installing an app that will be used for illegal purposes! And so, all the Mobdro users became part of a “residential proxy network” with more than 70 million IPs available in it. This was then enthusiastically and prolifically abused by spammers, who cheerfully paid for access to it because it was a gold mine.

Access to port 25 + a pandemic = a perfect storm

Many networks still allow their users unrestricted access to port 25 – even on carrier-grade NAT IPs shared by thousands of people. Additionally, some ISPs continue to ship routers to end-users with port 25 open by default. Android has permissive policies for installing 3rd party apps. Many small businesses don’t have the knowledge or resources to employ good network security. Together, these lax policies allow residential proxy networks to thrive, abusing the network resources of millions of people without any thought or consideration as to how they or their businesses may be affected.

As the pandemic took hold in 2020, all of the above was amplified, forcing businesses and people to work remotely without preparation or precautions. Suddenly, people were working from home and connecting their insecure personal devices and networks to their jobs. The spam from those devices started flowing out of their work email server, which then got blocked. This caused big problems for small businesses struggling to survive, and the sheer volume of people at home in lockdown perpetuated the issue – what do people stuck at home do? Sign up for free VPNs, stream media, play games, and go online shopping!

As our investigations progressed, people started asking questions about what else that carefully hidden little bit of code could have been doing, and the answer is… we don’t know. All Spamhaus saw was the spam. Spam can be a mere nuisance, or it can be maliciously laden with malware. The proxy that had been slipped behind people’s firewalls seems like a very tempting proposition for miscreants who intend to do more than send basic “pills” spam. On its face, this bears a lot of similarity to how many types of malware spreads. Once the malware is safely inside the network, it starts finding additional vulnerabilities and exploits them. Who is to say that criminals do not use proxy networks in a similar way?

This is ridiculous! How do I keep myself safe?

It would be wonderful if Android made installing such third-party apps harder, but in the meantime, the best advice we can offer is not to install them. You should ONLY install apps from the official stores. Network operators could do a lot to help mitigate the damage by limiting outbound port 25 to email server access only. When it comes to the use of CGNATs, this is especially true, where one person’s compromised Android device can affect the daily business of thousands of other people who are sharing those IPs. ISPs could start shipping routers without port 25 access, and they could provide clear, accessible documentation available for home users on how to close outbound port 25 on existing routers.

It would also be worth considering the legality and ethics of creating and using such a network. It is a reasonable assumption that most people whose resources are being used in this way are not aware of it, and if they were, they would terminate the agreement. ISPs don’t appreciate being abused in this way either, but these proxy networks are so spread out that the effect is rarely localized enough to create the kind of problems that spark change. Getting the kind of global level of cooperation required to put a stop to it is unlikely since there are much more imminent threats that grab the focus.

These proxy networks are nominally legal, to be sure – after all, people do click “agree” on the EULAs even though they have no idea what the agreement says, but what they are then used for is often wholly illegal. It’s a very clever and utterly ethically bankrupt business model, and it is all about the money, in the end – the modern, digital way of making “money for nothing” and then laughing all the way to the bank.

There oughta be a law.