Deluged by spam

[SteveRiley]

Today I started receiving several spammy emails. Here's the first one:



You'll note that it appears to be from me! But a quick look at the headers reveals that the sender address was forged, a common spam tactic:

Return-Path: <bristlesgesu195@dilos.com>
Delivered-To: s...@rileyz.net
Received: from localhost (localhost [127.0.0.1])
	by m92p.rileyz.local (Postfix) with ESMTP id EF39D16A23CE
	for <s...@rileyz.net>; Wed,  6 Nov 2013 14:30:47 -0800 (PST)
X-Virus-Scanned: Debian amavisd-new at m92p.rileyz.local
Received: from m92p.rileyz.local ([127.0.0.1])
	by localhost (m92p.rileyz.local [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id SbP0JHn5Ks3J for <s...@rileyz.net>;
	Wed,  6 Nov 2013 14:30:43 -0800 (PST)
Received: from mail-23-ewr.dyndns.com (mxout-042-ewr.mailhop.org [216.146.33.42])
	by m92p.rileyz.local (Postfix) with ESMTP id 4857816A234B
	for <s...@rileyz.net>; Wed,  6 Nov 2013 14:30:41 -0800 (PST)
X-Mail-Handler: MailHop by DynDNS
X-Originating-IP: 200.68.68.1
Received: from mx.brihetyasociados.com (unknown [200.68.68.1])
	by mail-23-ewr.dyndns.com (Postfix) with ESMTP id AD88C42C9E
	for <s...@rileyz.net>; Wed,  6 Nov 2013 22:30:40 +0000 (UTC)
Received: from 200.68.68.1(helo=rileyz.net)
	by rileyz.net with esmtpa (Exim 4.69)
	(envelope-from )
	id 1MMQOP-3037wz-X8
	for <s...@rileyz.net>; Wed, 6 Nov 2013 19:30:39 -0300
From: <s...@rileyz.net>
To: <s...@rileyz.net>
Subject: We talked for a long time, and you went and said goodbye! 
Date: Wed, 6 Nov 2013 19:30:39 -0300
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Mailer: achwdk-67
Message-ID: <4306586829.RN4QO2PB368394@qkcdqlqqgkolk.foaiojn.ua>

Hi, I'm cute girl 
I dream to have sex with strangers, my name is Lisa.
Can we get started? I'm on this dating site - come in to me!
http://bangmedia.go2cloud.org/aff_ad?campaign_id=1544&aff_id=548

23 of these have appeared in my inbox today. The subject will vary, but it's usually some plea to talk more, or concern that I forgot the person, or regret that I have lost my love. The person's name varies from among a half dozen or so. The URL is exactly the same, every time. You'll note that in this example, the sending server EHLO'ed using my own domain; this isn't always the case.

Upon examining my mail logs, I was shocked to see that this crap has been hitting me nonstop since 28 October!

root@m92p:/var/log# [B]ll mail.log*[/B]
-rw-r----- 1 root adm 1139901 Nov  7 01:20 mail.log
-rw-r----- 1 root adm 1451517 Nov  3 06:25 mail.log.1
-rw-r----- 1 root adm  166014 Oct 27 06:25 mail.log.2.gz
-rw-r----- 1 root adm  126534 Oct 20 06:30 mail.log.3.gz
-rw-r----- 1 root adm  130798 Oct 13 06:35 mail.log.4.gz

root@m92p:/var/log# [B]zcat mail.log.2.gz | grep reject | wc -l[/B]
0

root@m92p:/var/log# [B]grep reject mail.log.1 | wc -l[/B]
404

root@m92p:/var/log# [B]grep reject mail.log | wc -l[/B]
557

I built this server on 6 October. The logs are small, as you'd expect on a server used by only two people. Coincidentally, after the log rolled over on 28 October, that's when the onslaught started. The first arrived at 2:18:

Oct 28 02:18:36 m92p postfix/smtpd[31082]: NOQUEUE: reject: RCPT from mxout-030-ewr.mailhop.org[216.146.33.30]: 450 4.1.8 <iu@bhjygrp.org>: Sender address rejected: Domain not found; from=<iu@bhjygrp.org> to=<s...@rileyz.net> proto=ESMTP helo=<mail-12-ewr.dyndns.com>

My log is overflowing with these; 961 so far. Some botnet is going mad, sending zillions of these to who knows how many people, mostly using invented domain names. Decent Postfix configurations will block those. Of the few that get through, their domain names are valid.

In the last few hours, Spamassassin has begun scoring them, and the score is increasing...apparently the various online block lists are figuring out whatever pattern exists here. The most recent four have been scored thusly:

X-Spam-Status: No, score=2.284 tagged_above=2 required=6.31
	tests=[RCVD_IN_DNSWL_NONE=-0.0001, T_SURBL_MULTI1=0.01,
	T_URIBL_BLACK_OVERLAP=0.01, URIBL_BLACK=1.775, URIBL_JP_SURBL=0.336,
	URIBL_WS_SURBL=0.1533]

X-Spam-Status: No, score=3.529 tagged_above=2 required=6.31
	tests=[RCVD_IN_BL_SPAMCOP_NET=1.246, RCVD_IN_DNSWL_NONE=-0.0001,
	SPF_PASS=-0.001, T_SURBL_MULTI1=0.01, T_URIBL_BLACK_OVERLAP=0.01,
	URIBL_BLACK=1.775, URIBL_JP_SURBL=0.336, URIBL_WS_SURBL=0.1533]

X-Spam-Status: No, score=5.328 tagged_above=2 required=6.31
	tests=[RCVD_IN_DNSWL_NONE=-0.0001, SPF_FAIL=0.919,
	TO_EQ_FM_DOM_SPF_FAIL=0.001, TO_EQ_FM_SPF_FAIL=2.123,
	T_SURBL_MULTI1=0.01, T_URIBL_BLACK_OVERLAP=0.01,
	UNPARSEABLE_RELAY=0.001, URIBL_BLACK=1.775, URIBL_JP_SURBL=0.336,
	URIBL_WS_SURBL=0.1533]

X-Spam-Status: No, score=2.284 tagged_above=2 required=6.31
	tests=[RCVD_IN_DNSWL_NONE=-0.0001, T_SURBL_MULTI1=0.01,
	T_URIBL_BLACK_OVERLAP=0.01, URIBL_BLACK=1.775, URIBL_JP_SURBL=0.336,
	URIBL_WS_SURBL=0.1533]

Fun times...

[GreyGeek]

Somebody in Capital Federal, Brazil is playing games with you.
I had similar msgs for a week or two late this last summer and marked them as spam in gmail. They quit coming.
I post comments using disqus on several sites and I'm guessing that someone screen scraped my email address or hacked the user list.

[Feathers McGraw]

Spam email is in incredibly annoying, yet I find it fascinating.

How exactly did they make the email appear to be from you, without having to authorise?

I noticed someone had been using my email server to spam people recently. Luckily, the volume of emails sent was pretty low. I found three spam accounts that were created in the short space of time it took me to set up Citadel. I think I must have been hit by a script in that vulnerable window :/ so unlucky (or was it? There must be so many scripts out there).

Feathers

[SteveRiley]

How exactly did they make the email appear to be from you, without having to authorise?

Because SMTP has no built-in mechanism for preventing spoofed envelopes (what SMTP uses to route email) or for preventing spoofed message headers (what you see in the From:, To:, Date:, Subject:, header in your email program). See http://tools.ietf.org/html/rfc4021#section-2.1.2 for the official explanation.

I noticed someone had been using my email server to spam people recently. Luckily, the volume of emails sent was pretty low. I found three spam accounts that were created in the short space of time it took me to set up Citadel. I think I must have been hit by a script in that vulnerable window :/ so unlucky (or was it? There must be so many scripts out there).

Whoa, wait...you found new accounts created in your Citadel user database?

[Feathers McGraw]

Thanks for that link!

Yeah, they were created just after citadel was set up, and just before I created my own accounts. Very disturbing.

One called "backup", one called "ftp daemon", and one called "mail", all of which appear to have been created from an IP address in Germany.

Only "backup" has been used, first it sent out my IP and the usernames it had created to some addresses, bordered by some random hashes of letters and numbers, then (only recently) it started spamming.

Have dropped the permissions on the accounts right down, and put a password on them to prevent login (there was no password before). Didn't want to delete them yet because it would destroy the evidence!

Feathers

[SteveRiley]

I ran Citadel briefly for a few weeks a couple years ago when I was first learning email. Never did I see anything like that. Without the ability to get my own hands on your machine and really take a close look, I'm reluctant to immediately assign blame to Citadel.

Holy crapoli, though. If I were you, I wouldn't trust that machine anymore. I'd pave and rebuild. Think about it -- something that you don't know has managed to worm its way into your machine, create mail accounts, and start sending spam. That's really bad, dude. Your machine is currently in an unpredictable state.

I, uh, did a short experiment. My typing is the bold:

steve@t520:~$ [B]telnet samhobbs.co.uk 80[/B]
Trying 195.166.151.235...
Connected to samhobbs.co.uk.
Escape character is '^]'.
[B]get / http/1.1
host: samhobbs.co.uk[/B]

HTTP/1.0 301 Moved Permanently
Date: Mon, 11 Nov 2013 08:05:06 GMT
Server: Apache
X-Powered-By: PHP/5.4.4-14+deb7u5
X-Pingback: http://www.samhobbs.co.uk/wordpress/xmlrpc.php
Location: http://www.samhobbs.co.uk/
Vary: Accept-Encoding
Content-Length: 0
Connection: close
Content-Type: text/html; charset=UTF-8

Connection closed by foreign host.

So you're running WordPress. I hope you keep your WP updated -- it's known for being somewhat prone to various vulnerabilities.

---

Also...

steve@t520:~$ [B]nmap samhobbs.co.uk[/B]

Starting Nmap 6.40 ( http://nmap.org ) at 2013-11-10 15:56 PST
Nmap scan report for samhobbs.co.uk (195.166.151.235)
Host is up (0.18s latency).
rDNS record for 195.166.151.235: samhobbs.plus.com
Not shown: 988 closed ports
PORT     STATE    SERVICE
25/tcp   filtered smtp
80/tcp   open     http
135/tcp  filtered msrpc
139/tcp  filtered netbios-ssn
143/tcp  open     imap
443/tcp  open     https
445/tcp  filtered microsoft-ds
465/tcp  open     smtps
587/tcp  open     submission
993/tcp  open     imaps
2222/tcp open     EtherNet/IP-1
5222/tcp open     xmpp-client

Nmap done: 1 IP address (1 host up) scanned in 39.88 seconds

You probably shouldn't be opening unencrypted IMAP (on 143/tcp) to the world. If you ever happen to log in to your IMAP server over that socket on the Internet, you will have transmitted your password in clear text. On my network, only IMAPS (on 993/tcp) is allowed inbound. In fact, I don't run Dovecot on 143/tcp at all. So, if you can, configure Citadel not to use IMAP (only IMAPS) and remove the rule for 143/tcp in your router.

---

Also also...

steve@t520:~$ [B]telnet samhobbs.co.uk 2222[/B]
Trying 195.166.151.235...
Connected to samhobbs.co.uk.
Escape character is '^]'.
SSH-2.0-OpenSSH_6.0p1 Debian-4
^]
telnet> quit
Connection closed.

Hope you're using a sufficiently complex password such that's resistant to dictionary or rainbow table attacks...

---

Also also also...fear not. These experiments are only simple probes. I'm not a bad guy

[Feathers McGraw]

Thanks Steve,

It’s worth its weight in gold to have someone who actually knows what they’re doing take a look at this kind of thing.

In order:

RE compromising the whole system: Citadel is using its own inbuilt authentication mechanism, so managing to create an account in Citadel does not equal obtaining a system user account. Should I still be worried about the rest of the system? Being a Pi, it’s not the end of the world if I have to rip it apart and build it again.

RE wordpress: I keep it running the newest version (I check regularly, and they’ve actually recently brought out an auto update feature, which is nice).

RE 143/tcp, it’s open because I’ve been using STARTTLS… I take your point about accidentally sending login in plaintext, my clients machines are set up to use STARTTLS always, but perhaps it isn’t particularly sensible. Will look into switching to 993 (by the sound of things I may be reinstalling/configuring anyway).

RE port 2222: SSH only permits publickey login. I’m surprised it didn’t close the connection immediately (try ssh foo@samhobbs.co.uk).

Feathers

[Snowhog]

I agree. Given the 'unknown' total state of affairs on your Pi, I'd nuke it and start anew. You've learned quite a bit, so it will allow you to apply that knowledge with the result being a better, more secure, POP on the Internet.

[Feathers McGraw]

IMAP

But yes, I think you're right. Repetition is a useful way to make sure you _really_ understand things. I find that the first attempt is just muddling through.

May try postfix + dovecot + squirrelmail again this time.

Should be fun (read: incredibly frustrating, then incredibly satisfying)

Feathers

[SteveRiley]

RE compromising the whole system: Citadel is using its own inbuilt authentication mechanism, so managing to create an account in Citadel does not equal obtaining a system user account.

Regardless, an external agent compromised something and created accounts. Perhaps through some obscure vulnerability in Citadel's web server, I'm really not sure, as I didn't keep it installed long enough to learn about it.

Should I still be worried about the rest of the system? Being a Pi, it’s not the end of the world if I have to rip it apart and build it again.

Yes, worry.

RE 143/tcp, it’s open because I’ve been using STARTTLS… I take your point about accidentally sending login in plaintext, my clients machines are set up to use STARTTLS always, but perhaps it isn’t particularly sensible. Will look into switching to 993 (by the sound of things I may be reinstalling/configuring anyway).

There's an adjacent lesson to learn here. Don't make it easier for your or anyone else to make mistakes. The majority of attacks succeed because of configuration vulnerabilities.

RE port 2222: SSH only permits publickey login. I’m surprised it didn’t close the connection immediately (try ssh foo@samhobbs.co.uk).

That's because telnet just opens a socket and relays data back and forth. I hadn't actually tried to establish an SSH session, thus OpenSSH didn't know whether to refuse me or not.

May try postfix + dovecot + squirrelmail again this time.

Should be fun (read: incredibly frustrating, then incredibly satisfying)

Yep, that was exactly my sequence, too. Tell you what, though, you'll learn more during all that than you ever will pick up with something like Citadel. A good understanding of Postfix and Dovecot can be reusable in other situations. For example, you can answer questions on Linux forums

[Feathers McGraw]

Fair enough. It's neater to only have Apache running anyway, I originally tried to set it up Citadel to run on Apache, but it wasn't having any of it (or I was being stupid)!

I do feel sorry for the Citadel devs, try searching for Citadel - it's used by so many other companies that they're very unlikely to turn up in search results. Not sure if that's bad luck or poor decision making. Finding support isn't easy for the same reason.

I'm currently re-building.

When email programs search for settings, what are they doing? Are they trying different configurations in a pre-determined order, or is there a file that email servers present to clients as a guide?

Is there a method similar to disabling password authorisation for SSH and accepting publickey login only, but for FTP? I don't have a WAN facing FTP server, but even so I'd rather not have an FTP server using normal passwords. I did see that SFTP sends the login and password encrypted, which is better but not quite what I'm after.

Yeah, I did get that impression. I learned quite a bit just from trying Postfix + Dovecot + Squirrelmail the first time round, shouldn't have given up so easily!

Feathers

[oshunluvr]

Can't one connect ftp through an ssh tunnel?

[Feathers McGraw]

I think the confusion here (mine, at least) is SFTP vs FTP over SSL, which are apparently not the same thing!

[Teunis]

Good stuff to read about.
I've never dabbled in building my own server, mainly because I have a more than excellent ISP (xs4all.nl) but it has always intrigued me so one day I might use this knowledge.

Because my ISP also has options for sFTP, I use Filezilla to connect to the shell server and get redirected to my account.

http://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol

At the moment I mainly use WebDav via Dolphin to access my space, rsync is also an option.

About various mail servers, the ISP has both Squirelmail and Roundcube available and are actively developing and contributing, the last two years mainly to the latter and since a couple of weeks it's looking really good.

The thing I would have most use for is a private and home based shell or proxy server but I understand that's one of the hardest things to protect...

To come back to Steve's original posting, they also deliver good spam filtering (presently DNSBL + Cloudmark), I've been publicly using my mail address since 1997 and hardly ever see spam outside the dedicated folder nor false-positives.

Oh yes:

teunis@13-10-W520:~$ telnet shell.xs4all.nl 80
Trying 194.109.21.8...
Connected to shell.xs4all.nl.
Escape character is '^]'.
SSH-2.0-OpenSSH_5.5p1 Debian-6+squeeze4

[Feathers McGraw]

The thing I would have most use for is a private and home based shell or proxy server but I understand that's one of the hardest things to protect...

Tell me about it! I had an unfortunate encounter with Apache's proxy module due to a stupid configuration error. I'm sure you're not as retarded as I am but it kind of highlights how easy is is to mess this kind of thing up, especially if you're just blundering your way through! Wrote about it here:

http://www.samhobbs.co.uk/2013/10/be...he2-mod_proxy/

Steve has mentioned Roundcube before, sounds like a nice program.

Feathers