Deluged by spam

[Feathers McGraw]

I think I'd be the same - although spam is really annoying, it's also interesting!

[SteveRiley]

BTW... I'm thinking of rebuilding, again. Remember how I gushed over iRedMail? Well, turns out the thing is pretty rigid. You can't use updated versions of the major packages if they aren't in your distribution's "stable" repository. So I'm stuck on Postfix 2.9 and Apache 2.2. I also think that it's just too heavy -- Cluebringer (Postfix policy configuration) is largely useless for me, and Amavis is overkill. I only want Spamassassin, whereas Amavis acts as a wrapper around Spamassassin and ClamAV. I am not concerned about viruses in emails these days.

So... I'll start with a VM first. Debian 7 bumped up to unstable. Then, since I'm starting over, I'll use Nginx rather than Apache, and Exim rather than Postfix. I'll also switch to MySQL, because PostgreSQL is, alas, just not as widely supported. I'm rather curious about DBMail, too; this will be the replacement for Dovecot.

[Feathers McGraw]

Are there any features you're missing with your current setup that you could have with more recent versions of those packages? Or are you just curious?

I found that when I was setting up Postfix and Dovecot there were quite a few parameters that weren't supported because the version in the repos is fairly "old".

For example, all of my sending restrictions have to go in smtpd_recipient_restrictions instead of smtpd_relay_restrictions, apparently they added the relay restrictions in newer versions so you can't accidentally create a permissive policy.

Sounds interesting, let me know how it goes, I'd like to appreciate the differences between those new programs and postfix/dovecot/squirrelmail!

I feel like I have a pretty good understanding of those now, just finished writing a tutorial for setting them up on a Pi because I couldn't find one when I set out

Feathers

[SteveRiley]

I like torturing myself learning new things. And I like updated packages.

I spent all day yesterday trying to wrap my brain around Nginx. Yuck. Nginx began life as a reverse proxy program, and over time has morphed into a web server. Its configuration is entirely around URLs/URIs, not files. It seems to prefer virtual sites in the pattern application.example.com; to make this work via SSL you either need a certificate per app or a wildcard certificate for the domain. Both these choices are expensive. I much prefer the form example.com/application. I could not grok how to get Nginx to behave this way when application is a symlink in the server's root directory to a real location outside the root. The resulting pages were wonky.

Of course, at least part of my problem might be that I'm also subjecting myself to learning a different distro. I want to build this server once and just keep smoothly upgrading it, without ever rebuilding again. Thus, a rolling-release distro is logical, and I decided on Arch. But I don't think that's a contributor, as I've used Arch off and on and feel comfortable with it to a certain degree.

I uninstalled Nginx and replaced it with Lighttpd. Ran into some weird errors that I traced down to a typo in the Arch wiki, which I fixed. Once I got that sorted, Lightthpd started working flawlessly.

[Feathers McGraw]

That's interesting, I wasn't sure whether to try Nginx or Apache2 first when I was setting out, I'm glad I learned to use Apache now! Isn't nginx supposed to be more lightweight, or was that Lighttpd? I seem to remember that was the main benefit of not using Apache for a RasPi.

I'm actually the opposite, I prefer to have different services on their own subdomains, there's less chance of a conflict that way. Although it's unlikely to happen, if I was using the /service format and I created a page in Wordpress called "squirrelmail" then Apache would kick up a fuss. Seems neater to split things up by subdomain.

Why are wildcard certificates expensive? Do companies charge more for a wildcard certificate than a normal one, or are you talking about buying a wildcard cert in addition to your already existing normal cert?

I use a wildcard cert myself, for *.samhobbs.co.uk, since www.samhobbs.co.uk matches this too. I rewrite samhobbs.co.uk to www.samhobbs.co.uk, and this way everything can use the same cert.

I've had a few typo errors myself (usually my typing). I really like the detailed warnings Apache2 gives you if you mess something up like this, it makes it really easy to find the error and fix it.

Feathers

[SteveRiley]

Isn't nginx supposed to be more lightweight, or was that Lighttpd? I seem to remember that was the main benefit of not using Apache for a RasPi.

Lighttpd seems to use the least amount of RAM. Apache 2.4 has some new modes, none of which I'm familiar with yet. Even though it's been out for quite a while, most distributions have stuck with 2.2.

Why are wildcard certificates expensive? Do companies charge more for a wildcard certificate than a normal one, or are you talking about buying a wildcard cert in addition to your already existing normal cert?

When I went shopping for an SSL certificate, the price for wildcard domain validation was two to three times as much as simple domain validation. So I bought the simple.

I've had a few typo errors myself (usually my typing). I really like the detailed warnings Apache2 gives you if you mess something up like this, it makes it really easy to find the error and fix it.

Nginx's logging is terse. And so is its documentation. Lighttpd has the easiest config syntax and is reasonably well documented. But it isn't updated much -- a new version came out in September, the previous one was a year ago. So I dunno...I'm not really a fan of using old or infrequently-updated software (*), so I may end up going back to Apache and seeing what's up with 2.4.

(*) Yes, I know coreutils are older than dirt. I guess I don't care about that!

[Feathers McGraw]

What's the reasoning behind the additional expense? Is it more work to verify that you own all the subdomains or something?

Or are they just milking it? Lol.

A spam related question:

I have pages and pages of this in my logs:

Dec 22 18:51:27 samhobbs postfix/smtpd[18054]: connect from unknown[218.59.209.140]
Dec 22 18:51:27 samhobbs postfix/smtpd[18054]: lost connection after UNKNOWN from unknown[218.59.209.140]
Dec 22 18:51:27 samhobbs postfix/smtpd[18054]: disconnect from unknown[218.59.209.140]

I'm guessing this is a badly configured spam bot, as the HAM is getting through just fine. And the IP address is in China, of course...

I'm just curious as to what the "lost connection after UNKNOWN from unknown" means - does this mean it tried to send a command that wasn't valid SMTP, or is it something more specific?

Is there a way to block repeated connection attempts of this type? It seems to be one connection per second for 5-15 minutes, which has to be wasting loads of resources.

Feathers

[SteveRiley]

What's the reasoning behind the additional expense? Is it more work to verify that you own all the subdomains or something? Or are they just milking it? Lol.

Domain validation is minimal trust -- it's how these CAs do their "SSL in minutes" thing. You enter the domain in question during the registration process. The CA sends emails to {postmaster|hostmaster|webmaster|admin} @example.com (the standard list). I've configured "postmaster" on my domain. In the email is a link to click; once you do that, they sign your CSR and make your certificate available for download.

There are no additional steps for a wildcard domain validation certificate; after all, you're simply validating the domain. The CAs charge more for wildcard certs just because they're, yep, milking it

A spam related question. I have pages and pages of this in my logs: ... I'm guessing this is a badly configured spam bot, as the HAM is getting through just fine. And the IP address is in China, of course... . I'm just curious as to what the "lost connection after UNKNOWN from unknown" means - does this mean it tried to send a command that wasn't valid SMTP, or is it something more specific?

"connect from unknown[218.59.209.140]" means that the DNS reverse lookup on the IP address failed (try it: dig -x 208.59.209.140). "lost connection after UNKNOWN" is a clue that a badly configured spambot is knocking on your door. It's trying something completely weird that Postfix doesn't understand.

Is there a way to block repeated connection attempts of this type? It seems to be one connection per second for 5-15 minutes, which has to be wasting loads of resources.

Yes: learn about fail2ban.

[Feathers McGraw]

Domain validation is minimal trust -- it's how these CAs do their "SSL in minutes" thing. You enter the domain in question during the registration process. The CA sends emails to {postmaster|hostmaster|webmaster|admin} @example.com (the standard list). I've configured "postmaster" on my domain. In the email is a link to click; once you do that, they sign your CSR and make your certificate available for download.

There are no additional steps for a wildcard domain validation certificate; after all, you're simply validating the domain. The CAs charge more for wildcard certs just because they're, yep, milking it

I remember that process from CAcert, I assumed the process was more stringent for paid certs...

Since the level of validation/security is the same, why don't all GNU/Linux distributions include CAcert's root certificates by default? It can't be much effort to do, and there's no reason for free software distributions to buy into the racket.

As an aside, how many of those email addresses have to be real for standards compliance? I've configured all of them already, as well as root, just curious.

"connect from unknown[218.59.209.140]" means that the DNS reverse lookup on the IP address failed (try it: dig -x 208.59.209.140). "lost connection after UNKNOWN" is a clue that a badly configured spambot is knocking on your door. It's trying something completely weird that Postfix doesn't understand.

I didn't realise postfix ALWAYS does a rDNS lookup, thought it would only bother when you've told it to block senders with no PTR record. I'm especially glad I got my PTR record sorted now:

feathers-mcgraw@62-West-Wallaby-Street:~$ dig samhobbs.co.uk

; <<>> DiG 9.9.2-P1 <<>> samhobbs.co.uk
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46541
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;samhobbs.co.uk.                        IN      A

;; ANSWER SECTION:
samhobbs.co.uk.         3600    IN      A       195.166.151.235

;; Query time: 60 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
;; WHEN: Tue Dec 24 00:22:53 2013
;; MSG SIZE  rcvd: 48

feathers-mcgraw@62-West-Wallaby-Street:~$ dig -x 195.166.151.235

; <<>> DiG 9.9.2-P1 <<>> -x 195.166.151.235
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27491
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;235.151.166.195.in-addr.arpa.  IN      PTR

;; ANSWER SECTION:
235.151.166.195.in-addr.arpa. 43200 IN  PTR     samhobbs.co.uk.

;; Query time: 52 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
;; WHEN: Tue Dec 24 00:23:05 2013
;; MSG SIZE  rcvd: 74

Yes: learn about fail2ban.

Thanks, will have a read

[SteveRiley]

I remember that process from CAcert, I assumed the process was more stringent for paid certs...Since the level of validation/security is the same, why don't all GNU/Linux distributions include CAcert's root certificates by default? It can't be much effort to do, and there's no reason for free software distributions to buy into the racket.

The only thing CAcert offers is domain validation certificates. Other CAs offer organization validation, which involves further steps to prove your identity to the CA. Because the other CAs have the infrastructure to support stronger validation and permit periodic auditing, their roots are baked into nearly all operating systems. These CAs are selling simpler domain validated certs as a way to make a little extra money.

CAcert, being a community-based CA, lacks the funding to pay for the stronger security infrastructure that high validation requires. Several Linux distributions include the CAcert root -- which, being community-based themselves, makes a certain amount of sense. You won't find CAcert's root in anything from Apple, Android, Mozilla (that's right, they don't trust CAcert); Debian may drop it, too.

As an aside, how many of those email addresses have to be real for standards compliance? I've configured all of them already, as well as root, just curious.

I dunno. I only have "postmaster" because that's what IRedMail sets up automatically.

[Feathers McGraw]

The only thing CAcert offers is domain validation certificates. Other CAs offer organization validation, which involves further steps to prove your identity to the CA. Because the other CAs have the infrastructure to support stronger validation and permit periodic auditing, their roots are baked into nearly all operating systems. These CAs are selling simpler domain validated certs as a way to make a little extra money.

Ah, that'll be why CAcert strips some information from the cert before they sign it then!

CAcert, being a community-based CA, lacks the funding to pay for the stronger security infrastructure that high validation requires. Several Linux distributions include the CAcert root -- which, being community-based themselves, makes a certain amount of sense. You won't find CAcert's root in anything from Apple, Android, Mozilla (that's right, they don't trust CAcert); Debian may drop it, too.

I have found Mozilla's approach a little bit odd. I use Firefox for Android and Thunderbird on Kubuntu, and neither of these respects any changes you make to the OS's own certs - they have their own cert database that you have to edit separately.

I wish there was a way to make them use the OS' certs, but I haven't found one.

I can see why they might have done it - if you're just a user on a computer and you don't have admin privileges then their way could be pretty handy...for me, though, it's just a pain.

Feathers

[Feathers McGraw]

This is pretty relevant to this thread, so I won't bother making a whole new thread about it!

I've managed to get Spamassassin up and running. I've set the bar to be marked at spam pretty low (2.5) and some ham is being marked.

I'll have to do some training. Was thinking of adding a CRON job (once a week or similar) to learn based on which folder I put stuff in (i.e. if it's in the spam folder and I've left it there, then it really is spam! If it's in the inbox and I haven't spammed it then it's ham.)

Spam is marked with a header like this:

[***** SPAM 2.6 *****] Dovecot Tutorial

If possible, I'd like to have it delivered to the right folder (spam or inbox) based on its header before I see it. Any ideas? I've done some searching and it seems that Spamassassin will only mark the emails in a certain way, it leaves actually delivering that email to other programs (currently sendmail). Here's the relevant line from /etc/postfix/master.cf:

spamassassin    unix  -       n       n       -       -       pipe user=debian-spamd argv=/usr/bin/spamc -f -e /usr/sbin/sendmail -oi -f ${sender} ${recipient}

Feathers

[Snowhog]

See ManualWhitelist - SpamAssassin

You should be able to adapt this to other than 'users'.

[Feathers McGraw]

Snowhog, that link isn't working for me

[SteveRiley]

If possible, I'd like to have it delivered to the right folder (spam or inbox) based on its header before I see it. Any ideas? I've done some searching and it seems that Spamassassin will only mark the emails in a certain way, it leaves actually delivering that email to other programs (currently sendmail).

Postfix calling Sendmail? That's clunky. Maildrop or Procmail are probably better for this, but they are royal PITAs to set up. I would recommend simply creating a Sieve rule in your mail client to do this. That way, Dovecot remains in charge of delivery, and the Sieve rule, which actually runs on the server, moves the mail for you.