Dholdrums

So a couple of years ago, I got tired of all the brute force ssh attacks on my servers.

I wasn't particularly worried about anybody getting in: on most of my boxes, there are only one or two allowed logins, and I restrict who can log in using the AllowUser parameter in /etc/ssh/sshd_config. There are also other safeguards in place.

Still, if someone has unsuccessfully tried 30 times to log in as "root" on my box, I'd like to avoid having them come into contact with any of my hosts, so enter SSHGuard.

SSHGuard, along with its more well known brethren, DenyHosts and Fail2ban, all work roughly the same way. They monitor your logs, and if they see some predetermined number of failed logins, they block the ip.

The devil is in the details. They all vary on what kind of logs they can parse. They all can parse ssh logs, but some can also handle Apache logs, mail logs, ftp logs etc. They all seem to be able to talk to iptables directly, but others can also understand pf, ipfw, etc.

For a couple of years, I used DenyHosts on my FreeBSD box. It was a pain to setup, and as I found a few months ago, didn't work properly. DenyHosts parsed auth.log well enough, the problem was that it didn't speak pf. So in order to get it to work with pf, I had to write wrapper scripts. The interface was easy enough, you give a path to a script to add an IP and another path to remove an IP. The scripts (de)populated some file and would call pf to reload the table. The following would be the relevant pf rules:

table <denyhosts> persist file "/var/db/denyhosts/pftable"

block in log quick on $EXT_NIC from <denyhosts>

This all would have been well and good until I recently noticed that my blacklist file was growing at an ever steady rate, and included duplicates. Basically, DenyHosts kept adding the same IPs, and never took any out. There were easily thousands of entries.

For a while I thought it was a problem with my scripts, and I debugged the hell out of it (not easy when the only way to test it is to wait for someone to brute force your ssh server) and finally realized that it was some bug in DenyHosts. I probably should have filed a bug, or brought it to the attention of the developers, but by that time, I was pretty disgusted with everything, so I just turned it off altogether

A few months ago, someone was asking on the FreeBSD mailing list about how to secure their server and someone suggested SSHGuard. I don't know how I didn't find out about this project earlier. It's been around since 2007 at least.

SSHGuard has the following features that I liked:

• parses many types logs, not just ssh
• can talk to pf directly
• has a port (security/sshguard-pf) which works out of the box with little to no configuration
• written in C, so no dependencies on scripts demanding the interpreter (other programs seem to eat a lot of resources)
• didn't I mention that the needed configuration is tiny?

Here's what I did to get it working:

• installed the port
• added the following pf rules:

table <sshguard> persist

block in log quick on $EXT_NIC proto tcp from <sshguard> \
to any port 22 label "ssh bruteforce"

• added the following to /etc/rc.conf

sshguard_enable="yes"
sshguard_watch_logs="/var/log/auth.log"

• created the file /usr/local/etc/sshguard.whitelist and put in my whitelist entries (SSHGuard is really flexible about the format of this file)

And that was pretty much it.

My one minor gripe is that SSHGuard's blacklist DB file is not easily parsable, so I'm stuck to dumping the pf table (pfctl -t sshguard -T show) to look at blacklisted entries. Not a big deal.

And that was it. SSHGuard has worked perfectly for me. I will soon change the pf rule to block blacklisted IPs access to everything on my hosts, not just port 22.

So it took a few months, but I finally got around to converting my DNS slave. It took about an hour to play around with NSD and figure out how to make it read from a bind master. And the actual cutover was less than five minutes.

Probably the part I like best is how clean the config file is. Take a look at this:

server:
        ip-address: 0.0.0.0
        do-ip4: yes
        do-ip6: no
        logfile: "/var/log/nsd.log"

remote-control:
        control-enable: yes

key:
        name: MYKEY
        algorithm: hmac-md5
        secret: "my+secret+in+base64="

zone:
        name: example.org
        zonefile: /var/db/nsd/example.org
        request-xfr: 1.2.3.4 MYKEY

It's insane how easy this was. Sometime soon I will update my master and then I'll be done with bind for good.

In the What's New page for the upcoming FreeBSD 10, I see that the venerable (infamous?) bind is being replaced with LDNS and unbound, I found the related project NSD, also created by Stichting NLnet Labs.

This is good news. I've always been looking for a bind replacement because it is so large and unwieldy.

Late last year I started researching NSD to replace my bind server, and I've more or less decided I would switch over to it in 2014. But interestingly NSD doesn't have the thing I wanted the most: a clean zone file format. The config files are all sane now, but NSD more or less keeps BIND's zone file format. This page has some examples. Contrast this with DJB's tinydns which has an excellent zone file format, really short and succinct and hard to type in wrong.

Nonetheless, a couple of the primary ROOT servers are run using NSD, so that's very good to hear.

Will report back on my NSD adventures after I switch over.

A while back, someone asked if anyone had successfully taught themselves Unix. (It was in the context of systems administration, so I'm assuming the question from a Unix sysadmin POV.)

An interesting question.

I was reflecting back on a few years of Unix experience and decided to put pen to paper (so to speak) and jot down what it would take to teach yourself Unix.

(NB: the question of whether or not I have been successful at it is hard to answer. I suppose you would have to ask my colleagues or supervisors that. But in any case, my observations follow.)

One of my University professors taught a Unix sysadmin course for grad students. I was an undergrad at the time, and did not take it, but followed it closely. The material didn't seem very hard, and I wish I had been able to take it. I don't think the fact that it was a grad-level course was what held me back at the time, but more likely a lack of funds. But I think more than coursework or access to books, the primary factor in being able to teach yourself Unix is a hunger to do so. If you are not interested in the subject, then no amount of classes you take will help you. If you have a sincere desire to learn the subject, then lack of materials will not hinder you.

Almost all my computer-related college courses had to do with Unix in some shape or form. If you paid proper attention, they Unix part of it was soaked into your bones via osmosis.

But that was many years ago.

There are a lot more formal courses now on Unix and Unix systems administration nowadays. Certainly there are more books and online materials. So the young person who is looking to enter this field certainly has more resources to access that they can take advantage of.

If you're like me, you use some local ISP for your primary, non-work, Internet connection. Since I'm in NYC, I use Verizon's FIOS service. Although it is pricey, the speeds I get are decent (by American standards).

However, Verizon does one thing that they do that I absolutely hate: THEY HIJACK YOUR DNS.

What is DNS hijacking? Wikipedia has a fairly good entry on the subject.

Basically, if you make an invalid DNS request, instead of returning an error (NXDOMAIN) to your browser as required by the Internet standards, they instead route the page to one of their own pages, usually to display advertising. This breaks all sorts of things, these are described in gory detail in the article above—but such a blatant violation of an Internet RFCs is enough to justify an outcry.

Presuming you use some sort of Unix box, there's a couple of solutions for this:

• You can change the permissions flags on /etc/resolv.conf to be immutable (chflags on FreeBSD, chattr on Linux), although I consider this to be a bit heavy handed.
• You can tell your DHCP client to not update resolv.conf (via dhclient.conf)
• But I think the easiest is way to solve this problem is to override the settings on your wireless router to send hardcoded DNS settings to DHCP clients instead of accepting the ones sent by the ISP. The added advantage here is that it works for all operating systems and clients in your subnet.

Which DNS servers to use instead? Google offers a service called Public DNS that won't hijack your DNS:

• 8.8.8.8
• 8.8.4.4

Although you should keep in mind that Google can and will use your name lookup data for advertising purposes.

Lastly, this article assumes this is for a home or personal network. If you are running a business or other organization, you really should be using your own DNS servers.

Sometimes people report a huge size discrepancy between the output of du and df. There are two main reasons for this:

• open file descriptors
• mounted filesystems hiding unmounted filesystems

With regards to open file descriptors, check that any processes that are running are not holding open files. Shut down Apache, or Oracle, or MySQL, or Postgres, etc. That should usually do it.

The other thing you can check is to see that mounted filesystems are not masking (or hiding unmounted filesystems). Unmount all filesystems and check again, you will see what has been hiding behind the mounted FS.