Posted by Thomas

Pelican, Good for websites too

I recently decided to publicly mention a small project I've been hacking away on.

It was more or less in the I'm just hacking on some code state for a while. And then in the past year, I cleaned it up a bit, added an interface, and made it (barely) functional -- if a bit feature-incomplete.

So I wanted to show it to other people, and what better way to do that than at a lightning talk at a conference?

But I wanted to have more than a GitHub page to point to, and I really didn't want to spend the time to author up a fancy website (although I can understand some HTML/CSS, I'm not a web designer by any means), so I decided to give Pelican a shot at this job.

Pelican, although mostly designed for blogs, also does a decent job at static web pages.

I got it up and running in about an hour or so. I spent most of that time finding a good theme which I liked.

I'm very happy with the results: recalc.org.

So my overall consensus is that if you need a fairly simple website, and like the [edit with text editor] and then [run make to see your output] way Pelican works, it's the tool for you.

Tagged , ,

Banner Ads in your Presentation? Not Cool Man.

I attended a conference a while back, and sat in on one of the main presentations.

It was a great talk, had great content, and the speaker was well known, having written a widely used piece of software.

Overall, I really enjoyed the talk and learned a good amount.

But he did one thing that I found extremely distasteful.

On the bottom of all his slides was a banner ad for his company, it took up a quarter of the screen.

Tacky.

SSHGuard

So a couple of years ago, I got tired of all the brute force ssh attacks on my servers.

I wasn't particularly worried about anybody getting in: on most of my boxes, there are only one or two allowed logins, and I restrict who can log in using the AllowUser parameter in /etc/ssh/sshd_config. There are also other safeguards in place.

Still, if someone has unsuccessfully tried 30 times to log in as "root" on my box, I'd like to avoid having them come into contact with any of my hosts, so enter SSHGuard.

SSHGuard, along with its more well known brethren, DenyHosts and Fail2ban, all work roughly the same way. They monitor your logs, and if they see some predetermined number of failed logins, they block the ip.

The devil is in the details. They all vary on what kind of logs they can parse. They all can parse ssh logs, but some can also handle Apache logs, mail logs, ftp logs etc. They all seem to be able to talk to iptables directly, but others can also understand pf, ipfw, etc.

For a couple of years, I used DenyHosts on my FreeBSD box. It was a pain to setup, and as I found a few months ago, didn't work properly. DenyHosts parsed auth.log well enough, the problem was that it didn't speak pf. So in order to get it to work with pf, I had to write wrapper scripts. The interface was easy enough, you give a path to a script to add an IP and another path to remove an IP. The scripts (de)populated some file and would call pf to reload the table. The following would be the relevant pf rules:

table <denyhosts> persist file "/var/db/denyhosts/pftable"

block in log quick on $EXT_NIC from <denyhosts>

This all would have been well and good until I recently noticed that my blacklist file was growing at an ever steady rate, and included duplicates. Basically, DenyHosts kept adding the same IPs, and never took any out. There were easily thousands of entries.

For a while I thought it was a problem with my scripts, and I debugged the hell out of it (not easy when the only way to test it is to wait for someone to brute force your ssh server) and finally realized that it was some bug in DenyHosts. I probably should have filed a bug, or brought it to the attention of the developers, but by that time, I was pretty disgusted with everything, so I just turned it off altogether

A few months ago, someone was asking on the FreeBSD mailing list about how to secure their server and someone suggested SSHGuard. I don't know how I didn't find out about this project earlier. It's been around since 2007 at least.

SSHGuard has the following features that I liked:

  • parses many types logs, not just ssh
  • can talk to pf directly
  • has a port (security/sshguard-pf) which works out of the box with little to no configuration
  • written in C, so no dependencies on scripts demanding the interpreter (other programs seem to eat a lot of resources)
  • didn't I mention that the needed configuration is tiny?

Here's what I did to get it working:

  • installed the port
  • added the following pf rules:
table <sshguard> persist

block in log quick on $EXT_NIC proto tcp from <sshguard> \
to any port 22 label "ssh bruteforce"
  • added the following to /etc/rc.conf
sshguard_enable="yes"
sshguard_watch_logs="/var/log/auth.log"
  • created the file /usr/local/etc/sshguard.whitelist and put in my whitelist entries (SSHGuard is really flexible about the format of this file)

And that was pretty much it.

My one minor gripe is that SSHGuard's blacklist DB file is not easily parsable, so I'm stuck to dumping the pf table (pfctl -t sshguard -T show) to look at blacklisted entries. Not a big deal.

And that was it. SSHGuard has worked perfectly for me. I will soon change the pf rule to block blacklisted IPs access to everything on my hosts, not just port 22.

Tagged , ,

moved to pelican

Finally done!

This blog is now statically generated by Pelican!

It was incredibly easy to move over. The WordPress XML import more or less worked without a hitch, and I'm very satisfied with the end result.

I will craft my own theme one of these days, but until then, one of the precooked ones is fine.

Tagged ,

moving to pelican

In the 3 years or so that this blog has been up, I've always been uneasy about it. A couple of reasons for this:

Firstly, Wordpress, the underlying platform, consists of a bunch of PHP scripts wrapped around a MySQL database. By far the most frequent security issues I've seen come sites running PHP. Drupal, Joomla, et al. are all easily hackable and WordPress is no exception. I update WordPress religiously on my system and I'm still afraid that someone is going to use some zero-day PHP exploit to hack me.

Less dangerously, but more annoyingly, editing articles for the blog requires that I log into my site local instance of WordPress and use the app to edit the article. This is not terribly hard, but just hard enough that I don't like doing it. Why can't I just fire up vi (startup time = ~300ms) and start writing? Hence in 3 years, I've written maybe 10 articles total. In fact, I've just played with Pelican for a bit and it was incredibly easy to add and edit articles. I'm now writing this post in WordPress and by comparison it is slow as molasses, and inelegant to type this into a web window.

I started looking at static site generators a while back. The first was Blosxom, which came out 12 years ago apparently. I first investigated it in 2004. It had promise back then, but was still rough around the edges.

I have since discovered Pelican, thanks to dotCommie. It's written in Python, uses Markdown/reStructured Text as input, and with the magic of templates (it uses Jinja), generates totally static content. Which is all I need. I've been playing around with it and like what I see so far. I will switch the entire site over when I find some free time.

Tagged ,

nsd is bloody awesome

So it took a few months, but I finally got around to converting my DNS slave. It took about an hour to play around with NSD and figure out how to make it read from a bind master. And the actual cutover was less than five minutes.

Probably the part I like best is how clean the config file is. Take a look at this:

server:
        ip-address: 0.0.0.0
        do-ip4: yes
        do-ip6: no
        logfile: "/var/log/nsd.log"

remote-control:
        control-enable: yes

key:
        name: MYKEY
        algorithm: hmac-md5
        secret: "my+secret+in+base64="

zone:
        name: example.org
        zonefile: /var/db/nsd/example.org
        request-xfr: 1.2.3.4 MYKEY

It's insane how easy this was. Sometime soon I will update my master and then I'll be done with bind for good.

Tagged , , ,

switching from bind

In the What's New page for the upcoming FreeBSD 10, I see that the venerable (infamous?) bind is being replaced with LDNS and unbound, I found the related project NSD, also created by Stichting NLnet Labs.

This is good news. I've always been looking for a bind replacement because it is so large and unwieldy.

Late last year I started researching NSD to replace my bind server, and I've more or less decided I would switch over to it in 2014. But interestingly NSD doesn't have the thing I wanted the most: a clean zone file format. The config files are all sane now, but NSD more or less keeps BIND's zone file format. This page has some examples. Contrast this with DJB's tinydns which has an excellent zone file format, really short and succinct and hard to type in wrong.

Nonetheless, a couple of the primary ROOT servers are run using NSD, so that's very good to hear.

Will report back on my NSD adventures after I switch over.

Tagged , , , ,

fat, sick, and nearly dead is awesome

Over the weekend I decided to watch some movies and went over into my long dormant Netflix streaming account. Some time earlier I had added some documentaries on veganism, but had never gotten around to watching them.

I ended up watching the documentary "Fat, Sick, and Nearly Dead". Without giving too much away, I want to say that this is one of the greatest documentaries I have ever seen. I immediately watched it again after watching it the first time. And today, I went ahead and showed it to a friend.

It's basically about an overweight Australian guy who decides to start eating (or drinking in this case) healthy, and the lengths he goes to achieve that. The real star of the show is someone he accidentally befriends in Winslow, Arizona during a cross country trip through the US. Do watch it, it is amazing.

I ordered a juicer. It is coming Wednesday. Do yourself a favor and watch this film. It will change your life.

A couple of postscripts:

  • the film has a website: www.fatsickandnearlydead.com and it can be seen for free on Hulu (albeit with commercials)
  • the juice fast portrayed in the film is apparently called "Rebooting" and there is a reboot website as well
  • I haven't bought into Juicing as the end-all, cure-all, I think a much better place to start is the book Eat To Live, authored by Dr. Joel Fuhrman (who is featured prominently in this film), but it's certainly a start
Tagged , , ,

tv makes you stupid^H^H^H unproductive

I discovered a productivity tip this week that I just had to share.

So recently I had some vacation time, and I was mostly using it to stay home and fix up my place and take care of some errands. A college friend of mine invited me down to her place in the Washington DC metro area, so I went down on Christmas day and crashed with her for a week.

Now I should mention that apart from two days when my friend was working, she had a really packed schedule for us. Movies, dinner parties, theater, restaurants, salsa dancing, etc. Not that I minded, I think we had a great time.

But towards the end of the week, I noticed and pointed out to her something very interesting: I got a LOT of stuff done when I was staying with her. And I mean a lot. I upgraded one of my servers, I finished a graphics design project for a charity I work with. I wrote some code. I read. And so on.

In fact, I was so much more productive during my vacation, that I got more work done than when I was actually at work.

And I was really puzzled by this.

Even my friend was surprised, and we had a long discussions about why this was so. My first theory was that Virginia and the DC Metro area in general was so much more stress-free and relaxing than NYC, that I was in a mode to be productive. Her theory was that I was not at home and didn't have the usual worries I would have had I been there, so that relaxed mode put me in a state of super-productivity.

I wasn't sure what the reason was, but in any case, I was very pleased that I got so much stuff done on my break.

Then, two nights ago, I drove home. I came into my house and sat down. And it hit my like a TON of bricks.

My television was making me stupid and unproductive.

You see, as soon as I came in, I grabbed a soda from the fridge, plopped my arse on the couch, and started watching shows I had recorded on my DVR. This is something I usually do when I come home from work.

My friend in Virginia didn't have DVR. She didn't even have cable. I had 500+ channels coming in on Verizon FIOS, I think she had 3 or 4 terrestrial channels. I had a large screen tv, she had a dinky little 12 inch. When I was at her place, there wouldn't have been anything to watch, or watch it in any style, so I didn't watch any television.

I realized that so much of my life at home revolved around me spending HOURS sitting in front of the television. When I was at my friend's place, I didn't have that luxury (if I could use that word), so I did the next best thing, I did work.

How many times over the last year(s) had I just wasted in front of my television for no good reason? How much of my life have I sacrificed at this altar?

Now pardon me while I go to kill my tv.

Tagged

How to Teach Yourself Unix Systems Administration

A while back, someone asked if anyone had successfully taught themselves Unix. (It was in the context of systems administration, so I'm assuming the question from a Unix sysadmin POV.)

An interesting question.

I was reflecting back on a few years of Unix experience and decided to put pen to paper (so to speak) and jot down what it would take to teach yourself Unix.

(NB: the question of whether or not I have been successful at it is hard to answer. I suppose you would have to ask my colleagues or supervisors that. But in any case, my observations follow.)

One of my University professors taught a Unix sysadmin course for grad students. I was an undergrad at the time, and did not take it, but followed it closely. The material didn't seem very hard, and I wish I had been able to take it. I don't think the fact that it was a grad-level course was what held me back at the time, but more likely a lack of funds. But I think more than coursework or access to books, the primary factor in being able to teach yourself Unix is a hunger to do so. If you are not interested in the subject, then no amount of classes you take will help you. If you have a sincere desire to learn the subject, then lack of materials will not hinder you.

Almost all my computer-related college courses had to do with Unix in some shape or form. If you paid proper attention, they Unix part of it was soaked into your bones via osmosis.

But that was many years ago.

There are a lot more formal courses now on Unix and Unix systems administration nowadays. Certainly there are more books and online materials. So the young person who is looking to enter this field certainly has more resources to access that they can take advantage of.

Get a Unix Job

If you are in college or about to get into college, definitely try and hit up a college Unix lab. They are always looking for cheap labor and you will learn a LOT. Also, they don't require a glamorous resume. If you know a bit and are hungry to learn, you can probably snag the job.

Get Unix books

If you are serious and have some money to spend, go and get the Evi Nemeth (Prentice Hall) and Aileen Frisch (O'Reilly) Unix Sysadmin books. They are both excellent. The one by Tom Limoncelli is pretty good too.

These are all sysadmin books by the way. There are books that have solely to do with Unix that you should also get.

Run Unix

I think it goes without saying that you should already be running a Linux or a BSD on your home PC already. Don't dual boot! That only will tempt you to boot back into Windows when you find you can't do something in Unix. Kill Windows and jump into Unix cold turkey. Start off with something easy like Ubuntu or Fedora, and then work your way to something more powerful like Debian or FreeBSD (which have steeper learning curves).

Google It

You have something now that I did not have when I started out -- Google. Use that to your advantage. If you are not sure of something, Google it, odds are there have been tons of people who have gone before you and have had the same doubts.

Join a User Group

Definitely join your local LUG or BSD user group. Subscribe to their mailing lists and stay on it.

Learn a Programming Language

In addition to learning Unix, if you don't how already, you should be teaching yourself how to program. Not just shell scripting, although that is a start. Pick one of Python, Ruby, or Perl, and learn it inside and out. A good Unix person is a good programmer. Unix is so powerful because it gives you a rich suite of tools to program and is probably the most programmer-friendly of all the modern operating systems. Even if you never use Unix in the future, you will find that the ability to program your way out of certain problems to be invaluable. As a sysadmin, not a single day goes by where I don't write or use some dozen line throwaway script that just makes my life easier. You won't believe me now, just do it, and trust me on that.

Join a Professional Organization

There will be things that you just won't be able to teach yourself, either because it (1) takes experience or (2) is very expensive. #1 will come with time and with jobs of greater complexity and responsibilities. It won't hurt to to join LOPSA or Usenix SAGE later on, they have a lot of people talking about best practices and such which is very useful, even if you don't plan on becoming a sysadmin. As for #2, stuff like SANs and NASs and multipath and fiber channel are harder to learn on your own with homebrew equipment unless you've got a job where you can play around with stuff like that, but it will come as you graduate to better jobs.

Don't Quit

It will all be a bit obtuse and difficult at first. Unix has a fairly steep learning curve. Stick with it. One day, probably a year or so into it, things will just "click". You will see the beauty of the command line tools and how awesome it is to glue them together with pipes. You will behold the elegance of putting everything in text files and the value of a really powerful text editor and shell. Later on, you will invariably find yourself on a Windows or Mac box and discover that things that are easy and powerful on that Unix box of yours is either impossible or very hard to do on non-Unices -- this is less true with Mac nowadays since it is Unix under the hood, but you get the gist. Once you reach that point, you will have "gotten" Unix and will begin to love it.

Learn to Love It

I think it was Paul Graham who talked about this in one of his essays: during the dot com boom someone came to him and told them they were going to start a company and that it was going to be based on Windows. The person assured him that the company was going to be a success because they hired Windows "experts". Graham thought to himself that the company was doomed to failure. The reason being that in order to be an expert at something, you had to have used it for several hundreds of hours (the equivalent of 10-man years); not just at work or school, but in your own free time. In order to become an expert at an OS, you needed to love it and use it all the time. Graham knew people loved Unix and used it all the time. On the other hand, he knew that no one had any real love for Windows, so these so-called experts probably weren't. Sure enough, the company went under a short time later.

Tagged