userlame.com is no longer running on my personal network after probably 8 years or so. It now lives in a real-life datacenter!

In other news, I'm moving things off of my network because I have some big plans ahead...

Please do tell me if you see anything funky on the site or something doesn't work.

Also, I fixed the date/time problem. I really can't believe I waited so long to make a fix as minor as that was.

The title says it all. I've pondered this [plasmasturm.org] myself many times, and always reached the same conclusion.

Someone local (in my city) just checked out the site today for the first time in over a month.

WHO ARE YOU MYSTERY PERSON? SHOW YOURSELF!

Really, I'm just curious.

Um, turns out there's a local root exploit in the linux kernel versions 2.6.17 to 2.6.24.1. I just tried it on my machines - yeah, it works. Read the slashdot discussion here [it.slashdot.org] and see the proof of concept code here [milw0rm.com].

Still reading about it now. This is huge. Hold me.

By a picture on their website [bradblog.com]!

Haaahahahaha. You posted a close-up picture of the real key?! Haaaaaaaaahahahahahaha.

Edit: apparently I didn't read this (the date specifically) too closely. Old news, but still new to me and made me laugh.

After months of rigorous testing, sweat, blood and tears I've finally perfected the formula.

Here it is.

Are you ready?

Just so I won't forget and maybe google will spider this and it'll help somebody. There's a (kinda) bug [bugs.gentoo.org] in gentoo's 2007.0 stage3. I'm using amd64, but it's due to the version of ssh which was put into the stage3, so I'm sure it's across platforms. /etc/skel/.ssh is created with permissions 0600 - so when you add a user, by default they won't be able to cd to ~/.ssh. That's a problem when using ssh keys and it frustrated me for a few minutes because I couldn't login in to new servers.

Fix by the following:


After this any new users can use their .ssh directory. If you've already added users and are having problems here's a quick fix:

I just read this [edition.cnn.com] today:



Agreed. So, when does the US become a modern, confident nation?

I meant to post this a while ago, but never got around to it. If you've purchased a ZyXEL GS2024, and it came with firmware V3.8(LT.0)C0 I pity you. I too bought this lovely lemon. Long story short, it sucked. I was really disappointed when I had my new switch running for about 5 minutes and thought I had to RMA it.

I pulled it out of the box, fired it up, and started playing with it, and no sooner had I upped the serial baud rate when the hardware error light came on. Then it started flashing. Not steadily either, just random on/off. The second command I checked out was reading the logs. I found a whooole bunch of errors in the log about the fan dropping below threshold and recovering (many times per second). From running the diagnostics on the switch the fan was peachy. Sigh.

I checked for new firmware, but this was the most recent. So recent in fact (I thought) that it wasn't even on the manufacturer's site. The manual for 3.8 was there and released only weeks before. I'm sure they actually pulled the firmware down when they realized what a bad bug it had. I googled around and found someone else complaining of the same errors after upgrading from 3.6 to 3.8. I can't find the site now, which is weird. I was there numerous times and posted on it. Hm. I'd post the log messages from the switch, but mine are gone and the only place I knew that had them was that site.

Anyway, I downgraded the firmware and have been playing around with it while running version 3.6. The only feature that I noticed was missing was the ability to use ssh keys. I also noticed logging in as admin wasn't automatically in enable mode in the earlier version. Whatever.

Well, earlier this month, ZyXEL put out a bugfixed version of 3.8 which is available on their site [us.zyxel.com]. If you were one of the lucky ones to get your GS-2024 shipped with firmware V3.8(LT.0)C0 - well, hopefully you've downgraded, but now you can go get a better (read: working) version of 3.8. The version name is very similar - V3.8(LT.1)C0

I would like to point out that it's a very nice switch and I'm quite pleased with it now. ZyXEL was very fast with a fix, but it was disappointing that a bug so painfully obvious was put out, and loaded on production models. Running it for more than a minute or two showed the problem.

Update: Found the forum post I was talking about here [fixya.com]. The fan log messages looked like:

I just noticed the dates...on every post. Hrrrrrrmmmmmm....looks like I screwed the db transfer and didn't notice. Too bad the old one has already been blown away!

At least the IDs kept em in order. Sigh. Now to figure out why new posts get the same date.

The title says it all. There are some very big things in the works. That's all I have time to say now. The big things are keeping me busy.

Oh, and there was some breakage around the site. That's all better. Oops.

I don't like yahoo very much these days. They are (were) a minor but annoying thorn in my side. Lemme splain.

Yahoo has this nice feature for its email client called a vacation response. Basically you type in what you want it to say - "out of the office riding goats" or whathaveyou, and when someone sends an email to your account it sends back an email with the subject "Yahoo! Auto Response" - the text of the email contains "out of the office riding goats" or whatever you typed in.

Great. I can tell people I'm not going to read their email for a while. Whoopie.

Some flaws (besides the obvious AUTO-RESPONSES ARE EVIL. DO NOT USE THEM):

1) It doesn't keep track of who it has sent a response to. If I sent two emails to an account with that activiated, I'd get two "Yahoo! Auto Response" replies. Not terrible, though it could possibly lead to some funny mail loops.

2) It's really easy to set up a yahoo account. It would be pretty easy to script an account setup that would present just the captchas to be solved. Then the script could set up an auto-response.

3) No anti-spam policies at all seem to be applied to email that will be auto-responded to. At least I know they aren't checking SPF.

Mix em all together and you get a delicious spam pie. I've written a proof of concept (not a script, you'll have to do it by hand - the rest is an exercise I'll kill the reader for following) so you can see it in action. Try it yourself [userlame.com]. But please, have some netiquette.

In short, spammers are sending email to fake accounts with spam vacation responses. They are faking lots and lots of from addresses on those emails and the spam vacation response is sent to the faked from address. Thus, I end up with a buttload of spam.

It also is completely valid mail. It comes from otherwise (I guess) sane outbound mailservers for yahoo. The email will have a valid domain DomainKey-Signature and everything. It was getting by all my anti-spam because it looks so real.

And so finally, a resolution...spamassassin. Here's a simple rule to send all that crap where it belongs. Adjust the score accordingly if you need to.

header YAHOO_AUTORESPONSE_SPAMZ Subject =~ /^Yahoo! Auto Response$/
score YAHOO_AUTORESPONSE_SPAMZ 5.0
describe YAHOO_AUTORESPONSE_SPAMZ Yahoo auto-response spam

I'm going to bed.

Edit: Yikes, that link was broken. Originally it was .html, and I moved it to .php but didn't update the link. My browser had the .html cached so I never noticed. Sigh.

Cheer me up.

Thanks.

Long story, buckle up.

So I'm building a new server. Nothing fancy, but eventually I'd like this box to take over for the server that's running this site. So I'm trying to actually do stuff up right, have real redundancy, harden the server some, etc. The box has been sitting around for a while just minus HDDs and PSU, and those I got. I picked up a couple identical small disks and want to do mirroring for redundancy. This site has been hobbling along on a wing and a prayer, and (fingers crossed) no data loss yet. Yet. I'd like to keep it that way. I also decided I'm going to give FreeBSD a shot. Supposedly it has great performance under load, is way stable, and a host of other good things. Now, I don't know crap about BSD and that's actually a big part of why I'm using it. userlame likes to learn.

Parts arrived and I got to putting things together when, genius that I am, I toasted the motherboard. Oops. Ok, so now I'm waiting on a new mobo. While I'm waiting, I decided I'd do some test installing and BSD learning and such on a VM. Well I've finally gotten the install part down now, and that's what this post is about. The actual install is easy as pie. Add mirroring into the mix, and it gets a bit ugly.

So. I learned about these "slices" and "partitions" BSD uses -- it's much different than anything I've dealt with before -- and I kind of have a handle on what's going on. I decided I'm going to have my disk layout as such: each disk with a single slice and partitions laid out like so:

/ (root) - mirrored between drives
swap - striped between drives
/tmp - striped
/usr - mirrored
/var - mirrored
/var/log - mirrored
/home - mirrored

Getting through the install process is easy like I said. I can very quickly get one disk partitioned and do a minimal install that takes about 20 seconds. But where do I go from here once I have one disk layed out correctly (how to mirror)? After monkeying around for hours on end, here's how I did it:

Note: The way I do this is likely very wrong and I don't advocate anyone follow these directions. Really, I don't know what I'm doing wrt BSD.

First, I installed everything to one disk. Then, I just ran through the exact same install process on the other disk. I found this was the easiest way to get things partitioned and correct bsdlabels on each disk slice. At this point the disk looks like this:

ad0s1a - /
ad0s1b - swap
ad0s1d - /var
ad0s1e - /tmp
ad0s1f - /usr
ad0s1g - /var/log
ad0s1h - /home

And the other disk is exactly the same, just ad1 instead of ad0. Now I ran into a chicken and egg problem. I couldn't get gmirror to label (or insert) a mounted partition. I set sysctl kern.geom.debugflags=16 and that didn't help. I also couldn't use gmirror when booted from the cd (different version of some sort missing a lot of functionality it seemed). I tried about a katrillion things before I came up with a solution.

First, I booted to ad0 which gave me a running system with all the above partitions. Then I got gmirror into the mix with "gmirror load". Then, for each mirror I wanted (root, var, usr, varlog, home) I labeled the second disk's partitions. For example the root partition was "gmirror label -v -b round-robin root /dev/ad1s1a" and /usr was "gmirror label -v -b round-robin usr /dev/ad1s1f". After doing this for each partition I had a device in /dev/mirror for each label name - such as /dev/mirror/root.

Next, I wiped those mirrors and layed out a new filesystem on each of them like "newfs -O2 -U /dev/mirror/usr". Next, I mounted the new mirror filesystems into /mnt/labelname such as "mkdir /mnt/varlog" then "mount /dev/mirror/varlog /mnt/varlog". After all the mirrors were mounted, I dumped the live filesystems into them like "dump -L -0 -f- /home | (cd /mnt/home && restore -r -v -f-)". After dumping each one, I unmounted the filesystem and removed the /mnt/labelname directory to be sure things went smooth.

One special note: while I had /mnt/root mounted and after running "dump -L -0 -f- / | (cd /mnt/root && restore -r -v -f-)" I needed to edit /etc/fstab and /boot/loader.conf before rebooting, so I edited these on both the live filesystem and the mnt'd filesystem. In /etc/fstab (and /mnt/root/etc/fstab), I changed everything from the real partitions (such as /dev/ad0s1a and /dev/ad0s1g) to the mirrored partitions (such as /dev/mirror/root and /dev/mirror/varlog). In /boot/loader.conf (and /mnt/root/boot/loader.conf - both empty files) I simply added the line geom_mirror_load="YES". I finished umounting /mnt/root (it was the last one I did), and rebooted. Sure enough, on rebooting I was running on my mirrored filesystems! :

/dev/mirror/root on / (ufs, local)
devfs on /dev (devfs, local)
/dev/mirror/home on /home (ufs, local, soft-updates)
/dev/ad0s1e on /tmp (ufs, local, soft-updates) [Still on real disk; we'll get there]
/dev/mirror/usr on /usr (ufs, local, soft-updates)
/dev/mirror/var on /var (ufs, local, soft-updates)
/dev/mirror/varlog on /var/log (ufs, local, soft-updates)

But hold on, they aren't really mirrored yet. They're still only running on one disk. Now that the partitions on ad0s1 aren't being mounted however, it's a simple matter to add those partitions to the mirror. Examples: "gmirror insert -v root /dev/ad0s1a" and "gmirror insert -v home /dev/ad0s1h". And ta-da that's it! A mirrored filesystem! (I don't have [code] tags so this formatting will be nasty):

# gmirror status
         Name       Status    Components
  mirror/root     COMPLETE    ad0s1a
                                        ad1s1a
   mirror/var     COMPLETE    ad0s1d
                                        ad1s1d
   mirror/usr     COMPLETE    ad0s1f
                                        ad1s1f
mirror/varlog     COMPLETE    ad0s1g
                                        ad1s1g
  mirror/home     COMPLETE    ad0s1h
                                        ad1s1h


I know I mentioned something about doing striping earlier though, so let's get to that. Once I had the mirroring stuff under my belt, this wasn't so tough. Let's do the swap first. We can't use any partitions that are in use so first we have to turn swap off. No problem. "swapoff -a". Then we just create the stripe - "gstripe load" to turn it on and "gstripe label -v swap /dev/ad0s1b /dev/ad1s1b" and that's it! "swapon /dev/stripe/swap" and we're in business. Don't forget to update /etc/fstab with the new swap device as well. /tmp isn't much harder. Just a "umount /tmp" followed by "gstripe label -v tmp /dev/ad0s1e /dev/ad1s1e" and that's done. Change that in /etc/fstab as well. Just one more thing to do - update /boot/loader.conf to load striping as well. Add the line (only contains our 1 line from earlier now) geom_stripe_load="YES" and that's it.

A reboot later, and I'm running with striped swap and /tmp, and the rest of my filesystems are mirrored. And it only took the better part of 9 hours to get there!

I'm f'in bored.

Wow. So I bought a motorcycle. My buddy just drove it to a parking lot for me today so's I could ride around some and get the feel for it. I've never ridden a motorcycle before ever. Oh. Man. That was sweet. I've got a long way to go in getting used to the thing and getting comfortable, but I think I'm gonna like this. Just toolin around in a parking lot was a blast! I even drove myself home. Totally lost my hat somewhere though. Worth. It.

This is the only pic I have right now:


UPDATE: A couple more pics:





UPDATE 2: Found my hat. I was sitting on it when I rode home, and sure enough it was still on the seat.

There will be some brief outages tonight both with the site and userlame.net in general as I move to a new network. I hope to minimize any downtime and I will try to make the failover work into the new environment.

Then I'm gonna have to rewrite that thing. Sigh. Here goes!

I've spent the last day and a half intermittantly trying to track down a bug I was hitting when restarting apache2. Buckle up, this is going to be a long problem explanation.

Basically, I was seeing incredibly slow startup times of apache. It could be from 30 seconds up to 5 minutes or more. And it wasn't just starting apache; it was slow stopping apache (-k stop), starting apache (-k start), and even testing the config (-t).

That's weird, but it gets weirder. This problem would only randomly (pun intended; read on) appear. Apache would restart just fine, and then I'd restart it again later and it would hang. Even the problem itself didn't remain the same. Some times it would get through configtest ok and hang at stop and start (or graceful). Sometimes it would get through configtest and stop and then hang on start. It also hung two places on starting. First it would block and hang immediately, then it would background and open a listen socket on 80 but block and hang before it started worker processes. You could connect to the server and send requests, but there you'd sit with no workers to service you. Now randomly mash together some of these places where it could hang and that's what I'd see. And not consistently; sometimes it was just fine.

I'm running gentoo and was using the init script /etc/init.d/apache2 to restart the server. I confirmed it wasn't the init scripts hanging by first running apache2ctl directly, then going straight to the apache2 binary. Same deal with all of them. I went through everything that I could think of that could be causing it. I'd never seen this problem before. I ran into the problem after installing mod_perl, but the issue remained after reverting everything to it's original state. I spent a lot of time believing that had caused it.

I'm going to skip all the crazy googling and wild goose chasing I did. I put my configs into all state of disarray even trying to isolate the problem.

Fast forward about 24 hours - in a strange moment of clarity I realized that I had run an strace on the init and apachectl scripts which didn't really show me much, but hadn't done an strace directly on apache2 (which would obviously give me better info). So I traced a configtest (strace /usr/sbin/apache2 -DPHP5 -DPERL -DDEV -d/usr/lib/apache2 -f/etc/apache2/httpd.conf -t) a couple times, until it hung. And there it was right before my eyes:



It was opening /dev/random and blocking! Shit! I confirmed that this was the problem by cat /proc/sys/kernel/random/entropy_avail and it was something like 33. I was short on entropy. Why it wasn't using urandom I didn't know, but armed with my new knowledge I hit google again and found this article [raptorized.com]. It pretty well explains it at the end...I stuck urandom into my USE flags in make.conf (surprised this isn't a default flag I guess), and re-emerged dev-libs/apr. Bam, problem gone.

Phew, that kinda sucked. The scoop is I'm working on my dev server porting the new site to mod_perl. Yeah right, that will help me actually work on it.

userlame.net is back....sort of. Got the network installed today and all is happy except for my upstream. There's something ungodly going on here and I'm waiting on my ISP to look into it. I've got my downstream running happily along, but my upstream is at like 10 Kb/s. Ouch. That's kilobit mind you.

Update 4/3 5:15pm CDT: Upstream is back to about normal. Things are looking good.

You read that right. The network is moving. Tomorrow-like. Kind of interesting that the last two front page posts which are months apart are just announcing downtime. I haven't even looked at this site in a long time.

Bleh.

So yeah, AT&T will do their shtuff, then some downtime which is planned to be 30-60 minutes (god I hope not more), then I'll be back in business. Until that time, the desktop/camserv is going down down down!