A little more wisdom

The longer it takes to fix a bug, the smaller the fix will be.

[I once spent two weeks finding a nasty bug in the Newton’s OS, which was manifested by a mysterious and well-nigh irreproducible scheduler freeze. The fix was to swap two instructions in a trap handler.

You’ll just have to take my word for this: I did the fix over then-Apple’s Xmas holiday week off. You had to get very-high-level-manager approval to work during this week, because they paid triple. The fact they were paying me lots more money had nothing to do with how long it took to find the problem. Honest.]


Operating system people get no respect. All the hoopla and credit goes to the folks six APIs above your stuff, who move pixels around and make “ZIP-BOOM” noises while wasting cycles with horrible abandon. No one cares about the coal miners who pull those cycles out of the raw earth to begin with, it’s all about the actors using the light made by your coal.

Don’t forget that.  Do, and you’ll wind up unhappy.

Bitterness equals 1 over the number of levels you are above the hardware. This, of course, makes hardware people infinitely bitter, which exactly matches observation.


“What we need in this company are more fat engineers with beards.”



Installer heck

I started using Microsoft’s Visual C++ in 1993 when the 1.0 version came out; when version 1.5 was released, I was using it to develop products at a startup. One of the things that was interesting about VC++ then was the installer; compared to prior versions of the Microsoft tools, VC++ was a breeze to install. Visual C++ actually had an installer team; MSDN published an interview with the manager of the group. “The installation experience is very important.” They got it.

Fast-forward 15 years and installers are still important, but God damn it, they suck hard.

I’m not sure why this is. Correction, I have conflicting theories. The first is, most teams foist off the installer work to junior engineers; the kind of people who will actually believe that typing in 60 character case-sensitive unlock codes is a good thing. Then there are paranoid freaks who have the need to seal everything up behind product keys, feature enabler codes, license.txt files and (of course) physical dongles, just to get to the EULA. Put these two types in a room together and they’ll produce a Fort Knox-class installer for Notepad that nobody will be able to run.  The transistor blows to protect the fuse; the software blows to protect product support.


Recently I had to download and install a package that had one of the craziest protection schemes I’ve ever seen. After clicking through the usual maze of agreements (“I accept” and “Okay fine, your company now owns my immortal soul”), the download page asked for the MAC address of my machine. Under separate cover, the site’s download manager emailed a license file that had to be placed into a very specific directory on my machine’s hard drive, whereupon the installer would run (instead of failing with messages like “00232-FATAL_LIC_NF”). The result? The installer unpacked a directory of source code. Obviously someone was told, “Protect this sample code at all costs,” and followed orders to the letter.

I promptly checked the stuff into source control. Problem solved. I wonder which corner of Hell I’m going to end up in?

Lucifer: “Listen up! Adulterers, follow the signs saying ‘Burning Tar Pits’. Money lenders and financial industry employees, you’re in the ‘Taco Bell’ line for the rest of eternity; pray you don’t reach the counter. And anyone who circumvented the spirit of a software license system –”

Me: “WHAT!?”

L: “… the Bubbling Acid Pits of Operating System Install Hell!.”

Me: “Wait, what about my other sins? Pride? Greed? And shit, I thought a lot about killing the guy who wrote the driver framework for –”

L (smiles evilly): “We’re gonna put you right next to Richard Stallman.”

Let us not talk about the installs from hell, where you hunt high and low for the 5996-qr3b-patch09_6_51a_.32 (but not 5996-qr3b-patch09_6_52b_.31). Eventually you find it buried on some web site (behind three username/password doors, each with different and conflicting rules for required and allowed password characters), start the download (apparently through a submarine line laid to the Antarctic and back, via the Panama canal) and discover that what you’ve downloaded is actually just a download manager for the next stage of the process. Which involves a physical trip to Sandusky, Ohio to prostrate yourself in the lobby of LoserSoft with a suitcase of cash and beg abjectly for a download key, followed by self-immolation. Gotta read the goddamn fine print on the EULA.

There was the memorable piece of Macintosh software that involved nearly fifty swaps of seven floppy disks. It. Took. A. While.

You want to know what my idea of the perfect installer is? A copy, from point A to point B. From the distribution disk (or a zip file) to some directory of my choosing. It used to be that a lot of Macintosh software worked this way (I don’t know if this practice continues). I know that most Window programs do not (don’t get me started on registry entries and use of complexified XML as a glorified substitute for .INI files).

Somewhere in this galaxy an advanced civilization has discovered the solution to all of this, and they are happily flitting around and not telling us what it is. But I suspect the reason we haven’t seen them is that their fleet of hyperdrive saucers is grounded until they discover how to install 5996-qr3b-patch09_6_51A_.33 (“Oh, Qualtar, I think that’s supposed to be a capital groknard, not a lower-case one…”)

Taking stock

“That’s three P-1 bugs this week, Bill.”

“Yes sir, I know.”

“All the managers had a meeting about it this morning. So . . . how do you want to do this? Go the HR route, or –”

“I’ll take the Yard.”

“You’re sure?”

“I talked it over with my family last night. Yes, I’m sure.”

“Right. Just between you and me, I’m happy you made that choice.”

“I don’t want to let the team down, with this final push. And all.”

“Okay, I’ll call security.”


HugeSoft’s pillory is located on the main campus to one side of the famous soccer field, near the center line, on a little mound raised up and covered with grass. It is easily visible and strategic to one of the company’s largest cafeterias. There are no trees, and there is usually no shade until late afternoon. It is a point of debate whether a day on the mound is easier in warm weather, or during one of the frequent rainy days in the area where HugeSoft is located. Three posts are set in a line, about ten feet apart. Usually only one or two are occupied. During heavy ship cycles there are waiting lists.

You’re tied up most of the day. The stint starts around 8am. HR meticulously video tapes the entire ordeal, from the traditional walk past the CEO’s office to the finishing-up dunking ceremony in the fountains. Anyone can say a safe word and be released immediately, no questions asked. Only half a dozen people have done this. People are led to a post and loosely tied to it; water is at hand, and on hot days sun-screen is applied. If it gets over 80, it’s over. It’s not the Middle Ages or early Colonial America, it’s no picnic, but it’s also not a joke.


Bob Kneeth is a Vice President. Sundar V. is a high-level engineer. Both of them have spent a day out in The Yard.

“You can easily have a bad month,” says Kneeth, “Everybody does now and then. You break the build a couple of times because you tried a shortcut and it backfired. Or you’re not getting enough sleep; that happens a lot. Then your boss and HR show up in your cubical, and frankly it’s kind of a relief.”

Sundar: “It’s really catharsis. If they wanted to get rid of you, they’d just fire your ass, and frankly that’d be a lot easier and cheaper. This tells the rest of company, ‘These guys, they fucked up. But they’re worth saving. Pay attention to what you do, because quality matters.'”

Kneeth (laughing): “The only really bad time is around lunch, when they throw stuff at you. HR’s there to make sure you don’t get hurt, and there are plenty of foam toys, but they can throw food at you if they want to.”

Sundar: “Mexican day is the worst.”

Kneeth: “Some people have allergies, and there are signs saying what you can toss.”

Sundar: “Yeah. I got off easy, because I’m vegan. One of my buddies is Jewish, and no one knew what the devil to toss at him because HR decided bagels were too hard.”


In the engineering groups it is traditional to bring a change of clothes and pull an all-nighter after spending the day on the Yard. In the morning, HR comes by and all records of the event are erased. It didn’t happen.

“It’s both funny and not funny at all,” says an anonymous engineer. “It’s a serious quality thing, and if you’ve been out there more than twice you’re probably going to be out on the street after that. But everyone makes mistakes, and this is a way of acknowledging that we’re human, and that even serious mistakes can be forgiven.”