Walter Jon Williams

Walter Jon Williams has written a new book, read the interview in Locus.  Link.

I think I like his earlier books.  Voice of the Whirlwind was great, Angel Station was pretty good, and Hard Wired was probably the last decent cyberpunk novel of the 80s.  Knight Moves pretty clearly channeled Zelazny, and the Drake Majistral books have Keith Laumer and Retief written all over them.

Aristoi started out slow, was great in the middle, but stopped just as things started to get interesting.  His “metropolitan” books were well written, cohesive, gritty and heavy on the mystery, but so heavy mystery-wise that he forgot to resolve things and there’s no sign that they’re ever going to get wrapped up.  Forget the “earthquake” mainstream novel.  And finally, forget that the Praxis novels ever existed.  They’re not quite Elron level, and maybe I would have liked them when I was twelve, but I just couldn’t finish them.  Whole-species psychological quirks are tough to pull off (Niven just barely squeaks by with Puppeteers and Kzinti), but the Praxis books start with ham and add bacon until they feel like page after page of prejudice fighting stupidity.

I have high hopes for his new book.  We’ll see.


Newton Storage History

I joined the Newton group in early June of 1992. I’d been at Apple since 1987 working on tools and then a database engine in the development tools group, but after a bunch of bad politics that I won’t dwell upon I wound up needing a new position pretty quickly. I managed to land a gig working on a device driver framework for the Newton.

I arrived in Newton just as the group had decided to do “Junior,” the product that was to become the first Newton MessagePad. Up to that point there were a bunch of people happily noodling away in Ralph (the object-oriented dialect of Scheme that the Newton tablet product was going to be based on), but the decision was made (over CES, I gather) to do Junior in C++ and what later would become NewtonScript. When I arrived there were a bunch of ex-LISP hackers wandering the hallways clutching C++ reference books and looking unhappy. Really unhappy. The expensive tablet product (“Senior”) wasn’t entirely shelved — they kept doing revs of its plastic — but it was soft-pedalled and eventually died.

I did a simple driver model for devices (in retrospect, probably way too simple), and inherited a heap manager that I eventually made work pretty well. I worked with some early storage stuff with Tim Harrington, another engineer in the kernel group.

At first the store inside the Newton itself just kept objects in a heap in the Newton’s battery backed RAM. We also worked out a scheme for keeping heaps on removable PCMCIA RAM cards, but the problem was that the heaps had to be located at particular virtual address to work, and the virtual address of the card could vary depending on the slot you plugged the card into. We could have simply relocated the heap structures, but the problem remained that the power could fail (or the user could remove the card) in the middle of store update, whereupon you’d lose all the data (or worse, it would become flakily corrupt). We needed a storage system that was address-independent and that would also survive restarts.

The group was starting to realize that the Newton platform was way over budget on RAM. There was some neat work done by the kernel team to reduce the stack and other memory needed for threads, and I did some work in the heap managers to minimize virtual memory usage and fragmentation, but it was clear that we wouldn’t have enough RAM to store as much user data as we would need to; even reducing the application heap, we’d need another 128K or more to make people (ahem: Marketing) happy.

So the hardware team added a single byte lane of RAM. Basically this was a region of memory where every fourth byte was valid (and the agreement between the hardware folks and the OS team was that they could populate more lanes of bytes, until they filled all four lanes and it became just like the rest of RAM). This was easy for the hardware folks to do (just solder another chip on the board), but the heap-based store couldn’t deal with byte lanes.

We also had to support Flash devices. The type of flash we planned to support could be read directly like RAM, but not written to without going through funny little state-machine gyrations. Given the wacky byte-lanes of RAM and the non-RAM-nature of Flash, it was clear we had to do a bunch of work on the storage layer. So I did that; the object store talked to the actual backing store through an abstraction and the callers had no idea what kind of wacky stuff the store was going through in order to make things persistent. In fact, for the internal store of the OMP we used 128K of byte-lane memory plus another 64K of ordinary memory “stolen” from the rest of the system.  The abstraction glued all this wacky memory together and the NewtonScript world never knew the difference.

But there were still windows where resets or power failures would cause the store to become corrupt; it wasn’t crash-proof or reset-proof. We all knew this, and I’d done some thinking about putting a transaction system in, but none of us thought we had time to fix things before ship.

One afternoon Michael Culbert, the Newton Technologist, came into my office to discuss battery contact bounce. I’d been nervous about the robustness of the storage system and had talked with him about it before, and now he confirmed my worst fears. Battery contact bounce is the jiggling of the power connectors when you toss a device onto a hard surface; the device can easily momentarily lose power and reset, and if the Newton was in the middle of a storage update all the data would be lost. As I said, we’d been talking about doing transactions in the stores for a while, and Michael finally had a good argument for why they were going to be critical.

“We have to do this,” I said, “Or we’re fucked.”

“We have to do this or we’re fucked,” he echoed. “I’ll go tell Fi [the software manager of Newton] and Gaston [the head of the division].”

“Okay. It’ll take a few weeks.”

“I know.”

This was maybe four months before the product was scheduled to ship. The first thing I did was to leave work for a couple of hours to visit some bookstores I liked, buy a couple of CDs I’d been thinking of getting, and take a long walk. Then I came back that evening, closed the door to my office and worked 18-20 hour days for the next five or six weeks.

A couple of evenings into this effort Gaston came into my office and tried some small talk. He wasn’t very good at it. I was concentrating on the design of the transaction log at that point, and really didn’t want to be disturbed.

Gaston said, “Fi tells me you’re the guy who’s going to save the product.”

“I guess so.”

“All by yourself? You don’t need any help?”

“Not right now.” Some things aren’t partitionable.  I knew what I had to do, practically down to each subroutine to write.

After Gaston left, I tracked down Fi and told her, “No offense, but please tell Gaston not to come around my office anymore. He makes me nervous.” She did get some other folks on the team to help out on some pieces that I was able to partition and farm out, but for the most part the things that had to be done simply couldn’t be parceled out to other people, there just wasn’t time to get them up to speed.  “Adding people to a late project makes it later.”

The only thing I remember clearly about the next couple of weeks was that I dreamed a whole lot about transaction systems. I stuck the changes into the system (we had mandatory code reviews; I spent the two days prior to checking-in talking myself hoarse, walking through the code with Greg Seitz, another kernel guy). I did my only all-nighter at Apple getting that build out, and I recall only a couple of bug fixes to it.

The upshot of the transaction system was that you could do a bunch of updates to the objects on a store, crash in the middle, and recover everything prior to the update that crashed. It was pretty cool.

Here’s roughly how it worked:

Imagine that you have a notepad entry, with a bunch of text. The NewtonScript storage layer breaks up the notes into a bunch of small objects, including the text itself, plus some extra objects for searching. There are objects containing structures that allow the text to be quickly searched, and so on.

Now say your original note was, “Buy milk,” and that you add “Bread, too!” to it. What the Newton does is create a whole set of shadow objects, just like the old ones but with your changes applied. Then, with a single write to a single byte somewhere in the store the storage system commits your changes, then goes back and deletes the outdated objects.

If the power fails before the commit point, the Newton just deletes the new objects and you haven’t lost much work (you just have to remember to buy bread). If the power fails after the commit point, the Newton just ensures that all of the old objects have been gotten rid of (so it can re-use the space they take up).

This sounds simple in principle, but in fact it’s quite difficult to get correct. Additionally, the flash storage we were using had the amusing characteristic that bits could go bad, which is a little like having to design a car with an engine that might lose a gear or two and still have to maintain freeway speeds without the driver even noticing.

Flash-based file systems in 1992 were kind of rocket science; the existing stuff from Microsoft had to be manually compacted, which as you can well imagine was a really awful pain the rear. The Newton’s flash store did all of this transparently and in real-time, too; during a technical presentation we got some questions about this (I always assumed they were from Microsofties):

“When do users have to compact the flash cards?”

“You don’t, it’s all done for you.”

There was some refusal to believe this, and we got the question several times. That was fun. Nowadays users just expect this stuff to work, but back then all the existing Flash file systems were pretty icky.  It felt great to have something cool out there.

After 1.0 shipped we added large binary support. The idea was to use the virtual memory system of the Newton OS to dynamically page in modifiable binary objects that could be significantly larger than the amount of RAM in the system (e.g., faxes). The first couple of cuts at this were pretty miserable until I had the idea of hooking the objects into the NewtonScript garbage collector.

The idea was that a large binary not associated with a persistent slot would live in a kind of limbo; if it was garbage-collected it would be deleted, and if the system crashed before it was deleted it would be cleaned-up as part of the normal transaction “undo” phase. But if a object was persisted and had a large binary as a slot, the large binary would be kept alive. It took a while to do, but the result was a system with only 512K of RAM that could receive a multi-megabyte FAX over a modem, or work with bitmap images larger than main memory, or play sounds that were likewise really large.

There was a lot of great technology hidden away in the guts of Newton.  I worked on some other pieces that were neat, but this is what I’m most happy about having shipped. It was a lot of fun to work on. Newton was one of those projects I always felt it was a privilege to be part of.

The source of the disease?

I ran across an option the other day that was something like:


Can you see what drives me crazy about this?  There’s this 26 character identifier that’s already impossible to type, and somebody decided it would be a good thing to shorten DEFAULT to DFLT.

What did this do?  Did it run faster?  Did it save disk space?  Did it save the world?  (Note that this system allows 128 character identifiers).  Why didn’t they shorten everything else, too?  (SVR_OPT_DFLT_ZNKRS_ON?  SV_OP_DF_ZKR_ON?)  Why why why why why?

My guess: It was probably historical.  Some ancient dinosaur of a reviewer got up on his mountain of saved-up dino poop and had a tirade about how CompuVac did this in the 60s and it was damned hard because they had to chip all their punch-cards out of fossilized dinosaur poop (after waiting for the poop to fossilize, you see) and by gum, by gum boys, this ain’t gonna change, it’s DFLT or,  or . . . (insert strangle-y purple sounds here).  Loosens collar, loosens tie.  By gum.

The donut-eaters in the rest of the room looked at each other, did a “gimme” and broke for lunch, chatting about the latest in blood pressure medications.  Wonderful stuff, maybe we should slip him some.

So DFLT it is, and pair programmers everywhere probably have daily conversations like:

“So, next: set server option default zonkers on.”

“Okay.  Um . . . syntax error.”

“What?  Oh.  That’s DFLT.”

“Right, sorry.  My fault.”

“Yup, you’re an idiot.”

Preserve us from dino-hood, please O Lord, and from becoming dinos ourselves.

Stormy Game

Material required:

– One email system
– A windstorm


– Each player is allowed one entry
– Play goes until a set time (e.g., 5pm, 10pm, the winds die down)
– The winning entry is the last successfully sent email to the other players before the windstorm takes the power out for an agreed-upon minimum time (e.g., one hour)


Buffy Loaf

A very simple meatloaf. Note the lack of egg and dairy.

2 lbs ground buffalo
1 onion, finely diced
1 green pepper, diced
1 small carrot
2 tsp oregano
2-3 cloves garlic, finely diced
1/2 cup ketchup
3/4 cup bread crumbs (or 1 package saltine crackers, finely crushed)
1/3 lb or so of bacon
Grind the carrot up in a food processor (6-7 one second pulses). If you don’t have a food processor, skip the carrot. Heat onion in pan until barely translucent, add green pepper, carrot and garlic and get things softened up. You don’t need to really cook these since they’re going into the loaf anyway.

Mash everything together (yes, the above mixture will be a little hot) and mix thoroughly; shape into a loaf in a pan. Get the loaf fairly tight because there is really no binder. Put bacon on top in strips. If possible, put the loaf into the fridge for a few hours so it will be firmer after baking.

Bake at 350 degrees for at least an hour (I take it out when the temp reaches 148 degrees, about 1:15 in our oven). Let sit at least 10 minutes.

The loaf will tend to fall apart, but it’ll be far more cohesive after a night in the fridge. The time resting in the fridge really seems to help.


5,325 pieces of blog spam and still counting.

Nitwit losers.

Look, if you’re going to comment on something, then comment.  Don’t just say, “Nice post, couldn’t have said it better myself” with a URL pointing to some stupid you-know-what-kind-of-pill-site because (guess what?) it’ll either be filtered or I’ll catch it in moderation.  I don’t know what blog spam commenters are paid, but it can’t be worth trying to think up something even halfway interesting.  Go flip burgers or something.


It’s been crunchy at work, and too much of my time has been taken up playing Bioshock and Halo 3.  (Bioshock, by the way, just rocks.  One or two of the best moments I’ve ever had in gaming were in there, ranking with Defender, Robotron, death matches in Quake, and hours or maybe all those months lost to Diablo 2).