Flash memories

The Mars rover Spirit has had a bit of trouble with its flash memory file system. This is amusing (to me) because I have a fair amount of experience in just this area, which has triggered a rather geeky reminisence about technology, solving the real problems of customers, economics and success in the marketplace. All this from some stupid bug in a file system 30 million miles away. 🙂

It was 1992 or thereabouts, in Cupertino. I had just joined the Newton team, was working on its data storage code, and there were some worries.

Anyone who’s stuck a disk into a computer and then forcibly removed the disk before the computer was ready to give it back has probably had Bad Things happen to their data. Anyone who’s shut down a computer by, say, pulling the plug has probably had Bad Things happen, too. Basically you’re mucking with the integrity of the data; it might wind up half-written, be subtley or catastrophically altered, or it could just get wiped out. Entire industries have been built around guaranteeing integrity of information, including notable database vendors and data recovery firms. Most modern file systems incorporate some kind of data safety. It wasn’t always the case.

The Newton supported removable PCMCIA cards, and furthermore it was a battery-operated device. The Newton’s object store was internal, but you could also store user data (notes, contacts, drawings, etc.) on the removable cards. The problem we had was how to guarantee that data stored on the cards was totally safe.

Imagine you’re Joe Newton, Consumer (not too bright, since you’ve just plonked down $900 for a Messagepad). You’ve scribbled a note onto your Newton and you turn the machine off. You pretty much expect your data to stay around. No excuses about “Well, the Newton wasn’t ready for that” or “You just have to wait a few seconds before you hit the power switch.” This is a consumer product and people expect it to just work. [No snide remarks, please. I know the Newton crashed in the marketplace — I’ll talk about some of the reasons for that in a minute]. [[And I am finessing some more technical points here, such as what “off” really means]].

There are other nasty things that Joe can do, including ripping the PCMCIA card out of the machine without notice, taking out the batteries, or even just dropping the unit a short distance (which may make the battery contacts bounce, causing the system to reset, possibly in the middle of an update). Since any data corruption can potentially take down the entire internal file system, this is bad news. If you’ve invested hundreds of hours into the data on your unit, having that data go away is Very Bad.

So the Newton has a transaction system built into it. If you rip out a PCMCIA card or reset the unit before data are completely updated, the system will roll back to some good, consistent state with all but the latest changes intact. It’s pretty nifty, and it’s a feature of the product that no one really sees. (If no one sees it, job accomplished!)

I’m going to take this in two directions now.

Rover Madness

Now, keeping track of several megabytes of storage on a PCMCIA card itself consumes memory. The Newton didn’t have a whole lot of RAM to work with — 512K in the base unit, but really only about 30K or so to play with as far as the storage system was concerned. The way the Newton storage system was designed, the larger the external memory, the more internal RAM was necessary to keep track of what was stored in the external memory. If you plugged a truly huge Flash card into the unit (“huge” meant 20Mb or so, in 1994), then a much greater burden was placed on the system’s RAM, which could cause lower levels of the operating system to run out of memory, which in truly extreme cases caused a reboot.

So: Flash card fills up. You plug it into a Newton, and the unit starts rebooting cyclically (reboot, look at the card, fill up RAM with management structures, run out of memory, panic, and reboot again).

Which is more or less what was happening on Spirit (without seeing the source for the Mars rovers, I’m making an educated guess).

Market Forces

Of course, Palm came along and kicked Newton’s butt. And far as I know, Palm didn’t have a whizzy transactional object store keeping the user data safe from catastrophe, they did something much better, mostly not addressable in the software.

They made it very easy for users to back up their data. With a Palm, you plonked the unit into a (supplied) cradle, pressed a button and it got backed up. How hard is that?

A Newton was a huge pain in the rear to back up. No cradle. You had to buy a copy of Newton Connection (it wasn’t included), you had to plug the unit into a serial cable (after locating the connector, which was hidden behind a flap), launch an application, click about a dozen buttons (some on the Newton, some on the desktop machine), and then wait. How hard is that? How often do you bother to back up data? How often do you even bother to use your “MessagePad” to do actual messaging, given the barrier to communication?

If you’ve got a safe backup (and hopefully, many of them), then you don’t need bulletproof guarantees about data integrity. Sure, it sucks to lose the information in the field, but when you get back home you just plop the unit in the cradle and (poof) you’re mostly restored.

This level of ease-of-use takes a whole-product view that some groups are just not very good at. It’s not enough to have smart people doing whizzy technology. And sometimes thinking about the product at a higher level than just “some cool software” makes the job a lot easier.

Palms were a third the price of Newtons, and a third the size and weight. The Palm development environment was a lot cheaper, and you didn’t have to write code in the whizzy but wacky NewtonScript language. NewtonScript was pretty neat, but it was strange, and it was a major impediment to folks who just wanted to port their C code and sell software. The Newton team viewed native code as being dangerous, so there was basically no story for porting any existing code to the platform, which resulted in a paucity of applications. In fact, Apple wanted 1% of developers’ profit on titles — you’d think that Apple would have been very supportive of Newton developers, giving away dev kits and making information public, but instead things were almost hostile.

I think that Apple thought the Newton was too precious to actually sell. And after months of being coy with the technology, folks just wandered away in search of something else they could use as platform. Without dealing with those crazy people in Cupertino.

There are other things that Palm got right that the Newton didn’t (like the handwriting recognition), but for something early and expensive, it doesn’t take much to kill a product line. Combine this with Apple’s classic lack of follow-through on things that weren’t a resounding success, and you have a recipe for failure.

It’s a shame; handled right, the Newton could have become a major player. Instead, Apple squandered its investment and handed the market to companies that didn’t have to sink nearly as much into development costs.

Maybe I’ll talk later about other stuff that went wrong, but it’s hardly interesting. The important less for me was that solving a hard problem is not the same thing as solving the right problem, and I’ve tried to remain pragmatic about things like that ever since.


Don’t miss the 2nd International Dylan Conference in New Zealand. Newton had been going on for a number of years before I joined the project; it was originally a $10,000 tablet running Dylan “to the metal.” [Note many of Apple’s failures start out in complete denial about cost, “Sure, somebody’s gonna pay ten grand for this cool thing!”]. I joined up just as the mgt. made the decision to re-write everything in C++ and actually ship something; there were a lot of long faces (and copies of Stroustrup) in the halls the first couple of weeks that I was there. (Dylan looks neat, but I’ve never programmed in it. I blame the emotional scarring from too debugging too much C++).

Here’s an excerable little paper on why states controlling the Internet is a good thing. Bastards.

Eolas patent

The US government might be relenting on the Eolas patent. It’s about time the USPTO admitted that the patent should never have been issued.

What’s wrong with the patent process? Three things: A low bar to entry caused by incompetence in the USPTO (“Hey guys, my patent on eating food with a beverage came through!”) combined with opportunistic zero-morals holding companies who acquire patents and take advantage of their monopoly to sue everyone in sight. The third element is the inability of the USPTO to admit they’ve been wrong in issuing a patent for something obvious. Everyone knows there are bogus patents out there, but what can you do? You pay the $25K, and it’s cheaper than going to court. Try to convince a jury (of peers? don’t get me started) that XOR is something obvious to a skilled computer programmer.

There are other things that are wrong, like the “laches” that make it possible to submit a patent for something like, say, matter transmission now, and keep updating the submission until the technology happens, whereupon you hold a monopoly on something basic with (bonus!) a recent filing date. Nifty, huh? This happened recently with a guy (well, he was dead, but the, uh, decent company who’d bought his patent was standing up for him) who filed an application in 1955 for a patent on computer vision, which was extended and re-re-re-re-extended until it was finally granted recently. Whereupon it was used to sue the makers of bar-code readers.

Somewhere I have a copy of the infamous “Network byte-order” patent. Yup, someone managed to patent the technique of shipping bytes around the world in big-endian format. A friend-of-a-friend, he’s reported to have said “I can’t believe they gave that to me.” Thankfully, this miserable example has expired, but plenty more have not.

Recursive Suess

One of my favorite Dr. Suess works is about the bee-meta-watchers. I’d like to link to a photo, but the only ones I’ve seen are of pretty bad quality (and the one on Geocities spawns evil bad popups).

Along with an interesting reference work, kind of a Suessian concordance. Check out the Zatz-it Nose-patting extension (three seater), used to caress the nose of a Zatz-it beast. Someone tell me software is literature and I’ll scream, it’s a lot more like Dr. Suess than (say) Kafka. Um. I guess.

Searches, oh we get searches

I was viewing some of FYYFF’s site statistics. I don’t get that many hits (maybe 50 a day are real people, and for all I know only a couple of my friends ever read this stuff). What was interesting was the searches that my pages matched. Basically, most people are pretty normal … but some of you are really sick.

First we have some pretty tame hits:

dsl hell
pacbell dsl tech phone number
nucore steel
free banana bread recipe
dungeon siege keyboard commands

I haven’t had DSL in years, but apparently things remain pretty hellish; the “DSL Hell” piece I wrote four+ years ago is still getting lots of hits. I wrote a piece on Richard Preston’s American Steel, which described NuCore Steel’s start-up mill in the midwest, in the mid 80s (I took the hard work and impossible technology as an analog of the project I was working on at Apple). There’s a killer ‘nanna bread recipe buried here somewhere. Etc.

Following those are some pretty understandable things, maybe a little specialized, but nothing to be ashamed of:

why programmers dont have girlfriends
how much do computer engineers get paid
phone interview questions software engineer
robert jordan wheel of time
rapid beeping computer

1) I do know geeks with girlfriends. 2) Whatever they’re worth. 3) Hey, phone screening can be fun, as long as neither party is a moron. 4) Someone needs to take Robert Jordan’s word processor away. 5) I don’t know, try re-seating some RAM chips, or getting the cat’s butt off the keyboard.

Then … there are the weird ones (these are the tamer examples):

meat thud
thwock fume
flogging of jesus
i left my job why
inventing a new toilet
internet toilet
strippers to prostitutes
penis enlarging spells
lost down my toilet

I dunno, but some of you are disturbed.

Sigh. Welcome to the Internet, land of $$$FREE$$$ Jesus-flogging, toilet-enlarging spells. Remember, in the 50s, when TeeVee was going to be the Great Educator, obsoleting classrooms and freeing up teachers to a life of gainful employment as Asteroid Express flight attendants?

Uh huh.

Happy New Year

It’s the year of the monkey. Ook.

It should also be noted that today is Boomtime, the 7 day of Chaos in the YOLD 3170 (4), according to the Discordian calendar. (There’s also supposed to be a New Discordian calendar, but I didn’t have any luck finding it, and you can never really tell with those guys anyway. Multiple popes? Huh).