This is a bit of a geeky post, but then this is a geeky computer blog, so … so be it. Regular readers of this blog will know that I’m a fan of Nethack, and have been playing it on and off for over fifteen years (mostly off, but recently I’ve rediscovered it again). Well, finally, after all this time, I can say the following:

Goodbye marcw the Demigod...

You went to your reward with 5941448 points,
The Book of the Dead (worth 10000 zorkmids and 25000 points)
Vorpal Blade (worth 4000 zorkmids and 10000 points)
The Heart of Ahriman (worth 2500 zorkmids and 6250 points)
The Bell of Opening (worth 5000 zorkmids and 12500 points)
The Candelabrum of Invocation (worth 5000 zorkmids and 12500 points)
       8 emeralds (worth 20000 zorkmids),
       2 diamonds (worth 8000 zorkmids),
       2 rubies (worth 7000 zorkmids),
       1 amulet of ESP (worth 150 zorkmids),
       1 amulet of unchanging (worth 150 zorkmids),
and 2497 pieces of gold, after 93897 moves.
You were level 22 with a maximum of 95 hit points when you ascended.

 No  Points     Name                                                   Hp [max]
  1    5941448  marcw-Bar-Hum-Mal-Neu ascended to demigod-hood.        95  [95]
  2    2623722  marcw-Val-Hum-Fem-Neu died on the Plane of Fire.
                Dissolved in molten lava (with the Amulet).           129 [287]
  3    1330849  marcw-Val-Hum-Fem-Neu choked on her food in Gehennom
                on level 33.  Choked on a disenchanter corpse.        248 [248]

Interestingly, I almost always play Valkyries, but decided to try Barbarians for a couple of games. The first game, I made it all the way down to level 24 without finding a single altar (except for a non-aligned one in the mines with a nasty priest next to it) before an Arch-Lich and Titan finished me off. The second game was this one. Most of the rest of the time, I do embarrassing things like choke on things or eat something I shouldn’t have. Need to be more careful, I suppose.

Well, that’s all there is to this post, but it’s exciting news for me. I’m still playing, and now trying other classes (read: dying a lot).

I recently spent two weeks converting JustLooking, my Mac OS X Image Viewing program, from NSImage to CoreImage and friends. This experience was overall much easier than I expected, and I have learned a bunch of things, some of which might have been handy to have known in advance.

The good news is that it mostly lives up to the hype. The bad news is that it’s not without tricks and traps of its own. Here are some notes and comments.

1. Once set up, properly configured, coded, and tuned, CoreImage rocks. Whereas before 10fps NSImage transitions between medium images dragged on an Intel Mac, 30fps transitions on 1GHz G4s are smooth like butter.
2. Proper configuration is a must. I first just converted my various NSImage calls to their CoreImage counterparts (AppKit makes it easy to put CoreImage into your program), and saw maybe a 10-15% performance improvement.
3. Using CoreImage pretty much requires you to start getting comfy with OpenGL and its usage in Mac Applications. Instead of NSView classes, you need to use NSOpenGLView. Going to do drawing in the view along with your CoreImage? You’re going to need to learn how to do this in OpenGL (or, if you’re like me, find some samples that do the same thing you’re doing and template heavily).
4. This one took me the longest time to figure out: In Interface Builder, if your window has a subclassed NSView called MyNSView, simply changing MyNSView’s superclass to be NSOpenGLView instead of NSView and changing your code files likewise is not enough. My views were drawing all sorts of weird crap all over the place and it was driving me crazy. The part I was missing? You have to go back to IB
   • Delete your NSView from the window.
   • Drag on a new NSOpenGLView
   • And finally mark its subclas as being MyNSView (which now inherits from NSOpenGLView instead of NSView).
5. NSOpenGLView is quirky. The weirdest of them all: It often doesn’t call your drawRect: when you first display it or tell it to change its contents. Thus, it seems as though a lot of people (myself included) have developed code that sets up a single-fire timer for some short period of time (0.1 seconds or so) to cause a redraw, which is respected.
6. I still cannot get an NSOpenGLView to be semi-transparent. It’s trivial to do this with an NSView—just call [[NSColor colorWithDeviceRed: 0 green: 0 blue: 0 alpha: 0.7] set]. In some situations, overriding - (BOOL)isOpaque might be required so you can return NO. Doing this in the NSOpenGLView, however, is a monumental effort, and something I have to figure out. I found a few examples on the Internet, but was never able to make one work. (If you have done this before, please please please send me mail or add a comment).
7. Any given CIImage instance you have is probably not an actual image. Instead, it’s a collection of instructions that the CoreImage code will later use to figure out the best way to render your final image to the screen, using all sorts of cool and clever optimisations. Applying resizing, scaling, and rotating transformations are all done at once, as none are actually computed until rendering time.
8. The filters you can use with CoreImage images are just awesome. From the banal like regular NSAffineTransforms, to impressive scaling filters, to all sorts of colour and distorting filters, you might be forgiven for thinking you could write you own little Photoshop clone in a few hours. The CILanczosScaleTransform filter, in particular, provides excellent results (sadly, it isn’t quite fast enough to use in real time).
9. The transitions available in CoreImage are all quite cool, and reasonably easy to use. Most simply require the setting of a few params and you’re good to go—one or two might require a mask image. The one thing the docs do not explain very well, however, is how to use the transitions. Oh sure, there’s a full page in the Programmer’s Guide, but it blithely leaves out a few of the key steps, leaving you scratching your head. Which brings me to my next point:
10. If you’re going to use a CoreImage transition, download and learn to love the CITransitionSelectorSample2 sample. Until you start playing with CoreImage (or have read this article), you might not have noticed that it uses an NSOpenGLView, that it fires a timer to make sure that NSOpenGLView always paints correctly, and more. And It shows very clearly how to use all of pre-canned transitions in Tiger.
11. The CoreImage classes are not without quirks, possibly bugs (although I’m such a newbie at this that I’m still willing to believe they were faults of my own). In my program I simultaneously do things like rotations, translations (so that images remain centred in the window), and scales to draw images on the screen. On a few occasions, I found that CoreImage would draw the images correctly for a few frames in a transition, and then suddenly start drawing them in unexpected places. Only by eliminating one of the transforms was I able to eliminate these problems.
12. The CIImage method imageWithContentsOfURL: is extremely useful for quickly loading in images, but won’t let you get at things like EXIF or other image meta-data. You’re far better off using CGImageSourceRef APIs to load in CGImages, and then pass these to the CIImage method imageWithCGImage:. These will even let you load in all the frames in an animated GIF (something which NSImage simply does not).
13. For thoses cases where you do want to use an NSView, such as printing, you’re still more than welcome to use CIImage objects. JustLooking does its printing in a regular NSView, and just uses AppKit extensions to the CIImage class to do the drawing. Extremely convenient!
14. This isn’t so much a CoreImage observation as much as an artifact of design changes I made in order to best use CoreImage: Drawing text in an NSOpenGLView is a monstrous pain in the butt. There are various classes and tutorials on how to do this, which is quite disheartening. I avoided tackling this issue and got around this by just using other NSViews.

In summary, I’m extremely glad I took the CoreImage leap. The system is extremely cool, and has pushed NSImage to a minor utility role in my programming world. If you do decide to go for it, take heart—it’s really not so bad, and internet searches will give you all the help you could want.

The Setup

Imagine, if you will, the following scenario:

• You design a whole new database schema for your cool new scalable web-application. You’re using MySQL and the InnoDB datbase engine for everything, because your schema is so cool it uses all sorts of foreign keys and transactions and the like.
• You quickly set up MySQL and get your application going with your new schema on your development staging machine.
• You get MySQL up and running on your live server, play around with it for a bit to make sure it’s working, and then set up a my.cnf file with all sorts of caching and security goodies in it.
• You do a backup from your dev machine, restore it to the live server, and ta-daa!!! Your web application is up and running on your live server.

What you might not have noticed, especially if you – like me – have a few thousands rows of data, is that MySQL might have screwed you along the way and not really told you all that clearly.

The Problem

After a few days of operation on my live server, I started to notice a few weird things—foreign keys weren’t being enforced properly, and there were some values in the database that probably shouldn’t have been possible. I furrowed my brows and put it on my list of stuff to investigate.

Well, yesterday, I added a new table to the database, and it went something like this:

mysql> CREATE TABLE Fudgecicles
>(
>  id INTEGER AUTO_INCREMENT PRIMARY KEY,
>  value VARCHAR(255) NOT NULL,
>)
>ENGINE = InnoDB;

Query OK, 0 rows affected, 1 warning (0.10 sec)

Where did that warning come from?

mysql> SHOW WARNINGS;

It is here that MySQL tells me:

+---------+------+-----------------------------------------------------+
| Level   | Code | Message                                             |
+---------+------+-----------------------------------------------------+
| Warning | 1266 | Using storage engine MyISAM for table 'Fudgecicles' |
+---------+------+-----------------------------------------------------+

Augh! No! Bad! Bad, MySQL, Bad! Why on earth would it do that? I didn’t misspell InnoDB or even use “incorrect” casing in the name. There’s nothing wrong with the schema I specified and I’ve done this hundreds of times before.

Well, after some research, I then tried the following:

mysql> SHOW ENGINES;

And MySQL helpfully gave me the following:

+------------+---------+----------------------------------------------------------------+
| Engine     | Support | Comment                                                        |
+------------+---------+----------------------------------------------------------------+
| MyISAM     | DEFAULT | Default engine as of MySQL 3.23 with great performance         |
| MEMORY     | YES     | Hash based, stored in memory, useful for temporary tables      |
| InnoDB     | DISABLED| Supports transactions, row-level locking, and foreign keys     |
| BerkeleyDB | NO      | Supports transactions and page-level locking                   |
| BLACKHOLE  | NO      | /dev/null storage engine (anything you write to it disappears) |
| EXAMPLE    | NO      | Example storage engine                                         |
| ARCHIVE    | YES     | Archive storage engine                                         |
| CSV        | NO      | CSV storage engine                                             |
| ndbcluster | NO      | Clustered, fault-tolerant, memory-based tables                 |
| FEDERATED  | NO      | Federated MySQL storage engine                                 |
| MRG_MYISAM | YES     | Collection of identical MyISAM tables                          |
| ISAM       | NO      | Obsolete storage engine                                        |
+------------+---------+----------------------------------------------------------------+
12 rows in set (0.00 sec)

The InnoDB database engine had been disabled somewhere along the way and I hadn’t even noticed. It is enabled, by default, on the standard Linux and Mac OS X MySQL binaries that I’ve been downloading. So something changed along the way that made this all stop.

Far worse, I then began to worry about my existing tables, all of which we supposed to be InnoDB. Upon executing the following for each of them:

mysql> SHOW CREATE TABLE Fudgecicles;

I would see something like:

| Table       | Create Table
| Fudgecicles | CREATE TABLE `Fudgecicles` (
  `id` int(11) NOT NULL auto_increment,
  `value` varchar(255) NOT NULL,
  PRIMARY KEY  (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8

Sure enough, every single one of my tables was MyISAM, and not the InnoDB it was supposed to be.

The Reason

The problem turns out to be that InnoDB is somewhat finicky about my.cnf settings, and will frequently refuse to operate if settings change in this file in such a way that makes any existing data incompatible with the way it would write new data.

In my case, it turns out that changing the settings for the InnoDB binary data and log file sizes were somehow incompatible with the existing binary data and log files (ibdata1 and ib_logfileX). Upon starting the server, InnoDB finds this inconsistent state and simply refuses to start up. This is the first problem.

The second, and far more serious problem, is that MySQL just switches the database persistence engine on you and only provides a little warning. If you’re loading in thousands – if not tens of thousands – of rows, those warnings are easily lost in the scroll-a-thon that ensues. This is bad behaviour. MySQL should simply refuse to create your table if your selected persistence engine is not available.

The Solution

Fixing this problem involved three parts. The first, and easiest, is to get the InnoDB engine back. You have two choices:

1. You can either revert to the original settings you had for the my.cnf file to remove the inconsistent state that is freaking out poor ole’ InnoDB.
2. You can delete the idbdata1 and ib_logfileX files with the inconsistent sizes, thus removing in a different way the same inconsistent state, and allowing InnoDB to startup with your new (and presumably better) configuration values in my.cnf.

I chose the second method.

You then stop and restart the MySQL database engine, and InnoDB will be back. You can verify that all is well in InnoDB land by executing:

mysql> SHOW InnoDB STATUS;

And you will receive a gloriously long and detailed set of information on how things are going.

Part two of the process involved converting my tables back to InnoDB.

mysql>  ALTER TABLE FishSticks ENGINE = InnoDB;

This proves to be tedious and time consuming, as you cannot simply go from table A to table Z doing the conversion – because of the foreign keys and references, they have to be done in the right order. When one of the ALTER TABLE statements fails, you can find out what happened by re-executing the SHOW InnoDB STATUS command – it will tell you why it wouldn’t convert the table to InnoDB.

But, eventually, I got them all done. It was then that I noticed that none of the foreign keys were set up properly any more.

So, the last step of the process is to re-establish the FOREIGN KEYs. I did this by, for each foreign key in each table I had, executing the following commands:

mysql> ALTER TABLE FishSticks DROP KEY [dead foreign key name];
mysql> ALTER TABLE FishSticks ADD FOREIGN KEY (keyname) REFERENCES Table (fieldname);

The good news is that saving my database and getting back to all sorts of InnoDB goodness only took about an hour in total, for about 30 tables or so.

However, if MySQL had simply reported an error a few days earlier instead of blithely just switching tables types behind the scenes, I might have avoided this whole mess in the first place. Oh well, at least I learned a few neat little commands I can play around with now! Lesson learned!

Here’s to hoping that this article helps some other folks solve the same problem a bit quicker!

I’m just beginning a short little contract here in China for a month or so to pay some bills and help a friend out. Part of doing work in the computer industry in China is that 1000$ USD for a computer is an outrageous sum of money. Those of us with Apples are either reckless spendthrifts or simply lucky beyond belief (Mac OS X itself is simply not understood … is that a program that runs on top of Windows?). So, this morning, when one of my coworker’s computer broke, I groaned. Not only was it likely to be a huge hassle, but the likelihood of having spare parts or another working computer were quite remote.

In short, he had no network. The cable lights simply did not come on. We switched cables. We unplugged and re-plugged in everything in the office. We uninstalled and re-installed drivers. We fiddled and diddled with Windows ad nauseum. We stole other departments’ switches. No joy.

Finally, in an act of desperation, I opened the computer up, tapped the network port aluminum housing a few times, looked sagely at at the current state of affordable PC hardware (it is pretty cool looking inside those things), and then tried again …. it works fine now.

What the—???

Thanks to Keith from the UK for pointing out something odd in my book that doesn’t seem to work as it did in earlier versions of PHP:

If you have a regular expression (I use the POSIX ones almost exclusively since they’re UTF-8 aware whereas the Perl ones were not when last I inquired), and you want to set a range for the number of matches on a particular expression you can use the syntax:

$expr = '[a-zA-Z]{5,50}';        // matches between 5 (incl) and 50 (incl) letters

Now, the problem is: what if you want to have the number of characters in the range be PHP variables that you can set in a configuration file or some such thing? Your first attempt, and what I used in my book, might be:

$expr = "[a-zA-Z]\{$min,$max}";        // double quotes for var expansion

And you would get a wonderfully annoying error message from the PHP engine:

Parse error: syntax error, unexpected ',', expecting '}' in Filename on line 5

No amount of backslashes will fix this problem. It turns out that the PHP parser consumes { and } characters when performing complex variable expansion, so …. all you have to do is add an extra set around each of the variables you wish to expand. PHP leaves the other two alone:

$expr = "[a-zA-Z]{{$min},{$max}}";        // extra { }s are consumed.

And what you are left with is a wonderfully working regular expression.

I recently started working on an image viewing program for Mac OS X using Cocoa, and one of the features I decided to add was the ability to rotate images in 90° increments. I did some searching on the internet, and found a few things:

• Some other people who were struggling with the same thing and posted some code snippets.
• Code for a rotation function that supports arbitrary rotation and scaling.
• Apple Documentation on Drawing

Neither of the first two was exactly what I wanted—the first didn’t quite work, while the second was too complicated and, in order to support arbitrary rotation, created an NSImage object that was way too large.

So, after reading the Apple Cocoa Drawing Guide on Using Transforms, I came up with the following code to rotate an NSImage object 90°. Please note that this function ONLY supports a clockwise or counterclockwise 90° rotation. It also returns an object that must be released by the callER. I’m not 100% sure if this is standard Objective-C/Cocoa methodology, as I’m still a bit new to all this stuff.

But, without any further ado, here is the rotation function:

- (NSImage *)rotateIndividualImage: (NSImage *)image clockwise: (BOOL)clockwise
{
    NSImage *existingImage = image;
    NSSize existingSize;

    /**
     * Get the size of the original image in its raw bitmap format.
     * The bestRepresentationForDevice: nil tells the NSImage to just
     * give us the raw image instead of it's wacky DPI-translated version.
     */
    existingSize.width = [[existingImage bestRepresentationForDevice: nil] pixelsWide];
    existingSize.height = [[existingImage bestRepresentationForDevice: nil] pixelsHigh];

    NSSize newSize = NSMakeSize(existingSize.height, existingSize.width);
    NSImage *rotatedImage = [[NSImage alloc] initWithSize:newSize];

    [rotatedImage lockFocus];

    /**
     * Apply the following transformations:
     *
     * - bring the rotation point to the centre of the image instead of
     *   the default lower, left corner (0,0).
     * - rotate it by 90 degrees, either clock or counter clockwise.
     * - re-translate the rotated image back down to the lower left corner
     *   so that it appears in the right place.
     */
    NSAffineTransform *rotateTF = [NSAffineTransform transform];
    NSPoint centerPoint = NSMakePoint(newSize.width / 2, newSize.height / 2);

    [rotateTF translateXBy: centerPoint.x yBy: centerPoint.y];
    [rotateTF rotateByDegrees: (clockwise) ? - 90 : 90];
    [rotateTF translateXBy: -centerPoint.y yBy: -centerPoint.x];
    [rotateTF concat];

    /**
     * We have to get the image representation to do its drawing directly,
     * because otherwise the stupid NSImage DPI thingie bites us in the butt
     * again.
     */
    NSRect r1 = NSMakeRect(0, 0, newSize.height, newSize.width);
    [[existingImage bestRepresentationForDevice: nil] drawInRect: r1];

    [rotatedImage unlockFocus];

    return rotatedImage;
}

The function works great and correctly rotates the incoming NSImage 90 degrees. However, People working with JPEGs should probably not save these data to disk in place of existing images, as the function works on the decompressed bits and resaving them can lose more image data. To truly solve this problem, you should use a “Lossless JPEG Image Rotation” algorithm.

Last night, after a nice weekend of varying activities (yoga, studying, yoga, sleeping), the plan was to lie in bed and read some Apple Developer Connection documentation for a while and generally learn more about some stuff I’ve been working on lately.

So, it was with some dismay that I sat down with Samantha’s PowerBook G4, clicked on Safari and got … nothing. The icon bounced twice and then stopped. System logs showed nothing, and the application appeared to be all fine in its folder in /Applications.

Uuuuuh. Now what? Oh yes, the System Console, in /Application/Utilities. It gave me the following very helpful message:

2006-08-20 22:32:29.825 Safari[249] Unable to load nib file: MainMenu, exiting

So what was wrong with mine? Some searching around the Intarwebs suggested that it might be a permissions thing. So I fired up the Disk Utility and that said that a bunch of permissions were all wrong (how did that happen?) and fixed them.

Still no dice with Safari. There was another button on the Disk Utility to “Verify Disk”. That did not go well. Lots of red text and scary messages, and finally “exiting”.

So, in the end, I had to actually boot the PowerBook into single user mode (when the machine is booting up, hold down CMD/Apple + S), and run

/sbin/fsck -fy

to get everything working again. It took running this program twice to get everything fixed, and even then, one of the sub-nib files in the MainMenu.nib/ directory for Safari.app was still corrupted, and had to be copied from somewhere else. Luckily, we’ve got another PowerBook G4 here in Beijing, or I would have had to harrass somebody overseas for the file.

None of this is particularly new, shocking, or difficult to figure out, especially for those who have run various flavours of Unix for some time. I’ve just grown so used to my PowerBook being extremely reliable (in three years of owning a Dell Laptop running XP, I never once had half the average uptime I get on my PowerBoook) that it’s so shocking when there is a problem, I’m taken off guard.

There is an interesting point to be made here though, and that is that there no way Samantha would have been able to figure out and do this by herself. She still would have had to go to an Apple Geek Lab (or whatever they’re called) – of which there are none here in Beijing – and get somebody to look at it for her, or beg a friend (who might not have known what s/he was doing and suggested “wiping and reinstalling everything”). As nice as it can be to use, personal computing software still has a long way to go.