The Google Development Model: A single, shared code base. Fluid team and project switching. Super transparent intranet. Pet projects encouraged. Energizing intrapersonal environment.

Adam Rifkin: Life In A TEXTAREA

It’s hard to relax when you’re living in a box. I’d estimate that I spend between 10% and 20% of my life in a browser, and half of that time is spent in a <TEXTAREA> like the one I’m typing in right at this moment to flesh out this post. The reason I cannot relax is that Web forms have not evolved in fifteen years — there’s still so few text editting features that I find myself manually searching and replacing sometimes. There are few activities more useless than eyeballing a <TEXTAREA> looking for text sequences when I know this is what computers were friggin’ invented to do.

I wholeheartedly agree! Browser developers: haven’t you heard our call? Save us!

I’ve been busy:

It’s getting there!

Now I regret not getting XWindows set up on my Linux machine. XScreenSaver now has a “Boing” mode!

I couldn’t believe it when, in 2005, installing the very latest version of Debian, I was confronted with the exact same screen telling me to enter vertical blank timeouts or whatever insane stuff it needs to know to get video to work that I was confronted with back in 1995 the last time I installed Linux. Back then I figured the arcane numbers out, I don’t know how. This time I said “Yeah… no”, hit CTRL-C, and have been satisifed with text mode. Until now!

Jorn Barger, creator of robot wisdom, the first weblog, has returned? As a writer for The Register?

(via)

The reverse engineering of Google Maps begins.

(via)

Have you noticed how all the episodes of Enterprise this season are just rehashes of episodes of original Trek? An upcoming one is going to be even crazier. They’re going for a mash-up! A combination of “THE THOLIAN WEB” and “MIRROR, MIRROR”! So get this: somehow the 22nd century Enterprise is going to ren-dez-vous with the 23rd century USS Defiant… IN THE MIRROR UNIVERSE! It’s got time travel .. it’s got evil versions of the crew (probably with beards)… it’s got Tholians… and it’s got a painstakingly detailed recreation of the original set! How can they go wrong?

So then.. what’s next?

• A Piece of Mudd’s Tribbles
• Return to Tomorrow is All Our Yesterdays
• Assignment: Operation: Spock’s Brain: Annihilate!
• Let Where No Man Has Gone Before Be Your Last Battlefield
• Spectre of [Balance of (Devil in the Dark) and (Dagger of the Mind)]
• For The City On The Edge Of Forever Is Hollow and I Have Touched the Squire of Gothos
• If Nobody is Going to Mourn for Adonis, Can We At Least Get a Requiem for Methuselah?

The September that never ended is ending.

(via)

The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software - a really good article on how Moore’s Law is (sort of) ending and the performance gains we’ve been enjoying for free are (sort of) going away. I see this as a good thing: it means that software writers who can figure out concurrency are going to be more and more in demand.

(via)

Feed on Feeds 0.1.9: That problem is fixed, this stuff is included. <stevejobs>Oh, by the way… this release is also about 10 times faster.</stevejobs>

See, I’m just now figuring out this whole “database index” thing. If you’re upgrading from a previous version, re-run the installer (install.php). It will add some indexes to the items table that really should have been there from the beginning. If you’ve already created indexes yourself, and run the installer, it may create duplicate indexes. Would that be bad? Try to imagine all life as you know it stopping instantaneously and every molecule in your body exploding at the speed of light. Or if that is too scary you could just go in with phpMyAdmin or whatever you used to create your renegade unauthorized indexes in the first place and delete them.

And it also means that if you give somebody a list of RSS items and a ‘mark-as-read’ button, they’re no longer going to tolerate the full-page-reload-and-scroll-to-the-top that certain unnamed aggregators have conditioned them to accept.

Here is one of the clearest descriptions I’ve ever read of how to deal with synchronizing game state across a slow, unreliable network. The increase in complexity they had to go through to get from LAN play (Halo 1) to WAN play (Halo 2) is pretty amazing.

FoF 0.1.8 users, please see this post at LaughingMeme. Magpie 0.7 has a bug (that I think was included in some code I originally sent to Kellan… sorry about that!) which will cause character encoding translation to fail if you don’t have iconv, but do have mbstring. If you have iconv, everything is fine. If you want to fix this right away, you can download the new version of ‘rss_parse.inc’ that Kellan has made available. Of course, I’ll include this fix in the next release of FoF.

What’s that thing they say? Release early and often? In that spirit, and in the context of these two earlier posts, here you go:

Snoopy + native PHP SSL + Digest Auth: ALPHA version

This was based on whatever you call the version of Snoopy that comes with MagpieRSS 0.7. It said 1.0 in the header but I’m not sure if it’s exactly the same as the one you can get from the Snoopy project at SF, or if Magpie’s Snoopy has drifted out of sync.

This is designed to be a drop-in replacement for Snoopy. If you happen to have FoF 0.1.8, this will work extra nicely. Be warned that older versions of FoF had a stupid bug preventing subscription to https addresses, so don’t bother trying it.

Change 1: If you ask this version of Snoopy for an https: URL, instead of trying to execute curl externally, it will check if you are running at least PHP 4.3 and have OpenSSL installed. If so, it will use PHP’s built in SSL implementation. If not, it will fail back to the old behavior and try to find curl to execute. There is also a flag you can set (use_curl) to force the use of curl if you’re happier with that.

Change 2: If you ask this version of Snoopy for a URL with a username/password (ie http://user:pass@example.com), it will first attempt Basic Auth, like always. But if that fails, and the server sends back a header with a Digest Auth challenge, Snoopy will attempt to respond appropriately by issuing another HTTP request with a Digest response. The “killer app” of this feature is subscription to friends-only LJ feeds.

I’d love to get some feedback on this. So download and execute away!

It works! I have Digest Authentication working in PHP. It’s the client side, which I think hasn’t been done before. Like I thought, it wasn’t that hard. It works for the simplest (but most common? only?) case, where qop=auth and algorithm=MD5. There was one tool I needed that I was really glad to have: tcpdump. Yet another reason why Macs rule: they come with a packet trace utility all installed and ready to go, just waiting there in /usr/sbin for the day you realize you need it.

I’ll release a version of Snoopy that uses PHP’s native SSL support (if available) and this new feature in… a while. I need to clean it up significantly first.

Now that Kellan has finally integrated my character encoding changes, I figure I might as well cause him some more trouble. I’m trying to get HTTPS and Digest Authentication working in Snoopy, therefore MagpieRSS, and therefore FoF.

There is currently support for HTTPS in Snoopy, (and in FoF 0.1.8, now that I’ve fixed a stupid bug) but it depends on executing ‘curl’ externally. Yuck. And Snoopy does Basic Authentication, but not Digest. Since one of the really common things people would like to do is subscribe to a friends-only LJ feed, supporting Digest would be nice.

I think I have HTTPS working already, using code copied from HTTP_Request. Assuming you have all the right dependencies (I think just OpenSSL), SSL magically “just works” by using fsockopen with “ssl://” prepended to the host. Kind of a strange way to do it, but PHP programmers are used to such things. It seems to work equally well connecting to hosts with “real” SSL certificates, and “bogus” self-signed ones, which I’m going to interpret as a feature in this case. (In Java you have to jump through UNBELIEVABLE hoops just to say “yes, yes, I know the certificate isn’t signed, or the host name doens’t match, BUT JUST CONNECT ANYWAY!”)

As for Digest Auth, that’s harder. I can’t find s single existing PHP implementation. Well, I found the server side, but not the client side. From reading the RFC, and some implementations in other languages, it doesn’t look that hard. So I’ll try that next! I’ll probably get it to just barely work with LJ, declare victory, and ship it.

By the way silverorange’s test private feeds have been really helpful in getting this to work. Thanks for those!

One other related thing people always request is for FoF to work behind an authenticated proxy. I’m having a really, really hard time caring about that one, mostly because I don’t have an easy way to test it, and also because it just seems so weird, so I’ll likely not do it.

I’ve had all these minor improvements in my private version of Feed on Feeds for the longest time, but didn’t think it was quite enough for a release. But the release of MagpieRSS 0.7 finally pushed it over the edge. So get 0.1.8 now!

Perl 2004 Advent Calendar! Which has become a messy-78 yearly tradition.

By the way I should also mention that this weblog is now over two years old. I’m not sure if that’s old enough to make me cool yet.

MagpieRSS 0.7 has been released! Magpie is the RSS library FoF is based on. Mainly this release includes the unfortunate code that you need to deal with XML character encodings in PHP. FoF has included a version of this code for a while, but I’m glad that it’s now in the main-line Magpie.

reBlog 1.0 has been released! reBlog is a system based on FoF for reading feeds and very easily (in one click) “re-blogging”, or posting to your own weblog, the items you find interesting. It looks like for this release Eyebeam’s teamed up with some designers to spruce up FoF’s famously, uh, “utilitarian” interface.

This is what some of the best web engineers in the world can come up for image printing and download prevention. Recall that what they’re trying to do is impossible, so the fact that they’ve even made it difficult is pretty impressive.

I wonder if the Mozilla guys will consider the techniques Google’s used here to be bugs, and fix them in the next release?

Neal Gafter has a blog! Created as a response to Bruce Eckel? Or to me?

From the latest NTK:

VIEWGLOB is a little GTK+ window that sits near your xterm, and listens to you tapping away on bash or zsh. When you “cd” to a new directory, it shows you the content listing of that directory. […] As the author Stephen Bach says, “no longer will you compulsively ls after every cd”.

I made the exact same thing, for the exact same reason (the “ls compulsion”) back in 1997 at Pratt & Whitney. Mine was super low-tech: the auto-ls window was just a cmdtool that started up, wrote its PID to a file, and then went into an infinite loop waiting for a signal. Then I used an alias to override my cd command to do the cd, write the path of the new current dir to a file, read the PID file, and use that to send a signal to the auto-ls window. The auto-ls window would wake up, read the path, execute ls, and then wait again. Sure, this new-fangled VIEWGLOB has more features, but my solution was written in about one one thousandth of the number of lines of code of VIEWGLOB’s Makefile.

I also hacked my .XDefaults files and whatever those other files were called so that my windows didn’t have any borders or widgets around the edges. You could only move, resize, or close them if you knew the obscure key combinations. Yeah… I was just EVER SO COOL back then.

Waxy.org [via] links metaaggregator (aggregates all the [via] links from here) by 90% Crud.

(recursive via)

Steve Ballmer is insane. Steve Ballmer has always been insane. (via)

I finally ran Software Update and got the latest Safari, v1.2.5. It seems much faster! The lag I used to have switching between tabs, and the slowness typing and scrolling are completely gone. Nice. But… THERE’S STILL NO UNDO! Typing in a textarea is absolutely treacherous. One accidental ⌘-A and all your text is irretrievably gone, reducing the usability of my 2004 Mac to that of a 1984 DOS-based word processor. Didn’t that lady throw that hammer through the scary guy on that big screen so I don’t have to deal with this?

And while you’re at it, make it so I can undo the closing of a tab, too.

Like your JavaDocs with community annotations? And with advertising? Then JDocs.com is for you!

(combined with the super bionic searching capability of javadocs.org this would be even cooler)

I should have one of those linkyblog things. Until then:

HTTP Caching and Cache-Busting - But what if I need a cache-buster-buster? (via)

How to handle international dates and times in PHP and MySQL - I’ve always wondered how to do this properly. I am a little concerned that he mentions that all bets might be off on Windows. (via)

Great Hackers - Interesting to me because he keeps mentioning ITA Software as one of the few places that currently employs and cultivates “Great Hackers”.

Looking over the release notes for Universal Feed Parser 3.3, I found this item: Optimized EBCDIC to ASCII conversion. Words fail me.

Hey all you people who want me to rework Feed on Feeds to use Universal Feed Parser: guess what! It’s done! But not by me! Ludovico Magnocavallo started with FoF, completely disassembled it, and then put it back together in a totally new form: SimpleAggregator 1.0. He’s refactored the code, integrated a template system, redone the UI, and through the use of UFP, now supports every feed format ever invented.

I haven’t actually run this new aggregator, because I don’t quite have the technology to line up all the requirements (All the requirements of FoF + InnoDB + PEAR + Python + some Python modules + whatever the requirements of UFP are). And that is the reason why I’m not going to go this way in FoF: I want to remain 100% PHP and keep the external requirements down to a minimum. But for those of you who already have really well configured servers, you might want to check it out.

Excellent work by Johnvey Hwang.

(via)

Try this live blog search (using XmlHttpRequest) at Bitflux! (via)

Mr. Mark has updated his Universal Feed Parser. Apparently “previous versions of Universal Feed Parser naively used a regular expression on the raw byte stream to find the encoding attribute”. Using a regex to find the encoding attribute? How shameful! What kind of moron would do that?

They’ve making the versioning of Java even more confusing: Tiger? 1.5? 5.0? But it’s also still J2SE?

(by the way: free wireless from the show floor! yeah!)

Feed on Feeds 0.1.7

This version fixes that stupid mistake with htmlspecialchars in the last one that broke the forward and back navigation links on view.php. And! It includes a experimental pre-alpha version of Magpie that really tries its best to get the whole character encoding thing right. It uses this method, except it tries iconv (thanks Phil!) before trying mb_convert_encoding. Anecdotal evidence has shown that if iconv is installed, it may be more reliable than mbstring. It’s still not clear just what percentage of servers out there is going to have either one of them installed. Future versions of FoF will analyze your system during install.php and let you know what you’ve got and what encodings are going to work. For now, if you find feeds that still are getting corrupted, send me the links!

I’m going to wait another day, at least, to release FoF 0.1.7. I’m still working with Kellan to figure out the approach to translating those charsets that is likely to work on the largest number of hosts. To iconv or to mbstring? That is the question!

Plus, it’s late at night, and rushed releases can be dangerous.

Zonble works fast! He has translated “PHP, XML, and Character Encodings: a tale of sadness, rage, and (data-)loss” into Chinese, here:

PHP、XML、以及字元編碼：一則關於悲情、憤怒以及傷逝（資料）的故事

Please note that since posting the original article, I’ve still found few more bugs and enhancements in the charset handling code. FoF 0.1.7 will incorporate them all, and I’ll go back and update the article with the final code.

I came to the same conclusion. Feed on Feeds will have labels, some day. I implemented nested folders one time and didn’t like it.

Ofoto adds 5 terabytes of storage every week. They’ll have a petabyte (1 Pb = 1,000 terabytes = 1,000,000 gigabytes) of storage soon. So will the Internet Archive. That’s an incredible amount of data!

Even more incredible: within 20 or 30 years your common desktop PC will come with a hard disk that size… and big archives like these will have smashed through the exabyte range and be up in the yottabytes. One yottabyte is 9,671,406,556,917,033,397,649,408 bits.

This prediction is not insane: my first computer, about 24 years ago, had 1 Kb, about 8,000 bits, of RAM. A new computer today has 1 Gb — a million fold increase. You get about a thousand fold increase in RAM size, hard disk size, and CPU speed almost every 10 years.

Update: This code has been finalized and debugged, and is now shipped as part of MagpieRSS 0.7! Sadness and rage no more!

So I have this little program, called Feed on Feeds. It’s an RSS and Atom aggregator. For a long time I’ve known that it doesn’t quite handle international characters that well, so I set out to fix it. I knew that somewhere between input feed and output HTML page, characters were getting messed up. I adopted a policy of “UTF-8 Everywhere”: since FoF has to deal with feeds in lots of different charsets, but display them all on one page, I’d translate everything into UTF-8. I UTF-ized everything in the display code, and made sure that the DB wasn’t mucking with the characters, finally closing in on the place where it seemed characters were being munged: the XML parser itself, called by MagpieRSS, the RSS and Atom parser used by FoF.

Here’s how Magpie was creating the XML parser:

$parser = xml_parser_create();

Nice! Simple! But it munges characters, especially numeric entities. After reading some PHP docs, I found that there are two things you can set in PHP’s XML parser: the source encoding, and the target encoding. You can set the target encoding this way:

$parser = xml_parser_create();
xml_parser_set_option($parser, XML_OPTION_TARGET_ENCODING, "UTF-8");

This means: “Whatever charset the XML is in, I want you to translate it into UTF-8. And if you happen to find any numeric entities in there, resolve them into UTF-8 characters, too.”

So I tried that. But it still wasn’t working. Some feeds were translated into UTF-8 properly, but others weren’t. Feeds already in UTF-8 were re-encoded, resulting in gibberish. Reading some more documentation and bug reports, I found that if you don’t set the source encoding, PHP assumes your XML is in ISO-8859-1! I was amazed that PHP’s XML parser didn’t examine the XML prolog to determine the encoding, and further shocked that they chose such an insane default. But anyway. You can set both source and target encodings this way:

$parser = xml_parser_create("EBCIDIC");
xml_parser_set_option($parser, XML_OPTION_TARGET_ENCODING, "UTF-8");

This means, “I’m about to give you some XML, in EBCIDIC. I want you to translate all those characters into UTF-8 while you’re parsing it. Don’t forget to turn any numeric entities you find into UTF-8, too.”

That works… but presents a problem. How do you know the charset the XML is in? The only answer I could come up with: scan the XML myself, and find the encoding!

$rx = '/<?xml.*encoding=['"](.*?)['"].*?>/m';

if (preg_match($rx, $xml, $m)) {
  $encoding = strtoupper($m[1]);
} else {
  $encoding = "UTF-8";
}

That regex finds the charset declaration in the XML prolog itself, and if found, saves it in the variable $encoding. If it wasn’t found, it assumes the XML is in UTF-8 already, which is the default for XML.

So the full code is now:

$rx = '/<?xml.*encoding=['"](.*?)['"].*?>/m';

if (preg_match($rx, $xml, $m)) {
  $encoding = strtoupper($m[1]);
} else {
  $encoding = "UTF-8";
}

$parser = xml_parser_create($encoding);
xml_parser_set_option($parser, XML_OPTION_TARGET_ENCODING, "UTF-8");

That, finally, worked. All my feeds were reliably translated into UTF-8. But that was just by coincidence. All the feeds I subscribe to were already in UTF-8 or ISO-8859-1. After making this release, people complained that feeds in ISO-8859-15 and BIG-5 weren’t working. Consulting the PHP docs again, and double checking in the source code because it was just so surprising I found that PHP 4.x only supports UTF-8, ISO-8859-1, and US-ASCII. So anybody out there who wants to subscribe to a feed in ISO-8859-anything-but-1 or BIG5 of SHIFT-JIS is still screwed.

Even PHP 5 won’t help here, when it is released: It sort-of supports a longer list of encodings, but not BIG5 or GB2312, the two main Chinese encodings.

So I searched the PHP docs some more, and came up with a potential solution: mbstring! The mbstring family of functions supports a huge long list of encodings, and can translate between them. So here’s the final solution: use a regex to find the source encoding. If PHP can handle it natively, fine. If not, lop off the XML prolog, replace it with one that says encoding="utf-8" and pass the whole XML file through mb_convert_encoding to convert it to UTF-8 before the parser even sees it. If mb_convert_encoding blows up (which it will if the source encoding is not recognized, or if the function completely doesn’t exist, which I’m told is highly probable, since it is an optional extension) just give up and pass the XML straight to the parser and avert your eyes as it makes mincemeat of the characters. At least I tried.

$rx = '/<?xml.*encoding=['"](.*?)['"].*?>/m';

if (preg_match($rx, $source, $m)) {
  $encoding = strtoupper($m[1]);
} else {
  $encoding = "UTF-8";
}

if($encoding == "UTF-8" || $encoding == "US-ASCII" || $encoding == "ISO-8859-1") {
  $parser = xml_parser_create($encoding);
} else {

  if(function_exists('mb_convert_encoding')) {
    $encoded_source = @mb_convert_encoding($source, "UTF-8", $encoding);
  }

  if($encoded_source != NULL) {
    $source = str_replace ( $m[0],'<?xml version="1.0" encoding="utf-8"?>', $encoded_source);
  }

  $parser = xml_parser_create("UTF-8");
}

xml_parser_set_option($parser, XML_OPTION_TARGET_ENCODING, "UTF-8");

Surprisingly, this hack on top of a hack wrapped up in a hack with extra hack on top… worked! It was able to parse ISO-8859-15, BIG-5, even GB2312 feeds just fine, and translate them all into UTF-8 for display on a single page. I have these changes in my local copy of FoF now, and I’m going to let them burn in for a few days before I release them to the wider world, who will probably point out, within minutes, the multiple and tragic ways that even this solution fails. But until then, I proclaim that this is the state of the art in PHP XML charset-aware parsing. I think this is as good as it gets in PHP 4.x.

Footnote: when I say PHP5 sort-of supports more encodings, this is what I mean: PHP5 (I looked at RC3, maybe these bugs will be fixed by the final release) is completely nuts. The XML parser supports a bunch more encodings, but they are really hard to get to. If you try to explicitly set the input encoding, the PHP code limits you to UTF-8, ISO-8859-1, or US-ASCII, even though libxml2, the underlying parser, supports many more. But, if you know the super secret codes, you can construct the parser this way:

$parser = xml_parser_create("");

Notice the difference? In PHP5 bizarro world, passing in an empty string means “do what you should have done all along, auto detect the stupid encoding!” But, there’s another problem: if you auto-detect the stupid encoding this way, the stupid target encoding is stupidly set to ISO-8859-1. I don’t know who would want that. And it goes against the documentation, which says by default the target encoding is set to the source encoding. And again, you are restricted artifically from setting the output encoding to anything other than UTF-8, ISO-8859-1, or US-ASCII. So you could, if you want, use a regex (yuck!) to find the source encoding, but you wouldn’t be allowed to set the target encoding to match. But, at least, you can do this:

$parser = xml_parser_create("");
xml_parser_set_option($parser, XML_OPTION_TARGET_ENCODING, "UTF-8");

Meaning, “Auto detect the source encoding, and then translate everything, including numeric entities, into UTF-8.”

At least you will be able to… when PHP5 comes out, and is installed on the server where your application needs to run, which for me (I get complaints that FoF won’t work on PHP 3) probably won’t be for several years.

Feed on Feeds 0.1.6

0.1.5 still had some pretty major bugs. I’ve fixed those, and tested this relase pretty thoroughly. Hopefully this one has no “SEVERITY ONE” issues.

By the way, I’m very very impressed with the “many eyes” effect on these last few releases. The community of FoF users found all the bugs almost immediately… within hours! And in the middle of the night! (at least, it was night here). And, had not only had they posted good bug reports to SF, but they even found the root causes, and produced patches! I wish the QA group at work was one tenth this good.

“Dear Steve, noticed you just checked in Boring.java. You’re using an unsynchronized Hashmap at line eight billion and seven. Funny thing, that will work most of the time, but it turns out that in cluster mode it will cause silent data loss. Checked in the fix for you, k thx bye.”

So thanks, everyone, for bearing with us during the technical difficulties, and helping out to solve them!