The release of PHP 5.3 is due sometime soon and with a feature freeze in place since the 24th of July and a pre-release alpha now available, it’s worth exploring some of the many additions and changes that are going to be introduced.

As PHP is the language I most frequently work in and one which I’ve done all sorts with (from web applications, to file exploration to media player scripting), I like to think I’m sensitive to deficiencies and oddities in the released implementations. Version 5.3 contains a lot of elements backported from the still distant version 6, the most glaring omission being end-to-end Unicode support without mb_* fudges or iconv; being able to use string-backed functions like array_unique() without suspicion will be a big help, but I digress.

The most high-profile addition is that of namespaces, gone will be the warts that dot current frameworks (e.g. Zend_Db_Table_Rowset) which will make different frameworks and modules far easier to use and far more friendly when you want them to play nicely together.

Static functions have also been promoted to all a lot of the meta-programming niceties that member functions have including true overloading support which will allow first level abstractions such as database wrappers to not require instantiation before being called (which I discovered around the same time as my get_class exploration). For instance, if using an ORM, doing People::getAllById() will now be easier to achieve. Along side this many of the magic methods have been tightened up to make them less ambiguous (__get can only be public and not static, signatures enforced etc.)

Looking through some of the other changes detailed in the PHP Wiki it seems that a selection of new functions surrounding garbage collection are now being exposed including checking whether it is enabled, and selectively enabling or disabling it. Whether this is a mistake (close by get_extension_funcs() is detailed as a new function but appears to have been in since PHP4) and these are bleed-throughs from the Zend Engine is unclear, but without some surrounding memory management facilities, it would seem unwise to disable or allow disabling of garbage collection.

On the extension front numerous ones have been standardised and moved into the PECL system which goes some way to neatening things up; the change some are talking about is the choice between a local MySQL library (mysqlnd) versus the native libmysql library that comes when compiling against a MySQL release. PHP and MySQL have always been bedfellows despite their conflicting release licenses (especially so since Sun gobbled up MySQL) so this seems like a smart move for all concerned with separate code-base, better engine integration and statistical analysis now possible (PDF details).

What all of this adds up to is a release that’s solid on paper, but the bum-rush for patches is sure to be as swift as any other PHP release. Especially with the OO enhancements though, it feels like these should have been included from day one, as not only will there now be a disjoint between PHP4 and PHP5 shared servers, but PHP5.2 and PHP5.3 as well. For someone who runs their own server this is not massive worry, especially when the list of backwards compatibility changes are so small, but for service providers (hosts, ISPs etc.) still dragging their feet over 4 > 5 > 5.2, this adds another step of complexity.

The real test will obviously be the frameworks and high profile applications that PHP utilises and with word that the Zend Framework won’t be supporting namespaces until its 2.0 release next year the lead time could be immense, especially when you consider phpBB, what was once considered the yardstick of PHP usage, still supports 4.3 with its most recent version, the playing field for cutting edge PHP seems less than agile.

Forms are perhaps the bane of web development for me; you can’t get them to look good, you can’t find a foolproof way to make them act well and lets not even start of trying to get them into a pacified state, free from the dangers of user input (surprise ending: form input will never be completely trustworthy). A lot of sites would appear to have aesthetically pleasing forms, however this is a careful ruse by them as they sidestep the problem of forms by having only one or two of them, and then they usually only have a few fields. The monstrosities I am required to deal with almost daily are things of grotesque beauty, veritable Rube Goldberg machines of complexity.

The long and the short of this diversion into why forms are evil (please, end my suffering quickly XForms) is that to get a form looking good, you have to spend a long time fiddling with things. Enough of this banter anyway, my fiddling with JavaScript (like the dirty little bastard child of C and Perl it is) produced a way of creating a tabbed form that defaults to a standard single form if a user prefers to use NoScript or an antiquated browser of yore.

So the following markup:

<form method="post" id="theForm">
	<fieldset>
		<legend>First tab</legend>
		<ol>

			<li><label for="formone">One</label> <input type="text" name="one" id="formone" /></li>
			<li><label for="formtwo">Two</label> <input type="text" name="two" id="formtwo" /></li>
			<li><label for="formthree">Three</label> <input type="text" name="three" id="formthree" /></li>
		</ol>
	</fieldset>
	<fieldset>

		<legend>Second tab</legend>
		<ol>
			<li><label for="formfour">Four</label> <input type="text" name="four" id="formfour" /></li>
			<li><label for="formfive">Five</label> <input type="text" name="five" id="formfive" /></li>
			<li><label for="formsix">Six</label> <input type="text" name="six" id="formsix" /></li>

		</ol>
	</fieldset>
	<fieldset>
		<legend>Third tab</legend>
		<ol>
			<li><label for="formseven">Seven</label> <input type="text" name="seven" id="formseven" /></li>
			<li><label for="formeight">Eight</label> <input type="text" name="eight" id="formeight" /></li>
			<li><label for="formnine">Nine</label> <input type="text" name="nine" id="formnine" /></li>
		</ol>
	</fieldset>
</form>

The default will be to display the first fieldset, with links in a list to display the other two. A trivial form like this certainly doesn’t require a tabbed layout, but a monstrosity that contains 27 input fields (some multiple choice) could do with a little information management when displayed to the user. The general markup is what I’ve settled on for the majority of my forms and is based almost entirely on Nick Rigby’s article on ALA but the styling isn’t what’s important here.

For this project, like any I undertake with JavaScript, I’ll be using the Prototype library (version 1.6 specifically for this snippet), this could be done without it with minimum fuss but Prototype is lovely so I usually already have it included.

The functionality of this project is pretty minimal, the building of a list of the available fieldsets lies at the core of it. When the script is invoked it will hide all but the first fieldset, build an unordered list of the fieldsets (taking the names from the <legend> elements) and then set up event listeners for that list to change the visible state of each fieldset.

First things first, set up the Javascript object and hiding of the fieldsets:

var tabbedForm = {
	init: function() {
		var formElem = $('theForm');
		if(formElem)
		{
			$A(formElem.getElementsByTagName('fieldset')).each(function(s, i) {
				var fieldsetId = s.identify();
				// hide all but the first
				if(i != 0)
				{
					s.hide();
				}
			});
		}
	}
};

Nothing spectacular, uses the oft ignored index property of the each() function to scry when it’s not the first in a list, there are plenty of other ways of achieving this. Next job is to build the list of available fieldsets and plop that into the document at some point, so augmenting the init() function:

init: function() {
	var formElem = $('theForm');
	if(formElem)
	{
		var listElem = document.createElement('ul');
		$A(formElem.getElementsByTagName('fieldset')).each(function(s, i) {
			var fieldsetId = s.identify();
			// hide all but the first
			if(i != 0)
			{
				s.hide();
			}

			var legendElem = s.down('legend');
			if(legendElem)
			{
				var listItemElem = document.createElement('li');
				var linkElem = document.createElement('a');
				linkElem.href = fieldsetId;
				linkElem.innerHTML = legendElem.innerHTML;
				Element.addClassName(linkElem, fieldsetId);
				Event.observe(linkElem, 'click', tabbedForm.tabClicked);

				listItemElem.appendChild(linkElem);
				listElem.appendChild(listItemElem);
			}
		});

		Element.insert(formElem, {before: listElem});
	}
}

An unordered list item is created, the for each fieldset, the <legend> element is nabbed and its value used as the title for each list item. Probably the only questionable part is making the link element point to the ID of the fieldset, this is just how I do things so that when a link is clicked, the ID is available. Other people I know put these sort of items within the Javascript object itself or in a classname or somesuch, whatever works for you; I don’t have to worry about non-Javascript users clicking the links because the entire structure is generated rather than marked up. I drop the completed unordered list above the form element which fits with the “tab” metaphor we’re aiming for.

The only remaining function is what happens when a link in the generated list is clicked which according to my event listener is called (cunningly enough), “tabClicked”:

tabClicked: function(evt) {
	Event.stop(evt);
	var linkElem = Event.findElement(evt, 'a');
	var formElem = $('theForm');
	if(linkElem && formElem)
	{
		var idToShow = linkElem.href.substr(linkElem.href.lastIndexOf('/')+1);
		$A(formElem.getElementsByTagName('fieldset')).each(function(s) {
			if(s.identify() == idToShow)
			{
				s.show();
			}
			else
			{
				s.hide();
			}
		});
	}
}

After stopping the link click event from bubbling up any further it grabs the clicked link element (I find it best not to take for granted which element has been clicked and just do a “findElement” to make sure we’re on the same page), pulls the ID from href attribute then iterates through the form’s fieldsets to find the one it refers to.

At this point the scripting is completed and a barebones proof of concept can be seen. Obviously with no style it’s not going to look like tabs, but with a little sliding-door tomfoolery, you’ll be tabbed up in no time. At this point you’ll likely want to expand on the functions above by dropping in some choice CSS classes, setting the active tab to “on” for appropriate styling and maybe even adding some other classes to let your stylesheet know things have been modified by a script (I find simply added a “scripted” class to the container element works wonders).

The beauty of this is it’s accessible (the form still works 100% without scripting) and it prevents a user from seeing just what a mammoth form they may be completing (blood of your first born? yes please).

I plucked these links from my RSS reader (RSS Owl, how I love thee) however there is one category still left to add, the ubiquitous “Stuff” category which more or less defines this site as well as my life so take from that what you will. I’m hoping to tweak the design at some point and possibly add RSS feed links (as that’s usually how I read them) but that’s for another time.

Note (2008-01-11): The full (official?) SDK for this file format has been located which includes both a packer and an unpacker as well as other tools I’m sure are useful for working on the file format. The full name of the file format is “Yaneurao” with the SDK going by the nomenclature of “yaneSDK” which is the stem for the file format signature of “yanepkDx”. There is already a .NET version of the SDK so if you’re interested in my deconstruction process then read on, otherwise I would recommend using the official/fully-featured SDKs.

The compression format was identified as LZSS and reading through several sites revealed that some of the data I had initially spotted but attributed to SHIFT JIS (or at one point a Unicode Byte Order Marker, perfect for a non-Unicode file) were the tell-tale signatures of LZSS; the gradual degradation into junk data was also typical of the algorithm as the further into the file the stream progresses, the more back references are present.

While I hadn’t heard of LZSS, it came as no surprise that it was a modified version of LZ77 which I had come across before though never toyed with. Having to dig through a dense PDF was not my idea of fun and my university days had proven that reading academic proofs rarely lead to workable implementations for me so I searched for a ready-made PHP version which (for reasons which will soon become glaringly apparent) didn’t prove fruitful. After coming up against dead-ends with other languages I settled on the defacto C version which seemed most other versions I found were based off.

Ignoring my original deconstruction script for the moment, I worked on the assumption that each individual file contained within the large .dat files were individually compressed given that each file had a readable opening section of bytes and (according to the LZSS spec) didn’t have any back references. Like with other implementations of algorithms I didn’t fully understand, I copied the C code more or less exactly, altering formatting to my tastes and altering code to take into account any PHP idioms that I could foresee. Checking things over, I pumped in one of the compressed files and, unsurprisingly, the output file was more or less blank. After rechecking the code and running it again, the output file was once again filled with spaces and some sporadic junk bytes that didn’t look familiar.

The script wasn’t even outputting the uncompressed data at the beginning of the file and the output was larger than the input but still not the size flagged in the original .dat files. After scratching my head for a while I set about spitting out some debug data to pinpoint what had gone wrong and where. The algorithm is broken down into roughly three main sections, in two main control structures. Putting in some basic output formatting to check each section was executing proved that each section was being run in a way that I could only assume was correct:

This assumption of course turned out to be false but I wouldn’t realise this until later in the day. The LZSS algorithm uses a number of constants to define things such as the size of the sliding buffer window, maximal reference length and minimal reference length (a change from the LZ77 algorithm to prevent the encoding being longer than the original) so I tweaked the values first with sensible then ridiculous values only to have the script spit out similarly broken output. The C algorithm also had several places where it used hex values to do bitwise operations, converting these to decimal (obviously) proved ineffective and I was ready by now to admit that I was stumped. I had been working on it for a while now so I took a break for lunch, during which I decided to ditch the C algorithm and start from scratch so that I actually understood what was going on.

This proved even more torturous so I switched back to my original script and started spitting out some fairly detailed output including: the section of the algorithm, the current byte location in the stream, the hex value of the most pertinent read byte and the binary value of that byte.

This more or less nailed down that the entire implementation was broken, the values it was generating from the very beginning were incorrect which of course meant all the back references and so forth were incorrect. Using the binary output and a bit of paper I worked out what the values were supposed to be and started following the values through the algorithm. This part was absolutely essential to working out what was wrong with the implementation of the algorithm as it elucidated what each part did:

1. The first section (which I had termed “FLAGS”) worked out whether a byte was a control byte and set a flags variable
2. The second section (which I had termed “AND1”) assumed it was reading a raw byte and simply wrote it to the output stream (and the buffer).
3. The third section (which I had termed “CONTROL”) read the two control bytes which formed a back reference and then read the appropriate data from the buffer and subsequently the output.

From my output it was apparent the meat of the algorithm, reading the raw data, wasn’t being done. Then, in that moment of lucid elation, I realised exactly what was going wrong.

PHP was grabbing a byte from the input file as a string, and being a loosely-typed language meant that when it came to doing bitwise operations, the underlying type was incredibly important. I’m more than willing to admit that this state of affairs was my own damn fault for prototyping this in a language that wasn’t built for algorithms and bit level operations and had I done this in a strongly-typed language, everything would have been dandy. Of course, had I simply dumped the implementation I found into a C file and compiled away, I wouldn’t really understand what was going on, so my retardedness didn’t go to waste.

Long story short, forcing the read bytes into an integer using the ord() function (and intval() just to make sure) solved the issue and the file I was working on transformed before my eyes.

Almost.

Turns out what “sage” had said in my comments on the original version of my unpacker was slightly wrong, the sliding window wasn’t 256 bytes (0x100) but the standard LZSS implementation window size of 4096 bytes which means that nothing really needed to be changed from the standard C implementation of the algorithm. As a proof of concept:

Sample LZSS compressed file, Sample uncompressed file

So I now present version 1.1 of the deconstructor script which is released under the same Creative Commons Attribution 2.0 Generic license.

deconstructor1.1.zip (1.4KB)

The usage is exactly the same:
php deconstructor.php data1.dat output\

The only difference will be the output spat out by the script which will tell you when a file has been decompressed and whether it succeeded or failed (done by checking the canonical size in the .dat file versus the output size).

To-do
At the moment the script outputs a file to a temporary name and then operates on that file. This isn’t optimal but I was having trouble getting my implementation to work in-stream, probably due to fatigue. I may or may not fix that for the PHP version as the next step is to drop the entire deconstructor into a C or C++ file and do a native compile so you don’t have to mess around with PHP and I feel like I’ve developed something in a big-boys’ language. If I get the time and the inclination I may do that over the weekend.

As well as the unpacker, I get the feeling that the friend who this is a favour for will require a repacker which will obviously mean doing the LZSS algorithm in reverse and also bundling everything into a .dat file. Should be an intriguing challenge to see if I’ve learned anything from this little endeavour.

Sample File 1, Sample File 2, Sample File 3

All screenshots taken from data1.data, sample file 1 and the window is resized for the most appropriate screenshot rather than general workability.

First thing I did was to crank open the lovely XVI32 hex editor and have a look at the sample files provided, their .dat extension more or less indicated they were a proprietary format and were unlikely to relinquish their secrets easily. What was known was that the files contained a header portion, a bundle of XML files in a contiguous stream and a lot of junk data. The XML files could be seen and their encoding was stated as SHIFT JIS and, after cursing its existence, I attributed the junk data to that which seemed like a good place to start.

The first eight bytes seemed to be a file signature, but Google searches for all or parts of the signature were fruitless which meant it was time to pick things apart.

The next four bytes were different for each file and at first I thought it was part of the block format that made up the header part of the file but the section repetition for the header block didn’t match up so after converting it to a variety of different number formats (I’m no hex wizard and I originally thought it was only a two byte short rather than a four byte integer or long) and assumed it was an unisgned long (32 bits) in Little Endian order.

The next section pattern repeated a number of times until the file obviously started with the embedded XML files. After a bit of byte counting and “duh” moments, the general format of the section is:

256 bytes - file path and name
4 bytes - unsigned long
4 bytes - unsigned long
4 bytes - unsigned long

At a total of 268 bytes for each block, this layout repeats for precisely the number of times specified by the very first unsigned long (after the file signature). So the entire header block consists of:

8 bytes - signature "yanepkDx"
4 bytes - number of header entries
(number of header entries * 268 bytes) - header entries

This was all well and good but didn’t really illuminate exactly what the three numbers were. After pulling out all the entries, a few things became clear:

• The first number in each block increases for each successive block
• The second number was always larger than or equal to the third number
• The first number plus the third number always equalled the first number of the block immediately after the current one

So without resorting to rocket science the first number is the absolute byte offset of the filename, the second number was a bit of a mystery, the third number is the length in bytes of the data in the file. After pushing this info through a script it became obvious this was the defacto format of the file, no complex tree structures or other nasties were awaiting; the XML files were pulled out without problem and within a few minutes their original file structure was recreated.

All done right? Wrong. My initial thought that the XML files were SHIFT JIS encoded was indeed correct, however it didn’t solve the junk that proliferated some the files.

Sample un-junked file, Sample junked file

Trying to shift the format into different encodings using known functions only seemed to jumble the junk around rather than get rid of it. It now became apparent that the data was more than likely compressed or otherwise encoded which illuminated what the mysterious second number was in each of the header blocks. The third number represented the packed size of the data, the second represented the unpacked size; this was obvious as the smaller, un-junked files had the same values for each, usually less than 150 bytes.

Running both the individual files and the larger .dat file through various decompressors proved less than useful as most of the time the file became so garbled that it sent a few hundred bell tones to my computer speaker making it sound like it was having a seizure. I tried various versions and functions of the gzip/zlib library, bzip2, LHA (of which I knew the Japanese were particularly fond of) and of course good old fashioned zip. It stood to reason that the compression wasn’t going to be processor intensive (very few game compression schemes are) which more or less ruled out predictive text algorithms (PPM et al) as well as ACE and 7z formats. The files also seemed to lack any form of dictionary entries as for each file the XML declaration was always in tact which meant that the compression seemed to start an arbitrary length into the file (which would explain why the smaller files were untouched).

This is unfortunately as far as I got after a mornings work and spent a decent amount of time attempting to track down information. The game the files comes from is Battle Moon Wars Act 3 and it seems that they use TYPE MOON characters, other games of which have been successfully translated which may be one avenue to investigate. The developers of the game are “Werk” and if any of their other games (either in the series or otherwise) had been pulled apart, it may give some indication as to where to go forward. There does seem to be information in someone’s brain as not only was an “unpacked” version of data1.dat unearthed, but forum posts indicate that work had already begun (if not already aborted) on the technical side of things.

For today at least I’m done with attempting to reverse-engineer arbitrary files and perhaps after sleeping on it some bright idea will be revealed to me that daylight failed to illuminate. For now there is the command line PHP script I quickly prototyped to deconstruct the .dat files (released under the Creative Commons Attribution 2.0 Generic License) and the promise of further work in the future:

deconstructor.zip (1KB)

Things should be self explanatory from the file; get a command line PHP interpreter set up and run “deconstructor.php” with the name of the file to tear apart and optionally an output folder e.g.
php deconstructor.php data1.dat output\

This is an open call for anyone who wants to help with the effort to scry the encoding/compression of the XML files whether you already know or want to take a stab at it, you are more than welcome.

I have tweaked a couple of items in the backend. The first is the RSS category links; as I have been known to post about whatever I feel like, taxonomy feeds are a good way of ignoring the general rambling I tend to do while squiffy or bored. The problem was that my permalink structure on Wordpress caused some problems with the automatic feed links which made me to hold off implementing them, the solution was simple and worked a charm. The other change was the aggregating of very old entries in the archive list; this was something that I had done when I programmed my own abortive attempt at a blog system but wasn’t something Wordpress did natively. I did think of putting it into a plugin but I considered that overkill as I was already monkeying around with the guts of Wordpress. I’d made the decision to stick on 2.2.3 and not upgrade to 2.3 until I was sure all the bugs in 2.3 had been rooted out; taking a quick glance at the trac listing for open tickets for 2.3.1 and beyond reveals very little that would stop me however I’d been burned by the near weekly 2.2.x updates of before. The reason there are very old entries is due to the merging of my old DeadJournal entries with this blog; primarily for posterity but (as I mention in the category description, another new addition) also to remind me of what my writing used to be like.

As far as future entries is concerned, I’ll finish my draft review of Paprika and start one on 5cm a Second which should provide enough time to start a new feature: the three episode taste test. While this anime season doesn’t look nearly as spectacular as the last one, it has a number of shows which may hold interest. Even with most series now on two episodes it’s pretty obvious which ones I’m going to stick with, however three is enough to get over the budget burn of the first and languid story/character drive of the second. At some point I’m also hoping to finish off my Cuba write up, although I need to retroactively populate previous entries with photos (already done for days one, two, three and four) and continue writing; each entry can take close to two hours to write and check which is basically the entire evening.

There are other things on the horizon as well with chaostangent.com in general. I recently attended the Future of Web Apps conference in London which was a bit of a mixed bag but generally awesome. Apart from the highly anti-social laptop usage and people continually trying to sell me things, most all of the developer talks I attended were excellent, the stand out one was by John Resig of the Mozilla Corporation which was both very informative and interesting. One thing the conference did do was give me some ideas for a project which has been sitting on the back burner for a while now. I’ve also been fiddling with Wordpress more and more having decided to stick with 2.2.3 for the time being so I can hopefully start integrating some of my ideas that I’d originally written off due to restrictions on the plugin architecture.

You can consider this post to be an interstitial between your (sometimes) regularly scheduled program, just to indicate that I’m not going to slip into another four week lull.

It surprised me then that AIR, the Adobe Integrated Runtime, seemed such a departure from what Adobe had released previously. Essentially it is a virtual machine that allows developers to use HTML, CSS and Javascript or Flash and Flex to develop “Rich Internet Applications” for the desktop. That’s the corporate rhetoric, and it took me the best part of a morning to figure out exactly what AIR was trying to do and the rest of the day wondering Why?

AIR is nothing more than a virtual machine in the same vein as the Java Virtual Machine: it provides a central target for development and, in theory, will allow programs developed for it to run identically across all platforms which support it. In writing this an admirable goal (Java’s maligned “Write once, debug anywhere” idiom notwithstanding), it’s not until you dig deeper until you realise just how ridiculous AIR appears. Ignoring the jargon and the acronym soup, AIR allows HTML, Javascript and Flash developers to create applications for your desktop; the end result is a program which installs and runs from your computer instead of the vicious wilds of the internet. The question is raised really as to why I would want a Flash developer to go anywhere near my desktop: a pantheon reserved for people with some semblance of UI design which the vast majority (not all) of Flash developers lack.

Both the sample and showcase applications are less than stellar, with such awe inspiring developments as an RSS reader! A desktop bookmarks organiser! An online storage solution! Or how about a good old fashioned map? What about some eternally useful desktop widgets? Testing them out I could only think: five years ago these would have been merely interesting.

“Rich Internet Application” seems like some kind of multilayered oxymoron, instantly conjuring up a swathe of gradients and pithy animations, married with protracted jargon like “mash-up” and “web 2.0” all the while robbing me of my standardised UI interactions. The web is only just taking its first formative steps into this field greater interactivity; we’re still testing this through a browser, not an experiencer. Until both the technologies and mindsets involved mature, we certainly don’t need to bring this frontier pursuit to the desktop.

Of course, the one good thing to come from this distasteful look at where Adobe is heading is the chance to engage with some of the other technologies both Adobe and others have developed. Flex was next on the list and after once again cutting through the advertising spiel and marketing jargon, it turned out to be a set of UI components for Flash. Obviously Flex has come a long way from the Server/Client setup that the late Macromedia used, and with a neat XML system, Flex attempts to bring Flash to application developers. Once again I fail to see why this is a good thing. Apart from the speed of prototyping, I can think of no Application Developer worth their title who would seriously consider Flash over more capable, accessible and mature technologies.

So two for two, Adobe seem to be pushing technologies which when isolated, are advanced and progressive, but seem to focus more on the wow than the why.

This trail ultimately led to the Microsoft “Flash-killer”, Silverlight. Essentially Silverlight is a combination of both Flex and Flash in a far more, if you’ll forgive the pun, flexible coating. Silverlight attempts to offer a genuine alternative to standard applications with a still evolving set of UI features but with the capacity for tried-and-tested programming languages to control them. Fundamentally, Silverlight suffers from the same downfalls as an AIR/Flash/HTML combo: accessibility and speed. However it seems that Microsoft has a far better grasp of their goal (a better form of online application) than Adobe.

Some of the criticism levelled at Microsoft over Silverlight include their lack of adherence to standards such as SVG. Ordinarily I would agree with this given the ongoing fiasco (to put it lightly) with Internet Explorer; however Silverlight and its competitors are still very much emerging technologies and it’s only the disparate parts which have been standardised rather than the entire package. Microsoft is a paid-up member of the W3C and could well have had a hand in those standards; but with increasingly vocal barbs being thrown about as to the practices of the W3C, it makes one wonder whether the standards are actually worth anything. Microsoft once said that just being a standard does not make it the best way to do things; this may seem like a spoiled brat reinventing the wheel (how’s that for metaphor mixing?) but in a small number of cases, this is very much true. You can be just as much hindered by those standards as empowered by them.

And then of course, at the very end of the rainbow was the recently released Google Gears which claimed to be able to run web applications offline. Once again, I fail to see the point in this beyond mobile devices, but kudos to Google for making the implementation rhetoric free and refreshingly simple to explain.

The benefits for doing this versus the work taken to achieve it weigh heavily on how much you value “nice” URLs in terms of memorability and search engine snuggliness. Request routing allows for arbitrary URLs, however, it adds an overhead to nearly every page request made to your application and also means your PHP system now needs to deal with the 404 error messages that the web server would have otherwise transparently sent. Despite it’s downside and moderately complex methodology, for large or commercial systems, the ability to tightly control the URLs can greatly aid extensibility and maintainability.The only other article I’ve read on the subject seemed to get the idea but it lacked a lot of nice functionality. The system mentioned in those articles can easily be achieved with some Apache mod_rewrite trickery without the hassle of involving PHP. Indeed, the system proposed allows for only one type of URL and doesn’t touch on the 404 handling. An ideal way would be the ability to route arbitrary as well as tokenised URLs. If you’ve programmed with RoR or Cake or even (to a certain extent) the grand-daddy of frameworks, Fusebox, then this is all probably sounding very familiar.

The way your PHP system is set up will ultimately define which tokens are available, but for most modern OO frameworks, they follow the MVC pattern which means for a request for most pages will be sent to a controller (or command) which will undertake an action on a particular set of arguments, usually a unique identifier, id.

Getting the URL into PHP

Before all that however, we need to get PHP to deal with requests which involves alleviating our webserver of the task. For a standard Apache, mod_rewrite and .htaccess system:

RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]

If a request does not reference a file and does not reference a directory, pass it to index.php. The same task can probably be done on lighttpd using a either url.redirct or url.rewrite-repeat. IIS apparently doesn’t do request based conditions out-of-the box and an additional ISAPI module must apparently be used.

In this example the request is passed to index.php, this is because I use a single entry point for my applications (which has a number of benefits such as application hiding, strict error checking and so forth), however this can easily be redirected to any script within your system; and as products like Wordpress have shown, this approach works in a multi-file as well as single entry point system.

With the URL now handed over, the first task is to strip down the URL into only the part we’re interested in. This means peeling off any query strings or base-directories which may have come along for the ride. I’ve come to the conclusion that there is no compact way of achieving this, and even the code snippet below omits some of the thornier details:

$parsedRequestURI = parse_url($_SERVER['REQUEST_URI']);
$requestURI = (!empty($parsedRequestURI['path'])) ? $parsedRequestURI['path'] : $_SERVER['REQUEST_URI'];
$baseDir = dirname($_SERVER['SCRIPT_NAME']);

if(substr($requestURI, 0, strlen($baseDir)) == $baseDir)
{
	$requestURI = substr($requestURI, strlen($baseDir));
}

If you’re using Windows, you’ll need to either find a different method for calculating the base directory or make sure the path separators are a forward slash not a back slash (a simple str_replace would suffice). There isn’t anything particularly taxing in that snippet and at the end of it all, $requestURI will have the URI that we want to focus on.

Our routes

The concept of a route revolves around being passed a URL and matching it to a specific part of our system. If your router isn’t going to be part of an MVC framework, then the system matching may simply be a static template or another script to run. The only part of the following explanation which needs to be adapted is what is spit out at the end for your system (or “dispatcher”) to deal with.

In my case, the routes is simply an array:

$routes = array('random' => array('command' => 'image', 'action' => 'random'),
	':command/:action/:id' => '', // standard routes
	':command/:action' => '',
	':command' => '');

The array key is the URL we wish to match against (complete with tokens) and the value is an array of options we wish to supply our system. Obviously for a fully tokenised URL, we want the options to be constructed from the URL (which sounds cyclic but bear with it) hence why those values are the empty string.

The crunch

We now have both parts necessary to route: the URL supplied by the user (e.g. mycommand/youraction/hisid) and a list of routes we’re going to match against. Our router has to perform one vital function: match the route given the URL. The converse of this function, get a URL from a route, will only be used internally but allows the PHP system to completely decouple itself from the concept of URLs (should it choose to do so).

The routeFromURL() function will break down our URL (and subsequently our list of URLs to match against) into parts, then iteratively attempt to match those parts until a full match is found. To break down the URL into parts:

$requestedRoute = array_merge(array_filter(explode('/', urldecode($url))));

This decodes any errant URL identifiers, splits it on the forward slash, filters out any blank entries then finally renumbers the index using the array_merge function. The renumbering of the keys is important when matching like-for-like on URLs.

foreach($this->routes AS $route => $options)
{
if(!empty($route))
{
$routeParts = array_merge(array_filter(explode('/', $route)));
$matched = false; // this specifies whether we've matched so far, if not, move on to the next route

// routes do not have to match absolutely, only the first parts of the route can match
if(count($routeParts) <= count($requestedRoute))
{
	for($i = 0; $i < count($routeParts); $i++)
	{
		// specialised matcher parts of the route
		if(substr($routeParts[$i], 0, 1) == ':')
		{
			$special = substr($routeParts[$i], 1);
			switch($special)
			{
				case 'command':
				case 'action':
				case 'id':
				case 'extra':
					$options[$special] = strtolower($requestedRoute[$i]);
					break;
			}
			$matched = true;
		}
		elseif($routeParts[$i] == $requestedRoute[$i])
		{
			$matched = true;
		}
		else
		{
			$matched = false;
			break;
		}
	}

	if($matched)
	{
		return $options;
	}
}

The iterations may be a little hard to follow in that snippet, essentially: for each route in our route table > for each part of that route > if it matches the equivalent part of the supplied URL > go on to the next part ELSE break and go on to the next route in our table.

At the end of this we’ll have an $options array which details our command (controller), action, id and anything extra. The comments in the code indicate that this isn’t a strict matcher and that the specificality of a match matters. All this means is that passed a URL of “this/is/a/route”, if the route table contains a match for “this/is”, this will match even if there is another route in the table after it (“this/is/a”) which is more specific. The easiest way to deal with this is to either strictly match the number of parts in the passed URL to the matched URL:

if(count($routeParts) <= count($requestedRoute))

becomes:

if(count($routeParts) == count($requestedRoute))

Or to simply order your route table with your most specific routes first. I’ve not had a problem with this behaviour, however your circumstances may dictate exact matching as a requirement.

The token aspect of this system allows for very flexible routes to be constructed which may be wholly tokenised (such as “:action/:id:/:command”), a hybrid (“image/show/:id”) or arbitrary (“show/me/the/money”). The number of tokens you can use is entirely customisable, the use of a switch statement prevents a large if block but for a large number of possible tokens, you may want to examine a better look-up system (perhaps store the tokens in an array).

Obviously in a production environment you’ll be doing error checking either within your router or in your dispatcher to make sure what is passed is sane. The dispatcher (where the $options array means something in the context of your system) is the place where 404 errors will be triggered as it’s only at that point when it becomes apparent if your system can handle the request or not. To trigger a 404 you need to send headers along with your PHP script:

header('HTTP/1.0 404 Not Found');

The content passed to the browser should be valid page describing the error, obviously this is the place where you can provide the ability for customisable 404 pages or even initiate a search to try and discern what the user was looking for.

getURLFromRoute

As mentioned before, the inverse of the function is to provide the router with a route and get back a URL. This allows your system to redirect pages or provide contextual URLs within pages. On the surface it sounds like a reasonably simple task, of course in practice it proves to be much more vexing.

Essentially we wish to match the route arrays to one another, and the route array which matches will then be our URL. Obviously the tokens muddle this slightly however the main iteration is the same as getRouteFromURL:

$specials = array('command','action','id','extra');
foreach($this->routes AS $route => $options)
{
	// if there are specials within the route, substitute them into the options array
	if(strpos($route, ':') !== FALSE)
	{
		$options = (!is_array($options)) ? array() : $options;
		foreach($specials AS $special)
		{
			if(strpos($route, ":{$special}") !== FALSE)
			{
				$options[$special] = (array_key_exists($special, $requestedRoute)) ? $requestedRoute[$special] : '';
			}
		}
	}

	if($requestedRoute == $options)
	{
		$replace = array_values($options);
		$search = array_map(create_function('$a', 'return ":{$a}";'), array_keys($options));

		$url = str_replace($search, $replace, $route);
		echo $url;
		return $url;
	}
}

The function works as such: iterate through the route table > if there are specials for a route, take the values from the requested route into our route array > if our constructed route array matches the requested route array, replace any tokens and return the URL ELSE cycle to the next route in the table.

Perhaps the only confounding part of that array is the presence of a lambda-style function which simple prefixes the $specials array values with a colon for easier use of str_replace.

Conclusion

As you can see, there’s a lot of code and lot of logic and pit-falls that go into provided a tokenised routing system, and this hasn’t even touched on the dispatching part of it (which can open a new can of worms with tasks like class Reflection and file existence checking).

Ordinarily something like this would go against my usual PHP coding methodology: less is better. I am however convinced that in the correct circumstances, the benefits can far outweigh the 0.03 seconds this adds to your execution time. Perhaps I’m becoming softer in my old age…

First, an explanation. If you have an abstract class you can define abstract methods which must be beefed out in any non-abstract subclasses but you can also define some standard methods (with bodies) which any subclasses will happily take with them. The easiest way to explain this is to just dump the code and go from there:

Edit: This has essentially been “solved” now and boiled down to me being a retard and not reading the documentation. The behaviour explained below is a bug with get_class when it’s called from an object’s method, to get the “real” class name of an instance, simply pass $this to the get_class method.

abstract class AbstractClass
{
	public function methodOne()
	{
		echo get_class()."\n";
	}
}

class ConcreteClass extends AbstractClass
{
	public function methodTwo()
	{
		echo get_class()."\n";
	}
}

$conc = new ConcreteClass();
$conc->methodOne();
$conc->methodTwo();

This is as simple as it gets, one abstract class, one “Concrete” class (because Concrete is cool kids) and two method invocations. Now those methods should output “ConcreteClass” twice, however, methodOne() outputs “AbstractClass” and methodTwo() outputs “ConcreteClass”. Looking at the definition for “get_class” reveals that what it returns is technically true ($conc is indeed an instance of AbstractClass) but this functionality is the opposite side of useful.

When I tried this back at the beginning of 2006, I cursed and chalked it up to PHP being PHP and got on with things. This simple case isn’t earth-shattering and can be worked around by putting a static function in AbstractClass which accepts an object and gets the class from that but that sort of roundabout route mangles an otherwise elegant solution.

It’s not hard to imagine a situation for this: an abstract “Model” class with methods for automagically building up SQL functions (pulling the name of the instantiating class as the name of the database table). No longer can you call $myModel->create() as the get_class() function would only return “Model” as the class rather than “MyModel”. The “solution” to this is, as mentioned above, to use static methods in Model, so $myModel->create() becomes Model::create($myModel) which is syntactically more convoluted and the static-charge of the method could easily cause problems further down the line.

When explaining this, I began to wonder whether I had gone space-crazy and perhaps my original implementation of this suffered from being too complex. Or perhaps it was indeed a bug and had been fixed. Neither were true. But then I wondered whether I was assuming something of PHP which perhaps didn’t hold true in other languages, perhaps I was imagining the functionality I wanted…

Ruby:

class AbstractClass
	def method_one
		puts self.class.to_s
	end
end

class ConcreteClass > AbstractClass
	def method_two
		puts self.class.to_s
	end
end

concrete = ConcreteClass.new
concrete.method_one
concrete.method_two

Java:

public abstract class AbstractClass
{
	public void methodOne()
	{
		System.out.println(this.getClass().getName());
	}
}

public class ConcreteClass extends AbstractClass
{
	public void methodTwo()
	{
		System.out.println(this.getClass().getName());
	}
}

public class Bimble
{
	public static void main(String[] args)
	{
		ConcreteClass conc = new ConcreteClass();
		conc.methodOne();
		conc.methodTwo();
	}
}

Both of these dirt-simple examples print out “ConcreteClass” twice and prove I wasn’t going crazy (at least not degenerating any further than is normal). I don’t really have much to say about the situation, PHP has always been a bastard child of OO and scripting but it’s foibles like these which build-up from irksome to tiresome.

The antagonist in this comedy of errors is not ignorance of the situation, but knowledge of the ineptitude surrounding it. In theory, Unicode is the great equaliser, the One Ring and so forth. In practice however things are different. For web projects I boiled down the dilemma into four places where Things Can Go Wrong. The first is the web-page markup itself, nestled cosily in the <head> tag is the content-type, oft forgotten and left to Dreamweaver to assign this is the encoding that is passed to your programming language of choice which is the second choke point. If all you do is pass the string to your database you may just be able to pull off the perfect murder, if however you wish to do any kind of modification to the string, then your programming language needs to know how to deal with the character encoding or how to convert to an encoding it can use. Once you spent enough time with the jesters, it’s time to pass things over to your database which, just to be pedantic, has two points of failure. The first is the connection encoding whereby the database tries to convert whatever it stores the data in to a suitable encoding for it’s client; and then there’s the minor issue of how your string is stored within the database.

Projects that don’t need to worry about languages other than English can well ignore character sets entirely and pretend that everything is buttercups and puppies in the world of ASCII. Move even a little though, even to Roman-based alphabets like French or Swedish and things break down. The ideal, the blue-sky pie would be UTF-8 from start to finish and back again, but this article wouldn’t exist if it were that simple.

The reality of the situation is that browsers are like panes of glass, you fit them correctly and you don’t need to worry about them. As long as you’re not using a database of antiquity then character encoding within the database is solid, the “major” databases have you covered. It’s a shame then that it all breaks down with the programming language/environment especially the scripting languages.

I’ve been informed that Perl, the most distinguished of scripting languages (what other language allows a power-user to be called a “monk”?), has everything neatly arranged and ready for surgery. Python shares a similar preparedness. It’s unfortunate then that PHP (the greatest scripting language evar!) and Ruby (the new greatest scripting language evar!) are so arse-backward despite their popularity.

PHP I don’t necessarily blame for it’s inadequacy; it’s something that I’ve come to expect from the Quasimodo bell-ringing approach it takes: volume over grace. Sure you have the multi-byte string module which is not included by default, or the iconv conversion module again not included by default, or the perpetually in development PHP 6. These are just plasters over a gaping tumour that is the lack of built-in character encoding support.

Ruby on the other hand, I held so much hope for. The fervent few claimed it would do so much for so little investment and yet where the lack of character encoding support grew from stagnation and sloth-like speed with PHP, Ruby has just omitted to deal with it entirely. As a language it is actively developed and yet once again character encoding support is resigned to labyrinthian workarounds and obscure modules (or “mixins” as I’ve been commanded to call them).

Were I a petty man I would blame the programmers of yore for their lack of foresight, but when ASCII was developed most computers had trouble rendering more colours than I have teeth so forgiveness I give retrospectively. What I can’t fathom is why two widely (read: cosmopolitan) used languages would be so flaccid when it comes to “simple” text. It’s not that I can read Chinese, or Japanese, or Korean, or Swedish or even French, but it irks me that were I wont to, so many obstacles stand in the way.