Showing posts with label html. Show all posts
Showing posts with label html. Show all posts

Friday, November 26, 2010

Tidy5 aka the future of HTML Tidy

UPDATE 2011-11-19: The most immediate of my concerns have been addressed by Björn Höhrmann who has submitted basic support for HTML5 in Tidy to a forked version available on Github.

I have been a long time fan of Tidy, a tool to clean up and do some basic checks on the code. However, the tool is not really being updated any more, and since I have moved to using HTML5 and ARIA on all my new projects, it has lost much of its usefulness.

I also see no momentum picking up and thus think it should be considered folding Tidy into html5lib. By that I mean using html5lib to get Tidy like functionality.

Today I wrote a mail that I cross posted to the discussion list for Tidy and the help list for WHATWG. This blog post is essentially a longer version of that email.

Tidy must go HTML5

Here is the deal with HTML5. Pretty soon every browser will have an HTML5 parser. Except for IE, browsers do not have multiple parsers.

This means that tokenization and DOM tree building will follow the rules defined in HTML5 – as opposed to not really following any rules at all, since HTML 4 never defined them.

Simply put, there is no opt out of HTML5. An HTML 4 or XHTML 1.x doctype is nothing more than a contract between developers. Technically all it does is to set the browser in standards compliance mode.

Thus, I do not see any future in a tool that does not rely on the HTML5 parsing algorithm. Tidy can not grow from its current code base, but needs to have the same html5lib at its core that is in the HTML5 validator, which basically is the same as the one being used in Firefox 4.

Additionally, Tidy suffers from:

  • Implementing WCAG 1 checks in a world that has gone WCAG 2.0.

  • Not recognizing ARIA, which is an extremely valuable technology on the script heavy pages of today.
  • Not recognizing SVG and MathML.

I know one can set up rules to enable Tidy to recognize more elements and attributes, but for full HTML5 + ARIA + SVG + MathML (and perhaps RDFa), that is simply not doable without superhuman efforts.

The merge

A basic Tidy5 implementation could look like this:

  1. Parse the tag soup into a DOM.
  2. Serialize HTML from that DOM.
  3. Compare the start and the end result.

Perhaps any error reporting can be made during the parsing process. Henri Sivonen could probably answer the question if that is possible.

However, there is also talk about having a lint like tool for HTML, that goes beyond what the validator does. So in addition to the above, there can be settings for stuff like:

  • Implicit close of elements. Tolerate, require or drop all closing tags?
  • Implicit elements – tolerate, require or drop (maybe require body but drop tbody...)?
  • Shortened attributes – tolerate, require or drop?
  • HTML 4 style type attributes on <script> and <style> – tolerate, require or drop?
  • Explicit closing of void elements – tolerate, require or drop?
  • Full XHTML syntax (convert both ways)
  • Indentation. Preferably with an option not to have block elements with a very short text content not to be broken up into 3 rows as in Tidy today.

Besides purification and linting, such a tool/library can be used for:

  • Security. This will require the possibility of white and/or blacklisting elements and attributes. And preferably also attribute values.
  • HTML post processing. This will enable authors to see indented code, that is explicit, while at the same time such "waste" can be removed before gzipping. This would be akin to JS minification and it could be performed on the fly from within PHP, Python, Java, Ruby, C#, server side JS or whatever. It can also be done manually before uploading from the development environment to production - or it could be integrated into the uploading tool!

Checking templates

The main feature that Tidy has today, is the ability to handle templates, by preservering/ignoring PHP or other server side code. To what extent the HTML5 parser can be modified to handle that feature I do not know.

From a maintenance and bug fixing point of view, I see huge wins in having a common base for Tidy, the HTML5 validator and HTML parsing in Gecko.

In fact, a very radical idea for Firefox (or any other browser using html5lib) would be to actually integrate these tidy-inspired features directly in their development tools, re-using the existing parser! A Firebug extension that lets me validate as well as tidy up my code directly within the browser would be super awesome!

But the actual possibility thereof is beyond my technical knowledge to evaluate, so I need to hear from people who know this stuff better than I do.

Integration with accessibility checking

Although automatic testing can not not substitute manual tests, they can give a developer an in the ball park idea about the accessibility of a page and fix the most obvious mistakes.

The fact that Tidy today do integrate WCAG 1.0 is better than nothing and any implementation of Tidy5 should strive to integrate WCAG 2.0 in a similar fashion. That really is a no brainer. Having to use only one tool and getting all errors in the same buffer (for programmers) or the same console (for manual checks) is certainly convenient.

OK, that was my two cents. What do you think?

Monday, July 13, 2009

No backwards compatibility = XHTML 2 was doomed to failure

XHTML 2 is the Itanium of web technologies. You remember Intel and HP celebrating the Itanium architecture as the new super-duper technology, that should leave all RISC-based competitors in the dust. VLIW (re-branded as EPIC) was touted as a disruptive innovation. The future was Itanium.

But it was not! Forget the marketing speak from Intel and its only Itanium customer worthy of being mentioned, HP. Itanium has flopped. Yes, other RISC architectures, are struggling. MIPS no longer power high end servers from Silicon Graphics. SPARC is loosing market share. Only IBM Power PC seems to be holding its ground in the server space. But the x86 architecture, that was supposed to die, is reigning more dominantly than ever.

Backwards compatibility is everything

Being backwards compatible is not only a nice feature. It is a prerequisite that simply seems non negotiable. Let's look at a few successful products to get an idea.

Windows 95 and DOS-based games

Before Windows 95 all high end games were run from the DOS-prompt. Windows 3.x was only a nuisance for game developers. In order to achieve the highest possible speeds they often tweaked the hardware interaction in every possible way. Getting these games to run under Windows 95 proved a challenge, to say the least. Microsoft solved this by special-casing game after game. The operating system would recognize a particular piece of software, know that it required special handling and adjust accordingly. Even in ways that broke protocol.

Punch cards

When were punch cards invented? 1725. When did they become a big success? 1890. When did they become surpassed by other technologies. In the early 1960's. When did IBM drop support for punch cards from their operating systems? Not for another 30 years. I would not even be surprised if it was possible to attach a punch card reader to a brand new z-series computer today, and actually have it work.

XHTML 2 was Dead Pre Arrival

It did not die as a markup language for the web. It never lived. The day the decisions was made not to be backwards compatible, it was doomed. It never really mattered that it had every conceivable shiny new feature. Technical merits are simply not enough. Therefore it simply does not matter how much you shout about them, or how much you disdain the fact that HTML 5, due to its legacy, is awful and badly designed.

Yes, I use PHP too, and no matter how much one shouts the relative technical merits of Ruby or Python, PHP seems not to grow weaker. Ugliness just is not that big a factor. A strong user base, re-use of code and know-how, ability to find advice and support, such things matter. Sociology always trump technology.

The future for XHTML 2

I have friends who prefer XHTML 2 to DocBook, for data storage on the server. Reading Steve Pembertons thoughts about the future, he seems to believe that is a viable niche and that it can make a comeback that way. And why not, in a controlled environment the improved semantics of XHTML 2 over legacy HTML may provide significant benefits. Pushing XHTML 2 as a progressive enhancement, server side, might work.

Encouraged by none other than Ian Hickson himself XHTML 2 will continue to be developed in a working group outside of the W3C. Can it make a comeback? Once upon a time the W3C decided to axe HTML. Many developers, myself included, thought it was the end of HTML. I was wrong. I therefore will not say, good bye XHTML 2, but a revoir.

Saturday, June 27, 2009

Validation and doctype myths and (inconvenient) truths

People like me who support web standards often talk about validation and doctypes. Yet, even within our camp, there seem to be a lot of confusion. I will try to address a few misconceptions, especially a few new ones that has come out of the ongoing debate about HTML 5 and accessibility and RDFa.

Background: How a browser processes markup

Most people tend to think in two stages. There is markup and there is a rendered page on the screen. In reality this is a a more complex process.

First comes parsing. The purpose of this is to convert the markup into an representation inside the program that is so to speak "understandable" to the computer and usable for rendering on the screen as well as to assistive technologies, like a screen reader. This internal representation is accessible also to manipulation through the DOM API. Indeed it is often talked about as the internal DOM representation of the document. From now one I will simply call it the DOM. Just bear in mind that I refer to this internal functionality as a whole, and not to the API in the rest of this article.

Parsing in itself is a multi-step process. It involves the simple mapping of the HTML markup, but also the applying of CSS, handling events, etc.

The rendering (painting on the screen) is in turned made from the DOM. As is the exposure of the page to assistive technologies.

Personally I've found it helpful to think about this process in three stages:

  1. The code (HTML, CSS, etc) arrives as a network stream.
  2. The DOM, constructed by the parsing.
  3. The perceivable results (on the screen, in the speakers, in the braille terminal, on paper from print, etc.)

With this knowledge we can formulate the purpose of validation:

  • Validation provides a secure mapping between the markup and the DOM. You as a developer know what you are going to get.
  • Validation provides easier development, including easier error detection and better maintainability through cleaner and consistent code.

Browsers have always had mechanisms to handle badly written HTML. Indeed they have reverse engineered each other in this regard so much that invalid, tag-soup, piece of shit code usually renders just fine. And the HTML 5 spec goes to great length to explain just how such code should be handled by a browser. If one has supreme knowledge about every little detail of how browsers work internally, one can therefore get predictable results even with code that does not validate at all.

Web developers should, however, not be required to have such in-depth knowledge. Validation is a tool that helps us stay within safe boundaries. Stepping outside them might work, but it will always lead to extra work in the end.

With this in mind we can formulate a few spin-off effects of validation, such as:

  • Validation is an act of courtesy towards other people who one day might be charged with taking over your code base, or indeed only be asked to take a look on a mailing list or forum, to help you solve a problem.
  • Validation is a mark of professionalism, a sign that you care about code quality.

But the main effect is that validation is a tool that helps you as a developer get to your desired results. I always tell my students to validate early and validate often. After every major change to the code, re-run the validator!

Or to put it differently. Valid code is not the end goal, it is a very good tool in order to reach the end goals of predictability, consistency, maintainability and effectiveness.

Myth: You must validate in order to be accessible

Wrong. Validation is advisable of course, but in a pure technical sense not a requirement. It is perfectly possible to write unsemantic code, without proper hooks for assistive technologies, that still validates. And it is perfectly possible to do the other way round, although validation — especially to a strict doctype — will be one help towards accessible web sites.

Validation can check for the presence of accessibility features, such as alt-attributes, table column headings, etc. It can never ensure that the contents within those attributes and elements have been written in a usable way. Valid code is a good starting point for accessibility, not a guarantee of accessibility.

This is especially true for the non-strict versions of HTML 4.01 and XHTML 1.0. These 4 (2 * 2) (X)HTML versions contain a lot of elements and attributes, that should not be used. The validator will complain that they are deprecated, but it will still give the page a green light. Anything but strict doctypes or conformant HTML 5 (see below) should have been verboten long ago for any professional web developer.

Myth: There is no penalty for not validating

There is one camp, primarily accessibility experts, who would like browsers not to render pages that contain markup errors, or at least give clear warnings that they do. Recently it has been advocated that e.g. the HTML 5 canvas element should not render anything to sighted users unless there is a fallback for the blind. They also have argued that any refusal to incorporate ARIA or RDFa into HTML 5 can simply be overridden because validation does not matter.

In this context it is true. As long as browsers and assistive technologies support ARIA and Google, Yahoo and other search providers will honour RDFa, it will work. User agent behaviour is always the bottom line, the true de facto standard in practice.

This takes us back to my main theme for this article. Things might work when using code that is not valid. But you can be much more confident that it will, if it validates. In my experience, the most common error that I catch using a validator is spelling errors in tag and attribute names. Such errors may wreck your page in many ways, maybe even in unseen ways because you have misspelled an ARIA attribute. And such errors are easier to spot if they are not hidden behind hundreds of other validation errors, that by themselves actually are benign.

Let me repeat that. Many validation errors are benign: An un-encoded &, an unnecessary closing tag, and, with HTML 4.01, forgetting to specify the type attribute for a script. In themselves these errors will not harm the execution of the parser, the construction of the DOM or the rendering of the page on the screen and exposure of its contents to assistive technologies.

Actually, one may find oneself in a position where it is beneficial not to be valid. This applies both to HTML and CSS. New features, often available only in their experimental first forms, can be quite useful and add to a page's usability, esthetics or accessibility.

However, the problem with benign errors is that they often obscure the malicious errors. A page that contain several hundred validation errors is much harder to debug, than one that only has a few. It is therefore imperative that one chooses the best possible validator for one's purpose. E.g. validators can be configured to ignore vendor specific CSS-rules or to include ARIA.

Actually, the main reason I use an HTML 5 doctype for all my new sites, and gradually change my own sites to do the same, is so that I can use a validator that supports these new technologies.

Myth: You can use JavaScript to cheat the validator

Technically this is not a myth. Yes you can. Most validators will work on the raw HTML and will not process any scripts. However, the purpose of validation is not validation. The purpose is predictability, consistency, maintainability and effectiveness. Inserting or altering markup through scripts, or to put it better making changes to the DOM, should be made in such a way as not to jeopardize the very reasons we wanted or code to be valid in the first place.

This is just like school. Cheating may get you the grade, but you won't get the benefit of the knowledge. Cheating the validator means that you render the validation practically useless.

There are tools available that will let you see the generated HTML, that is the HTML as the browser has understood it too be, post parsing and post scripts being run on the page. This is sort of like going back from step 2 above to the first step. That code should be equally valid, and not differ from the original input in any unexpected way.

Follow up question: Is it OK to use JavaScript instead of the target attribute?

One of the most common uses of JavaScript to cheat the validator is to replace this:

<a href="http://..." target="_blank">linktext</a>

With this:

<a href="http://...">linktext</a>
<script>
// JQuery code that attaches event to all presumed external links
$("a[href^=http").click(function() {
    window.open(this.href);
    return false;
});

I belive this is good practice. Not because we are cheating the validator, but because we are using DOM-scripting to handle behaviour. We are using the right tool for the right job.

Follow up question: Is it OK to use JavaScript to defeat browser bugs?

My first answer is, is that really necessary? For example, it is quite possible to have the object element work in all browsers, including Internet Explorer 6, to serve Flash or Java Applets. Most JavaScript techniques to include Flash on a page have been obtrusive and not degraded gracefully. And they have used outdated browser-sniffing, potentially making them unlikely to work as newer browser versions or alternate browsers like Chrome get released. Using unobtrusive DOM-script to enhance the plugin experience is of course OK.

There are however bugs and lacking support for modern standards in some browsers (yes, we all know I primarily talk about MSIE now) that can only be alleviated through scripting. Chose carefully what scripts to use, though! And remember that there are many users that might not get your scripts, perhaps since a corporate proxy has stripped out all content from your script elements.

Does the doctype matter?

Tied in to the question about validation is the choice of doctype. It serves two purposes:

  1. It declares what vocabulary a developer intends to use.
  2. It is the main way in which all sane browsers chose their rendering mode. I defer this topic to Henri Sivonen, while asking people to note that Internet Explorer 8 is not sane in any way...

As regards the first point the doctype is of value to other people with whom you co-operate, a social contract between all developers in the team. But its main value is helping the validator see what rules to validate against.

Except for rendering mode switching, the doctype does not in any really discernible way affect how the browser will treat the functionality of your markup. E.g. even if you declare a strict doctype, it will happily honour elements like <font>, attributes like target or even the marquee-element! To a (sane) browser, there is no such things as different versions of HTML. XHTML 1.0 strict or HTML 4.01 frameset or HTML 5 is all the same. Any content sent with the MIME declaration text/html is treated the same.

By dropping the DTD, HTML 5 makes explicit, what so far has been implicit. There are different editions of the HTML standard, but in practice (inside the UA) there is but one HTML.

Even if you have used XHTML syntax and have an XHTML DTD, the markup still will be parsed as usual. To trigger XML parsing, one must change the MIME declaration, which of course will fail miserably with Internet Explorer and thus never happens except for niche web sites. It has also been conclusively proven that there is no benefit in switching modes depending on the UA. The speed difference between HTML and true XHTML is first of all negligible and it only really concerns the first step (parsing), which from a performance perspective is only a fraction of time, compared to whats going on in step 2 (the DOM) and step 3 (rendering). (There may be good reasons to use XHTML, but they are related to workflow, tools and data-exchange.)

Myth: HTML 5 re-introduces bad markup

OK this is not exactly a validation or doctype myth, but it is related. the short answer is of course that HTML 5 does not force bad markup down anyones throat. All good practices are still doable.

This myth started in the early days of HTML 5, when authors started to look at the spec and saw monstrosities like <font>, or even <marquee>! The dual purpose of writing a spec on how to handle bad markup, part of the browser requirements, together with a spec about how to produce good markup quickly turned into a communications debacle. The core team behind HTML 5 is perhaps not the ones with the best people skills in the world. Communication has broken down repeatedly. (On the other hand they are unlikely to change any bad behaviour, perceived or factual, through being bashed.)

In practice, though, HTML 5 has conformance requirements — which basically is a new term to describe validity — that are even more far reaching than the ones in HTML 4.01 strict or XHTML 1.0 strict. Being conformant should ensure that you are closer to adhering to best practice and accessibility principles. Validation thus is not less important, it is even more important than ever. Just remember that validation never really has been about anything else but getting a predictable DOM from your markup and encouraging best practices. The conformance criteria in HTML 5 are being written in such a way as to take your code even further towards these lofty goals.

This is one of the reasons there is no DTD for HTML 5. These new rules for validity are sometimes so precise or require such processing logic, that they can not be expressed through an DTD. As a side effect we get a doctype one actually can learn by heart:

<!DOCTYPE html>

That will make my students very glad!