Modern Software Experience

2007-06-15

Apple claims

twice as fast

Apple claims that Safari is faster than other browser, and it cites benchmark results to prove it. The claims sound pretty impressive: Safari has always been the fastest browser on the Mac and now it’s the fastest browser on Windows, loading and drawing web pages up to twice as fast as Microsoft Internet Explorer 7 and up to 1.6 times faster than Mozilla Firefox 2.

Actual performance is considerably less impressive. Never mind the installation issues you have to overcome to make it display any fonts at all. Never mind security issues that have been revealed already (and fixed by the version 3.0.1 update). The issue at hand is Safari’s actual performance. Apple’s claims simply are not consistent with real-world experience.

practical experience

My practical experience suggests that Safari is a tad slower than Firefox and Opera. Everyone who tested Safari for themselves confirms that it is a pretty fast browser, but nowhere near as fast as Apple claims. When claims do not match practical experience, you have to question the benchmarks those claims are based on.

industry standard

Apple claims that the benchmark they used is an industry standard. That is a rather strong description for a proprietary benchmark of Ziff-Davies’s PC Magazine, that was last updated in 2003 and officially discontinued a year ago. So it seems that Steve Jobs is not basing his message on a real benchmark, but picking a "benchmark" that suits his message.

page loading

I hardly care how fast JavaScript is, I care how fast pages are displayed. Never mind that truly standard-based web sites do not use JavaScript at all. The simple fact of the matter is that JavaScript executes locally, within the browser, and that whether some JavaScript code executes in 0.011 or 0.012 milliseconds hardly matters. Total page loading and display time is what really matters.

One browser’s JavaScript can be ten times as fast as another browser, but differences in page loading time remain relatively small, as what really makes the difference there is the network connection - and the cache.
Yet, Apple still claims much higher numbers, that its fankids are happy to repeat as gospel. More critical observers have wisely wondered how the benchmark numbers for Safari can be so much better, even when the stopwatch times for different browsers are nearly identical. That just does not seem right.

debunked

onload fudging

Mark Tarquin Wilton-Jones debunked Apple’s misleading claims in the Safari and Page Loading Time article on his web site. He has been doing independent benchmarking of browsers for years, and had little trouble recognising what was going on.

Mark notes that the tests rely on the JavaScript onload() event, and that Safari simply does not produce that event at the same point in the download and rendering of pages as other browser do – and that explains why Apple’s JavaScript test number look so much better, even when actual performance is closely matched.

updates

2008-03-18 Apple repeats claims

Introducing Safari 3.1, Apple essential repeats its claims: "Safari loads web pages 1.9 times faster than IE 7 and 1.7 times faster than Firefox 2", once again basing these on the discontinued i-Bench benchmark.

2011-04-25

WIRED broke the links to its news item years ago. The broken link has been removed.

2012-05-17

Ziff-Davies broke de i-Bench. The broken link has been removed. The iBench project on SourceForge is not the same as i-Bench.

links