A little over a week ago, I wrote about how we tried to get all our users to update cached copies of our JavaScript. At the time, I’d only turned the code on for 3 IP addresses, and over the course of two days, the number of requests using a stale cache from these three IPs completely went away. I then decided to widen the net.

Rather than target specific IP addresses, I decided to target any user with a boomerang version older than 0.9.1334338538, i.e., the version we pushed out on April 13 this year. The intent was to target the oldest versions of our JavaScript while still limiting exposure to a relatively small percent of users.

My reasons for limiting exposure were two-fold. First, in case there was a problem, I wanted to limit the number of users that experienced it, and second, I knew we’d be pushing out a new version of the JavaScript soon, and didn’t want to force several million users onto the new JavaScript the day it went out. This new fix wasn’t a critical security update, so we could afford to wait the 7 days for caches to expire normally. Data for a day — I started this test on June 8, a Friday. Our normal traffic patterns show two things:

  1. Fewer users visit the site on weekends
  2. Some users only visit the site on weekends

This meant that a change on Friday would only have noticeable effects starting Monday. Looking at the data from Thursday, the 14th gives us an idea:

JavaScript timestamp JavaScript version June 14 June 4
Feb 1 17:33:00 0.9.1328117580 13 168
Mar 8 15:39:17 0.9.1331221157 3 53
Mar 16 18:16:32 0.9.1331921792 - 9
Mar 22 03:33:44 0.9.1332387224 - 25
Mar 27 16:39:11 0.9.1332866351 - 4
Mar 28 16:03:01 0.9.1332950581 - 28
Mar 28 19:59:40 0.9.1332964780 - 3
Mar 29 16:04:27 0.9.1333037067 1 6
Mar 30 22:23:14 0.9.1333146194 - 2
Mar 31 00:52:20 0.9.1333155140 7 161
Apr 9 17:34:33 0.9.1333992873 7 387
Apr 11 13:37:46 0.9.1334151466 - 17
Apr 11 16:46:14 0.9.1334162774 - 69
Apr 13 17:35:38 0.9.1334338538 166 388
Apr 19 21:26:08 0.9.1334870768 2 1
Apr 19 22:43:46 0.9.1334875426 78 241
Apr 28 18:11:26 0.9.1335636686 36 118
May 1 07:27:15 0.9.1335857235 89 169
May 1 22:17:11 0.9.1335910631 780 2411
May 23 08:10:48 0.9.1337760648 444 1529
May 25 20:41:28 0.9.1337978488 386573 3837537
Jun 12 21:20:26 0.9.1339536026 2462721 -

We targeted all versions older than the April 13th version, and the data shows a significant reduction in the number of beacons from those versions. The beacons that did come through resulted in the refresh code going out, which meant all subsequent requests used the new JavaScript.

Notice also that just two days after pushing out a new version (June 12), the majority of users were using the new version.

Extending to 8 days

Comparing single days isn’t fair because browsing patterns tend to vary by day. I decided to look at multiple days worth of data instead: the days from June 1-8 and June 9-16. This is what it looks like:

JavaScript timestamp JavaScript version June 1-8 June 9-16 June 17-24
Feb 1 17:33:00 0.9.1328117580 653 78 71
Mar 5 22:16:16 0.9.1330985776 2 - 10
Mar 6 16:49:20 0.9.1331052560 7 - -
Mar 6 21:31:58 0.9.1331069518 3 13 -
Mar 8 15:39:17 0.9.1331221157 149 44 62
Mar 16 18:16:32 0.9.1331921792 120 1 -
Mar 22 03:33:44 0.9.1332387224 359 73 21
Mar 27 16:39:11 0.9.1332866351 96 30 7
Mar 28 16:03:01 0.9.1332950581 61 2 3
Mar 28 19:59:40 0.9.1332964780 24 11 15
Mar 29 16:04:27 0.9.1333037067 449 1 32
Mar 30 22:23:14 0.9.1333146194 32 4 -
Mar 31 00:52:20 0.9.1333155140 383 109 25
Apr 9 17:34:33 0.9.1333992873 2180 210 75
Apr 11 13:37:46 0.9.1334151466 109 7 -
Apr 11 16:46:14 0.9.1334162774 127 34 52
Apr 13 17:35:38 0.9.1334338538 1780 975 983
Apr 19 21:26:08 0.9.1334870768 6 2 -
Apr 19 22:43:46 0.9.1334875426 1652 739 1326
Apr 28 18:11:26 0.9.1335636686 678 206 255
May 1 07:27:15 0.9.1335857235 533 287 242
May 1 22:17:11 0.9.1335910631 12978 5599 3264
May 23 08:10:48 0.9.1337760648 28903 3091 1506
May 25 20:41:28 0.9.1337978488 27224329 14998263 438175
Jun 12 21:20:26 0.9.1339536026 - 6832278 884109
Jun 14 19:34:15 0.9.1339702455 - 6120811 29425044

The number of beacons from versions older than April 13 between June 1-8 is 4754, while the number between June 9-16 is 617… an 87% reduction in targetted stale versions. While we’ve definitely made huge progress, we aren’t all the way there yet. I plan on running the code as is for a few more weeks and monitoring its progress as we go forward.

Update June 24: I’ve looked at numbers for the next 8 days, and I see 376 beacons from stale versions of the file. A 92% reduction from two weeks ago and a 39% reduction from the last week.

Problems encountered

While I haven’t seen evidence of the IE8 bug reported earlier, I did see some weird behaviour from Firefox 3.5.11 (yes, people still use that).

It turns out that Firefox 3.5.11 does not respect location.hash (and a few other things too). This meant that our refresh script went into an infinite loop since location.hash was never set (see the previous article for what our script looks like).

A simple solution was to switch the location.hash check into a BOOMR.version check. This of coure means that our refresh html needs the latest (or at least our desired) version number of boomerang hard coded in. There are ways around it, using the beacon again, but I’ll leave that as an excercise for the reader.

This is what the JavaScript looks like after the change:

if(!window.parent) {
	BOOMR.version += '-1';
}
else if(window.parent === window) {
	BOOMR.version += '-2';
}
if (!BOOMR.version.match(/^0\.9\.1339702455/) {
	location.reload(true);
}
else if(location.hostname == 'lognormal.net') {
	location.href="about:blank";
}

It’s a little crude, but gets the job done for now. Ideally I’d implement a partial semver check, but that might make the JavaScript too complicated. For now this suffices, along with the fact that this page will only be loaded if the JavaScript version were older than our specified version (so we won’t force reload versions that were already newer than the hardcoded version).

Conclusion

That’s it for this post. I’m happy with the way things are progressing, and hope it keeps up its momentum.