Updating cached copies of boomerang

First, a little background…

At LogNormal, we measure page load time (among other things) using JavaScript and beacon the results back to our servers. Our JavaScript is based on boomerang and we serve a different build of the JavaScript file for each of our clients. The JavaScript has a max age of 1 week from request and most clients honour this and make a conditional request a week later. Most, but not all. The JavaScript sends us a beacon when complete, and the beacon includes the version number of the JavaScript file.

This configuration has been active for a few months, and the first file served using this configuration was pushed out on February 1st.

Note that all times mentioned are UTC. What we see — Our log files for June 4th show the following:

That last row is the latest version of the file, but as you can see, we still receive beacons from the version we pushed out on Feb 1. I decided to look only at requests from the oldest version of our JavaScript. The first thing to do was see if there was a pattern to these requests. Looking at our logs showed that three IPs accounted for the majority of these requests, and this was common across several days of logs.

Further investigation showed that all three of these IPs were proxies, two of which belonged to the same company and used the bluecat proxy software. I was unable to identify the proxy software running on the third host.

Forcing a cache refresh

As luck would have it, Stoyan and Steve did some research on forcing clients to update a cached script and this was exactly what I needed as the next step.

Our first attempt at this resulted in some of our users complaining about the reload page opening in a new tab. This time I decided to limit the test only to these three IPs and see what happened. Additionally, I made some changes to the reload HTML to let me know if things went wrong, and to avoid infinite loops if the proxy refused to update the script.

This is what the JavaScript in our new reload file looked like:

if(!window.parent) {
	BOOMR.version += '-1';
}
else if(window.parent === window) {
	BOOMR.version += '-2';
}
if (location.hash === '') {
	location.hash = "check";
	location.reload(true);
}
else if(location.hostname == 'lognormal.net') {
	location.href="about:blank";
}

The first two conditions check for exceptional situations that tell us if the page loaded in a new tab, or something weird. The last case makes sure our beacon only fires once (by default it fires every 5 minutes) if the user leaves the page open for a long time.

I added this in the morning of June 5th (UTC). There had been no requests from the Feb 1 version of the JavaScript prior to adding in this change. Over the course of the day, we received the following requests (sanitized for privacy). The displayed fields are: Time, response code, request URL, referrer URL, User Agent.

Remember that these requests come from a Proxy, so we see many different “users” from the same IP.

Results

I looked at the logs again at the end of the day (UTC), and this is what we have for the Feb 1st version:

The number of requests from the Feb 1 version of our JavaScript went down significantly, however, we should really look at the ratio.

On June 4, 153 out of 168 or 91% of requests from the Feb 1 version of our JavaScript came from the three IPs I looked at.

On June 5, the ratio is 6 out of 35 or 17%, giving us a 81% reduction in stale cache hits.

I’ll look at the numbers again tomorrow and update this post at that time and if things look good, will extend it to a few more IPs and versions.

Have thoughts on what else I should be looking at? Let me know in the comments. Interested in trying out LogNormal, visit our plans page.

Update: June 7

On June 6, 0 out of 84 requests using the Feb 1 version of our JavaScript came from the three IPs I looked at. We still receive many requests from old versions of our JavaScript, and over time we will try to get all of them to load up the latest version.