Watch for the surprises
• Andy • Programming, Work life • Isaac Asimov, Mark Jason Dominus, metrics, MJD
Look for and act on the surprises around you ever day. That's where we have the most opportunity to make changes for the better.
Yesterday, amidst all the discussion of CERN's announcement that it seemed to have measured neutrinos moving faster than light, Mark Jason Dominus reminded me of the line often attributed to Asimov that the sound of scientific breakthrough is not "Eureka!" but "That's funny..."
Back in the late 90s, I had a new boss in the IT department. When he called his first staff meeting, he told us to bring some sort of metrics from our areas of responsibility. I'd recently created the company's intranet, so I ran some reports of hits by top-level directory from the Apache logs. My boss gave my stats a cursory glance, handed them back and asked "So what surprised you?" The question itself surprised me. "Now that I look at it," I answered, "I didn't expect that the /foo directory would be getting so many hits." Then came his crucial follow-up: "So what should we do about it?"
So what's going on around you that you didn't expect? Are you even looking? How? Where?
Look at your server log files. Try to absorb patterns. Who's requesting the same non-existent .GIF file every 11 minutes?
Run a profiler on your code. Why is the sort function being called so many times? Why would a simple string transformation function take so long to execute?
Measure your system performance with a tool like Munin or Cacti. Look for use spikes. What happens at 3:30am that thrashes the system? Why do the cache hits drop to near zero twice a day?
Always keep an eye open for the unexpected behavior, the strange blip, the neutrino that took 60 nanoseconds longer to get there than expected. Then follow up on what you find.