"Semantic speedup" is an interesting idea. There is generally a lot of data - all the changing labels, for example - which cannot be registered at high speeds of movement and just adds clutter to the visualization. The display space freed by removal of this data could then be used for other data more useful at these speeds.
Color needs to be used carefully in any visualization - the principles B. Rogowitz talked about and those mentioned in Tufte form a framework which can be used. This is mentioned in the context of a rather unfortunate choice of colors - red, green, yellow - for empty, full and mixed areas in the cache. Mixed areas contain both empty and full blocks and the visual cues should reflect this - a choice of blue, green, cyan or white, black, grey would be more appropriate.