And that's the problem. Stress does slow you down -- your brain, that is. Whether you have "a lot to accomplish" or not, you cannot escape the fact that stress causes stress hormones that can lead to the atrophy, shrinkage, of the brain areas (hippocampus, prefrontal cortex) vital brain region for episodic, spatial, and contextual memory (now see here).
Even for short term stresses that only last a few hours can impair brain-cell communication in areas associated with learning and memory (now see here). This isn't caused by cortisol, but through another pathway of corticotropin releasing hormones.
The solution? You cannot turn off the world, but you can work with it more effectively. Focus on high quality, and low quantity -- that includes your daily To-Do check list as well. If you need a rule, try this.