Don't silently hide failures, barf on startup. Also add more integration
tests that should catch some of these things.