These asserts were there because this shouldn't happen, I added them back for debug builds. We really should find the cause of any violations.