Pointer updates are atomic so drop the sync.RWMutex as it is not needed
anymore. This also fixes the race introduced with dfc71df0 (although I
believe this is the first time we properly tested that code path).