@Kent Dawson yeah that diagnosis tracks — bus contention is almost certainly your culprit. The inconsistency makes total sense too, because it's going to depend on exactly when the accel/gyro transaction lands relative to the GNSS buffer window, so you'll see it come and go rather than a hard floor.
DMA is the move if you haven't enabled it yet — it basically takes the CPU out of the loop for those sensor reads so the GNSS processing doesn't get starved waiting for a blocking transfer to finish. If you're already on DMA and still seeing it, I'd look at your interrupt priority config next, because if the sensor interrupt is preempting the GNSS handler you're in the same boat through a different door.