Yeah I ran into exactly this on a similar build - turns out the mag on that module is pretty sensitive to the switching noise from the IMU's higher-rate sampling, so even a solid calibration can get wrecked at runtime. Try adding some low-pass filtering on the mag output and if you haven't already, double-check you're not sharing power rails without decent decoupling caps close to the sensor. Made a huge difference for me once I cleaned that up.
