@Leonard Dominguez yeah this is almost certainly temperature-driven soft iron drift — outdoor deployments are brutal for this because the sensor's magnetic environment shifts as the PCB thermally expands and contracts through day/night cycles, and your startup calibration offsets become stale pretty quickly.
The fix that works best in practice is running an ongoing calibration loop in firmware — basically keep a rolling min/max buffer on all three axes while the system is running, and update your hard/soft iron correction matrix whenever the spread changes beyond a threshold. You don't need to recalibrate constantly, just let the calibration "age" gracefully rather than freezing it at boot.
One thing worth checking first though: if your module is mounted near any metal that heats up during the day (enclosure hardware, a nearby regulator, anything ferromagnetic), that's going to cause exactly the pattern you're describing — fine at night, drifts off by afternoon, resets at the next cold morning. If it's temperature-related you'll see the heading error correlate pretty directly with your temperature sensor readings, which you already have on the same module — so that's an easy cross-check before you go deep on firmware changes.
If you want, drop your current calibration code in here and I can help you build out a continuous correction loop — the AI Firmware Builder in the Dev Hub can also scaffold this pretty fast if you describe the behavior you're going for.