Hopefully this will help if you have an issue with the automated test system.

# A Test is Failing: Should I be Concerned?
With so many tests run using end-to-end connectivity and real modules on what is now quite a complex lump of `ubxlib` software there will _always_ be intermittent failures, that's just life.  When you have pushed a branch and the tests have run you should take a look at each test failure and satisfy yourself that it cannot be caused by you.  A list of the known intermittent failure modes can be found on the [wiki page](https://github.com/u-blox/ubxlib_priv/wiki/Known-Test-Failure-Modes) of the `ubxlib_priv` repo; check the failures against this, maybe run the testing again to check for persistent failures and, of course, please keep the list up to date.

# All of the HTTP and/or MQTT and/or Sockets Tests Are Failing: What Should I Do?
There are a few possibilities:
- If the failure is occuring both for Wifi and for cellular then:
  - If the failure is for just _one_ of HTTP or MQTT or sockets then the test application for that specific service, running on our echo test server, has likely failed.  FYI, I've only ever seen this happen for HTTP (the HTTP echo application very occasionally hits a segmentation fault (something like `sudo journalctl -u http_server -n 10000 | grep segmentation` on the echo server machine will find this)).  Currently only Rob can fix this.
  - Otherwise the echo test server itelf, or the internet connection to it, must have failed; currently only Rob can fix this (by restarting the application).
- If the failure is occuring for cellular _only_ (likely accompanied by other cellular-only failures to do with connectivity, e.g. DNS look-up) then:
  - Something must be up with the Nutaq box which provides the Cat-M1 cellular network to the test system.  This requires manual intervention: there is a sign attached to the test system rack which points out where the power switch of the Nutaq box is: you will need to ask Helen Dalton or Phil Ware in the Cambourne office to do a long-press of the power button to power the Nutaq box down (the fans never go off, so you don't know when it is off unfortunately; just hold the button down for 10 seconds), wait a few seconds and then press the power button once more to power it back up again.

# The Jenkins Web Interface is Not Responding: What Should I Do?
There could be a number of reasons for this, here they are in order of likeliness, most likely at the top:
- Internet connectivity into the Cambourne office is down: this happens rather more frequently than it should; watch for IT announcements for the longer outages.  There is nothing that can be done about it, unfortunately, the system will become available again when internet connectivity is restored.
- Jenkins has stopped working: if you are able to SSH into the test system then do so and `sudo reboot` the machine you are SSH'ed into (which is the Jenkins machine).  Otherwise ask someone in the Cambourne office (e.g. Helen or Phil) to power-cycle the machine labelled `10.10.2.200`.
- The power has failed at the Cambourne office: when power is restored manual intervention is required to return the test system to normal operation.  The required steps (only pressing power buttons) are written on a sign attached to the test rack, perhaps ask Helen Dalton or Phil Ware to follow those instructions for you.
- The intermediate machine through which we access the test system (`ubxlib-test-system.redirectme.net`) has gone down: currently only Rob can fix this.
- The SSH tunnel from the Jenkins machine to `ubxlib-test-system.redirectme.net` has gone down: this is on an auto-restart so it will fix itself, but if it doesn't for some reason best read [ACCESS.md](ACCESS.md) for hints as to what you might do.

# I Am Being Rejected By the Jenkins Web Interface, Please Let Me In!
See [ACCESS_REQUEST.md](ACCESS_REQUEST.md) for how to do this.  Certificates are valid for a year and must be regenerated manually: [ACCESS_REQUEST.md](ACCESS_REQUEST.md) should help you with renewing them also (they can be regenerated without requiring the user of the certificate to do anything).

# My Browser Says that the Jenkins Web Interface is a Malicious Site and Won't Let Me In, What Should I Do?
For some reason, even though the URL has a valid LetsEncrypt certificate, auto-renewed, if you send someone a link to it pasted into an e-mail or in Microsoft Teams then clicking on that link results in a "malicious site" message: should this occur just copy and paste the link into your browser instead.

# I've Pushed a Commit to `ubxlib` But No Tests Have Run: What Should I Do?
We use something called a SMEE server and a SMEE client to allow Github to trigger test runs in Jenkins when it receives a push, via a webhook.  In all cases when triggering via SMEE fails you can manually trigger a test run on a new push yourself by going to the `ubxlib` item on your Jenkins dashboard and, on the left-hand menu, pressing the `Scan Repository Now` button.

Sometimes automatic SMEE-based triggering will fail because the public SMEE server has gone down: the `Settings` -> `Webhooks` page of the [ubxlib_priv](https://github.com/u-blox/ubxlib_priv) repo will show a red warning sign on the relevant webhook (the one at the bottom, beginning "8Q") when this happens.  Also, if the outward TCP connection from the SMEE client to the SMEE server has broken (e.g. if the SMEE server has changed IP address or the network connection has gone down ,see "this happens more than it should" above) then it does not fix itself; this is a known problem with the SMEE client and, for this reason, the SMEE client is set to restart itself every 2 hours, on the hour.  If you wish to restart the SMEE client manually, follow the instructions in [SETUP.md](SETUP.md#manually-restarting-smee-client) to do so.  Any branches that were pushed while the link to SMEE was down will not be noticed, you will still have to do the manual triggering thing to have tests run on such pushes.

# The HW of one of the Test Instances Seems to Have Failed: What Should I Do?
All of the HW should power itself down when it is no longer required for testing but it may be that this mechanism gets out of sync with reality for some reason, leaving the equipment on so that it does not have its sleepy-bye-byes; this can be fixed by going to the Jenkins dashboard and running `ubxlib_clean_up`, which will wait for any tests to finish and then make sure that all of the boards are powered off.  Should the failure remain then you'll likely need to ask whoever is administering the test system to go and fix it manually.

# A Test Agent Is Out of Disk Space: What Should I Do?
The Raspberry Pi agents have 32 gigabyte SSDs, which should be enough, except for three reasons:

- if an updated version of certain toolchains (e.g. nRFConnect, where each installation is around 3 gigabytes in size) has been installed, the old one is _not_ deleted and hence it is possible to run out of space on the SSD.  The only way to fix this is to SSH into the Jenkins machine and, from there, SSH into the relevant Raspberry Pi(s) and delete unused toolchains from the `~/.ubxlibpkg` directory (with `rm -rf ~/.ubxlibpkg/blah`).  Each Raspberry Pi only needs the toolchains for the type of HW that is connected to it (e.g. if an instance is running an ESP32 board it does not need the STM32Cube or nRFConnect toolchains) but, since the SSD from one Raspberry Pi is usually copied to a new one, they generally each have all of the toolchains installed on them; therefore there is likely some "slack" that you can regain by deleting unused toolchains.  No harm is done if you accidentally delete a toolchain that _is_ required since the test system automatically installs what is needed when it is run.

- PlatformIO downloads and caches versions of target toolchains for itself and so, if anyone has experimented with different versions (by editing the versions called-up by [platformio.py](tasks/platformio.py) in a test commit), it is possible for that cache to occupy disk space unnecessarily also: either search [platformio.py](tasks/platformio.py) for `prune` to find lines that you might temporarily uncomment and run in a test commit to clear unneeded items from the cache, or, as above, SSH into the Jenkins machine and, from there, SSH into the relevant Raspberry Pi(s) and delete the entire PlatformIO installation with `rm -rf ~/.platformio`; PlatformIO will reinstall itself on the next test run.

- Re-building a Docker image can lead to stuff lying around which Docker itself seems unable to clean-up; disk may be retrieved by completely removing all of the generated Docker stuff by running `docker system prune -a` on a Pi and then building the `ubxlib_builder` image on that machine (via Jenkins, in the usual way) afresh.

# I Need to Update/Add a Target Toolchain: How Do I Do That?
In this directory you will find [u_packages.yml](u_packages.yml) which specifies the versions of all of the target toolchains.  When you're pretty sure the new version is all working, i.e. you have tested it locally, updating that file will bring the new version into the test system automatically.  Old versions are NOT deleted, you must do that manually to recover disk space (see the previous item for how).

# I Need to Re-build the Docker Image Which Runs the Target Toolchains: How Do I Do That?
If you need, say, a new Python package or certain third-party toolchain changes, that may require you to re-build the `ubxlib_builder` Ubuntu Docker image which runs all of the target builds and tests on all of the test agents.  You should make any updates to [Dockerfile](docker/builder/Dockerfile) on a branch, then you may trigger a re-build of the Docker image on any of the test agents by going to the Jenkins dashboard and selecting `ubxlib_agent_docker_build`.  You will be required to enter the name of the branch where you have made your changes and select the agent(s) that you wish the re-build to be done on.  WARNING: check the build-log by eye, a successful outcome means the build has been executed and NOT that it has succeeded.  Obviously test any changes quite carefully from your experimental branch on a single agent before going any further. Only when you are quite sure that all is good should you repeat the exercise for all of the agents.  This is a _manual_ process, i.e. you must do the above for all test agents to update all of their Docker images once all is good.

# I Need to Install an Additional Jenkins Plug-in: How Do I Do that?
Just go for it, through the Jenkins management screen as normal, noting the warning in [SETUP.md](SETUP.md#advice) about Jenkins being a nest of vipers.  PLEASE DO ALSO update the plug-in list in the `Dockerfile` as documented inside [SETUP.md](SETUP.md#install-jenkins-inside-docker) (i.e. you don't need to update the actual `Dockerfile`, just the documentation from which it was created) so that, should anyone need to re-build the system from scratch, they have an accurate record of what is required.

# Jenkins Says That Dangerous Script Signatures Have Been Approved: Is That OK?
Yes: Jenkins gets in a tizzy if quite normal things (e.g. opening a local HTTP connection) are allowed via a script, just ignore it, the test system is sufficiently locked-down that only users authenticated by certificate can get in in the first place.

# I Need to Fiddle With Something on the Single Windows Test Agent: How Do I Do That?
While you can SSH into all of the Linux agents from the Jenkins machine, the Windows agent is, unfortunately, locked down further and so you can't SSH into it (Jenkins can, you can't); either use the Jenkins script console on that agent or access it directly/physically by connecting a screen/keyboard.