Re-evaluate our current approach to CI

[jessebraham]

There are presently a number of issues with our current CI pipeline(s), many of which resulted naturally as a result of the growth of the repository. However, I feel we're reaching a bit of a tipping point here and need to re-evaluate this to better suit our needs and give us a bit more confidence in the checks being performed.

For starters, I suspect we're performing a lot of the checks multiple times. CI runs in PRs, the merge queue, and then again once merged into main. While not all checks are necessarily performed at each stage, it's quite common for a few PRs to be opened/updated in succession and cause quite a backlog due to the limit of 20 concurrent runners. Much of this congestion is likely avoidable.

Additionally, checks are not being performed consistently across packages. MSRV checks are not being run for each package (in which case, we can't really guarantee this version, can we?), and likewise we need to do better with checking documentation builds. I suspect running documentation builds nightly is adequate, and on failure we can have the workflow open an issue or something.

Another issue is with the nightly checks, in which new errors crop up quite often, and are easy to ignore. Again, opening an issue upon failure is probably a good option.

It would also be nice if we could automatically run the documentation workflow upon creation of a release.

Maybe we can utilize our self-hosted runner VMs as well; this is worth investigating.

We likely want to add a seperate check to only check the stable subset of esp-hal (from #2998)

We might also want to take the time (again :D) to look into semver checks, now that we will have a stable release to compare it to.

Regardless, there's a fair bit to discuss I think, and a pretty serious overall may be in order. If anybody else has thoughts regarding this please comment in this issue.

@MabezDev @tom-borcin can we please schedule a meeting to discuss this at some point in the future? Doesn't have to be right away, but sooner would be better than later probably.