what do you want to automate in CI?

[trombik]

while working on an experiment in GitHub Actions, there is quite few possibilities in GitHub Actions. in the past, you need to use Apps, GitHub Actions, or write a custom bot to automate something complicated. with recent tools available, such as github-script, you can build your own logic in workflows. writing a custom bot is another option but GitHub Actions is complicated enough for average developers. I don't want to add more complexity to the repository, and it is not very feasible to maintain another (far more complicated).

what I don't like in the CI is that the logic to skip workflow is using a GitHub Actions workflow. I introduced it, but it's quite difficult to achieve what I want to do. when the workflow does not do what I intended, the reason is not quite clear. you have to read the source and understand intentions, or ideas, of the author. in addition, the condition is not visible to PR authors.

it looks to me that many projects use labels for CI and automation. say, if a PR failed to build, then add "build:fail" to the PR. and when the PR has "build:fail", another workflow leave a comment explaining what to do to the PR author. this is easier to see the logic because all the conditions are visible in the PR, as labels.

while exploring the possibilities, I would like to hear your needs in CI. anything else you hope to be implemented?

[UncleRus]

...another workflow leave a comment explaining what to do to the PR author.

Cool! I love it.

[trombik]

after lengthy research, here is the summary of label-based workflows.

the key point is, all workflows that build or check something (like building examples, or running lint programs) run when a label, or labels, are added (or removed). workflows only needs to have a simple logic, does the PR have this label? if so, run the workflow. for instance, "Build doc" workflow runs if the PR has a label, area:doc. when the PR has area:ci, all checks should run because the PR potentially affects all the checks. labels are added and removed by labeler workflows. the workflows just add or remove labels from PRs, multi-labeler is the one already does it. there would be other labeler workflows, such as a labeler that add/remove ci/Build examples/failed labels when a check status is failed so that other workflow can leave a comment, or do something else.

with this model, the actual test workflows have very few logic, and the logic is separated from the workflows.

the current CI implementations run a workflow when an event is triggered, like when a pull request is opened, or a commit is pushed to a branch. this has a drawback, that is, events are not very granular. what I want to do is, when a PR is opened, and the changes include changes under components or .github/workflows, then run Build examples workflow. that means the workflow needs to include not only the actions to take but the logic to run. as the logic to run is not visible to users, or PR authors, one has to read the workflow why it runs, or not.

using labels has a drawback, too. labeler workflows need write access to the repository because it modifies PRs by adding or removing labels. PRs from forked repositories do not have write access. for that reason, some workflows must be in the default branch. that means, to test some workflows, the workflows must be merged to the default branch first, and you cannot test it before merging. this is to prevent external contributors from getting API access to the repository (otherwise, external contributors, or attackers, would have be able to do anything APIs can do, like stealing access token, modifying code in the default branch, or adding someone to the repository as a member). maybe, there is a way to test workflows, like mocking events, or running a GitHub Actions runner on the local machine. but I have no idea.

another drawback is the cost of API calls. GitHub has a limit of API calls per an hour, 1000 request. this is probably enough for the moment. API calls require a GitHub Actions runner and it takes time. events are almost immediate, but adding a label requires a runner to start, and to call APIs. from my experience, the latency is acceptable.

[trombik]

also, this model will affect how you work with the repository (at least me). here is what I have been doing:

1. update my fork
2. create a branch
3. hack
4. commit
5. push the commits to my fork
6. see if the checks pass
7. if the checks pass, create a PR
8. ask review and merge

this will not work because checks are controlled by labels. you can label PRs but a branch is not a PR.

1. update my fork
2. create a branch
3. hack
4. commit
5. push the commits to my fork (the checks will NOT run)
6. create a PR (the PR will be labeled, and the checks will run)
7. see if the checks pass
8. ask review and merge

that is, you will not be able to see if the checks pass until you create a PR. you need to create a PR to see the check results.

hope this is okay with others.

[UncleRus]

It's okay for me.

But we need detailed and clear instructions on how it all works now, how to run CI, to make sure that everything is built without errors...

In addition, I personally ask for myself to leave the possibility of automatic launch of CI on direct pushes to the repository. This does not mean that I am not going to do PRs to fix bugs, but it is necessary to be completely sure that the CI status badge on the main page of the repository reflects reality.

[trombik]

for starters, created a README.md for developers.
https://github.com/UncleRus/esp-idf-lib/blob/fc77dd11d6a799c186d4bd54e8ee182171b7fcc2/.github/workflows/README.md

will add more to CONTRIBUTING.md for users.

[UncleRus]

Good start. IMHO, CONTRIBUTING.md needs to add answers to the questions like

• What happens if a contributor creates a PR, what processes will start?
• What actions should the contributor take?
• Do he need to manually add labels or will they be added automatically?
• What labels do what?
  and so on. CI should be convenient and transparent for contributors.

[trombik]

the last question is hard to answer at the moment. this is my first attempt to implement label-driven workflows. I will document the labels that are in use, but the labels are subject to changes. let me think (and experiment) it over first.

[trombik]

here is the first implementation. this PR introduces a labeler that labels PRs by workflow conclusion. an example label is ci/Build examples/failure.

#368

in this PR, the new workflow does not actually run because the PR is from a fork. to see if this workflow run as intended, the PR must be merged first.

[trombik]

@UncleRus would you disable "Require status checks before merging"? the condition is not always required because, when changes in a PR is only "*.md" files, `Build examples" is not required to be successful anymore. I will propose a better solution to prevent not-tested-code from merging.

[UncleRus]

Ok

[trombik]

#374 has been merged. when #376 is also merged, the CI should work as documented.

[trombik]

the workflows almost works as intended. the logic is correct, and works only when someone with write access labels a PR, not the labeler.

here is why:
https://docs.github.com/en/actions/using-workflows/triggering-a-workflow#triggering-a-workflow-from-a-workflow

an easy workaround, as described in the above page, is to use a personal access token in labeler.yml.

I am trying to find another way to do it without a personal access token.

[trombik]

after several attempts to run workflows without personal access token, I came to conclusion that it is not practical or feasible.

it's possible to use workflow_run as chained workflows, but:

• worklofw_run runs on master branch and the statuses of checks do not show up in the PR (Ci tests on workflow_run do not show up in PRs #394)
• it is error-prone because the context is tied with master branch (if you simply checkout the tree, the tree is NOT the tree that should be tested, but master).
• workflow_run has extra privileges, and the author must be careful NOT to do something nasty

I do not like using Personal Access Token because of the reasons I mentioned. but, unless GitHub implement new feature to make it easy and safe to run workflows like this, PAT is better because:

• the logic is known to work in my previous commits
• the behaviors of the workflows are same as before

@UncleRus what do you think?

[UncleRus]

I guess we don't have much of a choice. Let it be PAT.

[UncleRus]

Sakurai-san, I'm in awe.
The new approach to CI is something incredible and like magic. It's amazing to see how the labeler puts down labels and starts the right flows. The assembly of examples related only to the changed components is just instantly compared to the old CI.
It is a masterpiece.

[trombik]

it's not a masterpiece, yet. i could not remove duplicated builds (building the same code). the CI builds the same code when a push to the branch adds new label to the PR, i.e. when a new commit touch another component. this happens because the triggering events in this case are labeled and synchronized. they are different events, and, thus, there would be two builds.

but in most cases, the CI works better than before.