lilybuild

Runs pipeline jobs in containers.

Focus will be given to podman containers.

Motivation

This originates as a trial to replace the current workflow in GitLab CI/CD on [Lily Islands][https://lily-is.land]. Using GitLab is unethical because (1) its official instance, gitlab.com, imposes a region lock by restricting users in regions under the rule of the Peking government from using their account on gitlab.com, and asking them migrate to another service provided by JiHu ^{1 2 3}; and (2) said company tries to threaten to sue GitLab CE users in order to ask them to pay, despite the software is free/libre. ⁴

If you are currently using GitLab, or are considering doing so, either their SaaS version, or a self-hosted version, we strongly recommend that you think twice.

Main objectives include:

• Parse GitLab CI file, and run them as buildbot builds, inside containers.
• Run CI from both a PR-style forge (forgejo) and a diff-style forge (phorge) at the same time, and unifying them.

Deployment

Only LilyBuildConfig is required. All others are optional.

lilybuild.config.LilyBuildConfig (main entry point)

Requirements:

• Buildbot Workers, using image from Containerfile.worker For the current moment:
  • The lilybuild directory in this repository needs to be mounted to /lilybuild inside the container
  • A ssh key pair without password needs to be available in /secrets/lilybuild-volume-helper-key{,.pub} inside the container
  • An image built by Containerfile.volume-helper is available under the name defined in volume_helper_image in lilybuild/podman-helper
• Repository urls (can be from any forge (of course, you can use GitLab with it, but not using it can be a highly ethical choice.))

lbc = LilyBuildConfig(
    c,
    ['workername1', 'workername2'],
    # The ones below are only needed if you need status report for phorge OR need to use `arc patch` to fetch the source.
    # Can be overriden by individual repos.
    phorge_base_url='https://other.server.example/',
    phorge_token=util.Secret('phorge-token'),
)

lbc.configure_factory_and_builder()
lbc.add_repo(
    1, # repo id
    'https://server.example/owner/repo', # canonical url
    alternative_urls=['git@other.server.example:owner/repo'], # can be empty
    do_poll=False, # set to True if you don't have a change hook
)
# add other repos
lbc.configure_pipeline_defs()

Builders

LilyBuildConfig creates 3 builders: lilybuild, lilybuild-job, lilybuild-force.

lilybuild runs a pipeline definition parsed from the CI file. It triggers lilybuild-job, which runs a single job as defined by the CI file. lilybuild-force allows you to run a pipeline on any defined repository by the "force" button. You may want to restrict who can force build.

Phorge sources

If the build comes from a Phorge change hook (TODO: document how to setup such a hook), it will try to fetch from the staging area defined in Phorge. If there is no staging area, or the staging area does not contain that diff (for example, because the user chooses to skip staging in arc diff, or the diff was submitted on the web form, probably to prevent a security patch from leaking too early), it will checkout from the canonical repo url, then try to run arc patch to fetch the source code.

If the build does not come from a Phorge change hook, it will fetch normally via the canonical repo url.

Requiring approval

By default, if the source code is fetched via arc patch, an approval is needed to run the pipeline (assuming anyone who can push to staging area is trusted). The approval is done by rebuilding the build. (TODO: make this configurable)

lilybuild.auth.ForgejoAuth

An auth provider for forgejo.

Use:

# Allow users to login via forgejo
c['www']['auth'] = ForgejoAuth('https://forgejo.server.example/', CLIENT_ID, CLIENT_SECRET)

lilybuild.auth.ForgejoAuthz

An authz provider that gives rights to people who have access to a forgejo repository to control the builds in that repository (rebuild/stop).

Must use ForgejoAuth as the auth provider.

This class extends buildbot.www.authz.authz.Authz and accepts any arguments accepted by it.

Use:

c['www']['authz'] = ForgejoAuthz(
    base_url='https://forgejo.server.example/',
    access_token=util.Secret('forgejo-authz-token'), # This needs to be a token of an **admin account** either of the forgejo server, or of all the registered repositories, and with `read:repository` access.
    # any other options...
)

License

This is licensed under the same one as buildbot, i.e. GPL v2.

---

1. https://www.reddit.com/r/gitlab/comments/1hj6ern/gitlab_can_no_longer_service_mainland_china_macao/ ↩︎

2. https://forum.gitlab.com/t/question-about-the-gitlab-can-no-longer-service-in-mainland-china-macao-and-hong-kong/120085 ↩︎

3. https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/issues/25191 ↩︎

4. https://www.sohu.com/a/835281881_122066678 ↩︎