Migrating Wayland & Weston to GitLab

Hi all, As some of you have seen, freedesktop.org is migrating its Git hosting to GitLab[0]. Whilst the documentation is still a little scratchy - partly deliberate whilst we've been bootstrapping our infrastructure and monitoring how some smaller pilot projects have gone using it - here is what it definitely means for Wayland (wearing my fd.o admin hat), and some of my suggestions of what we should do as a project (wearing my 'one of many Wayland contributors' hat). The latter suggestions are just my suggestion and opinion, which I'm putting out there for discussion. GitLab background -------------------------- [fd.o admin hat on] GitLab is a web service developed by gitlab.com, offering code hosting, online code review, issue tracking, CI integration, hosting of generated web pages, and so on. They offer a fully open-source Community Edition, as well as a proprietary subscription-based Enterprise Edition with several tiers. fd.o currently hosts a plethora of services, including a normal git daemon and SSH server for hosting, cgit for repository browsing, Bugzilla for issue tracking, Mailman for mailing lists, Patchwork for tracking patch submissions, web hosting (static pages or ikiwiki). We also host an instance of Phabricator, which we rolled out as a test to see if it could provide us with better workflows. Long story short, it didn't: its issue tracking is pretty good, but code review not, as it requires a specific 'git-phab' tool to work properly, and supporting microcommits and rebasing workflows is an explicit anti-goal. Its UI is also totally unsuited for the forge-like setup fd.o has, as well as community projects: it's designed for internal use by dedicated software teams. All of this came up during XDC in Helsinki, where after sitting down and talking we decided that the UI would need a lot of work to be useful. Since then, Phabricator's development model has also closed up with very little possibility for community contributions, which isn't good for us. It's probably notable that the vast majority of people who used phabricator.fd.o were from Collabora, where we use it for all our issue tracking internally. Last year, GNOME went through a similar search[1] for a more modern and integrated tool which would let them have issue tracking linked with code hosting, the ability to push branches and generate merge requests people could review via the web from those, CI linked with merge requests and repositories, etc. I participated in that from the point of view of giving Phabricator a fair run out, being more familiar with it than others. Over the course of the discussions, I ended up supporting their conclusion that GitLab was the best tool for them, mainly through seeing the huge strides they'd made between (at the time) 8.x to 9.x, and subsequently. GitLab also has a well-documented REST API, as well as external web hooks, which can be used to extend its behaviour in ways which aren't possible for most of our tools. Most of the functionality available through its web interface is also available through a documented API. Currently we deal with extensions with local changes to the codebase, which makes upgrades painful. One reason we're looking for a single, more modern, tool is to free up some admin time. Not only is GitLab more featureful than our existing suite of tools, but our biggest admin problem at the moment is exactly those tools. Tracking the releases and deployment of all those tools is hugely time-consuming, and all the duct tape which holds them together is incredibly fragile. Dealing with spam is also a massive problem across a lot of them, and that's even before you get to GDPR issues. Running services which don't support modern identity management is also a problem. Users hate having to sign up in multiple places (SSH account, Bugzilla, lists, Patchwork), where we maintain a username/password store. None of these services support external authentication providers (which users have complained of), none of them support 2FA (a genuine security concern), and dealing with PII erasure requests on the GDPR is basically unworkable. Late last year, some of the people responsible for GNOME's migration put us in touch with GitLab's community outreach team, who offered to support us in our migration if that's what we wanted to do. After going through the options, we have accepted a donation-in-kind from them, of paying for us to run a GitLab CE instance (we declined an EE license, or remote hosting) on Google cloud infrastructure. This will let us replace life-expired physical machines as we wind down the services which run on them. Repository migration ---------------------------- [admin hat still on] The first thing to happen is to migrate the central Git push point to GitLab. anongit.fd.o and cgit will still work as read-only mirrors, but you will not be able to push to git.fd.o. This would happen for all of wayland, wayland-protocols, weston and wayland-web. libinput we could move separately, wayland-build-tools is open to discussion (does it still work?), and wayland-java appears to be completely abandoned. All this would require is for everyone with push rights to set up their GitLab account with SSH keys as per the instructions at [0]. On a designated flag day, the push URL would become a GitLab one, and direct pushes to git.fd.o would be rejected. Users would be able to view and fork the repository at gitlab.fd.o, though not necessarily able to do any more than that. Since this is easy and low-impact, I'd propose to do this at the end of this week or early next week, if there are no objections. Issue tracking ------------------- [admin hat still on] Our plan of record is to deprecate Phabricator as rapidly as possible. Our deployment of it is not well maintained, seems to have no future (see above), and has so few users that it's certainly low-hanging fruit for retiring services. Wayland's use of Phabricator is mainly tracking and design discussion for issues filed by pq or myself. There is also some light wiki usage, e.g. around the weston_head design. Deprecating Bugzilla is a longer-term goal. As said, Bugzilla is fairly hopeless at handling spam - eliminating spam takes much more time than you might think - and is a constant source of security worries. We're not forcing anyone off either service at this point, but we do provide migration of open bugs for projects using either Phabricator or Bugzilla. The first project (GeoClue) moved its bugs from Bugzilla to GitLab over the weekend, and we should see a few more over the rest of the week, as well as bug migration from Phabricator. Compared to Bugzilla, issue tracking lets projects define their own custom milestones (releases), labels (tags), and so on, which can make it easier to track 'what needs to be done for 1.17?'. [admin hat off, contributor hat on] I believe we should start triaging our Phabricator and Bugzilla issues as soon as possible, and look to migrate them in a couple of weeks or so, when we have a more sensible set of open issues. As said before, having to register separately to Bugzilla is a common complaint from users; using GitLab would also give us issue templates we could work with. One point we haven't (yet) sorted out is reply-by-email to deal with GitLab issues, though anecdotally I don't think many people use that for Wayland and Weston, so it shouldn't be a blocker. I'm happy to do the actual migration as it needs a little bit of manual legwork, but doing triage of currently-open issues isn't something I can do on my own. I'm not sure how we should best co-ordinate that - personally I do like using the 'todos' when you have checkboxes in GitLab issues[2] to bounce things between each other without it getting lost - but there are a lot of ways to skin that particular cat. CI -- [admin hat on] One nice thing in GitLab is CI, where just like Travis and similar, you can create a .gitlab-ci.yml file which will cause the given pipeline to be run on all pushes to the repository, including merge requests. We currently support running arbitrary pipelines in Docker containers, which we can use for build tests and so on, up to 'make check' as well. Enabling support for running under VMs is WIP; running things on real dedicated hardware (e.g. 'does the KMS backend still work on Raspberry Pi') is a whole different matter though. This doesn't require any special privilege to set up: you can fork any repository, create a .gitlab-ci.yml test definition in your local fork, and run all kinds of test builds, capturing the output and any artifacts. CI is also plumbed through to GitLab Pages. Each CI job can generate downloadable 'artifacts', which are publicly available for a set period of time. The way Pages works is that you run a CI pipeline that generates a set of artifacts for Pages, which are then served from a given domain. So wayland-web in its current form would have a CI job which just captured all its files as artifacts, but it would be possible to dynamically generate it from something like Jekyll, Hugo, or whatever your generator of choice is. [admin hat off, developer hat on] It would be really cool to get a basic build test together for our repositories, in various configurations. Making it easier to fix our website would also be a massive leap forward in terms of accessibility. Code review ----------------- [admin hat on] GitLab also provides code reviews through merge requests[3], which are like GitHub's pull requests (push to a branch or separate repo, create a request for someone to review and merge that code), but much more attuned to our workflows. Specifically, GitHub is geared towards chronological commit streams (initial code, fix, fix, fix, fix, fix, more stuff, more fixes) which ultimately get squashed into one - which was replicated by Phabricator. GitLab does allow for the merge-ordered microcommits we have on the list today, including allowing you to rebase them before submitting new versions, and also not squashing the results. More usefully, it also allows you to create these with 'git push' then following the URL it offers you to create a MR. Being a real git branch allows reviewers to pull the pristine tree as the submitter had (without lossily running through patch files), and also for CI pipelines to be run on those branches so they can see problems without needing the reviewer as a proxy to 'make distcheck'. Comments on GitLab MRs can either be placed as a general comment on the MR, or directly associated with lines of code. These comments can either be comments (general chatter), or marked as an 'issue', which must be explicitly marked as resolved in order to move on. This makes it more difficult to forget review comments in subsequent revisions. [admin hat off, contributor hat on] Every time someone comes on to #wayland saying that they want to submit a patch series but can't figure out how to use git-send-email, it kills me. Every time I look at the state of Patchwork and realise that it's a wasteland because the commit -> Patchwork hooks are necessarily lossy, I despair a bit. That's if it even correctly identifies and groups the patches in the first place, which it doesn't despite two years of development specifically aimed at having it work well for dri-devel@ and intel-gfx at . Not having code review hooked into issue tracking means that even after we've demanded our contributors jump through the git-send-email hoops, their contributions can disappear into the void. I think having it hooked up to issue tracking, with assignable labels and target milestones, with discussions people can actually follow (rather than Mailman archives, where any discussion spanning a month boundary is split in two), would really lower the barrier to contributing. We can also put a CONTRIBUTING.md file which is shown to people when they create merge requests, outlining how we review and what we're looking for. CI means that we can pick up the mechanical failures early, rather than waiting for a human to come by, run the tests themselves, and inform someone who may no longer have time or interest at that point. Something I've been told anecdotally by people either contributing to or using wlc/wlroots/etc, is that one of the reasons they used that over libweston is because our archaic setup means it's pointlessly difficult to communicate with us. They've got a point. My proposal is that as of the repo migration, we trial GitLab merge requests on a couple of patchsets where we'd expect to see the need for complex structured comments and multiple revisions. These revisions should probably be shadow-posted to the list, to make sure we didn't miss anything. Once we'd been through a couple of rounds and we had a general consensus amongst reviewers that we had a good workflow we were happy to use, we would add that to CONTRIBUTING.md as well as README, informing would-be contributors that we now preferred to receive patches through GitLab, though would continue to accept patches through the list for a short time (informing those who did submit via the list that they should move to GitLab). Once the next release had passed, we would stop doing that, and have GitLab MRs as our sole point of code review. So - thoughts? Cheers, Daniel [0]: https://gitlab.freedesktop.org/freedesktop/freedesktop/wikis/home [1]: https://wiki.gnome.org/Initiatives/DevelopmentInfrastructure [2]: https://docs.gitlab.com/ce/workflow/todos.html [3]: https://docs.gitlab.com/ce/user/project/merge_requests/index.html