[openstack-dev] [All][Neutron] Improving development and review velocity - Is it time for a drastic change?

Have you been negatively impacted by slow development and review velocity? Read on. OpenStack has had a slow review velocity for as long as I can remember. This has a cascading effect where people take up multiple tasks, so that they can work on something while the other is being reviewed. This adds even more patches to ever growing queues. Features miss releases and bugs never get fixed. Even worse, we turn away new contributors due to an agonizing process. In the Neutron community, we've tried a few things over the years. Neutron's growing scope was identified and load balancing, VPN and firewall as a service were split out to their own repositories. Neutron core reviewers had less load, *aaS contributors could iterate faster, it was a win win. Following that, Neutron plugins were split off as well. Neutron core reviewers did not have the expertise or access to specialized hardware of vendors anyway, vendors could iterate faster, and everybody were happy. Finally, a specialization system was created. Areas of the Neutron code base were determined and a "Lieutenant" was chosen for each area. That lieutenant could then nominate core reviewers, and those reviewers were then expected to +2 only within their area. This led to doubling the core team, and for my money was a great success. Leading us to today. Today, I think it's clear we still have a grave problem. Patches sit idle for months, turning contributors away. I believe we've reached a tipping point, and now is the time for out of the box thinking. I am proposing two changes: 1) Changing what a core reviewer is. It is time to move to a system of trust: Everyone have +2 rights to begin with, and the system self-regulates by shaming offending individuals and eventually taking away rights for repeated errors in judgement. I've proposed a Neutron governance change here: https://review.openstack.org/300271 2) Now, transform yourself six to twelve months in the future. We now face a new problem. Patches are flying in. You're no longer working on a dozen patches in parallel. You push up something, it is reviewed promptly, and you move on to the next thing. Our next issue is then CI run-time. The time it takes to test (Check queue), approve and test a patch again (Gate queue) is simply too long. How do we cut this down? Again, by using a proven open source methodology of trust. As Neutron's testing lieutenant, I hereby propose that we remove the tests. Why deal with a problem you can avoid in the first place? The Neutron team has been putting out fires in the form of gate issues on a weekly basis, double so late in to a release cycle. The gate has so many false negatives, the tests are riddled with race conditions, we've clearly failed to get testing right. Needless to say, my proposal keeps pep8 in place. We all know how important a consistent style is. I've proposed a patch that removes Neutron's tests here: https://review.openstack.org/300272