Enforcing Stability in Hackage – Some Thoughts

Some time ago, I wrote a post here about the relationship between Hackage and the Haskell Platform. By key point was this: Hackage is not a mechanism for enforcing stability; it’s good that anyone, no matter what they write, can upload it to Hackage. If we try to do quality control as a prerequisite to Hackage, then the only consequence will be fewer packages in Hackage.

That’s still true. The recent dicussion here made me think, though. Is there a proper role for quality control in Hackage? I think there might be. I’ll start with some basic rules:

Nothing in such a system should make it harder to upload a package to Hackage. This includes asking package developers to commit to future maintenance. It wouldn’t be worth the loss of a convenient way to install packages… even packages that don’t work at all, and for which the author is merely sharing a partial proof of concept.

The system should avoid assigning personal responsibility unless necessary. This means that if someone goes away, the system should still work as well as it can, modulo that the one person isn’t contributing. If the community as a whole is waiting on someone specific to get around to something, the system is broken.

Such a system would have to be symbiotic with the Haskell Platform. The Haskell Platform is working, and a great direction to follow for the future of Haskell. Hackage shouldn’t be competing against the platform in any way. This is to me the least justifiable of any ground rule… I’m generally not a believer in being so scared of duplication that we don’t try new things… but when it comes to quality assurance, I think a different dynamic is in play versus software development. Quality assurance really is a bit of a chore, a lot less personal, and it’s fair to at least work at not wasting anyone’s time.

If something could be designed according to these ground rules, it could be a positive direction for Haskell to go.

I see several parts to the effort:

Part 1: Quality of Individual Packages

A lot of thought has already gone into this piece of the equation. Everyone is looking forward to the day when we have Hackage 2.0, which new social networky features. Here’s a few of the ideas.

Community rating of software. This is the obvious thing to add, but it also may turn out to be the most difficult, by far, to get right! Everyone wants to do a nice rating system to harness popular opinion and all that… but systems that work tend to be the result of years and years of tweaking by dedicated people who are willing to track down the answers to questions like “why is this recommending X over Y?”

We’re going to have to ask ourselves some difficult questions. Is the quality of a Haskell package something that can just be measured on a linear scale? Or does it depend on what packages you’re using it with? Does it depend on your API preferences? Or can we say that some APIs are just better than others?

Then we’re going to have to ask the hardest question: how do we get people to really rate the software on quality and stability, not level of excitement? There are a lot of exciting Haskell projects out there, which are simply bad ideas to depend on in a stable piece of software! In the Haskell community as it exists today, I suspect many of them would end up rated quite highly. Unfortunately, social ranking systems are very frequently about level of interest rather than level of usefulness. The front page of Reddit is not a source of useful, practical information.

So all in all, there are going to be a few long, hard years of asking ourselves hard questions, both about the best mathematical model to compile all of this data, and also about the social engineering needed to increase the signal to noise ratio of the rating system.

I’ll offer this one suggestion now: let’s allow users to rank projects on several axes: (a) whether the quality is good and the interface is stable and usable, and (b) how “awesome” the project is. Ask the question about awesomeness first, so we get that out of the way, since that’s the place people will be more eager to give their input. Then ask about the quality.

User-visible statistics. Adding to this subjective judgement, it would be nice to see some hard data. A number of data points would be useful in making the decision to use a package.

How many people have this package installed? Not the greatest question to ask, but an easy one! This is subject to all the same concerns as the community rating idea, though. Packages might be widely installed for a lot of reasons: among them, that the package is useful, and also that the package seemed like a really cool thing to check out.

How many packages depend on this one? How widely used are they? Definitely a much better question. However, it’s biased in its own way. Some packages just aren’t meant to be used by other libraries, while others are entirely meant for that purpose.

How frequently are there build failures in this package? In packages that depend on this one? In packages this one depends on? This could be a very interesting source of information.

How quickly is the public interface for this package changing? This is a different question from the subjective idea of package stability that Hackage already reports from the package author. Of course, it requires a distance function on public APIs of packages.

User reviews, comments, discussions. As a Sourceforge-esque feature to help Haskell package developers, this would be great! It would be especially great if, instead of two thousand isolated discussion forums, this functioned in some way as a Hackage-wide discussion forum, and certain projects and/or users could tap in by defining filters on tags, keywords, etc. for the discussions interesting to them. Of course, an obvious way to see discussion forums about packages would also have some consequences for quality assurance.

Together, these techniques could hopefully make a lot of progress on the problem of staring at the five packages for doing linear algebra in Haskell and wondering exactly how they differ from each other, and which one you should use. That’s a nice problem to solve. It can even be done without a huge risk of violating any of the ground rules mentioned above. All of the ideas mentioned above are done by others besides the package author, we’re never waiting on anyone, and it’s solving a problem that’s a bit removed from the Haskell Platform (though certainly quality decisions that go into the Haskell Platform could be influenced by the statistics and input provided here).

The fact is, though, in the whole quality assurance pictures, it’s the easy problem!

Part 2: The Integration/Stability Picture

The far more difficult problem is integration of packages and maintaining a stable collection of functioning packages. Given that packages A, B, and C work independently, how can we be sure that they work together nicely? That they don’t have incompatible version dependencies? Or different and conflicting assumptions? Or that their interfaces aren’t so wildly conflicting that it’s awkward to use them both at once?

It is certainly tempting to back off here, and just say that the Haskell Platform will fix this. (No offense to Don… I agree with most of what he’s saying in that link) I think that may be a mistake, however. The Haskell Platform encompasses a few dozen packages. Granted, that may increase over time, but: (a) aren’t there certain kinds of packages that won’t and shouldn’t ever become a part of the Haskell Platform? Do they not need integration testing? Really? Also, (b) what about the problem of whether the Haskell Platform’s processes scale to a large number of packages?

I’d argue that it might help to think, in general, about the problem of collections of Haskell packages that are independently developed, but for which there’s a community of people that want to use them together. Then the Haskell Platform becomes the limiting case where that community of people is the entire Haskell community. There may, though, be smaller such communities.

Okay, you’re being quite fair to ask for an example here. The Happstack community looks like a strong candidate. Happstack itself is written as a collection of loosely coupled packages, such as happstack-server, and happstack-state. There’s talk of adding more on top of this: modules to handle authentication, and sessions. Happstack also includes dependencies on other modules… some of which often break with respect to each other (generally a matter of version dependency situations that Cabal can’t or won’t resolve). Even more to the point, look at web development in many other languages, and you’ll notice that there are even larger collections of various frameworks and libraries available, and that very few languages handle the testing of all of those web development tools and libraries as a part of the standard library for the language.

I think we should be posing and discussing these questions.

How much of what the Haskell platform does is specific to it being the Haskell Platform? How much of it could apply to any collection of packages that are developed independently, but have a community of people that want to use them together?

What features for Hackage might be requested by the Haskell Platform? Can they be generalized to various different communities or package collections?

A proposal might look something like this: in addition to packages, Hackage could track aggregates, which just have a name, description, version, and a collection of other tools and libraries (or other aggregates?) of specific version numbers. Perhaps, as in the case of the Haskell Platform, library authors would need to propose their packages to various aggregates…. perhaps one could let the aggregate owner decide that kind of policy. Many of the part 1 features might be applied here as well.

Or that might be complete bunk, and useless. The point is, perhaps we need to be thinking about how specific subcommunities could do something like the Haskell Platform at the subcommunity level. And that seems like a place where a unified set of tools in Hackage could be a very good thing.

Conclusion

In essence, I think we’re moving the right direction here. The answers people are giving are good ones: that Hackage is going to develop some more social moderation features, but that the Haskell Platform is ultimately the answer to stability. But maybe we ought to be asking, not if Hackage should enforce stability or integration, but rather if it should provide more of the tools for more groups to do so.