The critical missing pieces and a path forward

2017/5/5 11:15:47 -0700, david.lloyd at redhat.com: > Fellow experts, > > We and other EC members and community members have, on several > occasions, communicated the various deficiencies we see in the > specification. This is an update to make sure that it is very clear > which few technical, objective criteria are being missed by the current > specification that actually pushes it over the line to unacceptability > to Red Hat in its current form. I can't speak for other EG or EC > members, but I believe this accurately summarizes our position. Thank you for the summary. We've discussed all of these issues before. Since then I've seen no new information that persuades me to change my views on them, but I'll summarize those views here for the sake of the public record. First, however, a broader point. Part of our collective role here is to act as responsible stewards of the Java Platform. If we are to take that at all seriously then we must think as carefully as we can about every change or addition that we make. Every change to existing functionality carries compatibility risk. Every addition places hard constraints upon how the platform can evolve in the future. That a change or addition appears useful to some is just the start of what must be a thorough analysis of its wider and long-term consequences. That a change or addition is "quite easy" to implement is completely irrelevant. History bears witness to this. In the early years of Java we sometimes chose to expose low-level functionality, or violate fundamental language invariants, or leave APIs open for extension without having designed them for that purpose. In each case it was convenient and expedient and, yes, "quite easy", but twenty years later we're still paying for many of those decisions. To cite just one example, consider the fact that `final` does not really mean "final", which I've mentioned before [1]. The core reflection API introduced in Java 1.1 brought the ability to change the value of `final` fields. This seemed useful, and implementing it was "quite easy" at the time, and it did not have any obvious impact on the rest of the system. In the long run, however, it has greatly complicated if not prevented JIT compiler optimizations based upon or enabled by constant folding -- which is an awful lot of them. In the context of a library or an application the consequences of any particular design choice are often apparent, and it's often possible to correct a bad choice over time. We're working here, however, on the Java Platform itself. Millions of developers and users and customers depend upon it, every day. In that context the consequences of any particular design choice are often difficult to predict -- and bad choices are sometimes impossible to correct. If responsible stewardship means anything then it means that we must reject any proposal that merely appears useful to some and is "quite easy" to implement. To act otherwise would be the height of irresponsibility. Now, on to your criteria. > The first criterion is to allow cycles to exist among modules at run > time. Our experience with deploying graphs which include third-party > modules leads me to believe that this is going to be real problem that > people encounter, which will be surprising and (in some cases) very > difficult to fix or work around. >From previous conversations my understanding is that your experience is based upon deploying graphs of existing unchanged non-modular JAR files, with all of their messy flaws, in a Java EE application server. From a purely technical perspective, it's impressive that you can do that. If we were aiming in this JSR only to describe and standardize that practice then I would agree that cycles should be allowed. A primary goal of this JSR is, however, to create a module system that's approachable by all developers -- not just developers of application servers -- for use in the construction, maintenance, and deployment of libraries and large applications. This demands that we think more broadly about how to help all developers write programs "in the large". One result of that thinking is that this module system imposes some constraints that are intended to make working systems easier to reason about, and to guide developers toward better practices. Adapting to these constraints will sometimes require changes to existing code, but we've assumed from the start that modularization is an active process undertaken by the maintainer of a component rather than something done after-the-fact by someone else who cannot modify the component. Forbidding cyclic module relationships is one of these new constraints. That such relationships are generally a bad idea is not just my personal view; see, e.g., Kirk Knoernschild's compelling arguments against them in his fine book, _Java Application Architecture_ [2], specifically in Â§4.4, "Cyclic Dependencies -- The Death Knell". As I wrote previously [3], if real experience with this module system as used directly by developers reveals that cycles should be allowed then in a future release we can revise the specification to allow them. If we do so now, however, then we can never go back on that choice. (For those not familiar with the details of the design: Cycles are not allowed when modules are resolved but it is possible to create them post-resolution via the API, if needed [4][5]. Cycles are also set up amongst all automatic modules, after resolution, to make it easier to treat existing JAR files as modules without change [6].) > On the other hand, it should be quite > easy to allow this within the module resolver code, especially given the > self-contained and all-at-once nature of its algorithm. The ease with which a change can be implemented has no relevance at all to whether it is the right change for the Java Platform for all time. (See above.) > The second criterion is to re-evaluate (piecewise if necessary) the > (still very small) module primitives patch. This will be a huge boon to > container developers and framework authors (yes, this includes us, as > well as others). This is a very small effort even if the entire patch > is taken The ease with which a change can be implemented has no relevance at all to whether it is the right change for the Java Platform for all time. (See above.) > (which is something that, as I've said before, we're open to > discussion on, and there is much we can discuss and compromise on, on a > point-by-point basis, if only such discussion could be allowed). This > bridge may open the path to allow the *entire* Java ecosystem to begin > to converge on Jigsaw in ways beyond the class path, and indeed might > mean the difference between unification and fragmentation. Mark himself > has admitted that the specification isn't perfect, which (in my opinion) > also strongly argues in favor of the sensible strategy of providing a > hook for extensibility. This concession could provide relief for _many_ > of the other problems that we and others have identified. As I wrote earlier [7], the primitives that you propose to expose were not designed for use outside of the reference implementation itself. To expose them would constrain the future evolution of this module system, and the rest of the platform, with a commitment to support them for the long term. They were not designed for that purpose. To design a proper set of such primitives, i.e., a "meta" module system, is not a goal of this JSR. If that had been a goal then we would have started by studying the internals of many existing module systems in order to identify a common set of primitives to which we would be willing to commit for the long term. That's not what we've done, however, and without having done that I can't be confident that what we have today is the right set of primitives, or even a good one. To commit to this set of primitives now would be irresponsible. > The third criterion is to provide package namespace isolation among > module path modules. I maintain that the easiest way to do this is to > isolate each module path module to its own class loader, which also > should greatly ease the transition for existing Java code that uses the > existing ClassLoader API. It is clear to myself and others that package > namespace isolation is a fundamental expectation of a module system; the > few existing widely-deployed module runtimes have this behavior. Our > own experience tells me that this is going to be a real problem for many > nontrivial applications. Not having this type of isolation dramatically > undermines the value of the system. I understand the concern here, but the impact of not isolating package namespaces by default is ameliorated by the fact that most developers use the broadly-accepted reverse-DNS convention for package names. If an application works properly on the class path today then it likely does not have conflicting packages to start with, since conflicting packages on the class path so often lead to trouble. This module system loads application modules into a single class loader by default so that developers have one less behavioral difference to worry about as they upgrade their components to explicit modules. An early prototype did load every module into its own loader by default. With that we found that a fair bit of existing code -- not just code in the JDK itself -- is written assuming that the grandparent of its loader is the boot loader, that its loader (or perhaps its loader's parent) is the thread-context loader, that code in nearby modules is in the same loader, and so on. In a future release we can explore loading every application module into its own loader, and maybe even make that the default behavior over time, but for now a more conservative approach is preferable. (If do you need to load a set of modules with each module in its own class loader then you can arrange for that in just a few lines of code via the API [8].) - Mark [1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2017-April/011983.html [2] http://www.kirkk.com/modularity/ [3] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2017-March/000646.html [4] http://cr.openjdk.java.net/~mr/jigsaw/spec/api/java/lang/Module.html#addReads-java.lang.Module- [5] http://cr.openjdk.java.net/~mr/jigsaw/spec/api/java/lang/ModuleLayer.Controller.html#addReads-java.lang.Module-java.lang.Module- [6] http://openjdk.java.net/projects/jigsaw/spec/sotms/#automatic-modules [7] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2017-February/000585.html [8] http://cr.openjdk.java.net/~mr/jigsaw/spec/api/java/lang/ModuleLayer.html#defineModulesWithManyLoaders-java.lang.module.Configuration-java.lang.ClassLoader-