Last month, the JPA 2.1 Expert Group sparked an interesting debate about the accessibility of the Technology Compatibility Kit (TCK) for JSRs, which are used to verify implementations of specifications for accuracy. VMware’s Oliver Gierke criticises the current practice – we spoke to him about the current situation and why he believes a transparent, open TCK process is the answer.

JAXenter: Hi Oliver. Currently there is a controversial discussion in the JPA 2.1 Expert Group on the availability of TCKs. What role do these TCKs actually play?

Oliver Gierke: Before I joined the JPA expert group, I expected the expert groups to define the TCKs alongside the specification, to make sure both of them actually align with each other and that they really implement what was specified.

Working on Spring Data JPA, I repeatedly stumbled over issues in various persistence providers that indicated core methods of the API were not being covered by the TCK or the tests were not strict enough.

I asked for ways of how to improve the TCK but didn’t receive an answer.

That was the reason I intensified my attempts to join the expert group. After joining the group, I was caught by surprise that being part of it does not mean you get access to the TCK either. All we can do is formulate a mail to point to problems and hope they get addressed somehow.

JAX: So TCKs are thus solely controlled and developed by Oracle?

Exactly. The expert group does not have access to the TCK until the final release of the spec. After that only licensees actually get access. At this point, the JSR has been concluded already and issues with the TCK are hard to identify. There’s only a vague backchannel for feedback: an email to the specification lead.

JAX: Can you maybe give an example to explain what is problematic about the current situation?

Let’s assume the following scenario: we’ve got a team of ten parties defining requirements for a piece of software. But the team does so without defining test cases. Besides that the development teams start implementing code based on the written specification. They can discuss issues with the spec team but not regarding the test suite they have to adhere to eventually. They’re essentially shooting into the dark entirely. In parallel to these activities, a third stakeholder implements test cases for the spec – completely isolated, without any feedback at all.

Now the specification gets approved, which means it’s set in stone from this point on. Exactly at that point in time, the implementors get access to the test cases and their primary task right now is to get their implementation running against the current TCK. At no point in time are there actual checks in how far the TCK adheres to what is defined in the spec, whether there untested parts of the spec and so on.

Would we run software projects this way nowadays? I am sure projects like this exist in the real world. Still, this is not the way a software project should be run, I think. Especially if it’s about a Java Standard that developers are going to code against for years – if not even decades.

To come back to the JSRs, we face the additional problem that implementations will usually be certified. If we discover glitches violating the specification which lead back to a glitch in TCK – what do we do? De-certify the implementation?

There’s no process for how to deal with issues in the TCK, or missing tests in it, and a way to correct the certifications. One more reason for why it’s crucial to come to a solid TCK in the first place.

So we potentially – but in my point of view also practically – face quite a gap between what was specified and what the implementors are required to adhere to passing the TCK. This is a problem especially for the portability of applications between persistence providers, ultimately ending in the question whether it makes sense to code against a method that is vaguely specified but is implemented with completely different semantics throughout all persistence providers effectively. This is the point where the value of a specification is undermined fundamentally.

And I think it’s fair to raise the question why one would want to work on a specification if the only authoritative instance for implementors – and developers using the API – is defined completely separately from the spec and there’s now way to control how they relate to each other.

JAX: Are these issues you describe specifically relevant for JPA? Or put in another way: Are there similar discussions going on in other expert groups?

The core aspect here is that every JSR is run based on a version of the Java Community Process (JCP) that defines the process of how to run a JSR, what needs to be produced at which point in time, and under which license they have to be published. Some specs (e.g. JPA) are very close to these rules. But I know about other specifications (e.g. CDI) that are working more openly. The TCK of the CDI spec has been publicly available under the Apache license through the course of the entire specification process. This avoids the gap between spec and TCK and integrates the community at the same time. That does of course not mean, that ambiguities in the spec can be avoided entirely but they are much less likely.

JAX: What do you suggest to improve the current situation?

I am shooting for two things. First of all, the spec and the TCK should be developed alongside each other, e.g. we could insist on no API being able to be specified without dedicated test cases being provided as well. This of course requires access for expert group members to the TCK. Implementors would especially benefit from that, as they would not have to implement for more than two years without being able to run tests against their code.

The second thing I’d welcome warmly is a process of how to deal with TCK glitches. There has to be a way to continuously improve the test suite.

JAX: Would you like to see the TCKs made completely open source and freely available to the community?

Exactly. The already mentioned CDI specification shows that this is possible. The TCK is available on GitHub - ‘forkable’ and ‘improveable’ for everyone. Why shouldn’t this work for other JSRs?

JAX: Oracle’s Linda DeMichiel wrote on the mailing list recently that there is no special team for the JPA TCKs at Oracle. She mentions that TCKs for other JSRs have been completed earlier. Were other expert groups served much earlier? It sounds as if the TCK team at Oracle is not really big, so that the provision just takes a lot of time.

That’s a very good point. It’s hard to find out details about who is working on the TCK for JPA 2.1 to what extend. Linda also mentioned that so far no one has really shown interest in helping out with the TCK. It’s important to note that I don’t want to finger point at someone for doing bad work.

My main point is that in my opinion the current process is flawed and significantly impacts the quality of the standard in a negative way. I know that a better TCK means more work in the first place and I am definitely willing to help out. Perhaps we may even find more volunteers via the Adopt-a-JSR-program. But I do think that it’s hard to attract people if it’s effectively impossible to help if the process is not open. Beyond that I think most developers using standards like JPA have not been aware of the gap and why the current state is actually problematic at all.

If you look at the situation realistically, do you think the current situation will change soon? Is Oracle ready for constructive talks?

I already mentioned that it’s not easy to get access to information – even as a member of the JPA expert group. On the other hand I see it working for other specifications. In this respect, I consider this discussion as opportunity to create transparency and hopefully create interest for participation in the community at the same time. Ultimately it’s about a well-designed and usable API. There are way too many APIs out there that developers go berserk about. So every tiny improvement is an important one.

So you would like to broaden the discussion on this issue – our readers can comment here and share their opinion. What else can they do to get involved in the discussion?

The preferred way is the user’s mailing list of course. Beyond that we maintain a JIRA instance to collect feedback for dedicated chunks of work.