Guest post by Colleen V. Chien, Professor, Santa Clara University Law School. This post is the third in a series about insights developed based on USPTO data.

“If we could further narrow this gap in prior art between examination and litigation, then the accuracy of the patent grant – and therefore, its reliability – would increase.”

– Director Andrei Iancu

As Congress charts its path for the rest of the year, many in the patent community are eagerly awaiting new legislation on patentable subject matter. But to the extent that the Supreme Court’s jurisprudence can be understood as a clumsy response to the proliferation of weak patents, reversing these decisions will not address the underlying root causes of poor patent quality. It is therefore crucial to address the question raised at a recent Hearing before the Senate Subcommittee on Intellectual Property: how can the quality of US patents be ensured?

This post focuses on an aspect of patent quality that, though receiving scant focused attention by scholars (with some exception, such as the excellent work of Professor Steve Yelderman, now clerking at the Supreme Court), comes up in the examination of over 90 percent of all patent applications.[1] No other issue comes close: the USPTO reports that only about a third of office actions include a rejection based on 35 U.S.C. § 112 and just 11% percent receive a rejection based on § 101. The aspect is the robustness of prior art vetting under §§ 102 (novelty) and 103 (nonobviousness). Below I argue that policymakers should pay more attention to the patent system’s most important tool for ensuring a patent’s quality: not § 101 or inter partes review, but prior art.

The task of vetting inventions in view of the prior art is old, but there are a number of new challenges. As Director Iancu has repeatedly acknowledged, the “ever-accelerating publication and accessibility explosions” strain the ability of examiners to find the best prior art during examination. Foreign patenting, which creates harder-to-evaluate foreign language prior art, is on the rise. In areas of rapid development like artificial intelligence, not unlike the early days of software in which a President’s Commission recommended prohibiting software patents because of the inability to vet them, prior art is being generated at an extraordinary rate – but in repositories like arxiv.org or Google scholar, not necessarily in filings at the Patent Office. As part of its continued focus on prior art, the Office has multiple initiatives to improve access to prior art, including the Collaborative Search and Search Feedback Pilots. It’s asked about prior art in its recent call for comments about patenting artificial intelligence (deadline extended to 11/8). But how can we know if current or piloted approaches are enough?

In a recent paper, Comparative Patent Quality, I argue that the age-old tool of benchmarking can yield valuable insights for tracking quality in patent examination, and in particular the robustness of considered prior art. In it, I describe and apply an approach for benchmarking examiner citation patterns exploiting the natural experiment that occurs when the same application is filed in the USPTO and a foreign patent office (such as the European Patent Office (EPO)). The paper specifically considers the extent to which examiners are considering the full range of prior art—not just patents but also non-patent literature (NPL)—when vetting applications.

A second project, Rigorous Policy Pilots, considers two other prior art benchmarks. The first is the patent applicant’s Information Disclosure Statement (IDS), in which an applicant must disclose to the USPTO all of the prior art or other information of which she is aware material to her application. Another is the Patent Trial and Appeals Board’s inter partes review decisions, in which administrative law judges cite what they believe to be the most relevant prior art.

These papers validate and quantify what Director Iancu has called, the “gap between the prior art found during initial examination and the prior art found,” in this case, in inter partes review. Non-patent literature (NPL) is not being cited by US examiners in the majority of cases, and US examiners are citing it less than European examiners. Within a random sample, 3.2% of the prior art relied upon by a U.S. Examiner (as provided in an 892 Form) was non-patent literature while the comparable rate in European Patent Office cases (as an X or Y reference in the search report) was 20%.[2]

Figure 1 shows the gap in NPL citation rates among Examiners, PTAB judges, and applicants vetting the same invention (906 patents whose claims have been invalidated in inter partes review). While US Examiners cited non-patent literature on average 13% of the time, the PTAB was more than three times more likely to do so (41%). (Figure 1) Applicants were more than five times (66%) more likely to include NPL in an IDS than Examiners (13%) were to cite it. Though the sample size does not permit fine-grained comparisons, the basic finding – that US examiners are citing NPL far less frequently than PTAB judges or applicants – held across all technical centers.[3]

These comparisons have their limits: in inter partes review, the challenger, not Examiner supplies the prior art, and the patents of IPR are not representative of patents in general. An applicant’s inclusion of a reference in an IDS is very different than an Examiner’s consideration, then application of a reference. European examination is closer to US examination, however, and there’s also a gap, though it’s smaller: among the patents analyzed above that had a European counterpart, EPO examiners cited NPL at a 34% rate, more than double the US examiner rate on the same applications of 14% of the time.[4] But applicants do not necessarily treat European and US “twin” applications identically, and European patent law is not identical to US patent law. Perhaps most importantly, correlation does not imply causation, and it can’t be assumed from these contrasts that a lack of robust vetting is responsible for the issuance of patents later proved to be invalid.[5]

Still, because inter partes review only revisits the patent’s validity in light of the prior art under §§ 102 (novelty) and 103 (obviousness), it squarely presents the question, were there references that the examiner missed? This analysis suggests that the robustness of examiner-cited prior art in general, and the consideration of NPL in particular, deserves attention. Work is ongoing to tease out the differences and their relationship to quality not only according to this metric, but also other aspects of robustness pertaining for example to foreign patents and the classes from which patent citations are drawn.

What can the USPTO do now to pay more attention to the robustness of prior art and narrow the gap? I recommend a few steps. First, the Office should make the “robustness of the prior art considered,” whether measured by NPL citation, foreign patent citation, diversity of references, or other measure, an explicit quality metric within existing and future prior art initiatives, consistent with the Office’s commitment to continuous improvement. Studying the link between the robustness of cited prior art and quality and tracking this explicitly as a quality metric would signal the importance of this aspect of prosecution.

Second, the USPTO, as it seeks to leverage artificial-intelligence to improve examination, should take into account the “bias” against non-patent literature currently embedded in examiner citations. If a tool is trained primarily on examiner citations it will reinforce this bias, growing, not narrowing, the gap. As to artificial intelligence applications in particular, the USPTO should also work to ensure that US examiners have access to the same references and resources (including time and information) that those in industry[6] and their international counterparts do. With many AI patent applications coming from non-US sources, high patent quality now can protect freedom to operate later, for US innovators.

Second, as the USPTO continues to evolve its approach towards prior art, including through pilots, it should do so with an eye towards rigorous evaluation. This means explicitly identifying the goal of the piloted policy (the “treatment,” like enhanced search capabilities or examiner collaboration), the theory of change behind it, and a way to measure whether or not the approach being tested has succeeded in increasing the robustness of prior art cited, using experimental or quasi-experimental approaches. To see if a new approach is working, the Office would compare the robustness of art in applications receiving and not receiving the treatment.

While treating otherwise identical applications differently can present non-trivial challenges, existing caselaw and the experiences of agencies including the USPTO, in its own randomized trademark audits, suggest that they are surmountable and the resulting knowledge, well-worth pursuing. The “hard evidence” generated by a rigorous policy pilot, implemented with sufficient power and controls, could be particularly important for justifying changes, perhaps costly, to examination and prior art processes. More importantly, applying rigorous approaches to prior art pilots would do more to support the discovery of what works to advance patent quality, using the patent system’s most important tool for doing so: prior art vetting.

Thanks to research assistant Nick Halkowski, 3L at SCU Law, for providing excellent data support.

[1] Author’s analysis, using the USPTO Office Action Dataset.

[2] Comparative Patent Quality, n 239.

[3] Across all TCs except TC2800, where the gap was 17%, the gap between Examiner and PTAB NPL citation among the 906 invalidated patents was of 20% or more . (N=74-186).

[4] Rigorous Policy Pilots, appendix Figure 2.

[5] While several excellent studies of patent quality have considered the role of prior art including Wasserman and Frakes’ studies of the time devoted to patent examination, and work by my colleagues Professors Brian Love and Christian Helmers uncovering the determinants of quality, none of that I am aware has explicitly considered the “gap”, or “relative” citation of non-patent literature citation by US examiners, as compared to PTAB judges, judges, applicants, and EPO examiners as is contemplated by this analysis. Some evidence suggests that the robustness of US examiner vetting, relative to EP examiner vetting, may have implications for outcomes. Among the US patents granted by the USPTO but fully invalidated in IPR, 202 had an EPO counterpart application. Of these, approximately 1/3 of the EPO applications never matured into patents, primarily because they were withdrawn or revoked. But while applications that did not proceed in the EPO but were granted in the US were 30% more likely to have NPL cited in the search report than their US counterpart, the difference in non-patent literature citation among cases where both offices granted the patent was much smaller, a third of that or 10%.

[6] These sources include, based on my conversations with researchers: DeepMind Research, Papers with Code, and conference proceedings from NIPS, KDD, ISLVRC, and many others.