Marko Topolnik Marko Topolnik, PhD. is a Java professional and an active contributor on Stack Overflow. Here he builds upon his previous article on Java 8 Streams to show how they can be leveraged to improve the scalability of your Spring-based RESTful Web Services.

Web applications have recently settled on the "rich client" type of architecture, which means that the V part of the MVC pattern (the View) runs on the client side. A corollary of this is that the server side can now focus on pure data model, a further consequence being that the Web landscape is arriving at something that was a buzzword ten years ago in the Enterprise segment: service-oriented architecture (SOA).

The server side today is almost invariably defined in terms of RESTful web services it provides; well-defined over-the-wire APIs are the norm. I find this development great: all that server-side HTML templating business never found a sweet spot in my heart. In the big picture this means that the focus of the Web is shifting from offering end products to providing value-adding services which the customers can mix and match in creating their own customized user-visible products.

##1 The role of Spring MVC in RESTful services

In the realm of server-side Java Spring has always been my favorite choice. It has had strong support for the previous-generation HTML web applications, defining a well-sliced framework where the concerns of model, view, and controller are effortlessly separated. This framework had to be just slightly adapted to accommodate RESTful services: the concept of an HTTP message converter was introduced, which is a strategy object handling the concern of conversion between POJOs and the raw HTTP request/response body.

HTTP responses are generated by message converters sitting inside a fully automated view layer where they are dispatched based on the requested MIME content type on the one end and the type of the POJO representing the response on the other. The application writer is mostly concerned with providing controller classes which handle the concern of routing HTTP requests to Java methods (including the dissection of the HTTP request into method arguments), and service classes which implement HTTP-agnostic business logic.

##2 Flow of control between Controller and View

On the response side the key point of integration between the controller method and the message converter is its return value; in other words, the view will pull data from the controller. There is nothing wrong with this in principle, but there is one area where it may cause trouble if specific care is not applied: generating huge responses. Even though most web service requests are lightweight in terms of data volume (some query parameters are passed in and typically a JSON object is returned), almost every application also has a few tougher ones: services returning arbitrarily large results of queries on customer's data.

Whatever service you provide to the customer, the volume of their data hosted on your server is probably getting larger each day and the customer occasionally wants that data in bulk to process (or simply keep) offline.

Traditionally, the best-practice workflow of a request would go as follows:

make one or more database queries; retrieve the data; transform the data into the final form as required by the API; accumulate the result into the return value; commit and release the database connection; hand over the data to the message converter.

Clearly, for large responses the above creates an O(n) demand on the JVM heap memory: the more data you serve, the closer you get to an out-of-memory condition. This doesn't only put a hard limit on the size of your response; it also clogs up the JVM for everyone else, putting all other concurrent requests in danger of draining the heap, even if their individual demands are modest.

One crude approach to working around this issue has been to break the view-controller barrier and have the controller take over the concern of generating the HTTP response, pushing data directly from the database into the raw HTTP response stream. The controller had to either dispense with all the flexibility offered by the standard view layer, or waste the project's complexity budget on reimplementing it on the wrong side.

Another workaround was to simply refuse to serve large responses, requiring the client to issue requests for a limited range of data, issuing as many requests as needed to retrieve the whole dataset. This resulted both in the loss of atomicity (data rows could be repeated or missing) and an overall increase of the server load because the database must compute all the result rows preceding the first row of the requested range.

##3 Implicit change of workflow using Java 8 Streams

The proper way to handle the above issues is rearranging the workflow this way:

issue the query; acquire a cursor over the result set without consuming it; package the cursor together with the transforming logic; commit the transaction, holding the cursor; pass the cursor+logic to the view layer as the return value; in the view layer, produce the HTTP response by consuming the result set and applying the logic to each row; release the cursor and database connection.

Note that the above uses a feature not required by the "eager" scenario: holdable result sets. Database and JDBC implementations vary on how they achieve holdability and we'll give an overview for some mainstream databases towards the end of this article.

With Java 8 streams the best part is that the above rearrangement of workflow can happen while the code stays almost exactly the same — no part of it must migrate to another layer. For the case of Hibernate queries, compare

public List<UserDto> userList() { return sf.getCurrentSession() .createQuery("select new UserDto(name, email) from User").list(); }

with

public Stream<UserDto> userStream() { return resultStream(sf.getCurrentSession() .createQuery("select new UserDto(name, email) from User")); }

The code looks almost exactly the same, but the workflow implied by it changed from eager to lazy. As always, the stream can be automatically parallelized at the flick of a switch; just keep in mind that this pays off only if your transformation logic is substantial enough to make CPU the bottleneck, as opposed to I/O.

If you are interested in bringing this kind of change to your application, read on to learn about the setup and support code you'll need.

##4 Setting up the Spring+Hibernate project for holdable result sets

At the moment of this writing the latest Spring release still makes it a bit difficult to fully support result sets whose holdability extends into the View layer; however thanks to the great Spring team, this will be improved almost immediately, as tracked by the issue SPR-12349. I shall therefore skip the workarounds that have been necessary so far. As of Spring 4.1.2 you'll have to set up only two things to make this work:

install the OpenSessionInViewInterceptor to extend the lifetime of the Hibernate session into the View layer;

achieve result set holdability by enabling the newly-added property on your HibernateTransactionManager : allowResultSetAccessAfterCompletion .

Here's a quick JavaConfig example for the OpenSessionInViewInterceptor part. Typically you already have a WebMvcConfig class or its equivalent, so merge these details into it.

@Configuration public class WebMvcConfig extends WebMvcConfigurationSupport { @Autowired private OpenSessionInViewInterceptor osivInterceptor; @Autowired @Bean public OpenSessionInViewInterceptor osivInterceptor(SessionFactory sf) { OpenSessionInViewInterceptor osiv = new OpenSessionInViewInterceptor(); osiv.setSessionFactory(sf); return osiv; } @Override protected void addInterceptors(InterceptorRegistry registry) { registry.addWebRequestInterceptor(osivInterceptor); } }

And here's the setup you need for the HibernateTransactionManager . Your application should already have the transaction manager bean configured, so just enable the mentioned property on it:

@Configuration @EnableTransactionManagement public class DataConfig { @Autowired @Bean public PlatformTransactionManager txManager(SessionFactory sf) { HibernateTransactionManager mgr = new HibernateTransactionManager(sf); mgr.setAllowResultAccessAfterCompletion(true); return mgr; } ... (the rest of your data config, including the LocalSessionFactoryBean) ... }

##5 resultStream(): get the result of a Hibernate query in a Stream

In my previous post I presented the abstract class FixedBatchSpliteratorBase which makes it easy to implement any spliterator of a sequential data source. Here's the specialization to Hibernate query results:

import static org.hibernate.ScrollMode.*; ... (non-static imports elided) ... public class ScrollableResultsSpliterator<T> extends FixedBatchSpliteratorBase<T> { private final ScrollableResults results; private boolean closed; private Boolean canUnwrap; public ScrollableResultsSpliterator( Class<T> clazz, int batchSize, ScrollableResults results) { super(ORDERED | NONNULL, batchSize); if (results == null) throw new NullPointerException("ScrollableResults must not be null"); this.results = results; } @SuppressWarnings("unchecked") @Override public boolean tryAdvance(Consumer<? super T> action) { if (closed) return false; if (!results.next()) { close(); return false; } if (canUnwrap == null) { Object[] r = results.get(); canUnwrap = r.length == 1; action.accept((T) (canUnwrap ? r[0] : r)); } else { action.accept((T) (canUnwrap ? results.get(0) : results.get())); } return true; } public void close() { if (!closed) { results.close(); closed = true; } } }

Now we add convenience factory methods:

public static <T> Stream<T> resultStream( Class<T> clazz, int batchSize, Query query) { return resultStream(new ScrollableResultsSpliterator<T>(clazz, batchSize, query)); } public static <T> Stream<T> resultStream( ScrollableResultsSpliterator<T> spliterator) { return StreamSupport.stream(spliterator, false) .onClose(spliterator::close); }

and that's it. Depending on need we may add more convenience, including factories/constructors for Criteria in addition to the presented Query , using the default batch size, etc.

Note that I have submitted the equivalent of the above class for inclusion into the Spring Framework. It is being considered for version 4.2 as a part of the wider package of Spring's support for Java 8 Streams. The state of this proposal is tracked by the Spring issue SPR-12388

##6 HTTP message converters for Streams

By now we have gone through all the setup needed to deliver a Stream to the view layer, but by default there will be no message converters set up to support it. For potentially huge result sets one of the better choices of wire format is the plain-old CSV because it can be parsed line by line without involving any long-term parser state.

This is different from XML or JSON, which both need the large collection to be couched inside an overarching structure. The client side will probably have an easier time handling CSV than the other options. However, for API consistency you may want to use the format used for everything else in your app/service.

Here is my message converter which can convert a stream of CsvRecord s into an HTTP response with the content type "text/csv". It has the opencsv library as its dependency.

import static java.nio.charset.StandardCharsets.UTF_8; import static org.springframework.core.annotation.AnnotationUtils.findAnnotation; ... (non-static imports elided) ... public class CsvMessageWriter extends AbstractHttpMessageConverter<Stream<? extends CsvRecord>> { public static final MediaType MEDIA_TYPE = new MediaType("text", "csv", UTF_8); private final char separator, quote; public CsvMessageWriter() { this(CSVWriter.DEFAULT_SEPARATOR, CSVWriter.DEFAULT_QUOTE_CHARACTER); } public CsvMessageWriter(char separator, char quote) { super(MEDIA_TYPE); this.separator = separator; this.quote = quote; } @Override public boolean canRead(Class<?> clazz, MediaType mediaType) { return false; } @Override public boolean canWrite(Class<?> clazz, MediaType mediaType) { return Stream.class.isAssignableFrom(clazz) && canWrite(mediaType); } @Override protected boolean supports(Class<?> clazz) { // should not be called, since we override canRead/Write throw new UnsupportedOperationException(); } @Override protected Stream<? extends CSVRecord> readInternal( Class<? extends Stream<? extends CSVRecord>> clazz, HttpInputMessage inputMessage) { throw new UnsupportedOperationException(); } @Override protected void writeInternal( Stream<? extends CSVRecord> stream, HttpOutputMessage m) throws IOException, HttpMessageNotWritableException { m.getHeaders().setContentType(getSupportedMediaTypes().get(0)); final boolean headerDone[] = {false}; try (CSVWriter out = new CSVWriter(new OutputStreamWriter(m.getBody(), UTF_8), separator, quote)) { stream.forEachOrdered(rec -> { if (!headerDone[0]) { headerDone[0] = true; Optional.ofNullable(findAnnotation(rec.getClass(), CsvHeader.class)) .map(CsvHeader::value).ifPresent(out::writeNext); } out.writeNext(rec.toStringArray()); }); } } }

The configuration needed to register the above with Spring MVC is here:

@Configuration public class WebMvcConfig extends WebMvcConfigurationSupport { @Override protected void configureMessageConverters(List<HttpMessageConverter<?>> converters) { converters.add(new CsvMessageWriter('\t', '"')); addDefaultHttpMessageConverters(converters); } ... }

The CsvRecord type is our custom interface:

public interface CsvRecord { String[] toStringArray(); }

The intent is to implement this interface in any class you choose to be the carrier of your result data. opencsv 's CSV writer takes a string array as the representation of a CSV row, so the toStringArray() method must provide that.

To handle the concern of the CSV header row, I have chosen to define a custom class-level annotation:

import static java.lang.annotation.ElementType.TYPE; import static java.lang.annotation.RetentionPolicy.RUNTIME; ... (non-static imports elided) ... @Retention(RUNTIME) @Target(TYPE) public @interface CsvHeader { String[] value(); }

If you want your HTTP response to contain a CSV header row, annotate the stream element class with @CsvHeader("COL_1", "COL_2", ...) . Note that the restriction of this design is that an empty result set will not have the header row because I read this off the first instance in the Stream.

To output a Stream as a JSON array, we can add a custom serializer to the Jackson-based JSON infrastructure offered by the Spring framework. This shows a factory method which will instantiate the standard Spring converter and add a Stream serializer to its ObjectMapper :

private MappingJackson2HttpMessageConverter jackson2HttpMessageConverter() { MappingJackson2HttpMessageConverter jackson = new MappingJackson2HttpMessageConverter(); ObjectMapper om = jackson.getObjectMapper(); JsonSerializer<?> streamSer = new StdSerializer<Stream<?>>(Stream.class, true) { @Override public void serialize( Stream<?> stream, JsonGenerator jgen, SerializerProvider provider ) throws IOException, JsonGenerationException { provider.findValueSerializer(Iterator.class, null) .serialize(stream.iterator(), jgen, provider); } }; om.registerModule(new SimpleModule("Streams API", unknownVersion(), asList(streamSer))); return jackson; }

We register this with the framework the same way as shown above for the CSV converter:

@Override protected void configureMessageConverters(List<HttpMessageConverter<?>> converters) { converters.add(jackson2HttpMessageConverter()); ... }

A further possible option with XML/JSON is delivering a series of XML documents/JSON objects, each representing one row of data, instead of one large entity. This would help to simplify parsing, but it would be quite non-standard: for example, I am not aware of a standard MIME type for such content.

##7 Support for cursor holdability in mainstream database/JDBC implementations

Each database has a different story for cursor holdability. Consult the below list to see how your target database fares:

Oracle supports nothing but holdable cursors.

MS SQL Server supports non-holdability, but prefers holdability.

PostgreSQL implements holdability as an afterthought, copying the entire result set into a temporary area.

PostgreSQL JDBC driver doesn't even make use of that afterthought: it creates a non-holdable cursor and eagerly prefetches the entire result set into JVM heap (so-called "client-side" cursor).

MySQL flat-out does not support holdability.

So, with MySQL you are out of luck and with PostgreSQL there is still an associated O(n) cost in heap usage. This may be improved if the protocol used by the JDBC driver (the "frontend/backend" protocol) is extended to support the creation of holdable cursors. In a typical scenario the allocation in the JDBC driver will be less than the allocation of DTOs in the service layer, but this concern must be kept in mind.

##8 Testing it out with VisualGC

Let us now verify our theoretical predictions on a real system. For this I have used an existing Spring project; I chose a Stream-returning method and added another which is exactly the same in every respect, except it returns a List. The RESTful service responds with identical JSON in both cases. I also took care to disable the OpenSessionInViewInterceptor for the list-returning method to get the realistic setting for a project without holdable result sets. I used VisualVM and its VisualGC plugin to monitor the heap activity while the REST request was in progress.

This is what the activity looks like with the list-returning method:

First, let's spend some time getting to know all the panes here. On the left side are the Spaces — the regions of JVM memory. The diagrams reflect the current size and occupancy of each space. Ignore Metaspace (it's not a part of the heap); we have the Old generation as a monolithic block and the New generation divided into three parts: Eden, S0, and S1 (S stands for Survivor).

Once an object reaches the Old generation, it's stuck there until a Major garbage collection cycle occurs (an expensive operation, "stopping the world" for at least half a second, often as much as 2 seconds). An important goal for application architecture is preventing objects from being "tenured" into the Old generation.

New objects are born into the Eden space; when Eden is full, a Minor GC occurs (a very fast operation, could take under a millisecond). Live objects go to the current Survivor space (only one of S0 and S1 is "current" at any time). On the next Minor GC, the still-surviving objects are copied to the other Survivor space (which has become the "current" one at that point), together with those coming fresh from Eden. Objects surviving a few Minor GCs are finally tenured to the Old generation. Tenuring may also happen when running out of Survivor space.

The right-hand pane shows the Graphs. For each item in the Space pane there is a corresponding item here (plus some more, which we shall not deal with), showing the history of its state. The state is updated at a chosen interval; these pictures were made using the highest setting (10 times per second).

Let us focus on the Eden graph. The request started close to the left edge of the graph. The initial period was quiet: the database was preparing the result set. Then the transfer began and the allocation rate immediately surged to a very high level: the typical see-saw pattern shows the Eden being filled up to the top, a Minor GC occurring, Eden being evacuated, then again filling over just a few tenths of a second.

Now follow along to the Survivor spaces: the very first Minor GC (the first dent in Eden's see-saw pattern) immediately filled Survivor 0 all the way to the top: a lot of objects are being retained. The next Minor GC copied the entire Survivor 0 to Survivor 1 (no survivors died in the meantime) and the new survivors from Eden had to overflow directly to the Old generation (that's the first step in the staircase pattern of the Old generation).

The same sequence of events keeps going on for another five Minor GC cycles and the Old generation fills almost to the top. The Eden fills up again, but a Minor GC will not cut it anymore because there's no more room in the Old generation. So a Major GC hits in (recognizable by the thick slab in the GC Time graph). The result is only a partial cleanup within Eden; all the objects in the Old generation are still reachable. We are now on the verge of an out-of-memory event.

Luckily, the response is now prepared and begins to be served to the client. Heap stays clogged up for the entire duration of the transfer of the HTTP response to the client. If any other HTTP requests arrive during this critical period, they will quite probably fail with an OutOfMemoryError .

Now let's move on to the graphs for the stream-returning method:

The pattern is markedly different: the Old generation is a flat line (no objects are getting tenured); the Survivor spaces show partial occupancy; and the see-saw pattern in Eden has a significantly lower density, which means that the allocation rate is much lower. That's because objects are being allocated only as the client consumes the response, which means that the I/O transfer rate directly steers the Eden allocation rate. If other requests arrived while serving this one, the see-saw pattern would simply intensify; the Old generation would stay as flat as here.

Also note the initial, gentler slope in Eden: I have performed my tests with PostgreSQL which, as explained, uses a "client-side" cursor when holdability is requested. This initial slope indicates data being transferred into the JDBC driver. Even though the rate of transfer in terms of rows per second is much higher than during the HTTP transfer, memory allocation rate is much lower.

I specifically chose such a web service where the data volume in the HTTP response is much larger than in the database result set in order to suppress the effects of this particularity of PostgreSQL. The height of that gentler slope is exactly the height of the slabs in the Survivor spaces (the result set is retained by the JDBC driver throughout the response delivery phase).

Finally, compare the Spaces panes on the two pictures above. Note how large the Survivor spaces are in the first picture: they grew in order to adapt to the high rate of object retention. I executed the same type of request about ten times before taking the screenshot, so the picture shows the steady state for that case. This arrangement is detrimental to overall performance because Minor GCs will happen more often with the shrunken Eden space.

##9 Conclusion

In this article we have both explained and experimentally confirmed the advantages of lazy streams applied to the MVC pattern in Spring RESTful services. Java 8 gives us an easy way to write code whose shape stays the same as with eager evaluation. Separation of concerns between layers is preserved and automatic parallelization is just a flick of a switch away. One caveat remains: do check the policy employed by your JDBC driver for holdable result sets.

PostgreSQL's driver uses a client-side cursor, which in the worst case may entail more retention than in the case of eager evaluation. Even though that worst case is quite unlikely in practice (you would have to fetch more than you deliver to the client), it is something to keep an eye on. I will try to raise an issue with the PostgreSQL team to support native holdable cursors over JDBC.