The NYTimes News Service/Syndicate division distributes content to other publications. Our syndication business is growing and, until recently, was managed with a legacy desktop-based CMS, which did not integrate well with our web publishing system.

We decided to shift away from this system, and our editors needed a new authoring and syndication tool. This application, which we named Blackbeard, needed a document editor with the ability to track changes and other features like real-time dashboards. Backbone.js (coupled with ice) has served us well in the past and was an easy choice. However, deciding on a back end framework was harder.

A new back end framework provided an opportunity to alleviate some of the challenges in our development model. The highlights from our wishlist will look familiar to many developers. We wanted the following:

No wasted time waiting for the application to be deployed and the application server to start.

Anyone should be able to have an instance of the app running locally in less than five minutes.

Faster idea prototyping.

Easy implementation of real-time features.

Efficient article locking.

We set up about building a framework which accomplished all of these things.

Multiple Users and Locking Articles

Typically, multiple Blackbeard users edit or review an article in a very short period of time. We chose a model where one user exclusively locks the article while editing it and allows others to view it in read-only mode.

In the past, we’ve implemented a lock as a row in a database table with a timestamp column. When a user opens an unlocked article, a row is inserted into the table. While this user has the article open (and has not gone idle), a periodic call updates the timestamp. Stale locks are deleted every few minutes. This mechanism works well but results in a lot of writes to the database, connection overhead and thread blocking from all the back end calls to update the timestamp. We wanted to implement locking without the connection and database overhead.

Choosing Play

Since the team’s expertise is in working with the JVM, we looked at languages and tools for that platform. We narrowed our choices down to two candidates: Grails and Play. While Grails has a much larger ecosystem, we found Play’s websocket and web service APIs much simpler to use and we loved Scala’s elegance and expressiveness.

Our experience with Play has been mostly positive. Play’s WebServices API allows us to seamlessly fetch data from other applications. Using Akka (with its Apache Camel integration), we’re able to listen for article events published on ActiveMQ and then update client dashboards with just a few lines of code and in real time.

Most of our concerns about Play’s plugin ecosystem proved to be nonissues. For example, Play lacked native or plugin support for LDAP authentication, but it was easy to use Spring’s LDAP library. Setting up a blanket authentication filter was similarly straightforward using Play’s filter API. Play either provides basic building blocks or allows you to use a number of Scala or Java libraries (or any JVM library for that matter) to suit your needs.

Play is RESTful, so we didn’t have to do anything special to serve and consume JSON or XML. We also loved Play’s auto compilation of LESS and CoffeeScript, which encouraged us to use them both a lot sooner than we would have otherwise.

Play’s action composition feature allows us to achieve route-specific controls, such as permissions and authorizations, without much code. For example, to restrict a route to a certain user role, we can compose an admin action:

def AdminAction ( f : User => Request [ AnyContent ] => Result ) = { Action { request => getUser ( request ) . map ( user => if ( user. hasAdminPrivileges ) f ( user ) ( request ) else Forbidden ) . getOrElse { Forbidden } } }

We then define the route to be restricted:

def createUser = AdminAction { user => implicit request => … }

Building an Actor Hierarchy

The Akka toolkit’s actor model provides a platform to build concurrent and scalable applications. This involves creating computation units called actors, which receive messages and act on them. The advantage of an actor system is that the framework can handle thread management and system fault tolerance.

The actor hierarchy we built has a supervisor actor (called the master) and many child actors. Each child represents a user currently connected to the system and has a reference to the WebSocket connection that communicates with the user. When a user starts working on an article, it is noted in the database and a message is sent to the corresponding child actor. From this point on, the child actor sends periodic pings (with the help of the Akka scheduler) to verify that the user is active. If the client detects that the user is actively working on the article, it replies to the server and no further action is taken by either side. If the client detects that the user has been inactive for a while, it ignores the ping. The server removes the database records and unlocks the article.

We use the same system for general broadcast purposes as well. If we want to notify all connected clients about an event, we simply invoke:

master ! MSG _ BROADCAST ( relevant _ json )

Lessons Learned

We learned a few lessons along the way. The actors that syndicated content were initially configured to execute in a Future using Play’s default execution context. This meant any network anomaly could result in both syndication and user requests being blocked because the controllers also shared the same execution context. The solution was simple — have the Futures execute in a different execution context, such as:

class FtpActor ( ) extends Actor with ActorLogging { implicit val ec : ExecutionContext = ExecutionContext. fromExecutor ( Executors. newCachedThreadPool ( ) ) def receive = { case Transmit ( obj ) => Future { blocking ( ftpProvider. upload ( obj ) ) } ... } }

Maintaining state in an actor carries risks because the actors can crash with their states. The actor system takes care of recreating them with no effects on other parts of the system that refer to that actor. At first glance, this is problematic for us, because each child actor keeps information about articles that a user is working on. But a quick read of the Akka documentation showed us that we could use the life cycle events (specifically the restart hooks) to ensure states are not lost.

In Conclusion

The Play framework and the Scala language have made it fairly easy (and fun) for us to build this tool. You can can learn more about the details of the implementation from Victor Chan’s presentation at the nescala conference. We also have more blog posts coming up about how Scala, Akka and other components of this ecosystem are being used at The Times to build scalable systems.