Some people like to work with new technologies because, well, they're new and by extension pose a particular kind of challenge. I am definitely one of those people; but beyond just trying new things out and learning along the way, I like to make choices with purpose.

Besides just being cool or the new thing, getting up and running with spark notebooks solves one particular problem: bridging the gap between rapid prototyping and building large-scale systems. This problem presents itself most strongly in a start up environment where you are constantly making trade-offs between getting a product out the door and limiting your technical debt. At first we used iPython notebooks for rapid prototyping and switched to Scala once a project was ready to go to production - because seriously python for production? Whatevs. Just kidding. Please don't burn my house down - I know there's lots of Python lovers out there and you have my respect.

But we wanted to take advantage of Scala's many virtues without switching between programming languages and adding complexity to our technical stack.

The basic premise is simple: