Im normally a kdb+ kind of guy when it comes to managing huge amounts of streaming and historical tick data, the performance is great, the app small and clean and the language Q terse with just enough to get the job done. On the downside Q is a bit cryptic, and the documentation is brutally terse.. though readable.

I decided to do a bit of googling to see whether another product was out there that might be useful, and came across StreamBase, which is the commercial outgrowth of some research projects at MIT – Aurora, Borealis and Medusa. These projects were led by Michael Stonebraker, who invented {|discovered?} ingres and postgresql databases in their original form. His short blurb on Stream processing – Data Torrents and Rivers – is a worthwhile introduction.

The StreamSQL language spec seems to be independent of StreamBase, as it has its own site which describes the language – StreamSQL.org.

StreamSQL does seem to fall short of being a fully independent spec, and I wanted to make some comment on this… because the world really does need an accessible stream processing language that acts in the same way as SQL – I love Q but I just dont see your average quant developer having time to grok it when they already have to learn C++/Perl/Python/Matlab/R and I guess soon ruby [until lisp becomes the 100year language].

Heres my Open Letter to the StreamSQL people –

Ok, StreamBase looks like a good product, StreamSQL looks like a good language…

The value of StreamSQL is that it can be vendor neutral and have several competing implementations – exactly analagous to SQL, right?

Your StreamSQL site is nice… and very annoying :]

1) Please put a formal spec of the complete language as a single text file on your site – and give it a version number. Its nice to have the pages describe the formal spec, but its also annoying for people who have read a _lot_ of BNF / language specs – I want to see the complete language on one page, and know that that is exactly the syntax of version 0.96.20080321 of the spec!

0) This needs to be done as a consortium – currently StreamSQL is ‘owned’ by streambase and this site just makes that obvious, by trying to not make it obvious – until you get past this, no-one will touch StreamSQL with a bargepole, and we will have N competing standards that vary by unimportant details… no matter what happens they will, over time, converge to a default that everyone will then use.. Now, would you rather that default be Q? Making StreamSQL the defacto standard is in your interests… you will sell more streambase if you fully open the spec… it should be a consortium, have a voting body, and have at least 2 vendors who support it – or a vendor and perhaps an open source product.

A side issue is tone down the marketing… yes you’re already doing that, but not enough to make this work. Consider having people involved who are in academia or open source but who are not being paid by StreamBase. [am I wrong in assuming your bloggers are employees?]

I think part of maturing as a standard is that you start to see more examples publicly available on this site – I know this is the microsoft way to make people register to get to see your samples, but in the year 2008, with all the open source movement victories… well, it is evil to do that the old way :]

ps. Congratulations on a nice spec and a great product – things will advance for all if the spec grows away from its parent, and gains real independence!

Maybe a ‘Stream Query Language’ is important enough that it should just be added to ansi SQL? – the reason I think this won’t happen is that its just too expensive for vendors to say they are SQL compliant.

I guess in time, we’ll see an opensource stream database which is language compatible with either StreamBase / StreamSQL or kdb+ / q. But it’s the same old war of BetaMax vs VHS or Bluray vs HD-DVD, only one can win.