News !

As you’ll see below, the Stamina competition is now closed. Congratulations to the winners once again!

This website has been officially closed on 21 of March 2018, meaning that all features related to the competition

itself have been disabled (login, compete, submissions, etc). You can still download the training and test sets, though. Please send an email to blambeau@gmail.com with any question.

However, we are currently in the process of updating this website to become an online benchmark for grammar induction (applied to software model synthesis). This means that you can still download training and test sets, register and participate, use the oracle (once registered), and … try to outperform the winning algorithm!.

Our new benchmarking oracle is more verbose as it reports your exact score after submitting a solution rather than a simple success/failure answer. We hope that you’ll use and enjoy it. Additional features will be added as soon as we can! Feel free to contact us with any question and/or idea.

The Stamina Organizers.

About

The STAMINA (STAte Machine INference Approaches) competition aims to identify the best technique for learning deterministic finite state machines (FSMs). It extends former competitions, especially Abbadingo, and focusses on the complexity of the learning with respect to the alphabet size. Stamina relies on an adapted generation protocol for target machines and strings. It aims to encourage the development of novel learning algorithms and to encourage cross-fertilization between the machine learning and software engineering communities. A prize of £700 (about $1146) will be awarded to the winner.

The most significant contributions will be selected by the competition organizers and their authors will also be invited to submit their work in a special issue of the Journal of Empirical Software Engineering.

Additional details are given on the protocol page; see also instructions to participate. Implementation of baseline learning algorithms is provided. You can also download the training and test samples, and create a competitor account (required to submit results).

The Competition is closed !

The competition ended on the 31st December 2010. Submissions are no longer taken into account for computing the grid below, even if the oracle stays available in your participant section! The offical winners are:

Marijn Heule (TU Delft) and Sicco Verwer (TU Eindhoven, photo)

with the DFASAT algorithm

Congratulations to them !!

Competition grid

The grids below provide feedback about the competition results. The first one presents cell winners and best challengers so far.

Cell winners and best challengers Sparsity of the training sample 100% 50% 25% 12.5% Alphabet size 2 MVdPyA/Equipo MVdPyA/Menor 5 DFASAT / DFASAT DFASAT / DFASAT 10 DFASAT / DFASAT DFASAT / DFASAT 20 DFASAT / DFASAT DFASAT / DFASAT 50 DFASAT / DFASAT DFASAT / DFASAT

This second grid presents statistics about challengers and submissions: # challengers with the cell broken / # challengers with at least one problem broken / total # of submissions for the cell.

Statistics about the submissions Sparsity of the training sample 100% 50% 25% 12.5% Alphabet size 2 2 / 5 / 138 0 / 4 / 411 0 / 0 / 82 0 / 0 / 44 5 1 / 1 / 153 0 / 1 / 188 0 / 0 / 22 0 / 0 / 18 10 1 / 1 / 128 0 / 1 / 100 0 / 0 / 16 0 / 0 / 16 20 1 / 1 / 80 0 / 1 / 94 0 / 0 / 11 0 / 0 / 10 50 1 / 1 / 164 1 / 1 / 122 0 / 0 / 59 0 / 0 / 17

Organizers & Contact

Please send an email to stamina@listes.uclouvain.be with any questions.