What is AutoMan?

AutoMan is the first fully automatic crowdprogramming system. AutoMan integrates human-based (“crowdsourced”) computations into a standard programming language as ordinary function calls that can be intermixed freely with traditional functions. This abstraction lets programmers focus on their programming logic. An AutoMan program specifies a confidence level for the overall computation and a budget. The AutoMan runtime system then transparently manages all details necessary for scheduling, pricing, and quality control. AutoMan automatically schedules human tasks for each computation until it achieves the desired confidence level; monitors, reprices, and restarts human tasks as necessary; and maximizes parallelism across human workers while staying under budget.

AutoMan is available as a library for Scala.

Where did it all come from?

AutoMan is being actively developed by Daniel Barowy at Williams College and Emery Berger at the PLASMA Laboratory at the University of Massachusetts Amherst. Portions of AutoMan were developed as a collaboration with researchers at Microsoft Research NYC.

Getting AutoMan

The easiest way to get AutoMan is via the Maven Central Repository. If you’re using SBT:

libraryDependencies += "edu.umass.cs" %% "automan" % "1.1.7"

or if you’re using Maven:

<dependency> <groupId>edu.umass.cs</groupId> <artifactId>automan_2.11</artifactId> <version>1.2.0</version> </dependency>

Sorry, we no longer support Scala 2.10 as AutoMan requires Java 8.

Getting the source

Visit our GitHub repository at automan-lang/AutoMan.

Reporting bugs

Please report bugs using this repository’s issue tracker.

License

AutoMan is licensed under the GPLv2, Copyright (C) 2011-2017 The University of Massachusetts, Amherst.

Using AutoMan

In your source file, import the Mechanical Turk adapter (Scala syntax):

import edu.umass.cs.automan.adapters.mturk._

After that, initialize the AutoMan runtime with an MTurk config:

implicit val mt = mturk ( access_key_id = "my key" , // your MTurk "access key" secret_access_key = "my secret" , // your MTurk "secret key" sandbox_mode = true // if true, run on MTurk sandbox )

and then define your task:

def which_one () = radio ( budget = 8.00 , text = "Which one of these does not belong?" , options = ( "Oscar the Grouch" , "Kermit the Frog" , "Spongebob Squarepants" , "Cookie Monster" , "The Count" ) )

You may then call which_one just like an ordinary function (which it is). Note that AutoMan functions immediately return an Outcome , but continue to execute asynchronously in the background.

To access return values, you must pattern-match on the Outcome ’s answer field, e.g.,

val outcome = which_one () // ... do some other stuff ... // then, when you want answers ... val answer = outcome . answer match { case Answer ( value , _ , _ ) => value case _ => throw new Exception ( "Oh no!" ) }

Other possible cases are LowConfidenceAnswer and OverBudgetAnswer . If you run out of money during a computation, a LowConfidenceAnswer will let you access to lower-confidence results. An OverBudgetAnswer signals that you didn’t have enough money in your budget to begin with.

Cleanup of AutoMan Resources

Note that, due to AutoMan’s design, you must inform it when to shut down, otherwise it will continue to execute indefinitely and your program will hang:

a . close ()

Alternately, you may wrap your program in an automan statement, and cleanup will happen automatically. This feature was inspired by the C# using statement:

automan ( a ) { ... your program ... }

We will add more documentation to this site in the near future. In the interim, please see the collection of sample programs in the apps directory.

Supported Question Types

Question Type Purpose Quality-Controlled Number of Answers Returned radio The user is asked to choose one of n options. yes 1 checkbox The user is asked to choose one of m of n options, where m <= n. yes 1 freetext The user is asked to enter a textual response, such that the response conforms to a simple pattern (a “picture clause”). yes 1 estimate The user is asked to enter a numeric (real-valued) response. yes 1 radios Same as radio , except that it returns the entire distribution. no sample size checkboxes Same as checkbox , except that it returns the entire distribution. no sample size freetexts Same as freetext , except that it returns the entire distribution. no sample size

Using AutoMan with a Different Crowdsourcing Backend

We currently only support Amazon’s Mechanical Turk. However, AutoMan was designed to accommodate arbitrary backends. If you are interested in seeing your crowdsourcing platform supported, please contact us.

Memoization

AutoMan saves all intermediate human-computed results by default. You may turn this feature off by setting logging = LogConfig.NO_LOGGING in your AutoMan config. You may also set the location of the database with database_path = "/path/to/your/database" . Note that the format of the database has changed from earlier versions of AutoMan from Apache Derby to H2.

Building AutoMan

You do not need to build AutoMan yourself, as it is available via Maven as a JAR. However, if you want to hack on AutoMan, or if you just like building stuff, the AutoMan source code includes an SBT build script. The build script builds the AutoMan JAR for you, including downloading all of AutoMan’s dependencies. The build script can also build the sample applications that are located in the apps directory. These applications are the ones used in our papers.

You can build the AutoMan JAR using the following commands:

$ cd libautoman $ sbt pack

The AutoMan JAR plus all of its dependencies will then be found in the libautoman/target/pack/lib/ folder.

Sample Applications

Sample applications can be found in the apps directory. Apps can also be built using pack . E.g.,

$ cd apps/simple_program $ sbt pack

Unix/DOS shell scripts for running the programs can then be found in apps/[the app]/target/pack/bin/ .

Acknowledgements

This material is based on work supported by National Science Foundation Grant Nos. CCF-1144520 and CCF-0953754 and DARPA Award N10AP2026. Microsoft Research also generously supported research and development by funding experiments on Mechanical Turk.