Welcome back EVE industrialists! Or anyone interested in using Ruby scripting and web APIs to do some data mining! This is the second post in my series on hacking EVE.

Our long-term goal is to use automation to determine which items will be most profitable for an EVE industrialist to build. In part 1 of this series, I talked about using Ruby, RSpec, ActiveRecord, and MySQL to read static data from a MySQL export file. This post will discuss using Nokogiri and Moneta to read dynamic market data and help us to determine accurate prices for our products.

An Easy but Inaccurate Method

While players have long requested that CCP provide official market data APIs… they do not. Fortunately we have some 3rd-party sites that do. I’ll be using EVE-Central’s API, although you could certainly use the EVE Marketdata’s API if you prefer.

EVE-Central provides two APIs that look promising. The first is the marketstat API. This API requires that you pass the typeId that you’re interested in like:

http://api.eve-central.com/api/marketstat?typeid=34 http://api.eve-central.com/api/marketstat?typeid=34

It then returns a document including statistics on the buy and sell orders for that item type:

<evec_api method = "marketstat_xml" version = "2.0" > <marketstat > <type id = "34" > <buy > <volume > 185093933839 </volume > <avg > 4.02 </avg > <max > 6.32 </max > <min > 1.30 </min > <stddev > 0.74 </stddev > <median > 4.26 </median > <percentile > 4.78 </percentile > </buy > <sell > <volume > 80623224020 </volume > <avg > 5.26 </avg > <max > 16.00 </max > <min > 2.88 </min > <stddev > 0.95 </stddev > <median > 4.96 </median > <percentile > 4.29 </percentile > </sell > <all > <volume > 273991794319 </volume > <avg > 4.29 </avg > <max > 12.50 </max > <min > 0.20 </min > <stddev > 1.07 </stddev > <median > 4.41 </median > <percentile > 1.30 </percentile > </all > </type > </marketstat > </evec_api > <evec_api method="marketstat_xml" version="2.0"> <marketstat> <type id="34"> <buy> <volume>185093933839</volume> <avg>4.02</avg> <max>6.32</max> <min>1.30</min> <stddev>0.74</stddev> <median>4.26</median> <percentile>4.78</percentile> </buy> <sell> <volume>80623224020</volume> <avg>5.26</avg> <max>16.00</max> <min>2.88</min> <stddev>0.95</stddev> <median>4.96</median> <percentile>4.29</percentile> </sell> <all> <volume>273991794319</volume> <avg>4.29</avg> <max>12.50</max> <min>0.20</min> <stddev>1.07</stddev> <median>4.41</median> <percentile>1.30</percentile> </all> </type> </marketstat> </evec_api>

Using Nokogiri, we can pretty easily download, parse, and extract the interesting bits of information here:

#!/usr/bin/ruby require 'nokogiri' require 'open-uri' abort ( "Please pass an item id to be looked up." ) unless ARGV. length > 0 typeID = ARGV [ 0 ] api = "http://api.eve-central.com/api/marketstat?typeid=#{typeID}" puts "Getting market data: #{typeID} ..." marketData = Nokogiri::XML ( open ( api ) ) price = marketData. css ( "all avg" ) [ 0 ] . content puts "The average market price for type #{typeID} is #{price}" #!/usr/bin/ruby require 'nokogiri' require 'open-uri' abort("Please pass an item id to be looked up.") unless ARGV.length > 0 typeID = ARGV[0] api = "http://api.eve-central.com/api/marketstat?typeid=#{typeID}" puts "Getting market data: #{typeID} ..." marketData = Nokogiri::XML(open(api)) price = marketData.css("all avg")[0].content puts "The average market price for type #{typeID} is #{price}"

And get output like the following:

$ ruby marketStat.rb 34 Getting market data: 34 ... The average market price for type 34 is 4.29 $ ruby marketStat.rb 34 Getting market data: 34 ... The average market price for type 34 is 4.29

If you’re looking for a ballpark number, say for a killboard, this may be all you need. I find that for modeling my actual build profits though, these numbers are fairly useless. There are a number of reasons for this, including:

I’m not looking for rough numbers — I’m looking for the most accurate model I can find. Five percent off is a big deal in terms of profit margins.

Players deliberately post wildly high and low market orders to throw off averages, so I want to filter out that noise.

There are only so many jumps I’m willing to make to gather build resources and take goods to market, so only some markets really matter to me.

I generally don’t want to wait 3 months for my items to sell, so I’m really only interested in the lower end of the sell orders.

To be fair to marketstat, you can specify a particular system or region that you want to see prices from and use the minimum sell order value as your price. Even using those changes however, I still tend to get prices that don’t match up particularly well to reality. What I want is to be able to:

Use data only from the markets in which I’m likely to buy or sell items. It’s not profitable to fly all over the universe to save an ISK or two; so for planning purposes, only the prices in the markets I use matter.

Average the low end of the sell orders, say the bottom five. I don’t like to count on selling products at prices much higher than that, as I prefer to move items quickly.

I am, however, likely to buy in the markets with the lowest prices and sell in the markets with the highest, so I want to take that into account.

A Useful Pricing Model

So what I’d like to do in code is this:

require './models/pricing/quick_look_data.rb' require './models/pricing/low_sell_orders_pricing_model.rb' require './models/pricing/composite_pricing_model.rb' require './models/pricing/persistant_pricing_model.rb' # First build a pricing model for each of the systems we care about. # We'll do better than these magic system and type id numbers in later posts. jita = LowSellOrdersPricingModel. new ( QuickLookData. new ( usesystem: 30000142 ) ) amarr = LowSellOrdersPricingModel. new ( QuickLookData. new ( usesystem: 30002187 ) ) # Next we'll build a model that gives us the composite data from those systems. markets = CompositePricingModel. new ( [ jita, amarr ] ) # Persist it so we don't wear out the eve-central server. pricing = PersistantPricingModel. new ( markets ) # And now we should be able to get useful pricing data. puts "Tritanium buy price: #{pricing.buy_price(34)}" puts "Tritanium sell price: #{pricing.sell_price(34)}" require './models/pricing/quick_look_data.rb' require './models/pricing/low_sell_orders_pricing_model.rb' require './models/pricing/composite_pricing_model.rb' require './models/pricing/persistant_pricing_model.rb' # First build a pricing model for each of the systems we care about. # We'll do better than these magic system and type id numbers in later posts. jita = LowSellOrdersPricingModel.new(QuickLookData.new(usesystem: 30000142)) amarr = LowSellOrdersPricingModel.new(QuickLookData.new(usesystem: 30002187)) # Next we'll build a model that gives us the composite data from those systems. markets = CompositePricingModel.new([jita, amarr]) # Persist it so we don't wear out the eve-central server. pricing = PersistantPricingModel.new(markets) # And now we should be able to get useful pricing data. puts "Tritanium buy price: #{pricing.buy_price(34)}" puts "Tritanium sell price: #{pricing.sell_price(34)}"

We’ll take the average of the lowest sell orders in each system we care about as the price of the item in that system, then have a composite pricing model that assumes we’ll buy in the cheapest system and sell in the most expensive one. Finally we’ll use a wrapper that persists the data we get back so that we can run our scripts more quickly. This won’t matter in our simple case but gets pretty important if you want to, say, find the profit margin of every buildable item in the EVE universe.

The Implementation

The full source of my tests and models is available on GitHub, but I’ll highlight the more interesting bits.

First, the code to build the quicklook api and open a stream to the eve-central server are pretty straightforward:

require 'uri/http' require 'cgi' require 'open-uri' class QuickLookData def initialize ( options = { } ) @query = options end def uri ( extraQueryArgs = { } ) args = { } args [ :host ] = 'api.eve-central.com' args [ :path ] = '/api/quicklook' query = @query . merge ( extraQueryArgs ) . map { | k,v | "#{CGI.escape(k.to_s)}=#{CGI.escape(v.to_s)}" } . join ( "&" ) args [ :query ] = query unless query. empty ? URI::HTTP . build ( args ) end def data ( args ) open ( uri ( args ) ) end end require 'uri/http' require 'cgi' require 'open-uri' class QuickLookData def initialize(options = {}) @query = options end def uri(extraQueryArgs = {}) args = {} args[:host] = 'api.eve-central.com' args[:path] = '/api/quicklook' query = @query.merge(extraQueryArgs).map {|k,v| "#{CGI.escape(k.to_s)}=#{CGI.escape(v.to_s)}"}. join("&") args[:query] = query unless query.empty? URI::HTTP.build(args) end def data(args) open(uri(args)) end end

I’ll call data(:typeid => 34) to load a stream for the tritanium quicklook entries, and initialize my QuickLookData instances with (:usesystem => systemId) to only load the entries from systems I care about. The open-uri open(uri) method is all I need to fetch the feed.

Next I’ll use that data in my LowSellOrdersPricingModel.price method:

def price ( id ) return @prices [ id ] if @prices . has_key ? ( id ) print "Getting market data: #{id} ..." marketData = Nokogiri::XML ( @dataSource. data ( :typeid => id ) ) puts " Done." prices = marketData. css ( "sell_orders order price" ) . collect { | n | n. content . to_f } . sort . take ( 5 ) price = prices. size > 0 ? prices. inject ( 0.0 ) { | sum, el | sum + el } / prices. size : 0 @prices [ id ] = price price end def price(id) return @prices[id] if @prices.has_key?(id) print "Getting market data: #{id} ..." marketData = Nokogiri::XML(@dataSource.data(:typeid => id)) puts " Done." prices = marketData.css("sell_orders order price"). collect {|n| n.content.to_f}. sort. take(5) price = prices.size > 0 ? prices.inject(0.0) {|sum, el| sum + el} / prices.size : 0 @prices[id] = price price end

I parse the document using the Nokogiri library, grab the sell orders I’m interested in via the css lookup method, and average the lowest five prices.

Next I use some basic functional programming to implement the composite model:

class CompositePricingModel def initialize ( < span class = "hiddenGrammarError" pre= "" > models = [ ] ) @models </ span > = models end def buy_price ( id ) return 0 if @models . empty ? @models . collect { | m | m. buy_price ( id ) } . min end def sell_price ( id ) return 0 if @models . empty ? @models . collect { | m | m. sell_price ( id ) } . max end end class CompositePricingModel def initialize(<span class="hiddenGrammarError" pre="">models = []) @models</span> = models end def buy_price(id) return 0 if @models.empty? @models.collect {|m| m.buy_price(id)}.min end def sell_price(id) return 0 if @models.empty? @models.collect {|m| m.sell_price(id)}.max end end

And finally I use Moneta to persist the values so that I can be a good EVE-Central API citizen. The code defaults to updating prices that are more than a day old:

require 'moneta' class PersistantPricingModel def initialize ( model, baseDir = './data' , expiration = 60 * 60 * 24 ) @model = model @buy_prices = Moneta. new ( : PStore , :file => baseDir + "/buy.pstore" , :expires => true ) @sell_prices = Moneta. new ( : PStore , :file => baseDir + "/sell.pstore" , :expires => true ) @expiration = expiration end def buy_price ( id ) return @buy_prices [ id ] if @buy_prices . key ? ( id ) price = @model . buy_price ( id ) @buy_prices . store ( id, price, :expires => @expiration ) price end def sell_price ( id ) return @sell_prices [ id ] if @sell_prices . key ? ( id ) price = @model . sell_price ( id ) @sell_prices . store ( id, price, :expires => @expiration ) price end end require 'moneta' class PersistantPricingModel def initialize(model, baseDir = './data', expiration = 60 * 60 * 24) @model = model @buy_prices = Moneta.new(:PStore, :file => baseDir + "/buy.pstore", :expires => true) @sell_prices = Moneta.new(:PStore, :file => baseDir + "/sell.pstore", :expires => true) @expiration = expiration end def buy_price(id) return @buy_prices[id] if @buy_prices.key?(id) price = @model.buy_price(id) @buy_prices.store(id, price, :expires => @expiration) price end def sell_price(id) return @sell_prices[id] if @sell_prices.key?(id) price = @model.sell_price(id) @sell_prices.store(id, price, :expires => @expiration) price end end

Our Results

Et voila, we have relevant pricing data for the markets we care about. At the time of writing, the output from our test script is:

$ ruby pricing.rb Getting market data: 34 ... Done. Getting market data: 34 ... Done. Tritanium buy price: 4.63 Tritanium sell price: 4.774 $ ruby pricing.rb Getting market data: 34 ... Done. Getting market data: 34 ... Done. Tritanium buy price: 4.63 Tritanium sell price: 4.774

The super-observant will notice that the actual price we can expect to buy tritanium at in our markets is almost 8% higher than the marketstat overall average and 12% lower than the marketstat sell average. This is actually a best case scenario as we’re using the most commonly bought and sold item in EVE – the margins for most items will be much worse.

Next time around, we’ll use the static data from Part 1 along with the dynamic pricing data we’ve found here to see what kind of profit we can expect to make on a specific item.

Hacking EVE