I wrote geoloc to help me query my access logs, fast.

geoloc is a command line tool for bulk geolocation queries written in C++. Once its binary database has been built, geoloc performs geolocation queries offline.

Examples:

Bulk lookup from apache access.log:

$ cat access.log | awk '{print $1}' | geoloc -f - | column -t 10.172.47.117 AU 02 Sydney -33.8001 151.3123 AS1610581 BIGCableCo 10.36.87.70 AU 07 Melbourne -37.8266 144.7834 AS1370775 Micronode+PTY+LTD 10.88.81.165 US CA San+Francisco 37.6777 -122.2221 AS49335653 Big+Flare,+Inc

Query some IPs:

$ geoloc -q 8.8.8.8 192.30.252.131 --headers | column -t ip country region city latitude longitude as_num as_text 8.8.8.8 US CA Mountain+View 37.3860 -122.0838 AS15169 Google+Inc. 192.30.252.131 US CA San+Francisco 37.7697 -122.3933 AS36459 GitHub,+Inc.

geoloc is designed to run fast and load fast:

$ wc -l /tmp/ip_list 1000000 /tmp/ip_list $ time geoloc -f /tmp/ip_list > /tmp/res1 real 0m6.131s user 0m5.662s sys 0m0.369s $ time geoloc -q 8.8.8.8 192.30.252.131 > /tmp/res2 real 0m0.010s user 0m0.002s sys 0m0.005s

Installation

The program is designed as a portable application, to run out of ~/bin , with the database stored in ~/var/db/geoloc/geodata.bin .

To install:

$ git clone https://github.com/loadzero/geoloc.git && cd geoloc $ ./configure $ make $ make install

The configure script will check for these dependencies:

iconv

unzip

wget

make

c++

During installation, data will be downloaded from MaxMind to create the database.

An update script will be installed into ~/bin/_geoloc_update.sh . Run this script when you would like to update your geolocation database. MaxMind updates their source data once a month.

I have tested on OSX 10.9.5 and Ubuntu 14.04. Other unices are likely to work with minimal or no changes. It is unlikely to work on windows, due to the use of mmap.

Design and Implementation

I plan to do a longer write-up on the design and implementation of the tool, to share some C++ tips and tricks.

The short version is that the code operates in two phases, packing and query. The packing phase is all about converting the data into a machine optimal format, namely relocatable sorted vectors. The query phase simply mmaps that data, and performs a std::upper_bound binary search on it to find the IPs.

There is an outline of the code, roughly in topological order here, that contains a summary of each module.

Attribution

This software includes GeoLite data created by MaxMind available from http://www.maxmind.com