The First Insights: Tweets in QGIS

curl -k -d @locations.txt https://stream.twitter.com/1/statuses/filter.json -uuser:password > tweets.json

Tweets in QGIS- Prerequisites

access token

access token secret

consumer key

consumer key secret

Gathering Tweets

import tweepy access_token = "put your token here" access_token_secret = "put your token secret here" consumer_key = "put your key here" consumer_secret = "put your key secret here" key = tweepy.OAuthHandler(consumer_key, consumer_secret) key.set_access_token(access_token, access_token_secret) # here come the tweepy part: class stream2lib(tweepy.StreamListener): output = {} def __init__(self, api=None): api = tweepy.API(key) self.api = api or API() self.n = 0 #we will start with zero tweets self.m = 10 #let's stop with 10 tweets def on_status(self, status): #we will parse the interesting information into a nice format self.output[status.id] = { 'tweet':status.text.encode('utf8'), #text could have non utf8 characters, so change this! 'user':status.user.screen_name.encode('utf8'), #user name should be utf8 conform as well 'geo':status.geo, #this is the point location of the device 'localization':status.user.location, #user location as part of the user profile (normally set fixed per user) 'time_zone':status.user.time_zone, #quite 'time':status.timestamp_ms} #the timestamp given in ms since 01.01.1970 #we will only care about tweets with geo if self.output[status.id]['geo']!=None: self.n = self.n+1 #we found a geotweet. but that's always true when calling the command with "locations=[-x,-y,x,y]" as below if self.n < self.m: return True else: return False stream = tweepy.streaming.Stream(key, stream2lib()) #initiate the stream stream.filter(locations=[-180,-90,180,90]) #filter the stream for tweets in this "box" tweetdic = stream2lib().output #copy it in a variable print tweetdic #just to be sure ;-)

Adding Tweets as Points to QGIS

vl = QgsVectorLayer("Point", "temporary_twitter_results", "memory") pr = vl.dataProvider() # changes are only possible when editing the layer vl.startEditing()

pr.addAttributes([QgsField("user_name", QVariant.String),QgsField("localization", QVariant.String), QgsField("tweet", QVariant.String), QgsField("time", QVariant.String)])

for tweet in tweetdic: if tweetdic[tweet]['geo'] != None: fet = QgsFeature() #it's a new feature fet.setGeometry(QgsGeometry.fromPoint(QgsPoint(tweetdic[tweet]['geo']['coordinates'][1],tweetdic[tweet]['geo']['coordinates'][0] ))) #use the coordinates for point location tweettime = datetime.datetime.utcfromtimestamp(float(tweetdic[tweet]['time'][:-3] + "." + tweetdic[tweet]['time'][11:13])).strftime('%Y-%m-%d %H:%M:%S:%f') #parse the time to fit YYYY-MM-DD HH:MM:SS:MS fet.setAttributes([tweetdic[tweet]['user'],tweetdic[tweet]['localization'],tweetdic[tweet]['tweet'],tweettime]) #set attributes of current tweet at current location pr.addFeatures([fet]) #and add the feature to the layer.

# commit to stop editing the layer vl.commitChanges() # update layer's extent when new features have been added # because change of extent in provider is not propagated to the layer vl.updateExtents() QgsMapLayerRegistry.instance().addMapLayer(vl)

A Warning

Anita showed some nice examples of tweets in QGIS in 2012. Since then it seemed to be quiet about the twitter-content in QGIS. Yet tweets can be an interesting source of information. Sometimes they can tell you something about the spatiotemporal dimensions regarding a keyword, the digital heartbeat of a defined region and many more. Yet we need to be careful with the data as it is completely biased. But how to get this data stream into QGIS?As in 2011/2012 the twitter API was easy to fetch with a simple line of code and you were able to stream data. Anita showed this in a nice little way:With this line you collected all tweets in a file for a defined region (as stated in locations.txt) and you can then simply scan the file in QGIS for tweets with a defined location, add those lines as features in a point shapefile and of you go. But soon after, Twitter changed its API policy and switched to a more advanced authentication system . The solutions of Anita wasn’t working any more. Still nice examples from tweets in a geographic context appeared online. Most extreme map is probably this here:And also the recent versions of ArcGIS added the possibility to add tweets to ArcGIS map products . So it’s time to add it to QGIS as well.At the moment the most reliable way to get access to the Twitter API from a python environment seems to be tweepy. The installation is quite easy under Ubuntu. But for an installation in Windows you need some information about environment parameters and only the user knows whether he has the OS4geow shell installed. Long story short: You need to install the tweepy module by hand prior reading streams from twitter. Furthermore the twitter API needs some tokens and keys: Make sure to have a twitter account and create an application so you’ll get your codes:Ones you have both let’s dive into the QGIS python console.The first part of gathering tweets is to call tweepy with a defined location:As you can see: we gather tweets and we filter the stream. Unfortunately the filtered stream can’t be stored in a variable so we added the output to the whole listener and we need to filter this for tweets with a coordinate afterwards. With the lines above we have a dictionary of tweets….As we have the tweets in a variable we can simply iterate over the dictionary and fill a virtual layer in QGIS with the point information. First we create this virtual layer:At the moment the layer doesn’t contains any attributes so let’s add them as well:And the next lines will iterate over the dictionary, uses the coordinates as point locations and tweet attributes as attributes for each feature if a tweet has a coordinate:And as we have finished the iteration let’s stop the editing and publish the layer to the current QGIS project:In the end you can collect as many tweets you would like. But be warned this solution might freeze your QGIS application for a few moments until a new tweet was found.Together with Anita’s TimeManager plugin you can now create nice videos:The whole script can be downloaded: tester2 . Furthermore I created a QGIS plugin called twitter2qgis or geotweet:The plugin is also under development via github . so please report any issues or contribute with knowledge/programming…You can collect a large number of tweets and also mine them. But be aware: The user who uses twitter is quite specific. The user who also allows Twitter to use current location is even more specific. If you do any analysis with those tweets: keep in mind these aspects and also think about reading these articles: