In the previous post, I had discussed about creating flight tracking with python. I was quite happy for that, because I can do something that I wanted to do long time ago, which I didn't have any clue how to do it before, till I'm learning python and found that it was possible and.... Yes, it is possible and I can make it!





If you read my post about creating a simple live flight tracking with python , it already discussed how to generate a figure that shows aircraft's position on a map. It was a simple figure with Open Street Map (OSM) basemap and red dots that represents position of aircrafts. The position will be updated every second by sending a request to ADS-B exchange data API . If you have not read the tutorial, I suggest you to read it, because it explains some details about ADS-B exchange service especially how to request flight data.







In this post, we will do the same, to create a live flight tracker. But we will make it more beautiful, with more advance approach using Pandas and Bokeh . So at the end of this tutorial you can make an almost realtime flight tracking application like figure 1 below.

Figure 1. Realtime flight tracking application with Pandas and Bokeh

Tutorial Outline

Getting data and convert to pandas data frame

Load basemap into bokeh

Plot aircraft position on the map

Add some intreactivity

Build a realtime flight tracking application

For this tutorial I'm using Jupyter Notebook with Python 3.5.6, Pandas 0.23.4, Bokeh 0.13 and some other libraries like numpy, json, ssl and urllib. We need to import all the required libraries, but we will do it one by one as we need it.

Getting Flight Data and Convert to Pandas Data frame

Let's start this tutorial with getting flight data from ADS-B exchange data API. In this case I want to get flight data around the United States (US). To do that, let's pick a geographic coordinate around the center of US (Latitude=37.51,Longitude=-94.22) and request the data within a range that covers all the US (3000 km). The query for the request will be:

For this step we need to import pandas library, urllib request, json and ssl. We will send the query, get the response, load as json and convert it to pandas data frame. The query response returns more than 50 columns of information/data. We won't use all the data, but just some information such as latitude(Lat), longitude(Long), Ground Altitude (GAlt), Call, From, To and Operator (Op). Therefore when converting the json to pandas data frame we just include those specified columns. At the end, using data frame head method we will see top five data as in figure 2.

#IMPORT LIBRARY import pandas as pd import urllib.request import json import ssl # HEADER ssl . _create_default_https_context = ssl . _create_unverified_context opener = urllib . request . build_opener() opener . addheaders = [( 'User-agent' , 'Mozilla/5.0' )] #SEND REQUEST, READ RESPONSE AND LOAD AS JSON fp = opener . open( 'http://public-api.adsbexchange.com/VirtualRadar/AircraftList.json?lat=37.51&lng=-94.22&fDstL=0&fDstU=3000' ) mybyte = fp . read() mystr = mybyte . decode( "utf8" ) js_str = json . loads(mystr) fp . close() #CONVERT TO PANDAS DATAFRAME flight_df = pd . DataFrame(js_str[ 'acList' ],columns = [ 'Lat' , 'Long' , 'GAlt' , 'Call' , 'From' , 'To' , 'Op' ]) flight_df = flight_df . fillna( 'No Data' ) #replace NAN with No Data flight_df . head()

Figure 2. Flight data response

Adding Basemap to Bokeh

In this step we will setup a bokeh figure with a basemap. For this purpose we need to import some bokeh plotting libraries and a pre-configured tile source. For this case I chose STAMEN_TONER tile source. You can see other tile sources at the Bokeh tile providers help page





I just want to view the US country when opening the page. For that, we need to specify a range coordinate (bounding box) that consist of minimum and maximum coordinate in web Mercator coordinate system (EPSG: 3857). Then setup the figure and define some properties like x_range, y_range, both x and y axis type, plot height and sizing mode. Lastly by using show class, it will open a temporary html page that showing a basemap like figure 3. Using the tools on the right side of the figure you can interact with the map to pan, zoom in, zoom out, and reset the view.

from bokeh.plotting import figure, show from bokeh.tile_providers import STAMEN_TONER # DEFINE COORDINATE RANGE IN WEB MERCATOR COORDINATE SYSTEM (EPSG:3857) x_range,y_range = ([ -15187814 , -6458032 ], [ 2505715 , 6567666 ]) #SETUP FIGURE p = figure(x_range = x_range,y_range = y_range,x_axis_type = 'mercator' ,y_axis_type = 'mercator' , sizing_mode = 'scale_width' ,plot_height =300 ) p . add_tile(STAMEN_TONER) #SHOW FIGURE show(p)

Figure 3. Bokeh figure with basemap

Plotting Aircraft Position on The Map

Now let's combine the skill from the steps above to plot aircraft position on the map using latitude and longitude. But hold on! Latitude and longitude is in geographic coordinate system (EPSG:4326). On the other hand, the map is in web mercator coordinate system. The coordinate system is not match to each other, so we can not plot the aircraft position directly on the map. The aircraft position must be converted to web mercator coordinate system. Therefore we need to make a function to convert the latitude and longitude to mercator y and x. Later in the code I called this function as wgs84_to_web_mercator.





To plot point in bokeh, firstly we have to convert the pandas data frame into bokeh column data source. After that, we add the point using bokeh figure circle method. At this step we specify some properties such as column name for x and y axis, column data source, point color, size, etc.



The code below is complete code for this step which is the combination from previous steps plus additional lines to convert geographic coordinate to web mercator and adding point to the map. If the code is executed, it should open a temporary html page with showing aircraft's position in blue dots over the basemap as in figure 4.





1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 #IMPORT LIBRARY import pandas as pd import urllib.request import json import ssl from bokeh.plotting import figure, show, ColumnDataSource from bokeh.tile_providers import STAMEN_TONER import numpy as np # HEADER ssl . _create_default_https_context = ssl . _create_unverified_context opener = urllib . request . build_opener() opener . addheaders = [( 'User-agent' , 'Mozilla/5.0' )] #SEND REQUEST, READ RESPONSE AND LOAD AS JSON fp = opener . open( 'http://public-api.adsbexchange.com/VirtualRadar/AircraftList.json?lat=37.51&lng=-94.22&fDstL=0&fDstU=3000' ) mybyte = fp . read() mystr = mybyte . decode( "utf8" ) js_str = json . loads(mystr) fp . close() #CONVERT TO PANDAS DATAFRAME flight_df = pd . DataFrame(js_str[ 'acList' ],columns = [ 'Lat' , 'Long' , 'GAlt' , 'Call' , 'From' , 'To' , 'Op' ]) flight_df = flight_df . fillna( 'No Data' ) #replace NAN with No Data # DEFINE COORDINATE RANGE IN WEB MERCATOR COORDINATE SYSTEM (EPSG:3857) x_range,y_range = ([ -15187814 , -6458032 ], [ 2505715 , 6567666 ]) #FUNCTION TO CONVERT GCS WGS84 TO WEB MERCATOR def wgs84_to_web_mercator (df, lon = "Long" , lat = "Lat" ): k = 6378137 df[ "x" ] = df[lon] * (k * np . pi /180.0 ) df[ "y" ] = np . log(np . tan(( 90 + df[lat]) * np . pi /360.0 )) * k return df wgs84_to_web_mercator(flight_df) #BOKEH COLUMN DATA SOURCE flight_source = ColumnDataSource(flight_df) #SETUP FIGURE p = figure(x_range = x_range,y_range = y_range,x_axis_type = 'mercator' ,y_axis_type = 'mercator' , sizing_mode = 'scale_width' ,plot_height =300 ) p . add_tile(STAMEN_TONER) p . circle( 'x' , 'y' ,source = flight_source,fill_color = "blue" ,size =10 ,fill_alpha =0.8 ,line_width =0.5 ) #SHOW FIGURE show(p)

Figure 4. Aircraft's position on the map

Adding Some Styles and Interactivity

All the aircraft's position have been plotted on the map, but we just see a bunch of blue dots without further information what it is exactly. In this part we will add some styles like labels, classify the points based on ground altitude and add hover tool that shows flight information in a pop up window for a selected aircraft.



First step, import all required libraries that are HoverTool, LinearColorMapper and LabelSet from bokeh models. For color palette we can import a built-in color palette from bokeh. I used the RdYlBu color palette. This color palette will be classified linearly into 11 classes, so the palette name becomes RdYlBu11. There many color palettes available in bokeh, you can check it at



Add the following code into the previous one at importing libraries section (put it at line 9).



from bokeh.models import HoverTool,LinearColorMapper,LabelSet from bokeh.palettes import RdYlBu11 as palette

Then we are setting the color palette, hover tool and label as in the code below. Place the code in line 39 after bokeh column data source.



# SET COLOR PALETTE color_mapper = LinearColorMapper(palette = palette) #SET HOVER my_hover = HoverTool() my_hover . tooltips = [( 'Op' , '@Op' ),( 'From' , '@From' ),( 'To' , '@To' ),( 'GAlt' , '@GAlt' )] #SET LABEL labels = LabelSet(x = 'x' , y = 'y' , text = 'Call' , level = 'glyph' , x_offset =5 , y_offset =5 , source = flight_source, render_mode = 'canvas' ,background_fill_color = 'skyblue' ,text_font_size = "8pt" )

At the end, add the hover tool and labels with the following code. Put this code at line 45 before showing the figure.



#ADD HOVER TOOL AND LABELS p . add_tools(my_hover) p . add_layout(labels)

After finishing the code by adding color palette, hover and labelling, execute the code. You should get a map like figure 5 below.



Figure 5. Aircraft's position with label and hover tool It works, but too many labels are overlapping to each other. It's not really nice and too many labels will slow down the server when rendering the point on the map. It would be good for static map but not preferred for dynamic map which is updated in a short interval.

Build Real time Flight Tracking Application

The last section of this tutorial is to build an almost real time flight tracking application. We will create an application which is running o web page that will be refreshed each second, so that we can track aircraft's position time by time.



All the basic techniques already explained in the previous step. So in this section we just extend our knowledge to compose the application that works with bokeh server. In general, the code frame will look like this:



#IMPORT LIBRARY import pandas as pd ....... #HEADER ........ # COORDINATE CONVERSION FUNCTION ................... #FLIGHT TRACKING FUNCTION def flight_track (doc): .......... # UPDATING FLIGHT DATA def update (): #SEND REQUEST, READ RESPONSE AND LOAD AS JSON ........... #CONVERT TO PANDAS DATAFRAME ..................... # CONVERT TO BOKEH DATASOURCE AND STREAMING #CALLBACK UPDATE IN AN INTERVAL ............ #PLOT THE AIRCRAFT POSITION WITH BOKEH .............. # SERVER CODE ..................

Using all the codes that we discussed. Try to complete the code frame above. If you get stuck of course you can cheat the complete code at the end of this tutorial.



Before seeing the full code for the tracking application, let's me explain some parts of the code that not explained before.

In the import section, we add some bokeh libraries such as Server, Application and FunctionHandler (line 9-12). These libraries will be used for the server.

In line 30, we initiate a bokeh column data source in a dict. It will be used later for streaming the updated flight data response.

In streaming the updated data at line 50, we are using the length of data frame's row as the number of roll back property, to store only the updated data in the column data source. Otherwise, the data will be appended with the old data, causing the number of data will grow in each updating step.

At line 53 the updated function is calling for every 1000 millisecond (1 second).

The last three line is the server related codes, where we define application function handler, port number to be used for the application and start the server when the code is running. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 #IMPORT LIBRARY import urllib.request import json import ssl import pandas as pd from bokeh.plotting import figure,ColumnDataSource from bokeh.models import HoverTool,WMTSTileSource,LinearColorMapper,LabelSet from bokeh.palettes import RdYlBu11 as palette from bokeh.server.server import Server from bokeh.application import Application from bokeh.tile_providers import STAMEN_TONER from bokeh.application.handlers.function import FunctionHandler import numpy as np #HEADER ssl . _create_default_https_context = ssl . _create_unverified_context opener = urllib . request . build_opener() opener . addheaders = [( 'User-agent' , 'Mozilla/5.0' )] # COORDINATE CONVERSION FUNCTION def wgs84_to_web_mercator (df, lon = "Long" , lat = "Lat" ): k = 6378137 df[ "x" ] = df[lon] * (k * np . pi /180.0 ) df[ "y" ] = np . log(np . tan(( 90 + df[lat]) * np . pi /360.0 )) * k return df #FLIGHT TRACKING FUNCTION def flight_track (doc): # initiate bokeh column data source flight_source = ColumnDataSource({ 'Long' :[], 'Lat' :[], 'Op' :[], 'From' :[], 'To' :[], 'GAlt' :[], 'x' :[], 'y' :[], 'Call' :[]}) # UPDATING FLIGHT DATA def update (): #SEND REQUEST, READ RESPONSE AND LOAD AS JSON fp = opener . open( 'http://public-api.adsbexchange.com/VirtualRadar/AircraftList.json?lat=40.639722&lng=-73.778889&fDstL=0&fDstU=400' ) mybyte = fp . read() mystr = mybyte . decode( "utf8" ) js_str = json . loads(mystr) fp . close() #CONVERT TO PANDAS DATAFRAME flight_data = js_str[ 'acList' ] flight_df = pd . DataFrame(flight_data,columns = [ 'Long' , 'Lat' , 'Op' , 'From' , 'To' , 'GAlt' , 'Call' ]) wgs84_to_web_mercator(flight_df) flight_df = flight_df . fillna( 'No Data' ) # CONVERT TO BOKEH DATASOURCE AND STREAMING n_roll = len (flight_df . index) flight_source . stream(flight_df . to_dict(orient = 'list' ),n_roll) #CALLBACK UPATE IN AN INTERVAL doc . add_periodic_callback(update, 1000 ) #PLOT THE AIRCRAFT POSITION WITH BOKEH x_range,y_range = ([ -8570951 , -7864902 ], [ 4799227 , 5130103 ]) #bounding box for east US p = figure(x_range = x_range,y_range = y_range,x_axis_type = 'mercator' ,y_axis_type = 'mercator' ,sizing_mode = 'scale_width' ,plot_height =300 ) my_hover = HoverTool() color_mapper = LinearColorMapper(palette = palette) my_hover . tooltips = [( 'Op' , '@Op' ),( 'From' , '@From' ),( 'To' , '@To' ),( 'GAlt' , '@GAlt' )] labels = LabelSet(x = 'x' , y = 'y' , text = 'Call' , level = 'glyph' , x_offset =5 , y_offset =5 , source = flight_source, render_mode = 'canvas' ,background_fill_color = 'skyblue' ,text_font_size = "8pt" ) p . add_tile(STAMEN_TONER) p . circle( 'x' , 'y' ,source = flight_source,fill_color = { 'field' : 'GAlt' , 'transform' : color_mapper},hover_color = 'yellow' ,size =10 ,fill_alpha =0.8 ,line_width =0.5 ) p . add_tools(my_hover) p . add_layout(labels) doc . title = 'REAL TIME FLIGHT TRACKING WITH PANDAS AND BOKEH' doc . add_root(p) # SERVER CODE apps = { '/' : Application(FunctionHandler(flight_track))} server = Server(apps, port =1002 ) #define an unused port server . start()

When the code is executed, it will run the Tornado web server. If you see no error for a while, it means everything is going well. Finally open a web browser and type localhost:portnumber (eg. localhost:1002). You should see the flight tracking application running in the web browser as in figure below.



Sometimes when running the flight tracking application in a web page, you will get a page internal error. If this happen, open Anaconda command prompt where the jupyter notebook is running, and see for the error. If nothing goes wrong, try to restart the kernel and run the code again. It happened a lot for me during I wrote this application. Trust me!



Finally, I'd like to say thank you very much for reading this tutorial. I' m not expert in python programming, so please feel free to improve the code. There are many opportunities and possibilities we could do in python. I suggest to explore more and take some All the aircraft's position have been plotted on the map, but we just see a bunch of blue dots without further information what it is exactly. In this part we will add some styles like labels, classify the points based on ground altitude and add hover tool that shows flight information in a pop up window for a selected aircraft.First step, import all required libraries that are HoverTool, LinearColorMapper and LabelSet from bokeh models. For color palette we can import a built-in color palette from bokeh. I used the RdYlBu color palette. This color palette will be classified linearly into 11 classes, so the palette name becomes RdYlBu11. There many color palettes available in bokeh, you can check it at bokeh reference page Add the following code into the previous one at importing libraries section (put it at line 9).Then we are setting the color palette, hover tool and label as in the code below. Place the code in line 39 after bokeh column data source.At the end, add the hover tool and labels with the following code. Put this code at line 45 before showing the figure.After finishing the code by adding color palette, hover and labelling, execute the code. You should get a map like figure 5 below.It works, but too many labels are overlapping to each other. It's not really nice and too many labels will slow down the server when rendering the point on the map. It would be good for static map but not preferred for dynamic map which is updated in a short interval.The last section of this tutorial is to build an almost real time flight tracking application. We will create an application which is running o web page that will be refreshed each second, so that we can track aircraft's position time by time.All the basic techniques already explained in the previous step. So in this section we just extend our knowledge to compose the application that works with bokeh server. In general, the code frame will look like this:Using all the codes that we discussed. Try to complete the code frame above. If you get stuck of course you can cheat the complete code at the end of this tutorial.Before seeing the full code for the tracking application, let's me explain some parts of the code that not explained before.When the code is executed, it will run the Tornado web server. If you see no error for a while, it means everything is going well. Finally open a web browser and type localhost:portnumber (eg. localhost:1002). You should see the flight tracking application running in the web browser as in figure below.Sometimes when running the flight tracking application in a web page, you will get a page internal error. If this happen, open Anaconda command prompt where the jupyter notebook is running, and see for the error. If nothing goes wrong, try to restart the kernel and run the code again. It happened a lot for me during I wrote this application. Trust me!Finally, I'd like to say thank you very much for reading this tutorial. I' m not expert in python programming, so please feel free to improve the code. There are many opportunities and possibilities we could do in python. I suggest to explore more and take some Pandas courses or Bokeh courses and play with your imagination.

This tutorial consist of several parts. Getting through this tutorial, we will learn learn how to:Before proceeding to the next step, make sure you can get the data, because we will use it in the next part.

 Python tracking Tutorial