Kibana GeoIP example: How to index geographical location of IP addresses into Elasticsearch

Updated: Sep 14, 2020



The relation between your IP address and geolocation is very simple. There are numerous websites available as of today like Maxmind, IP2Location, IPstack , Software77 etc where you can track the geolocation of an IP address. What's the benefit? It's very simple, it gives you another dimension to analyze your data.


Let's say my data predicts that most of the users traffic is coming from 96.67.149.166. It doesn't make complete sense until I say most of the traffic is coming from New Jersey.

When I say geolocation it includes multiple attributes like city, state, country, continent, region, currency, country flag, country language, latitude, longitude etc. Most of the websites which provide geolocation are paid sites.


But there are few like IPstack which provides you free access token to make calls to their rest API's. Still there are limitations like how many rest API calls you can make per day and also how many types of attributes you can pull. Suppose I want to showcase specific city in the report and API provides limited access to country and continent only, then obviously that data is useless for me.


Now the best part is Elastic stack provides you free plugin called "GeoIP" which grants you access to lookup millions of IP addresses. You would be thinking from where it gets the location details? The answer is Maxmind which I referred earlier. GeoIP plugin internally does a lookup from stored copy of Maxmind database which keeps on updating and creates number of extra fields with geo coordinates (longitude & latitude). These geo coordinates can be used to plot maps in Kibana.



ELK Stack Installation


I am installing ELK stack on Mac OS, for installation on Linux machine refer this. ELK installation is very easy on Mac with Homebrew. It's hardly few minutes task if done properly.


1. Homebrew Installation


Run this command on your terminal. If you have already installed Homebrew move to the next step, or if this command doesn't work - copy it from here.


$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"


2. Java Installation


Check if java is installed on your machine.


$ java -version

java version "9.0.1"


If java is not installed, run following steps to install java.


$ brew tap caskroom/cask

$ brew cask install java

$ brew cask info java


3. Elasticsearch Installation


$ brew tap elastic/tap

$ brew install elastic/tap/elasticsearch-full

$ elasticsearch


If you see all INFO without any error, that means installation went fine. Let this run, don't kill the process.

Now, simply open localhost:9200 in your local browser. You will see elasticsearch version.


[TIP] You might face permission issue if you are not logged in with root user. To enable root user on Mac you can follow this. It's due to security reasons that root user is disabled by default on Mac.


However another solution is to change folder permission itself. Run these commands if you want to change folder permissions,


$ sudo chown -R $(whoami) /usr/local/include /usr/local/lib/pkgconfig

$ chmod u+w /usr/local/include /usr/local/lib/pkgconfig


Install xcode if it's missing,

$ xcode-select --install



4. Kibana Installation


$ brew install elastic/tap/kibana-full

$ kibana


Let this process run, don't kill. Now, open localhost:5601 in your local browser to check if kibana is running properly,


5. Logstash Installation


$ brew install elastic/tap/logstash-full



Configuring Logstash for GeoIP


Let's begin with few sample IP addresses as listed below. I generated this sample data from browserling.com so please ignore if there is some known ip address in this list. Honestly speaking even I don't know where these IP addresses will point to when we generate the maps.


Sample Data


1. Copy paste these records into a flat file with "ipaddress" header (sampleip.csv).


ipaddress

0.42.56.104

82.67.74.30

55.159.212.43

108.218.89.226

189.65.42.171

62.218.183.66

210.116.94.157

80.243.180.223

169.44.232.173

232.117.72.103

242.14.158.127