Collecting geocoded tweets with R and Java

Number of tweets in different languages posted
around Germany

There are many thing one can do with tweets (sentiment analysis, maps, ...). This entry shows you how you can access the publicly available API using Java and how to analyse the data using R. For my purpose I am collecting geocoded  tweets around Germany. To collect the tweets I wrote a little java program (see below) which uses the twitter library twitter4J. This can be run on a machine in the background. Here is the java program which collects the tweets and stores them to disk. Except about 1 GB per week.

Collecting the geocoded tweets

Using this script I collected approx 1.3 Mio tweets in a weeks. The tweets are stored one line per tweet and one file per hour e.g. 2013-05-21T19_51_03.json. The content of the file would look like:

{"created_at":"Tue May 21 17:51:09 +0000 2013","id":336901993555709952,"id_str":"336901993555709952","text":"@OmegaBlue69 ... {"created_at":"Tue May 21 17:51:10 +0000 2013","id":336901996680450048,"id_str":"336901996680450048","text":"Sweet1 ....

Handling the json-file

The first task extracts the relevant information from these files. The following script reads the json files line by line and writes the coordinates, languages and for each tweet to a text-file e.g. "2013-05-21T19_51_03.coords.txt" using rjson.

Putting it all together

The next script picks up all text files with coordinate information, merges infrequent levels and does the color-coding. Finally it creates a simple barplot and stores the data in a data.frame all.data and the colors in a vector cols
In another blog I describe how this data can be used to create a zoomable map.

Creating a zoomable map of tweets with R

Languages tweeted around Germany: red, blue, green, 

yellow, grey are for German, French, English, Dutch and 

other  respectively. See here for a zoomable version.

Motivated by the project twitter languages of New York I wanted to do map of tweets too.
For a different purpose (sentiment analysis) I am collecting tweets around Germany anyway. In another blog entry I describe how I collected the data.

With the package openmap, it is easy to create a map of the languages tweeted. Such a map is shown on the left. However, in post-google-map times if the user sees a map, he just starts to spin his mouse wheel to zoom into the map. Lets see how one can create such a map.

Creating the map

I assume that the data is in the following format, I need coordinates and colors. See the blog how to extract the data from the json-files and do the color coding.

>head(all.data)#The lat/long data approx. 300k
       long      lat  lang
1  4.901844 52.37762    en
2  6.255914 52.51602    nl
3 13.736128 51.04736    en

....

> head(cols)#The colors (appox. 300k)
[1] "#FF000080" "#00FF0080" "#0000FF80" ...

Zoomable maps are created with socalled tiles. You start with one resolution (zoomlevel), say zoomlevel 4. You create one tile. In the next zoom level this tile is split in 4 tiles and so on. These tiles are have to be placed in a certain directory structure. To create a zoomable map you simply have to create this directory structure with those tiles. There are plenty of javascript libraries with take all those tiles and create a map. I use one called leaflet (see below). Usually these libraries require 256x256 png images.

So lets see how this works. The region of the above figure and zoom-level 4 corresponds to the path 4/8/5.png. For the next zoom level 5 one has to create the 4 files 5/16/10.png, 5/16/11.png, 5/17/10.png and 5/17/12.png. The directory corresponds to x-axis (longitude) and the file name to the y-axis (latitude), the figure below shows how the split is done.

The tile on the left is split into 4 tiles shown on the right.

In the next zoom level we would have to render 16 images starting with 6\32\20.png.

The script below starts with the tile 4\8\5.png and recursively creates the images up to the desired zoom levels. To get the bounding boxes in terms of longitude and latitude the function tile2boundingBox is used. The maps are obtained using the map function of the OpenStreetMap package. The points have to be transformed  using projectMercator(lat = sd$lat, lon = sd$lon)and can then be drawn using e.g. the points function.

Once you have created the tiles, its just a few lines of javascript to have the maps ready. See below for the index.html file using leaftlet library.

The main file creating the tile

The java script part