Tagged: government

Interactive timeline of the PRISM scandal

Share Button


Purpose of this visualization

This is an interactive timeline of events about the Prism scandal, chronicled by selected media in online news articles, giving a summarized view of events as they unfolded. It’s intended as a parody of a NSA software to track people and analyze their metadata. It consists of these parts:

  • the chronological order of articles, visualized as a timeline,
  • a network of people, places and organizations that appear in the articles,
  • geographic information that the articles refer to, and
  • a bar graph showing wordcounts of interesting words, associated with the main theme.

In the bottom part there is a timeline displaying the published articles in chronological order. Articles are accessible by clicking on a title and then using the Go to article link in the popup bubble.

In the center background there is a rotating globe, displaying major cities referred by articles. Labels next to cities contain titles of all articles visible in timeline view that refer to specific city.

Center foreground contains a network showing interconnectedness of various entities recognized in the article text. Entities appearing in the same paragraph are connected. The network is additive, which means that the more frequently entities appear in same paragraphs, the stronger is the bond between them.

In the right corner, there is a small bar graph containing frequencies of selected words, giving an idea of Snowden’s options at the time. It’s just  a word count of shown words in visible articles, not a semantic analysis.

As the timeline is moved, new articles appear, and network is updated with new data, giving a quick overview of who and how frequently was involved in discussion, and who was related to them.

Hovering the mouse over a network node shows only the portion of the network that the node is directly connected to, making this useful for detailed exploration of relationships between those entities.

Launch the PRISM Scandal visualization!  Use Chrome if possible. It won’t work in IE.

PRISM Scandal Visualization Window
PRISM Scandal Visualization Window

Interacting with the visualization

Use the mouse to drag timeline left or right, or rotate the middle wheel for the same effect. For quicker navigation use the Quick jump menu. The subnetworks will load and unload automatically, and the whole network will try to stabilize so that it accurately reflects frequencies of terms and bond stregths between them. If it doesn’t stabilize well, click the Reorder network button or double-click in the middle of the timeline.

It’s possible to zoom the network in and out to get a better idea of shown names and connections. To do that, position the mouse pointer over the network area and drag or zoom with the wheel. Node size corresponds to term frequency in visible articles, while the bond thickness corresponds to its weight, that is to say frequency of said bond.

Click the article titles in the timeline to display more information and links. To change the publisher, click the Publisher menu and select a desired one. This will load new set of events into the timeline. To automatically move the timeline, use the Play button.

Data sources

News articles containing keywords Snowden, Prism, NSA, Wikileaks and Julian Assange were scraped from selected media and stored locally for processing. The articles themselves are linked from the timeline. Their content, apart from titles, is not accesible in this visualization for copyright issues. Geographical database with city names and corresponding latitudes and longitudes was obtained as a free download at GeoNames. The media an/or publishing houses were selected to give a balanced set of worldviews. These are, in alphabetical order:

Processing the articles

First a dictionary of all capitalized word sequences and their permutations was constructed by processing all articles in the database. This is essentially a dictionary of all people, states, cities and organizations appearing in the whole database. Then, the title and body text of each article was scanned for these dictionary entries and city names, so that an article abstract was constructed, containing of title, publishing date, a link to article, a wordcount of selected words, a subnetwork of connected entities, and a list of cities along with latitudes and longitudes.

Constructing the network

Subgraphs in an article

The article subnetwork was constructed so that entities in the same sentence (connecting in a paragraph shown in the picture) are connected with a set weight. Nodes not connected with any other nodes are dropped at this point, since their inclusion would lead to a largely unconnected network, which is visually unappealing and cumbersome to navigate.

All web scraping and text processing was done in Java locally, there were around 10,000 articles processed in the latest count. See picture below.



There does not exist a live server database that this visualization would query.  The entity dictionaries are here (names) and here (selected words).

Constructing the visualization

The visualization was constructed entirely in HTML5 and JavaScript. Four major libraries were used:

  • Sigma.js for displaying the networks. The latest version does not contain some key functionality for dynamically and additively loading and unloading of subgraphs into the main graph, so the source code was updated with required methods. Separate article on that topic is upcoming.
  • Three.js for rotating Earth and all geographically-related work.
  • Simile Timeline for the timeline.
  • Flot for the bar graph.

If anyone is interested in Java code for web scraping / networking / constructing timeline input files, drop me a note. My email is on the About page.

Enhanced by Zemanta

Social network diagrams of Slovenian governments between 1991 and 2013

Share Button

Such a young country, but already so messed up. One is inclined to think that all is lost, and one would not be far from the truth. Much ink has already been spilled on sad state of affairs in Slovenia, its fall from grace in European Union, the precipitous decline of living standard of its citizenry and its bleak outlook for the future. Did I mention the rampant corruption of its ruling class and top managers? Best not. This was, after all, supposed to be the next Switzerland.

Blaming the ruling class in mere abstract terms may give one a fleeting satisfaction, but who were the people who led us off the cliff? Someone did govern here, or was at least giving an appearance of governing. Prime ministers are known: Lojze Peterle, Janez Drnovšek, Tone Rop, Andrej Bajuk, Janez Janša, Borut Pahor and currently Alenka Bratušek. These are the main culprits for the downward spiral, of which one can only hope we already passed the first half. Names of their accomplices – the ministers, secretaries, etc. – have a tendency to drift into oblivion, as majority of people preoccupy themselves with the daily grind.

So who were they and how are they connected? Here’s a diagram showing all the government members  from 2001 on. I call it “loyalty diagram”, since it was constructed in a way that it shows who is close to whom, and who is hardly loyal to any alliance. The rationale in short is:

  • Ministers are considered to be very loyal to the prime minister (although I know they are not).
  • Secretaries a lot less, since they are essentially experts and not politicians.
  • Secretaries are less loyal to ministers as are ministers to prime minister, but still a lot, since it’s they who appoint them.
  • Secretaries are loyal to each other, since they are bureaucrats who like their positions and will in theory support each other, although in practice there exist many party rivalries.

Click the link or image below to launch the interactive diagram, which can be searched, panned, and zoomed, and which shows details for every staff member on the government. Red dots are prime ministers, bright blue ministers, dark blue secretaries. Every person is marked with a color of the highest position occupied.

Launch the interactive loyalty diagram

Social network of staff in Slovenian governments 2001-2013
Social network of staff in Slovenian governments 2001-2013

A few lines of commentary:

  • There are a select few of loyal party cadres that every prime minister carries with him, or her, which very rarely, if at all, work with anyone else. These are the dark blue and bright blue dots in close proximity of red dots (prime ministers).
  • Node radius is proportional to how many times the individual sat in a government over the years. For example, Janez Janša was not only prime minister twice, he also served in other capacities, most notably as Minister of Defense in 1994 and was taking on  more and more departmental duties as his government in 2012 slowly disintegrated.
  • There is a big cluster of common cadres between Janez Drnovšek’s and Anton Rop’s governments. It seems that a lot of secretaries are passed on into the next mandate, except in case of shift between left- and right-wing governments, which perform a purge on inauguration.
  • Anton Rop had most secretaries and the biggest government. If anything, the governments are getting slimmer with time.
  • People in the middle of diagram are generally dragged there because of many ties with different prime ministers and ministers, so they are either the most politically promiscuous, or (theoretically) the best experts in their fields, a theory swiftly disproven considering they took on ministerial duties in vastly different departments. These are the most die-hard bureaucrats who mostly didn’t do much else in life except being politicians. For the sake of argument, let’s suppose there are exceptions even between them.

Here is how the social network of government actors evolved over time:

Growth of social network of Slovenian government members 2001-2013 from Marko O’Hara on Vimeo.


Next diagram shows connections of same cadres to their respective fields of work. Green dots are government offices, other colors are the same as in diagram above. Here one can see, for example:

  • Who is walking in corridors of true power: prime ministers like to keep close Department of Defence, Department of Finance and Department of Internal Affairs. People close to these offices are the movers and shakers.
  • How different the governments of Slovenia truly were: departments were clumped together with other departments over time, split and again clumped with other departments. There’s hardly a department which survived this period without being split or clumped, most notably Department of Defense.
  • Who held which functions, and how are different departments connected with various people.

Launch the interactive diagram of employment by government office

Social network of members and government offices in Slovenian governments 2001-2013
Social network of members and government offices in Slovenian governments 2001-2013

Here is a short video of how all this evolved over time:

Growth of Slovenian government members and ministries 2001-2013 from Marko O’Hara on Vimeo.


Data sources

All data was kindly provided by Government of Republic of Slovenia. Download CSV version here. If anyone wants original documents, e-mail me. My address is on About page.


Graphs were constructed in Java and exported to Gephi for visualization, then again exported to web-friendly sigma.js format.

Correction: it’s actually from 1991 to 2013.

Corruption visualized: Global Corruption Barometer 2013 on world map

Share Button

Interactive map of data from Global Corruption Barometer 2013 (Transparency report), showing corruption levels per country for political parties, educational sector, private companies, media, civil servants, judicial and medical institutions, military, NGOs, parliament, police and religious institutions.

Launch interactive map

Global Corruption Barometer 2013 on interactive world map
Global Corruption Barometer 2013 on interactive world map

Made with TileMill, data: Transparency International.