On the basis of previous post, Transparency International Slovenia asked me to collaborate on some projects. This is one of them, and it was launched today on a separate site: kdovpliva.si (English: whoinfluences.si).
It’s an attempt to visualize several networks of lobbyists, their companies, politicians and state institutions. Perhaps the most interesting part is the network of lobbying contacts, which was constructed with data containing around 700 reported contacts between 2011 and late 2014.
As you may imagine, not every lobbying contact is reported. For those who are, records are kept at the Komisija za preprečevanje korupcije (Commission for prevention of corruption, a state institution). Transparency International Slovenia obtained those records as PDF files, since the institution refused to provide them in a machine-readable format. They hired a few volunteers to copy and paste the information in spreadsheets, then handed them to me to visualize them.
You can see the results below. Click here or the image to open the site in a new window. It’s in Slovenian. For methodology, continue reading below the image.
The meaning of every network is determined by the nature of its nodes and connections. Here, we have four node types:
those who were lobbied – state officials
organizations on which behalf lobbying was performed
state institutions at which the abovementioned officials work
Lobbying contact is initiated by a company or an organization, which employs a lobbyist to to the work. These people then contact state officials of a sufficient influence, who work at appropriate state institution.
So an organization is connected to the lobbyist with a weight of 2, the lobbyist to a state official with a weight of 1, and state official to her institution with a weight of 2. The weights signify the approximate loyalty between these entities. We presupposed that lobbyists are more loyal to their clients than they are to the state officials, with which they must be in a promiscuous relationship. Furthermore, the state officials are also supposed to be more loyal to their employers than to the lobbyists, although this is a daring supposition. But let’s say they are, or at least that they should be.
After some processing, the network emerged. Immediately apparent are the interest groups, centered around seats of power. Here’s an image of the pharmaceutical lobby. It’s centered on the Public Agency for Pharmaceuticals and Medicine. Main actors of influence are companies such as Merck, Novartis, Eli Lilly, Aventis, etc.
A click on the agency node brings up a panel with some details, such as a list of companies (font size indicates the frequency of contact), lobbying purposes and a timeline of lobbying contacts. Here we can see that Novartis and Krka were most active companies, and that they lobbied for purposes of pricing and to limit potential competition by producers of generic drugs.
You can explore the network by yourself to see the other interest groups.
Some advice from Information Commissioner
Unfortunately, we had to omit lobbyists’ names for reasons of supposed privacy. The Information Commissioner strongly advised us not to display them on the basis of some EU ruling. I’m not an expert in EU law, and perhaps there are good reasons for this. On the other hand, there may not be. I fail to see why this information would not be in public interest, since these decisions have an impact on a significant number of taxpayers, if not all of them.
Anyway, we have the names. After all, we had to use them to connect the network. They are present in raw data, just not displayed.
We’re are probably going to continue developing this project, as new information comes to light and new rulings regarding privacy are issued.
This is an attempt at visualizing different conspiracy theories. The visualization tries to show interconnectedness of actors, organizations and concepts in each one, so a network graph was chosen as a mode of presentation. The presented theories are: The Antigravity Drive, Chemtrails, The Cabal (American deep state from JFK assassination to 9/11), The Illuminati/New World Order, and the most recent, the Malaysian Airlines Flight MH370 disappearance. In a way, it’s a progression from the previous network visualization about the PRISM scandal, which was once also considered a conspiracy theory.
I chose this topic because those theories always attracted me as a means of alternative explanation of things that I couldn’t understand in official versions of events. That is not to say that I necessarily believe in any of them. For example, I’d be hard pressed to believe in the Moon Landing Hoax theory, which I first included here because of relative ease of gathering source material, but later discarded because of its relatively low value. The Flt 370 theory has extremely low credibility too, and I wonder what I’ll think when this post is a year old.
A conspiracy, according to Wikipedia ” … may also refer to a group of people who make an agreement to form a partnership in which each member becomes the agent or partner of every other member and engage in planning or agreeing to commit some act.“. This is a pretty broad definition. It can apply to a government, a company, or every group of people who are trying to further an agenda, be it good or bad for their natural or social environment. But anything labelled as a conspiracy almost always has an evil association, for example “A civil conspiracy or collusion is an agreement between two or more parties to deprive a third party of legal rights or deceive a third party to obtain an illegal objective.” (Wikipedia – civil conspiracy), or “In criminal law, a conspiracy is an agreement between two or more persons to commit a crime at some time in the future.” (Wikipedia – criminal conspiracy).
A conspiracy theory is therefore an attempt at explaining a real or imagined conspiracy. In this sense, even official stories of various incidents are conspiracy theories, unless they are well founded in evidence and irrefutable facts. In a free society, a kind of market then forms of conspiracy theories, in which those with better means, but also more vested interests, compete for public’s attention with other bodies of citizenry, whose interests and aims can differ significantly. For example, a government can execute a false flag attack, as the Nazis did in Poland at the beginning of WW2, and spin a theory that the other party did it, in order to go to war and grab land. The public may then be motivated to concoct a variety of counter theories with various motives – simply seeking the truth, overthrowing the government by exposing the lies it tells, furthering some commercial agenda, for example selling books, or purely personal paranoid agendas, which serve no one else than the authors and their need to sustain their delusions.
Let me briefly explain the theories I used in this visualization. First two are quite believable.
The Cabal: the story of American deep state and events from JFK assassination to 9/11 attacks
A story of how the Nazi regime allegedly developed a form of anti gravity propulsion in total secrecy, made possible by a strictly compartmentalized environment, imposed on the German war production efforts by the SS. The technology was then seized by the US military and other allies after the war and developed further in utmost secrecy. The first such machines ever seen were so-called foo fighters. These balls of light, sighted and documented by various US Air Force pilots, flew in parallel with bombers and fighter planes, and frequently executed seemingly impossible air maneuvres. Also mentioned is a mythical machine The Glocke (The Bell), which ran on red mercury and was responsible for death of several scientists due to extreme radiation it produced, and the discoveries of Viktor Schauberger. His implosion engine, which drew heavily on vortex physics, was allegedly successful, and produces two flying prototypes. The US military immediately grabbed and classified much of this work, and it stays secret until now. It’s said to be employed in B-2 bomber and various flying craft sighted around Area 51 in Nevada. The story also goes to mention modern experiments in anti gravity physics, notably performed by Evgeniy Podkletnov, which allegedly succeeded in reducing gravity over a spinning superconducting electromagnet for two percent.
How a handful of secret societies dominate the world. The plot allegedly has its roots in The Bavarian Illuminati society, started in the eighteen century by Adam Weisshaupt. They were eradicated, but some claim they survived in a covert form, forging an alliance with international bankers. Most big world events since then were planned in advance, among them both the advent of Communism, Nazism and Zionism, World Wars, and the third too. Says Pike: “The Third World War must be fomented by taking advantage of the differences caused by the “agentur” of the “Illuminati” between the political Zionists and the leaders of Islamic World. The war must be conducted in such a way that Islam (the Moslem Arabic World) and political Zionism (the State of Israel) mutually destroy each other. Meanwhile the other nations, once more divided on this issue will be constrained to fight to the point of complete physical, moral, spiritual and economical exhaustion…We shall unleash the Nihilists and the atheists, and we shall provoke a formidable social cataclysm which in all its horror will show clearly to the nations the effect of absolute atheism, origin of savagery and of the most bloody turmoil.”
In recent times, the organizations that further Illuminati goals are Council for Foreign Relations, Trilateral Commission and the Bilderbergers. Here are some books: The Illuminati: Facts & Fiction by Mark Dice, and The Illuminati original by Adam Weisshaupt.
Malaysian Airlines Flight MH370 disappearance
A recent theory about the whereabouts of the missing plane. On it, there seemed to be an awful lot of technical personnel, involved in developing military hardware. They supposedly worked for a company named Freescale Semiconductors, which was in a patent wrestle with the Rothschild family. Acording to the story, Israeli agents and elements of US military hijacked the plane and secretly flew it to Diego Garcia military base in the Indian Ocean to debrief the experts and possibly use the plane in another 9/11-style attack in the future.
Construction and visualization of visualization networks
A few words for technologically minded. The networks were constructed by text-mining the source material, isolating known entities in sentences by means of massive dictionaries, connecting them in subnetworks (each sentence – one subnetwork), and finally adding them in the master network for that topic. Only sentence-length subnetworks were constructed, although it would be probably more fruitful to connect entities in paragraphs too. That would yield a too convoluted master network, so I stayed with sentences for clarity.
The dictionaries were automatically generated from source texts, then edited, Many synonyms had to be added, since my dictionary generating technique relies more on brute force than on semantic aspects of text. Again, the connections are not semantic, which means that if there was a sentence “The Illuminati are NOT connected with the CFR”, Illuminati and CFR would still be connected. Here I’m relying on the power of statistics: in majority of sentences there mostly appear connected entities. For the minority in which they are not, the bonds between them are too weak to influence the big picture.
I did try to process volumes of texts with a natural language processing framework, namely Apache OpenNLP, but got frustrated with the amount of work that would be needed for this little hobby project. I’d need to train the classifiers to extract named entities, which is no small feat, and I’d probably not use them again. To gain some insight in types of connections between these entities, I tried parsing the sentences into parse trees, then extract relationships, but parsing tech is not very accurate. It would probably do, again relying on power of statistics, but the sheer amount of relationship types would add little to visual value of the graphs, so I decided that I’d do this with a simpler project first. The logic I wrote is still in project source code, so if anyone is interested, mail me (About page) and I’ll send it your way. Same goes for the graph files and the categorized dictionaries.
Finally, the topic networks were exported as subgraphs, so that every node in the network is represented by a subgraph. These subgraphs are added into – or removed from – the master graph by the client. The networks in Browser are managed by sigma.js. Preliminary analysis was done in Gephi, I recommend Network Graph Analysis and Visualization with Gephi by Ken Cherven.
Additionally, geographic entities were extracted for each node. These are represented on a small map in the bottom of the screen. Map is managed by d3.js.
Interacting with visualization
There are two modes – reading the story or exploring on your own. Switch between them by clicking a button on top right of the graph. While read the story, the graph will change in real time as you scroll the text down. If you choose to explore, you can click on terms, and their subgraphs will be interactively added to the master graph.
Clicking on a graph node will expand it (load its associated nodes and display them, if previously not loaded), or delete it, if it was already loaded, at the same time showing the text from which its existence was text-mined.
There’s no way for the user to control the map. It’s there for informative and decorative purposes.
This is an interactive timeline of events about the Prism scandal, chronicled by selected media in online news articles, giving a summarized view of events as they unfolded. It’s intended as a parody of a NSA software to track people and analyze their metadata. It consists of these parts:
the chronological order of articles, visualized as a timeline,
a network of people, places and organizations that appear in the articles,
geographic information that the articles refer to, and
a bar graph showing wordcounts of interesting words, associated with the main theme.
In the bottom part there is a timeline displaying the published articles in chronological order. Articles are accessible by clicking on a title and then using the Go to article link in the popup bubble.
In the center background there is a rotating globe, displaying major cities referred by articles. Labels next to cities contain titles of all articles visible in timeline view that refer to specific city.
Center foreground contains a network showing interconnectedness of various entities recognized in the article text. Entities appearing in the same paragraph are connected. The network is additive, which means that the more frequently entities appear in same paragraphs, the stronger is the bond between them.
In the right corner, there is a small bar graph containing frequencies of selected words, giving an idea of Snowden’s options at the time. It’s just a word count of shown words in visible articles, not a semantic analysis.
As the timeline is moved, new articles appear, and network is updated with new data, giving a quick overview of who and how frequently was involved in discussion, and who was related to them.
Hovering the mouse over a network node shows only the portion of the network that the node is directly connected to, making this useful for detailed exploration of relationships between those entities.
Use the mouse to drag timeline left or right, or rotate the middle wheel for the same effect. For quicker navigation use the Quick jump menu. The subnetworks will load and unload automatically, and the whole network will try to stabilize so that it accurately reflects frequencies of terms and bond stregths between them. If it doesn’t stabilize well, click the Reorder network button or double-click in the middle of the timeline.
It’s possible to zoom the network in and out to get a better idea of shown names and connections. To do that, position the mouse pointer over the network area and drag or zoom with the wheel. Node size corresponds to term frequency in visible articles, while the bond thickness corresponds to its weight, that is to say frequency of said bond.
Click the article titles in the timeline to display more information and links. To change the publisher, click the Publisher menu and select a desired one. This will load new set of events into the timeline. To automatically move the timeline, use the Play button.
News articles containing keywords Snowden, Prism, NSA, Wikileaks and Julian Assange were scraped from selected media and stored locally for processing. The articles themselves are linked from the timeline. Their content, apart from titles, is not accesible in this visualization for copyright issues. Geographical database with city names and corresponding latitudes and longitudes was obtained as a free download at GeoNames. The media an/or publishing houses were selected to give a balanced set of worldviews. These are, in alphabetical order:
First a dictionary of all capitalized word sequences and their permutations was constructed by processing all articles in the database. This is essentially a dictionary of all people, states, cities and organizations appearing in the whole database. Then, the title and body text of each article was scanned for these dictionary entries and city names, so that an article abstract was constructed, containing of title, publishing date, a link to article, a wordcount of selected words, a subnetwork of connected entities, and a list of cities along with latitudes and longitudes.
Constructing the network
The article subnetwork was constructed so that entities in the same sentence (connecting in a paragraph shown in the picture) are connected with a set weight. Nodes not connected with any other nodes are dropped at this point, since their inclusion would lead to a largely unconnected network, which is visually unappealing and cumbersome to navigate.
All web scraping and text processing was done in Java locally, there were around 10,000 articles processed in the latest count. See picture below.
There does not exist a live server database that this visualization would query. The entity dictionaries are here (names) and here (selected words).
Constructing the visualization
Sigma.js for displaying the networks. The latest version does not contain some key functionality for dynamically and additively loading and unloading of subgraphs into the main graph, so the source code was updated with required methods. Separate article on that topic is upcoming.
Three.js for rotating Earth and all geographically-related work.
Such a young country, but already so messed up. One is inclined to think that all is lost, and one would not be far from the truth. Much ink has already been spilled on sad state of affairs in Slovenia, its fall from grace in European Union, the precipitous decline of living standard of its citizenry and its bleak outlook for the future. Did I mention the rampant corruption of its ruling class and top managers? Best not. This was, after all, supposed to be the next Switzerland.
Blaming the ruling class in mere abstract terms may give one a fleeting satisfaction, but who were the people who led us off the cliff? Someone did govern here, or was at least giving an appearance of governing. Prime ministers are known: Lojze Peterle, Janez Drnovšek, Tone Rop, Andrej Bajuk, Janez Janša, Borut Pahor and currently Alenka Bratušek. These are the main culprits for the downward spiral, of which one can only hope we already passed the first half. Names of their accomplices – the ministers, secretaries, etc. – have a tendency to drift into oblivion, as majority of people preoccupy themselves with the daily grind.
So who were they and how are they connected? Here’s a diagram showing all the government members from 2001 on. I call it “loyalty diagram”, since it was constructed in a way that it shows who is close to whom, and who is hardly loyal to any alliance. The rationale in short is:
Ministers are considered to be very loyal to the prime minister (although I know they are not).
Secretaries a lot less, since they are essentially experts and not politicians.
Secretaries are less loyal to ministers as are ministers to prime minister, but still a lot, since it’s they who appoint them.
Secretaries are loyal to each other, since they are bureaucrats who like their positions and will in theory support each other, although in practice there exist many party rivalries.
Click the link or image below to launch the interactive diagram, which can be searched, panned, and zoomed, and which shows details for every staff member on the government. Red dots are prime ministers, bright blue ministers, dark blue secretaries. Every person is marked with a color of the highest position occupied.
There are a select few of loyal party cadres that every prime minister carries with him, or her, which very rarely, if at all, work with anyone else. These are the dark blue and bright blue dots in close proximity of red dots (prime ministers).
Node radius is proportional to how many times the individual sat in a government over the years. For example, Janez Janša was not only prime minister twice, he also served in other capacities, most notably as Minister of Defense in 1994 and was taking on more and more departmental duties as his government in 2012 slowly disintegrated.
There is a big cluster of common cadres between Janez Drnovšek’s and Anton Rop’s governments. It seems that a lot of secretaries are passed on into the next mandate, except in case of shift between left- and right-wing governments, which perform a purge on inauguration.
Anton Rop had most secretaries and the biggest government. If anything, the governments are getting slimmer with time.
People in the middle of diagram are generally dragged there because of many ties with different prime ministers and ministers, so they are either the most politically promiscuous, or (theoretically) the best experts in their fields, a theory swiftly disproven considering they took on ministerial duties in vastly different departments. These are the most die-hard bureaucrats who mostly didn’t do much else in life except being politicians. For the sake of argument, let’s suppose there are exceptions even between them.
Here is how the social network of government actors evolved over time:
Next diagram shows connections of same cadres to their respective fields of work. Green dots are government offices, other colors are the same as in diagram above. Here one can see, for example:
Who is walking in corridors of true power: prime ministers like to keep close Department of Defence, Department of Finance and Department of Internal Affairs. People close to these offices are the movers and shakers.
How different the governments of Slovenia truly were: departments were clumped together with other departments over time, split and again clumped with other departments. There’s hardly a department which survived this period without being split or clumped, most notably Department of Defense.
Who held which functions, and how are different departments connected with various people.
A better title for post would probably be “What kinds booze to drink together to get drunk in style, according to those who write, compile, publish, test and enjoy cocktail recipes”. Continuing from previous posts, I wanted to see how does it look a network of ingredients of all possible cocktail recipes, and if it’s possible to divide them into sensible groups, so that they would be instantly recognizable and even helpful to experienced and casual drinkers alike.
To do this, more than 25,000 recipes from Drinksmixer.com and Drinksnation.com were scraped, a network was constructed with Gephi, and visualized here below. Dot size reflects the count of that particular ingredient in all analyzed recipes. Dots of same color frequently appear together in recipes. One could say that one can hardly make a mistake if one combines three ingredients of the same color and drinks the concoction.
The map below is interactive, try panning and zooming with mouse or use the control in the upper left-hand corner.
I see five major groups of ingredients, but your alcohol proof may vary. Actually I suspected something like that:
ice is in its own group. For some reason it also contains tequila,
milky drinks are in their own group (gray-blue),
salty and spicy drinks are also in an easily recognizable group (pink),
blue group is dominated by vodka and rum,
green group mostly has gin and tangy juices, and
red group mostly contains fruit schnappses and liqueurs.
For a more mobile-friendly, searchable map with advanced interactivity, click here (Sigma.js). Clicking on an ingredient on this map will show a list of all connected ingredients. Clicking on an element in the list will show a subgraph.
Most recipes contained preferred brands for spirits and fruit juices, so I constructed another diagram. It shows which brands are usually grouped together in drinks.