The title should actually be “An exploration of dimensionality reduction techniques on voting dataset from Slovenian parliament”.
I’ve long been procrastinating with proper and comprehensive study of various machine learning techniques, especially those related to neural networks. I feel I made a few baby steps towards that goal with this research, which is actually a writeup of a project I made for a local newspaper in collaboration with excellent designer Aljaž Vindiš (follow him on Twitter).
The dataset comes from another project that I’m collaborating on with Transparency International Slovenia and Institut Jožef Stefan. Zakonodajni monitor is a platform for inspecting the legislative process and for following the activity of parliamentary representatives, intended mostly for journalists and researchers. Among other things, it contains records of every vote cast in parliamentary sessions by every representative, which is then used for various statistics and visualizations. It also has an API for public access to that data, although I have it in a local database too, making it somewhat easier for me to explore it.
This project is an attempt to visualize relationships between representatives and parties in two dimensional space, or on a line, to better understand the dynamics of power in Slovenian politics. It’s a part of my ongoing collaboration with Dnevnik newspaper for data analysis and visualization. Since the project was not supposed to be interactive from the start, one important constraint was that the results should be fit for a paper version of the newspaper.
Dataset
Each representative has a great many properties in the database, but among them is a vote vector, containing a record of her or his votes so far. A “yea” vote is 1, “nay” is -1, and abstinence for whatever reason is 0. At the time of this project, there were a little less than 650 votings in this parliamentary term, so input data for each representative was a vector with approximately 650 dimensions. Our objective was to construct one- or twodimensional visualization, which would hopefully confirm our existing knowledge about alliances between parties and individuals in the parliament, and, if possible, reveal new and interesting information.
To effectively communicate this information, we had to employ some dimensionality reduction techniques, of which we tried three:
- PCA (principal components analysis),
- autoencoder,
- t-SNE
In the end, we decided on t-SNE because it’s fast and convenient, but other two methods, with the exception of PCA in two dimensions, gave very similar results.
What is “dimensionality reduction”, you might ask? It’s a set of techniques to make sense of complex data. A shadow is a simple natural reduction technique, because it’s a projection from three dimensional space into two. Going on with this analogy, if you want to recognize a person from its shadow, the position of the sun matters a great deal. For example, sun directly over person’s head doesn’t give us much information about the person’s shape. It’s necessary to find a proper angle.
These various techniques have much to to with proper positioning of the “sun” in relation to data, to retain maximum possible amount of information in the projection. Of course, if you project from 650 dimensions into one, a lot of information is lost. Also, in many cases it’s not immediately clear what is the exact meaning of the axes in the projection. Read on, I’ll try to elaborate below.
Autoencoder
We started with an autoencoder. An autoencoder is a form of artificial neural network that is often used for dimensionality reduction. It is a deep neural network with many layers that essentially tries to teach itself identity, that is to say, it’s trained to generalize patterns in data in by compressing the knowledge in some way, and then recreating it. We used an autoencoder with 650 inputs, two layers of 100 neurons each, then a bottleneck layer with two neurons only, followed by an inverted structure acting as a decoder. When training was complete, every representative’s vector was again propagated through the network, and activations of the two bottleneck neurons saved as a coordinate pair. These were then plotted on a 2D canvas, resulting in a image shown below.
Legend for clarification:
- brown (SMC), blue (DeSUS) and red (SD) are leftist position parties with heavy majority in the parliament. Much could be said about their leftism, but let’s leave at that.
- violet (SDS) and green (NSi) are rightmost opposition parties. They are vehemently anti-communist (SDS) and catholic-conservative (NSi)
- rose is oppositional ZAAB, which is a party of former prime minister Alenka Bratušek. It leans to the left.
- grey is oppositional ZL, which is Slovenia’s version of Syriza.
The dataset was relatively small, so the autoencoder was implemented in JavaScript with Karpathy’s excellent convnet.js library. Training took two hours on a i7 machine with 16GB RAM.
As a small branch of this project, we also tried to arrange the representatives on an ideological spectrum. For this, a similar neural network was used, but we first trained it with the most left-and right-leaning representatives to obtain extremes, then fed the others through it and plotted the regression scores in one dimension. This arrangement is somewhat different than the final one.
Principal component analysis
Next on was an attempt to validate our results with PCA. Principal components analysis is a (quote) “technique, used to emphasize variation and bring out strong patterns in a dataset. It’s often used to make data easy to explore and visualize” . It’s essentially a method for projecting data from multidimensional space to a lowerdimensional (say 2D) one, where we try to retain as much information as possible. The first axis is chosen so that the variance along it is maximized, maximizing the information, the others follow in a similar fashion, with the constraint that they must be orthogonal.
We ran PCA for one- and twodimensional solutions, giving solutions on images below.
Here’s the one-dimensional variant:
t-SNE
Finally, we used t-SNE algorithm with one- and twodimensional solutions. t-SNE (or “t-distributed stochastic neighbor embedding”) is another technique for dimensionality reduction, well-suited for visualizing complex datasets in 2D or 3D. You’ll mostly see it in articles dealing with classification of complex data, for instance images and words, where you can see nice plots of similarily-themed images or words with similar meaning clustered together. Here we used it on our voting data, and the results were quite good. First we tried a 2D visualization. It’s roughly similar to the one derived from autoencoder.
Dot sizes correspond to voting attendance. You can see that the representatives with lower attendance are drawn to the center. Also, note that the violet group (SDS party), which is the true and fervent opposition, is relatively close to those with lower attendance. This is simply because the opposition frequently employs obstruction as a parliamentary tactics, or are simply not there due to other reasons.
The neutral control point is the azure rectangle in the center. It’s simply a hypotethical rep that always abstained.
See the voting records for the opposition (yellow is absent, red is a vote against, blue a vote in favor):
And here are records for some ministers:
Compare these with the position:
Partly confirming validity and possible artefacts, we moved on. What we realized so far was that the absences introduced errors in position, and that these errors tend to draw those absent towards the center, possibly confusing the arrangement in a way that some people could wonder: what does this clearly positional rep do close to the opposition? Is (s)he leaning towards them in voting? No, this is simply an artefact that absences introduce into the positioning due to the way these methods work.
Then we decided that we’d maybe like a simpler visualization, one that is more suitable for a paper medium. So we ran t-SNE again in one dimension, then we used a “beeswarm” layout to sketch things out. The beeswarm is essentially a one-dimensional layout, in which some clustered elements are pushed onto a plane to avoid overcrowding on the single axis.
Finally, and mostly for aesthetic reasons, we converted that into hex-binned layout. Number of hexagons corresponds to number of dots above, but voting attendance is encoded in opacity, and party affiliation is represented by color. Here is a sketch:
Here’s a closer view of the opposition:
As a final step, we removed everyone who was present at less that 200 voting sessions, and also added three control points:
- neutral: a hypothetical representative who always abstains,
- all yea: a hypothetical representative who always votes in favor of the proposition,
- all nay: a hypothetical representative who always votes against the proposition
The neutral control point neatly bisects the space between the position and opposition, not counting the mostly absent representatives from the position. The other two would be relevant if all propositions came from the position – it would then be at the extreme pro-government position. In reality, many acts are proposed by the opposition, so they are just not relevant. In the image below, these are the azure hexagons. Neutral is the leftmost one.
And here is a finished version, expertly done by a pro designer:
Added bonus: visualization of tSNE in 3D:
Closeup on opposition:
Interpretation
So, what does this visualization really show? I’d like to say that since the acts subject to voting are mostly put forward by the governing coalition, it’s an arrangement of representatives on a continuum of support for government policy. But that is simply not so, as many acts are proposed by the opposition. It’s more like that the arrangement depends on an individual’s position in relation to majority’s vote, which might or might not relate to the above.
This often coincides with arrangement on ideological spectrum, but it’s not the same. You might wonder what is a cluster of weakly-colored representatives in the right-middle. These are mostly ministers that cast a few votes in the beginning of the parliamentary term, but which then left to be members of actual government. They still were members at one point in time, so we included them in our research, but we might have easily dropped them, since they don’t really figure in day-to-day parliamentary work.
Most of the errors and contra-intuitive positioning are due to gaps in representatives’ voting records. These methods compare voting records component-wise, so if, for example, we have two members of the same party, who substituted for each other (one was there when another wasn’t, as is the case with the ministers), we can only compare their available records with everyone else, but not among themselves.
t-SNE was also done in JavaScript, with another one of Karpathy’s libraries (tsnejs).
Here’s a final look at the data: a hierarchical clustering of all the representatives, including those mostly absent.
Original datasets and code are available by request. My mail address is on the About page.
