neural networks Archives

by virostatiq - August 19, 2015August 19, 2015

Could computers vote instead of parliamentary representatives?

Haha, what a funny question. Of course they can’t. How can one teach a computer all the intricacies of lawmaking process, and trust it well enough to let it vote? This must surely be a recipe for disaster.

Yet, as I realized in previous research, the parties mostly demand ruthless discipline from their parliamentary representatives at voting time, simply to be able to actually govern in Slovenian multiparty democracy, where there’s never an absolute winner. This leads to coalition governments, where every vote counts towards a majority.

That means that in a polarized parliament, one could theoretically predict a representative’s vote by examining the votes cast by all other representatives. If an opposition party proposes to vote on an act, it’s very likely that members of government block will uniformly, or at least predominantly, vote against it, and vice versa. There are few exceptions to that rule, namely some profoundly ethical decisions, in which majority parties will let their members vote by conscience. But they are few and far apart.

Fun with neural networks

I decided to test this out by modeling some representatives by neural networks, and training the networks with a few voting sessions and their outcomes in the beginning of the parliamentary term.

Model for each representative was fed votes by every other rep except him- or herself as input, and his or her vote as desired output. This was repeated and repeated again for all hundred training sessions, until the model converged (loss fell under 0.05).

It was then shown voting sessions iz hasn’t seen yet, and tasked to predict the outcomes.

The results are shown in images below. For each representative, the image contains:

name and party,
training vector (the votes he/she cast in first 100 voting sessions – red for “against”, blue for “in favor”, yellow for absence for whatever reason),
actual votes (400 votes the network hasn’t seen and was trying to predict),
predicted votes (how the neural network thought the representative would vote), and
difference indicator (with red rectangles for wrong prediction, green rectangles for correct prediction, and yellow rectangles for absence)

I didn’t bother too much with statistics, to see who was the most predictable, neither did I try to predict voting for every rep.

In short, those with the mainly green bottom strip were the most predictable.

Government coalition

DeSUS

SMC

Opposition

NSi

IMNS

SDS

ZAAB

A cursory examination of results yields several realizations:

even in best predictions with lowest error rate, the model doesn’t predict absences well, especially for representatives with low incidence of absence in training data. This is intuitively understandable on two levels: first, it’s hard for the network to generalize something it didn’t observe, and second, absences can happen on a human whim, which is unreachable for a mathematical model. For representatives of opposition parties, who frequently engage in obstruction as a valid tactics, the model fares a little better.
the model predicts best the voting behavior of majjority party (SMC) members.
the model utterly fails to predict anything for representatives whowere absent in training period (duh).

So, could we substitute the actual representatives with simple neural networks? Not with this methodology. The problem is that we need votes of everyone else in the same session to predict the vote of modeled rep, so at the time of prediction, we already have their vote. We don’t have a way of inferring votes from scratch, or from previous votes.

We could, in theory, try to predict each rep’s vote independently from others by training the network on proposed acts’ texts. I speculate that a deeper network could correlate vectorized keywords in training texts with voting outcomes, and then be able to predict voting for each rep independently based on previous unseen texts. Maybe I’ll do that when I get the texts and learn a bit more. It’s still ANN 101 period for me.

I used a simple perceptron with 98 inputs (there have been 99 representatives in this term, counting also current ministers and substitutes), a hidden layer of 60 neurons, and a softmax classifier on the end.

As usual, I used Karpathy’s convnetjs for modeling, and d3 for visualization. Dataset comes from Zakonodajni monitor.

by virostatiq - August 14, 2015August 19, 2015

Similarities between representatives in Slovenian parliament

The title should actually be “An exploration of dimensionality reduction techniques on voting dataset from Slovenian parliament”.

I’ve long been procrastinating with proper and comprehensive study of various machine learning techniques, especially those related to neural networks. I feel I made a few baby steps towards that goal with this research, which is actually a writeup of a project I made for a local newspaper in collaboration with excellent designer Aljaž Vindiš (follow him on Twitter).

The dataset comes from another project that I’m collaborating on with Transparency International Slovenia and Institut Jožef Stefan. Zakonodajni monitor is a platform for inspecting the legislative process and for following the activity of parliamentary representatives, intended mostly for journalists and researchers. Among other things, it contains records of every vote cast in parliamentary sessions by every representative, which is then used for various statistics and visualizations. It also has an API for public access to that data, although I have it in a local database too, making it somewhat easier for me to explore it.

This project is an attempt to visualize relationships between representatives and parties in two dimensional space, or on a line, to better understand the dynamics of power in Slovenian politics. It’s a part of my ongoing collaboration with Dnevnik newspaper for data analysis and visualization. Since the project was not supposed to be interactive from the start, one important constraint was that the results should be fit for a paper version of the newspaper.

Dataset

Each representative has a great many properties in the database, but among them is a vote vector, containing a record of her or his votes so far. A “yea” vote is 1, “nay” is -1, and abstinence for whatever reason is 0. At the time of this project, there were a little less than 650 votings in this parliamentary term, so input data for each representative was a vector with approximately 650 dimensions. Our objective was to construct one- or twodimensional visualization, which would hopefully confirm our existing knowledge about alliances between parties and individuals in the parliament, and, if possible, reveal new and interesting information.

To effectively communicate this information, we had to employ some dimensionality reduction techniques, of which we tried three:

PCA (principal components analysis),
autoencoder,
t-SNE

In the end, we decided on t-SNE because it’s fast and convenient, but other two methods, with the exception of PCA in two dimensions, gave very similar results.

What is “dimensionality reduction”, you might ask? It’s a set of techniques to make sense of complex data. A shadow is a simple natural reduction technique, because it’s a projection from three dimensional space into two. Going on with this analogy, if you want to recognize a person from its shadow, the position of the sun matters a great deal. For example, sun directly over person’s head doesn’t give us much information about the person’s shape. It’s necessary to find a proper angle.

These various techniques have much to to with proper positioning of the “sun” in relation to data, to retain maximum possible amount of information in the projection. Of course, if you project from 650 dimensions into one, a lot of information is lost. Also, in many cases it’s not immediately clear what is the exact meaning of the axes in the projection. Read on, I’ll try to elaborate below.

Autoencoder

We started with an autoencoder. An autoencoder is a form of artificial neural network that is often used for dimensionality reduction. It is a deep neural network with many layers that essentially tries to teach itself identity, that is to say, it’s trained to generalize patterns in data in by compressing the knowledge in some way, and then recreating it. We used an autoencoder with 650 inputs, two layers of 100 neurons each, then a bottleneck layer with two neurons only, followed by an inverted structure acting as a decoder. When training was complete, every representative’s vector was again propagated through the network, and activations of the two bottleneck neurons saved as a coordinate pair. These were then plotted on a 2D canvas, resulting in a image shown below.

Legend for clarification:

brown (SMC), blue (DeSUS) and red (SD) are leftist position parties with heavy majority in the parliament. Much could be said about their leftism, but let’s leave at that.
violet (SDS) and green (NSi) are rightmost opposition parties. They are vehemently anti-communist (SDS) and catholic-conservative (NSi)
rose is oppositional ZAAB, which is a party of former prime minister Alenka Bratušek. It leans to the left.
grey is oppositional ZL, which is Slovenia’s version of Syriza.

The dataset was relatively small, so the autoencoder was implemented in JavaScript with Karpathy’s excellent convnet.js library. Training took two hours on a i7 machine with 16GB RAM.
As a small branch of this project, we also tried to arrange the representatives on an ideological spectrum. For this, a similar neural network was used, but we first trained it with the most left-and right-leaning representatives to obtain extremes, then fed the others through it and plotted the regression scores in one dimension. This arrangement is somewhat different than the final one.

Principal component analysis

Next on was an attempt to validate our results with PCA. Principal components analysis is a (quote) “technique, used to emphasize variation and bring out strong patterns in a dataset. It’s often used to make data easy to explore and visualize” . It’s essentially a method for projecting data from multidimensional space to a lowerdimensional (say 2D) one, where we try to retain as much information as possible. The first axis is chosen so that the variance along it is maximized, maximizing the information, the others follow in a similar fashion, with the constraint that they must be orthogonal.
We ran PCA for one- and twodimensional solutions, giving solutions on images below.

Here’s the one-dimensional variant:

t-SNE

Finally, we used t-SNE algorithm with one- and twodimensional solutions. t-SNE (or “t-distributed stochastic neighbor embedding”) is another technique for dimensionality reduction, well-suited for visualizing complex datasets in 2D or 3D. You’ll mostly see it in articles dealing with classification of complex data, for instance images and words, where you can see nice plots of similarily-themed images or words with similar meaning clustered together. Here we used it on our voting data, and the results were quite good. First we tried a 2D visualization. It’s roughly similar to the one derived from autoencoder.

Dot sizes correspond to voting attendance. You can see that the representatives with lower attendance are drawn to the center. Also, note that the violet group (SDS party), which is the true and fervent opposition, is relatively close to those with lower attendance. This is simply because the opposition frequently employs obstruction as a parliamentary tactics, or are simply not there due to other reasons.

The neutral control point is the azure rectangle in the center. It’s simply a hypotethical rep that always abstained.

See the voting records for the opposition (yellow is absent, red is a vote against, blue a vote in favor):

And here are records for some ministers:

Compare these with the position:

Partly confirming validity and possible artefacts, we moved on. What we realized so far was that the absences introduced errors in position, and that these errors tend to draw those absent towards the center, possibly confusing the arrangement in a way that some people could wonder: what does this clearly positional rep do close to the opposition? Is (s)he leaning towards them in voting? No, this is simply an artefact that absences introduce into the positioning due to the way these methods work.

Then we decided that we’d maybe like a simpler visualization, one that is more suitable for a paper medium. So we ran t-SNE again in one dimension, then we used a “beeswarm” layout to sketch things out. The beeswarm is essentially a one-dimensional layout, in which some clustered elements are pushed onto a plane to avoid overcrowding on the single axis.

Finally, and mostly for aesthetic reasons, we converted that into hex-binned layout. Number of hexagons corresponds to number of dots above, but voting attendance is encoded in opacity, and party affiliation is represented by color. Here is a sketch:

Here’s a closer view of the opposition:

As a final step, we removed everyone who was present at less that 200 voting sessions, and also added three control points:

neutral: a hypothetical representative who always abstains,
all yea: a hypothetical representative who always votes in favor of the proposition,
all nay: a hypothetical representative who always votes against the proposition

The neutral control point neatly bisects the space between the position and opposition, not counting the mostly absent representatives from the position. The other two would be relevant if all propositions came from the position – it would then be at the extreme pro-government position. In reality, many acts are proposed by the opposition, so they are just not relevant. In the image below, these are the azure hexagons. Neutral is the leftmost one.

And here is a finished version, expertly done by a pro designer:

Added bonus: visualization of tSNE in 3D:

Closeup on opposition:

Interpretation

So, what does this visualization really show? I’d like to say that since the acts subject to voting are mostly put forward by the governing coalition, it’s an arrangement of representatives on a continuum of support for government policy. But that is simply not so, as many acts are proposed by the opposition. It’s more like that the arrangement depends on an individual’s position in relation to majority’s vote, which might or might not relate to the above.

This often coincides with arrangement on ideological spectrum, but it’s not the same. You might wonder what is a cluster of weakly-colored representatives in the right-middle. These are mostly ministers that cast a few votes in the beginning of the parliamentary term, but which then left to be members of actual government. They still were members at one point in time, so we included them in our research, but we might have easily dropped them, since they don’t really figure in day-to-day parliamentary work.

Most of the errors and contra-intuitive positioning are due to gaps in representatives’ voting records. These methods compare voting records component-wise, so if, for example, we have two members of the same party, who substituted for each other (one was there when another wasn’t, as is the case with the ministers), we can only compare their available records with everyone else, but not among themselves.

t-SNE was also done in JavaScript, with another one of Karpathy’s libraries (tsnejs).

Here’s a final look at the data: a hierarchical clustering of all the representatives, including those mostly absent.

Original datasets and code are available by request. My mail address is on the About page.