d3 Archives - Virostatiq

by virostatiq - June 15, 2016June 15, 2016

State projects and county budget visualizations – a collaboration with Transparency International Slovenia

These two projects are a result of recent collaboration with Transparency International Slovenia. The datasets were provided by the state, and I was asked to develop visualizations that would structure the information in an accessible way. Much help was also provided by members of Institut Jožef Štefan.

State project browser

The first project is a browser of all projects, initiated by state institutions, from 1991 on. The idea was to let users discover, where and for what purposes the money goes in their county. The dataset and visualization allow for exploration by various categories, as well as time.

The projects in the dataset also contain projects that are still in the planning phase, and won’t be completed until year 2025. With this tool, citizens can hopefully inspect the planned expenditures for roads, water sources, and other categories of infrastructure, culture and other fields of development, and compare that with their own expectations.

It allows browsing and filtering of projects by statistical regions and counties, as well as displaying the timeline of all projects, which is basically an expandable version of a Gantt chart.

To see the interactive project website, click here, or click the image below.

The original data is provided on the project’s “About” page.

County budget browser

The new project is a straightforward visualization of county budgets. The budgets are displayed as dynamic, zoomable hierarchical (“sunburst”) diagrams. They react to each other, allowing a side-by-side comparison of budgets of two user-selected counties.

The visualization enables users to delve into expenses and incomes of all Slovenian counties on separate tabs.

To see the interactive project website, click here, or click the image below.

Technology and design

The data cleanup and preparation was done with some Python scripts. The sunburst diagram accepts hierarchical data in a tree format, so this provided an interesting exercise of converting a tabular dataset into a nested dictionary of optional depth.

The visualizations were done in d3, which is really an indispensable tool for any serious work in online visualization.

Both projects were minimalistically, yet expertly designed by Tomaž Plahuta (Bitnik, Eno).

Check out the projects and let me know your opinion in the comments!

by virostatiq - January 8, 2016January 8, 2016

Maps of various name suffix densities in USA and Slovenia

This post and maps were inspired by Moritz Stefaner’s -ach, -ingen, -zell. I firmly believe in giving credits to whom they are due, so there it is.

That said, I embarked into a similar adventure, first for Slovenia. Etimology of Slovenian towns and other populated places may differ a little from German one, so I was naturally curious what it would be like on a map. I had several geo files for Slovenia around, and also a comprehensive list of all populated places with coordinates, making this a relatively short endeavour.

In addition to common ~~suffices~~ suffixes, I also extracted common prefixes. This is because many Slovenian place names begin with “Gornja” (Upper) or “Velika” (Great), so I wanted to see if there are meaningful spatial distributions of these names. It turns out that they are.

For example, this one. By columns: “gornja” (a variant of “upper) vs “dolnja” (variant of “lower”), “zgornja” and “dolnja” (another couple of variations on the same dichotomy), the “velika” and “mala” (“great” and “small”). It’s apparent that places with those prefixes have characteristic spatial distributions. Why, I don’t know. Dialects of Slovenian language vary wildly, to the point that some of them are virtually incomprehensible to me.

To see the interactive version with more maps, click here, or click the image. Switch between prefixes and suffixes using links in the upper left square.

Distribution of places with some common prefixesprefixes — Distribution of places with some common prefixes

Having written the code and downloaded the geonames.org database, it was just a matter of changing a few things to produce a similar map of a similar distribution in the USA. I colored it a litlle differently, but it’s basically the same thing.

Again, click here or the image for interactive version. Note that you can click on a little link above each map to display the list of place names.

Then, a friend and coworker of mine said that he always wondered about the distributions of U.S. towns with borrowed names from European places. That would effectively show distributions of immigration in early history of USA, with exception of Spanish names, which tend to be on the Mexican border because of history, and some random noise in string matchings.

Here’s an image. Click here or map for interactive version.

Check out the maps! Some technical details: the maps were drawn with d3, and hexagons produced with the hex-binning plugin.

Name matching was not a big challenge, but I did want to find unique suffixes. So I wrote some software to first isolate the most frequently occurring seven character suffixes, then I gradually shortened them until big dropoff, say, more than 50 places occurred. That way I prevented near duplicates to be included, for example “-ville”, “-ille”, “-lle”, which have approximate same distribution, so only one of them has a place on the map.

The biggest challenge was in fact generating a hex grid within the borders, and then fitting the data inside it. That’s the reason the pages need some time to load. I brute-forced that by generating points inside the bounding box and checking if withing the polygon in question with turf.js, then setting all hexagon lenghts to zero, and finally filling them with real data.

Hope you enjoy the maps!

by virostatiq - September 1, 2015September 2, 2015

Map of apartment building management companies in Ljubljana

Some time ago, I did an analysis of building manager companies for dnevnik.si. It was an interesting project, because there’s much controversy surrounding this topic. Some of these companies have been in bitter disputes with the residents and competing companies over their business, generating a lot of newspaper coverage.

We did a simple analysis to deterimine market shares and relative winners and losers in comparison to market leader. Here’s an image of final page in the newspaper. Click it to open the PDF.

upravniki_dnevnik — Newspaper page in Dnevnik.si

We also did a lot of other stuff that didn’t find their way into the final article. I’m now publishing an interactive map, showing the distribution of these companies across town. There’s usually one per apartment building. For this map, I only processed the apartment buildings with more than one apartment, because that’s where business is.

Click on the image below to open the map in a new tab, or click here.

Click for interactive map of apartment building managers

Click on a building, and a popup will display the address and relevant history. It’s also possible to choose a company, and the map will refresh, displaying only the buildings in the selected company’s management. There’s also a button to display only those buildings where managers changed in the interval from years 2005 to 2015.

There’s also a graphic showing how many buildings a company gained from other companies. Here’s a screenshot. Click will take you to the same page as image above.

Who gained boildings from whom in Ljubljana — Who gained buildings from whom in Ljubljana

The original dataset came from GURS, but then we were able to acquire an update from the city government. We decided to use that, because they gave us three different updates, each five years apart, from which we were able to calculate the differences. You can see them in table below.

total flats: how many apartments are managed by a company,
gained: how many flats were gained by the company from 2005 to 2015,
lost: how many flats were lost by the company from 2005 to 2015.

Original data is here.

Company	Total flats	Gained	Lost
SPL D.D., FRANKOPANSKA ULICA 18A, 1519 LJUBLJANA	107130	38	429
INSTA D.O.O., KOPRSKA ULICA 94, 1000 LJUBLJANA	2338	0	183
METALKA STANOVANJSKE STORITVE D.O.O., INDUSTRIJSKA CESTA 1, 1000 LJUBLJANA	22529	669	382
SZ RUSKI CAR D.O.O., GLINŠKOVA PLOŠČAD 22, 1000 LJUBLJANA	5515	256	182
PRACTIC D.O.O., LJUBLJANA, CELOVŠKA CESTA 147, 1000 LJUBLJANA	8355	92	511
VALINA UPRAVLJANJE, D.O.O., LITIJSKA CESTA 45, 1000 LJUBLJANA	480	72	10
DONF D.O.O., MAJORJA LAVRIČA ULICA 12, 1000 LJUBLJANA	6891	167	741
STANDOM ENGINEERING OTRIN D.O.O., BRNČIČEVA ULICA 13, 1231 LJUBLJANA – ČRNUČE	5661	113	20
AKTIVA UPRAVLJANJE D.O.O., LJUBLJANSKA CESTA 12F, 1236 TRZIN	8764	335	52
ŽELVA D.O.O., SAMOVA ULICA 9, 1000 LJUBLJANA	13550	187	236
UPRADOM D.O.O., LINHARTOVA CESTA 13, 1000 LJUBLJANA	3141	187	0
MASTRA D.O.O., ZALOŠKA CESTA 153, 1120 LJUBLJANA	6566	198	260
VALOR D.O.O., LJUBLJANA, ROZMANOVA ULICA 1, 1000 LJUBLJANA	351	26	42
M.I.S. D.O.O. LJUBLJANA, PROLETARSKA CESTA 4, 1000 LJUBLJANA	3999	318	22
LORA, D.O.O., ZALOŠKA CESTA 163, 1000 LJUBLJANA	5084	0	24
TENIT, D.O.O., LJUBLJANA, GAŠPERŠIČEVA ULICA 23, 1120 LJUBLJANA	2631	154	146
TABOR UPRAVLJANJE IN VZDRŽEVANJE D.O.O., TABOR 9, 1000 LJUBLJANA	4729	71	0
UVG D.O.O., LINHARTOVA CESTA 17, 1000 LJUBLJANA	7189	304	503
MANICOM D.O.O., POVŠETOVA ULICA 65, 1000 LJUBLJANA	118	0	34
KOVAČEVIČ MARIJA, MAJDE VRHOVNIKOVE ULICA 18, 1000 LJUBLJANA	48	0	0
PATE, D.O.O., TOPNIŠKA ULICA 19, 1000 LJUBLJANA	1867	0	800
STANDOM D.O.O., ULICA ANE ZIHERLOVE 6, 1000 LJUBLJANA	5306	48	462
SKUPNOST LASTNIKOV POT K RIBNIKU 16-22, POT K RIBNIKU 16, 1000 LJUBLJANA	656	0	0
GDM D.O.O., MIVKA 19, 1000 LJUBLJANA	38	0	38
HIŠNI SERVIS MATIČ D.O.O. LJUBLJANA-ČRNUČE, CESTA 24. JUNIJA 74, 1231 LJUBLJANA – ČRNUČE	994	173	28
DISTOR D.O.O., KAVČIČEVA ULICA 58, 1000 LJUBLJANA	2792	102	109
HEKER D.O.O., LJUBLJANA, CELOVŠKA CESTA 144, 1000 LJUBLJANA	3119	33	96
VEGRIM D.O.O., LJUBLJANA, MEDVEDOVA CESTA 25, 1000 LJUBLJANA	3519	11	18
T & S D.O.O., LJUBLJANA, PARMOVA ULICA 53, 1000 LJUBLJANA	944	50	14
MAVSAR D.O.O., GORAZDOVA ULICA 19, 1000 LJUBLJANA	1505	0	47
G.V. CENTER D.O.O. LJUBLJANA, BRNČIČEVA ULICA 31, 1231 LJUBLJANA – ČRNUČE	7701	16	23
EMONA PLUS D.O.O., ŠMARTINSKA CESTA 130, 1528 LJUBLJANA	4855	1096	60
MAG DOM GORAZD MAKAROVIČ S.P., PODPEČ 21, 1312 VIDEM-DOBREPOLJE	166	14	0
STANOVANJSKA ZADRUGA TOMAČEVO JARŠE , Z.O.O., TOMAČEVO 1, 1000 LJUBLJANA	8272	32	10
KOMSTAN D.O.O., LJUBJANA, CELOVŠKA CESTA 291, 1000 LJUBLJANA	786	0	460
DOMO, D.O.O., MAJORJA LAVRIČA ULICA 12, 1000 LJUBLJANA	7649	845	42
LORA – LJUBLJANA, D.O.O., ZALOŠKA CESTA 163, 1000 LJUBLJANA	3119	20	215
AL-MA LJUBLJANA, BRODARJEV TRG 3, 1000 LJUBLJANA	233	0	233
SCH – GROUP INVEST D.O.O., DUNAJSKA CESTA 63, 1000 LJUBLJANA	10	0	0
STANOVANJSKA ZADRUGA RTV Z.O.O., LINHARTOVA CESTA 17, 1000 LJUBLJANA	1521	0	952
REKO D.O.O., PARMOVA ULICA 39, 1000 LJUBLJANA	37	0	17
REYAN D.O.O. LJUBLJANA, NEUBERGERJEVA ULICA 3, 1000 LJUBLJANA	3889	60	167
FINANCE-OPERATIVA D.O.O., KNEZA KOCLJA ULICA 37, 1000 LJUBLJANA	3105	255	52
DOM EFEKT D.O.O., OBIRSKA ULICA 1A, 1000 LJUBLJANA	5988	0	0
EMONA UPRAVLJANJE D.O.O., ŠMARTINSKA CESTA 130, 1000 LJUBLJANA	2367	0	286
S & H D.O.O., LJUBLJANA, MLINSKA POT 19, 1231 LJUBLJANA – ČRNUČE	32	0	32
UPSO D.O.O., PRUŠNIKOVA ULICA 95, 1210 LJUBLJANA – ŠENTVID	1980	328	0
GOSPODAR D.O.O., TRŽAŠKA CESTA 42, 1000 LJUBLJANA	13859	332	6
ALTARA D.O.O., ULICA HERMANA POTOČNIKA 19, 1000 LJUBLJANA	56	56	0
LESNINA LGM SI D.O.O., PROLETARSKA CESTA 4, 1000 LJUBLJANA	266	12	14
KIDREAL D.O.O., ZALOŠKA CESTA 1, 1000 LJUBLJANA	780	0	0
STUPRA D.O.O. LJUBLJANA, KRAKOVSKI NASIP 22, 1000 LJUBLJANA	72	0	16
DOMCOMMERCE D.O.O., PUHOVA ULICA 1, 1000 LJUBLJANA	186	0	60
TERCA D.O.O., ŠENTRUPERT 124, 8232 ŠENTRUPERT	677	77	0
FERDOM D.O.O., DOLENJSKA CESTA 242, 1000 LJUBLJANA	154	154	0
TABOR LJUBLJANA, D.D., TABOR 9, 1000 LJUBLJANA	288	130	93
KREATOR POSLOVNI CENTER, D.O.O., LITIJSKA CESTA 38, 1119 LJUBLJANA	318	0	0
PRIMA PLUS D.O.O., SLOVENČEVA ULICA 93, 1000 LJUBLJANA	1696	72	34
GRUDA MARTIN PILIH S.P., VIDEM 1, 1380 CERKNICA	96	96	0
PUN NEPREMIČNINE D.O.O. LJUBLJANA, KERSNIKOVA ULICA 10A, 1000 LJUBLJANA	2140	38	410
TEVIS – AGENCIJA ZA KADRE, D.O.O., KOTNIKOVA ULICA 28, 1000 LJUBLJANA	706	22	0
IMOVINA, D.O.O., LJUBLJANA, GOSPOSVETSKA CESTA 10, 1000 LJUBLJANA	214	16	0
NAŠA SOSESKA D.O.O., BRILEJEVA ULICA 8, 1000 LJUBLJANA	1898	743	0
MARBONA D.O.O. LJUBLJANA, ZALOŠKA CESTA 269, 1000 LJUBLJANA	455	199	0
DOMPLAN, D.D., BLEIWEISOVA 14, 4000 KRANJ	1160	24	0
SINTAL-EKO D.O.O., LITOSTROJSKA CESTA 38, 1000 LJUBLJANA	993	22	13
CIGALE & CO. D.O.O., GORIŠKA ULICA 63, 1000 LJUBLJANA	1996	0	54
MINISOPH, D.O.O., LJUBLJANA, GOSPOSKA ULICA 3, 1000 LJUBLJANA	80	0	0
FIDELIS D.O.O. LJUBLJANA, ZIHERLOVA ULICA 39, 1000 LJUBLJANA	138	0	101
KRIČKA ANA-IVANA S.P. UPRAVLJANJE S STAN. ZGRADBAMI, CESTA V MESTNI LOG 38, 1000 LJUBLJANA	364	0	262
SEGA IGOR, ČERNETOVA ULICA 26, 1000 LJUBLJANA	10	0	0
JUSTIN GABRIJEL, FLANDROVA ULICA 10, 1210 LJUBLJANA – ŠENTVID	8	0	0
SKUPNOST LASTNIKOV STANOVANJ LINHARTOVA 66, LINHARTOVA CESTA 66, 1112 LJUBLJANA	112	0	0
LIKAR IN SIN D.O.O., EFENKOVA CESTA 61, 3320 VELENJE	78	0	78
KATIN D.O.O., ROJČEVA ULICA 26, 1000 LJUBLJANA	25	0	25
SO – 06 SMOLIČ STANISLAV D. N. O., JAKČEVA ULICA 2, 1000 LJUBLJANA	167	0	167
ARTTEL D.O.O., STANEŽIČE 2, 1210 LJUBLJANA – ŠENTVID	821	0	345
AGING 1, D.O.O., DUNAJSKA CESTA 109, 1000 LJUBLJANA	66	0	0
DIMIC PETER, ULICA STANETA SEVERJA 2A, 1000 LJUBLJANA	12	0	0
SKUPNOST LASTNIKOV BRATOVŠEVA PLOŠČAD 35 IN 37, BRATOVŠEVA PLOŠČAD 37, 1000 LJUBLJANA	116	0	0
ZORKO D.O.O. LJUBLJANA, AGROKOMBINATSKA CESTA 12, 1000 LJUBLJANA	8	0	0
DEAN DARABOŠ S.P., NOVE FUŽINE 33, 1000 LJUBLJANA	429	405	0
SIMON ZORE S.P., KRANJSKA CESTA 4, 4240 RADOVLJICA	26	26	0
IZHOD D.O.O. LJUBLJANA, PERIČEVA ULICA 13, 1000 LJUBLJANA	155	0	25
MMC & CO. D.O.O. LJUBLJANA, TABOR 9, 1000 LJUBLJANA	499	0	159
SKUPNOST LASTNIKOV HIŠE MUCHERJEVA 3, MUCHERJEVA ULICA 3, 1000 LJUBLJANA	16	0	16
DONOR D.O.O., BRILEJEVA ULICA 14, 1000 LJUBLJANA	117	5	20
SPO D.O.O., BLAŽEVA ULICA 3, 4220 ŠKOFJA LOKA	65	11	0
DOP D.O.O. LJUBLJANA, VOJKOVA CESTA 50, 1000 LJUBLJANA	150	0	0
SKUPNOST ETAŽNIH LASTNIKOV VOGELNA 8 LJUBLJANA, VOGELNA ULICA 8, 1000 LJUBLJANA	112	0	0
SKUPNOST LASTNIKOV STANOVANJ LINHARTOVA 62, LINHARTOVA CESTA 62, 1000 LJUBLJANA	112	0	0
ZVEZDA 18-21 D.O.O., ZVEZDA 18, 1210 LJUBLJANA – ŠENTVID	92	0	92
MAHIČ JASMIR, PAPIRNIŠKA POT 2, 1261 LJUBLJANA – DOBRUNJE	40	0	0
STANISLAV TAJČMAN S.P. UPRAVLJANJE STAN. HIŠ, PRVOMAJSKA ULICA 10, 1000 LJUBLJANA	120	0	0
SKUPNOST LASTNIKOV BRATOV UČAKAR 108/110, ULICA BRATOV UČAKAR 110, 1000 LJUBLJANA	96	0	96
LAPUH MILAN, VOJKOVA CESTA 11, 1000 LJUBLJANA	16	0	0
ODA NEPROFITNA STANOVANJSKA ZADRUGA Z.O.O., DUNAJSKA CESTA 101, 1000 LJUBLJANA	112	0	0
B TECH D.O.O., CESTA ANDREJA BITENCA 68, 1000 LJUBLJANA	43	31	0
MILAVC ZDENKO, VRHOVNIKOVA ULICA 11, 1000 LJUBLJANA	18	0	0
DROFENIK D.O.O., STEGNE 7, 1000 LJUBLJANA	250	202	0
LUTOVAC PLUS LJUBLJANA, IGRIŠKA ULICA 3, 1000 LJUBLJANA	18	0	0
MARIJA ZALOŽNIK S.P., CESTA V MESTNI LOG 38, 1000 LJUBLJANA	116	106	0
PINACEA D.O.O., MORAVČE PRI GABROVKI 64, 1274 GABROVKA	82	0	0
DOMA, D.O.O., GROSUPLJE, ZALOG PRI ŠKOFLJICI 2, 1291 ŠKOFLJICA	80	80	0
HAAD LJUTIČ&CO. K.D., KAŠELJSKA CESTA 121, 1260 LJUBLJANA – POLJE	12	0	0
ŠANJUG BARBKA, MAJDE VRHOVNIKOVE ULICA 22, 1000 LJUBLJANA	48	0	0
STANOVANJSKA ZADRUGA NAŠA HIŠA Z.B.O., JAKČEVA ULICA 39, 1000 LJUBLJANA	154	0	154
STUPAR STEVO, ŠMARTINSKA CESTA 15, 1000 LJUBLJANA	10	0	0
STANOVANJSKA ZADRUGA MARIBOR Z.O.O., KREKOVA ULICA 18, 2000 MARIBOR	32	0	0
BRANDT UPRAVLJANJE D.O.O., POLJANSKI NASIP 32, 1000 LJUBLJANA	16	0	0
KOMMUNIO D.O.O., METELKOVA ULICA 7B, 1000 LJUBLJANA	39	39	0
PRIMUS PROJEKT D.O.O., SREDNJEVAŠKA ULICA 85, 1291 ŠKOFLJICA	54	6	0
PRIMOS D.O.O. LJUBLJANA, SVETČEVA ULICA 14, 1000 LJUBLJANA	8	0	0
ISIS D.O.O., SLOVENSKA CESTA 17, 1000 LJUBLJANA	20	0	0
PECHER ANDREJ, VIDEMSKA ULICA 1, 1000 LJUBLJANA	6	0	0
DOLHAR MARIJA, FLANDROVA ULICA 8, 1210 LJUBLJANA – ŠENTVID	10	0	0
SZ GLINŠKOVA PLOŠČAD 22 Z.O.O. LJUBLJANA, GLINŠKOVA PLOŠČAD 22, 1000 LJUBLJANA	50	0	50
FINISTUS – BLATNIK&BLATNIK D.N.O, BEBLERJEV TRG 3, 1000 LJUBLJANA	190	0	0
BINO D.O.O., JAKČEVA ULICA 43, 1000 LJUBLJANA	308	0	0
PETAN JANEZ, KREMŽARJEVA 10, 1000 LJUBLJANA	6	0	0
DOM D.O.O., DELPINOVA ULICA 8, 5000 NOVA GORICA	20	0	0
MEZEK JANEZ, AGROKOMBINATSKA CESTA 4, 1000 LJUBLJANA	44	0	0
TERČELJ ANDREJ, TRUBARJEVA CESTA 64, 1000 LJUBLJANA	8	8	0
ELDERA, D.O.O., MEDENSKA CESTA 81E, 1000 LJUBLJANA	77	0	17
DOM SISTEMI D.O.O., ULICA LOJZETA HROVATA 3A, 4000 KRANJ	60	60	0
NATUM D.O.O. LJUBLJANA, CESTA V PODBORŠT 10, 1231 LJUBLJANA – ČRNUČE	10	0	0
UPIS D.O.O., DUNAJSKA CESTA 437, 1291 ŠKOFLJICA	5	5	0
SKUPNOST LASTNIKOV RUSJANOV TRG 9, RUSJANOV TRG 9, 1000 LJUBLJANA	558	0	0
KITAK, SI.INVEST K.D., LJUBLJANA, BRODARJEV TRG 7, 1000 LJUBLJANA	7	7	0
SREČKO KITAK S.P., BRODARJEV TRG 7, 1000 LJUBLJANA	30	0	0
SKUPNOST LASTNIKOV TRNOVSKA ULICA 8, TRNOVSKA ULICA 8, 1000 LJUBLJANA	51	0	51
ŽAVBI VINCENC, TOPNIŠKA ULICA 14, 1000 LJUBLJANA	26	0	26
HORVAT VLADIMIR, FLANDROVA ULICA 6, 1210 LJUBLJANA – ŠENTVID	16	0	0
ATAC D.O.O. LJUBLJANA, DUNAJSKA CESTA 104, 1000 LJUBLJANA	8	0	0
SKUPNOST ETAŽNIH LASTNIKOV STANOVANJSKE STAVBE SMOLETOVA 18, LJUBLJANA, SMOLETOVA ULICA 18, 1000 LJUBLJANA	98	0	0
SKUPNOST LASTNIKOV GORNJI TRG 20, GORNJI TRG 20, 1000 LJUBLJANA	14	0	0
SKUPNOST ETAŽNIH LASTNIKOV STANOVANJSKE STAVBE VOGELNA 10 LJUBLJANA, VOGELNA ULICA 10, 1000 LJUBLJANA	54	0	54
K”ITAK D.O.O., BRODARJEV TRG 7, 1000 LJUBLJANA	34	0	0
BLAZNIK VOJKO, FLANDROVA ULICA 7, 1210 LJUBLJANA – ŠENTVID	8	0	0
SKUPNOST LASTNIKOV, PAVŠIČEVA 6 LJUBLJANA, PAVŠIČEVA ULICA 6, 1000 LJUBLJANA	152	0	0
SKOMINA MARIJA, TRŽAŠKA CESTA 65, 1000 LJUBLJANA	9	9	0
UNIH D.O.O., FAJFARJEVA ULICA 33, 1000 LJUBLJANA	64	64	0
BOJAN GLAVAN S.P., VIKRČE 22, 1211 LJUBLJANA-ŠMARTNO	84	0	0

by virostatiq - August 19, 2015August 19, 2015

Could computers vote instead of parliamentary representatives?

Haha, what a funny question. Of course they can’t. How can one teach a computer all the intricacies of lawmaking process, and trust it well enough to let it vote? This must surely be a recipe for disaster.

Yet, as I realized in previous research, the parties mostly demand ruthless discipline from their parliamentary representatives at voting time, simply to be able to actually govern in Slovenian multiparty democracy, where there’s never an absolute winner. This leads to coalition governments, where every vote counts towards a majority.

That means that in a polarized parliament, one could theoretically predict a representative’s vote by examining the votes cast by all other representatives. If an opposition party proposes to vote on an act, it’s very likely that members of government block will uniformly, or at least predominantly, vote against it, and vice versa. There are few exceptions to that rule, namely some profoundly ethical decisions, in which majority parties will let their members vote by conscience. But they are few and far apart.

Fun with neural networks

I decided to test this out by modeling some representatives by neural networks, and training the networks with a few voting sessions and their outcomes in the beginning of the parliamentary term.

Model for each representative was fed votes by every other rep except him- or herself as input, and his or her vote as desired output. This was repeated and repeated again for all hundred training sessions, until the model converged (loss fell under 0.05).

It was then shown voting sessions iz hasn’t seen yet, and tasked to predict the outcomes.

The results are shown in images below. For each representative, the image contains:

name and party,
training vector (the votes he/she cast in first 100 voting sessions – red for “against”, blue for “in favor”, yellow for absence for whatever reason),
actual votes (400 votes the network hasn’t seen and was trying to predict),
predicted votes (how the neural network thought the representative would vote), and
difference indicator (with red rectangles for wrong prediction, green rectangles for correct prediction, and yellow rectangles for absence)

I didn’t bother too much with statistics, to see who was the most predictable, neither did I try to predict voting for every rep.

In short, those with the mainly green bottom strip were the most predictable.

Government coalition

DeSUS

SMC

Opposition

NSi

IMNS

SDS

ZAAB

A cursory examination of results yields several realizations:

even in best predictions with lowest error rate, the model doesn’t predict absences well, especially for representatives with low incidence of absence in training data. This is intuitively understandable on two levels: first, it’s hard for the network to generalize something it didn’t observe, and second, absences can happen on a human whim, which is unreachable for a mathematical model. For representatives of opposition parties, who frequently engage in obstruction as a valid tactics, the model fares a little better.
the model predicts best the voting behavior of majjority party (SMC) members.
the model utterly fails to predict anything for representatives whowere absent in training period (duh).

So, could we substitute the actual representatives with simple neural networks? Not with this methodology. The problem is that we need votes of everyone else in the same session to predict the vote of modeled rep, so at the time of prediction, we already have their vote. We don’t have a way of inferring votes from scratch, or from previous votes.

We could, in theory, try to predict each rep’s vote independently from others by training the network on proposed acts’ texts. I speculate that a deeper network could correlate vectorized keywords in training texts with voting outcomes, and then be able to predict voting for each rep independently based on previous unseen texts. Maybe I’ll do that when I get the texts and learn a bit more. It’s still ANN 101 period for me.

I used a simple perceptron with 98 inputs (there have been 99 representatives in this term, counting also current ministers and substitutes), a hidden layer of 60 neurons, and a softmax classifier on the end.

As usual, I used Karpathy’s convnetjs for modeling, and d3 for visualization. Dataset comes from Zakonodajni monitor.

by virostatiq - August 19, 2015August 19, 2015

Grouping entries in Slovenian Wikipedia by contributors

Some time ago, I helped Miha Mazzini extract some data from Slovenian Wikipedia. For that, I needed to write a comprehensive parser, extracting not only titles and text, but also number of overall and per-contributor revisions, along with contributor usernames.

So, for each entry, I got a list of contributing accounts and number of edits that were performed by that account. I wondered: how are the areas of expertise distributed among all those contributors? Are some of them specialized mainly for science, some others for politics, and so on? And, perhaps more interestingly, where do these areas overlap? Do we have experts for sport, who also happen to curate political entries?

To find out, I extracted all entries with more than 25 edits, vectorized them by contributing accounts, and ran t-SNE to cluster them spatially and prepare the visualization. When t-SNE layout was complete, k-means clustering was run on the x,y coordinates only to be able to distinguish those areas by color. It must be said that these colored groups don’t always coincide with semantic grouping, so take it with a grain of salt. It’s there mainly to make the map look better and to improve legibility. Font size is in proportion with number of revisions that an entry had so far.

Here’s the entire map as one big 9000 x 6000 image. Click the image to display, then zoom into it with mouse. You can find cropped clusters and some commentary below.

Wikipedia entries by contributing authors – entire map (click for enlargement)

Turns out the areas of expertise are pretty well delineated.

Let’s look at some clusters. Here we have some geographic Wikipedia entries, mainly countries and some historical persons. That sounds logical – editing an entry about a great ruler probably causes one to contribute to an entry about his or her country.

Here are some famous Slovenian people, mostly writers, intermixed with some towns. In the lower left quadrant, there are alo some Slovenian politicians. It sounds funny that the late Communist ruler Josip Broz Tito is so close to Janez Janša, who is the current leader of right-wing opposition. It appears that there is a number of people who edit both entries. I wonder why. Here’s an article by Miles Mathis about editing of Wikipedia. I don’t know what to think of it, but I surely read a lot about autocratic rule of (English) Wikipedia editors to give it some credence. I don’t know about Slovenian version, and I don’t want to speculate, but this is as good an opportunity as any to start thinking about it.

Here is a cluster of lists. It seems that there exists an entire group of people who curate them, regardlessly of their content.

Here’s a funny cluster dealing mostly with public transit in Slovenia. It almost seems that there are some bureaucrats in the government that edit these entries on taxpayer dime. I could probably find that out, if I traced the IPs in the edit logs. If someone hires me as an Internet detective, I might do that, but I made these pictures for fun.

This is an interesting cluster. It appears that many same people edit entries about Euroviviion Contest, parliamentary elections, World cup in basketball and World cups in skiing, along with two new parties in Slovenian parliament: ZAAB, which is the remainder of majority party in the last parliamentary term, and SMC, which is a new majority party. Both parties, along with Pozitivna Slovenija (former majority party) were founded hastily right before elections, and won them by a big landslide. I wonder how would political analysts comment on their (speculatively) members’ love for sports contests.

Here we have many religious personalities, mostly many popes.

A grouping of entries about Slovenian popular music.

Some entries about historical scientists and natural sciences.

More geographical entries, along with some entries about Slovenian highways.

Here’s the center of the map. It would follow logic that entries with many non-specialized contributors are drawn towards it. It’s generally more chaotic that the outskirts, but here are great many contemporary and historical art personalities grouped together..

Another snapshot from the center, mostly consisting of entries about worker’s rights and things related to work.

So here it is. For the technically minded – everything was done in JavaScript with Andrej Karpathy‘s tsnejs library, clusterfck for k-means, and d3 for drawing.

I also made an inverted map, on which the contributors were shown, grouped by the entries they made revisions to. It’s not so interesting for general public, but if someone wants to see it, it’s available by request.