February 2016 - Virostatiq

by virostatiq - February 3, 2016

Mapping parking infractions in Manhattan, NYC, by car make

This is a technical explanation of procedure to map parking infractions in Manhattan for every available car make. To see the interactive visualization, click here, or click the image below. Otherwise read on.

Last year I published an Android app to enable Slovenian drivers to better avoid areas frequently inspected by parking wardens. It works by geolocating the user and then plotting issued paring tickets in the vicinity, with a breakdown by month, time of day and temperature on another screen. It was not a huge hit, but it did reasonably well for such a small country and no marketing budget.

I was thinking of making a version for New York City, but then abandoned the project. These visualizations are all that remains of it.

I started with downloading the data from New York Open Data repository. It’s here. The data is relatively rich, but it’s not geocoded. Luck had it that Mapbox just rolled out a batch geocoder at that time, and it was free with no quotas. So I quickly sent around 100,000 adresses through it and saved the results in a database for later use. The processed result is now available on Downloads page in form of JSON files, one per car make.

The actual drawing procedure was easier than I thought. I downloaded street data from New York GIS Clearinghouse and edited out everything but Manhattan with QGis.

First I tried a promising matrix approach, but I was unable to rotate the heatmap so that it would make sense. Here’s an example for Audi:

As you can see, it is a heatmap, but doesn’t look very good.

So I wrote a Python script that went through all street segments and awarded a point if there was an infraction closer that 100 meters from the relevant segment. Then I just used matplotlib to draw all the street segments, coloring them according to the maximum segment value.

A result for Audi now looks like this:

All that remained was drawing required images for animated GIFs, each for every hour for every car make. This was done with minimal modifications to original script (I learned Python multithreading in the process). The resulting images were then converted to animated GIFs with ImageMagic.

The whole procedure took approximately 12h of calculating and rendering time on a i7-6700 with 32 GB RAM. I guess I could shave several hours from that time, but I just let it run overnight.

See interactive version here, and tell me what you think in the comment section, if you feel like it.

by virostatiq - February 1, 2016

Books I read in 2015

This is a result of

find *.mobi  -newermt "2015-01-01 00:00"

command in my Kindle Documents folder. If anything, it shows I’m a compulsive reader, preferring Nordic crime novels, cyberpunk sci-fi and historical and natural sciences non-fiction. I probably read ten more programming-oriented books in PDFs, which I’m not including here.

My favorites were novels by Don Winslow and Richard K. Morgan. I also read most of the essays by Miles Mathis, about which I’m still forming an opinion.

127 books in all, in no particular order:

Acid Dreams – Martin A. Lee
A Killing Winter – Tom Callaghan
Altered Carbon – Richard K. Morgan
A Nasty Piece of Work_ A Novel – Robert Littell
Angelica’s Smile – Andrea Camilleri
Anne Holt – What is Mine (Punishment)
A Noble Killing – Barbara Nadel
Artificial Intelligence for Humans, Volume 2- Heaton, Jeff
Artificial Intelligence for Humans, Volume 3 – Jeff Heaton
Aurora – Kim Stanley Robinson
Autumn Killing_ A Thriller – Mons Kallentoft
Axis – Robert Charles Wilson
Beyond Words_ What Animals Think and Feel – Carl Safina
Black Man – Richard K. Morgan
Blackwater – Kerstin Ekman
Blessed Are Those Who Thirst – Anne Holt
Blind Goddess_ A Hanne Wilhelmsen Novel – Anne Holt
Blood on Snow_ A Novel – Jo Nesbo
Broken Angels – Richard K. Morgan
Burned – Thomas Enger
Cibola Burn_ Book Four of the E – James S. A. Corey
Closed for Winter – Jorn Lier Horst
Cold Hearts – Gunnar Staalesen
Complicity – Iain Banks
Consider Phlebas – Iain M. Banks
Dark Secrets – Michael Hjorth
Deadline – Barbara Nadel
Dead of Night – Barbara Nadel
Death by Design – Barbara Nadel
Death of the Demon_ A Hanne Wilhelmsen N – Anne Holt
Destination Void – Frank Herbert
Dope, Inc. Britain Opium War against the U – LaRouche
Drug War Capitalism – Dawn Paley
Epiphany of the Long Sun – Gene Wolfe
Excession – Iain M. Banks
Feersum Endjinn – Iain M. Banks
Inspector of the Dead – David Morrell
Inversions – Iain M. Banks
Killing Pablo – Mark Bowden
Legends – Robert Littell
Litany of the Long Sun – Gene Wolfe
Look to Windward – Iain M. Banks
Magicians of the Gods_ The Forg – Graham Hancock
Mao – Jung Chang
Marid Audran 01 – When Gravity Fails – George Alec Effinger
Marid Audran 02 – A Fire in the Sun – George Alec Effinger
Marid Audran 03 – The Exile Kiss – George Alec Effinger
Market Forces – Richard K. Morgan
Matter – Iain M. Banks
Midnight Sun_ Blood on Snow 2 – Jo Nesbo
Mirrors – Eduardo Galeano
Mother Russia – Robert Littell
Murder as a Fine Art – David Morrell
Nemesis Games – James S. A. Corey
Pierced – Thomas Enger
Poseidon’s Wake – Alastair Reynolds
Pretty Dead Things – Barbara Nadel
Satori – Don Winslow
Savage Continent – Lowe, Keith
Scarred_ A Novel – Thomas Enger
Seveneves – Neal Stephenson
Shadow and Claw – Gene Wolfe
Shibumi – Trevanian
Silenced – Kristina Ohlsson
Smilla’s Sense of Snow – Peter Hoeg
Snowblind – Jonasson, Ragnar
Spin – Wilson, Robert Charles Charles
Spring Tide – Borjlind, Cilla
Summer Death_ A Thriller – Mons Kallentoft
Superintelligence – Nick Bostrom
Sweet Reason – Robert Littell
Sword and Citadel – Gene Wolfe
The Abrupt Physics of Dying – Paul E. Hardisty
The Algebraist – Iain M. Banks
The Cartel_ A Novel – Don Winslow
The Caveman – Jorn Lier Horst
The Cold Commands – Richard K. Morgan
The Company – Robert Littell
The Consorts of Death – Gunnar Staalesen
The Dark Defiles (Land Fit for Heroes #3) – Richard K. Morgan
The Dosadi Experiment – Frank Herbert
The Eiger Sanction – Trevanian
The Ethnic Cleansing of Palestine – Ilan Pappe
The Fifth Profession – David Morrell
The Fire Witness_ A Novel – Lars Kepler
The Girl in the Spider’s Web (Millennium s – David Lagercrantz
The Gulag Archipelago, Volume 1_ An Expe – Solzhenitsyn, Aleksandr
The Gulag Archipelago, Volume 2_ An Expe – Solzhenitsyn, Aleksandr
The Hanging Girl – Jussi Adler-Olsen
The Healer – Antti Tuomainen
The House of the Scorpion – Farmer, Nancy
The Hydrogen Sonata – Iain M. Banks
The Intruder – Hakan Ostlundh
The Jesus Incident – Frank Herbert
The Lady from Zagreb – Philip Kerr
The Life-Changing Magic of Tidying Up_ The – Kondo, Marie
The Loo Sanction – Trevanian
The Lord of Opium – Farmer, Nancy
The Master Algorithm – Pedro Domingos
The New York Trilogy – Paul Auster
The Nightmare_ A Novel – Lars Kepler
The Once and Future Spy – Robert Littell
The Patience of the Spider – Andrea Camilleri
The Player of Games – Iain M. Banks
The Power of the Dog – Don Winslow
The Real Odessa – Goni, Uki
The Shape of Water – Andrea Camilleri
The Snack Thief – Andrea Camilleri
The State Of The Art – Iain M. Banks
The Steel Remains – Richard K. Morgan
The Summer of Katya – Trevanian
The Urth of the New Sun – Gene Wolfe
The Viper – Hakan Ostlundh
The White Plague – Frank Herbert
The Winter of Frankie Machine – Don Winslow
The Writing on the Wall – Gunnar Staalesen
Third Voice – Borjlind, Cilla
Thirteen – Richard K. Morgan
Vicious Circle – Robert Littell
Visiting Professor_ Novel of Chao – Robert Littell
Vortex – Robert Charles Wilson
We Shall Inherit the Wind – Gunnar Staalesen
‘What Do You Care What Other People Think_ – Richard P Feynman
Whipping Star – Frank Herbert
Woken Furies – Richard K. Morgan
Getting MEAN – Simon Holmes
Yours Until Death – Gunnar Staalesen