Tagged: processing

Presence of faces in House of Cards TV Series by episode

Share Button

I was wondering if presence of faces in video content was an indicator of anything, and if so, of what. So I decided to scan episodes of a popular TV series and analyze them, second by second, for number of faces in video frames, and then compare charts of various episodes. Here is the result of this research.

I decided to analyze House Of Cards, partly because it’s a great series, but also because it’s character focused, so there are many scenes with a lot of people. I built an interactive viewer, which allows to see which faces were recognized at a particular point in time in Episode 3, which contains a variety of scenes with many people in them.

Launch the viewer, or continue reading for short description of technology.

LAunch the House of Cards Face Recognition Interactive Viewer

Technology

To pull this off, I used the OpenCV computer vision library, which has a good capability to recognize faces. As the computer watches TV, this tool scans every frame for faces, and, if it finds any, communicates the relevant rectangles, so they can be drawn or extracted and saved.

Here’s a screenshot of a scene in church. It’s immediately apparent that the tool does not do such a good job, for many faces remain unrecognized. Still, many are recognized.

Recognized faces in House of Cards

Recognized faces in the church scene, Episode 3

In this frame below, more faces are recognized.

hoc-0201

There are also many false positives. The computer sometimes thinks that something is a face, where it most certainly it’s not, as in this picture below. If one looks carefully, one can sometimes see something face-like in these rectangles.

hoc-0018

To construct the viewer, I extracted individual faces from frames so I could display them on the page. They are of various sizes and look like this:

1 0 2 0 1 0

To construct the charts, I just counted the faces in each seconds, then displayed the time series for each episode.

Results

This is the final chart. It’s a series of timelines that show how many faces were recognized per second. Why are some lines orange, and some yellow?

As video frames scanning progressed, some faces were recognized in only one frame in entire second – there are 23 of them. Some other faces were recognized in more frames, ans others in yet more frames. I thought this to be a good indicator of face detection reliability, but that’s not so. If it tells anything, it’s how steady the camera was in that section.

House of Cards face recognition charts by episode
House of Cards face recognition charts by episode

My inspiration was small multiples, a visualization technique which allows for easier comparison of several datasets from the same domain. Wikipedia says:

A small multiple (sometimes called trellis chart, lattice chart, grid chart, or panel chart) is a series or grid of small similar graphics or charts, allowing them to be easily compared. The term was popularized by Edward Tufte.

According to Tufte (Envisioning Information, p. 67):

At the heart of quantitative reasoning is a single question: Compared to what? Small multiple designs, multivariate and data bountiful, answer directly by visually enforcing comparisons of changes, of the differences among objects, of the scope of alternatives. For a wide range of problems in data presentation, small multiples are the best design solution.

 

As always, if anyone is interested in code, mail me. My address is on About page.

Slovenian business activity by city as animated heatmaps

Share Button

A few months ago, while researching business times of various categories of establishments in Slovenia,  I thought it would be nice to somehow visualize a map with a graphical representation of density of open establishments. I decided on heatmap style, although I later discover that my chosen implementation had some drawbacks.

Getting the data

Data with business hours of commercial establishments is traditionally not open for many reasons, two of them being that (1) this information can be commercially exploited, and (2) the opening hours can be subject to frequent changes, which can tax the database owner with considerable effort should the database stay current and reliable.

First I toyed with the idea of crawling entire  directory of odpiralnicasi.com, then I actually thought about making a version for London, Amsterdam or San Francisco with Yelp data, for which I would have to crawl an entire Yelp city directory, a task I’m not sure it would succeed. Yelp would probably block my IP before I could harvest a significant portion of what interested me.

So I decided I would use the Najdi.si maps business directory. Disclosure: I work there, so I have access to the database with various business data, which is being kept current.

For every company, I took out only the name, geo coordinates, business hours and business category, then I constructed the animated maps. Before I delve into that, a short video of economic activity in Slovenia in course of a typical Monday.

Economic activity in Slovenia from Marko O’Hara on Vimeo.

The animated chart you see on the bottom shows the number of active establishments in various economic categories, such as Restaurants and catering, Industry, Shopping, etc. The full list is:

  • blue: Computers and IT,
  • red: Restaurants and catering,
  • green: Home and garden,
  • yellow: Beauty and health,
  • pink: General business,
  • orange: Free time,
  • violet: Industry,
  • magenta: Culture and schooling

Rendering the maps and constructing the visualization

Rendering one frame in one city at a specific time is just a matter of setting appropriate latitude, longitude and zoom level on the map, selecting the desired time and plotting on the map all establishments  that are open at that time. I used Processing to do that, and for the heat map part I used this excellent example by Philipp Seifried. As a finishing touch, I made maps to switch between day and night styles at appropriate times.

To do entire video, I had to write a parallel rendering queue lest the rendering of a single video took an eternity – Eclipse project available by email request.

To complicate things a bit I decided to include up to four different places on the same map, so the viewer could compare opening hours in Ljubljana in different economic categories, or see how different cities woke up and went to sleep at different times.

A typical frame looks like this:

Video frame / comparison of business activity in Nova Gorica, Koper, Celje and Novo Mesto at noon
Video frame / comparison of business activity in Nova Gorica, Koper, Celje and Novo Mesto at noon

Here’s an example for different economic activities in Ljubljana:

Economic activity in Ljubljana – four categories from Marko O’Hara on Vimeo.

  • top left: General business
  • top right: Restaurants and catering
  • bottom left: Industry,
  • bottom right:Beauty and health

Here’s a comparison between Ljubljana and the city of Maribor:

opentimes ljmb.mp4 from Marko O’Hara on Vimeo.

  • left: Ljubljana
  • right: Maribor

And here a comparison of business activity in Nova Gorica, Koper, Celje and Novo Mesto:

opentimes kpnmceng.mp4 from Marko O’Hara on Vimeo.

 

  • top left: Nova Gorica
  • top right: Koper
  • bottom left: Novo Mesto,
  • bottom right:Celje

 

Commentary

I mostly did this to be able to visually compare levels of business activity in Ljubljana. First of all, the heatmap technique I employed here turned out to be somewhat unreliable for video purposes, because it colors the dots relative to the highest concentration. But concentration and absolute numbers of active businesses change from frame to frame, so it seems that at night there’s more activity that during the day.

Even so it’s still clear that restaurants, bars and clubs are still pretty much open when other activity starts to die down.

This is Ljubljana at noon, again:

  • top left: General business
  • top right: Restaurants and catering
  • bottom left: Industry,
  • bottom right:Beauty and health

The big spot in the northeast is the mall region, where untold number of business operate in ten or more big malls. Business concentration there dwarfs everything else in the city, except maybe in industrial category.

lj at 11 h

Below is Ljubljana at eight o’clock in the evening. Pretty much everything has closed down except for eating and drinking, and maybe the cinema theater in the mall.

lj20h

Below: Ljubljana at ten o’clock in the evening. Some businesses don’t close down at all. I double checked the primary data source and it’s true. There are cleaning services that stay open during the night, etc.

lj22h

I’m relatively satisfied with results except for the heatmap issue. I may correct that if I get the data for a bigger city.

Building ages in Ljubljana, Slovenia

Share Button

Such is the beauty of open data that when I saw the excellent Portland: The Age of a City by Justin Palmer, I immediately wanted to do something similar, but for my town. The people at the government office (GURS) were kind enough to provide me with the files, and after some coding, here it is.

It’s an exploration of how the city grew through the last century. Blue is old, violet younger, res still younger, bright red the youngest.

Launch the interactive map showing structure ages in Ljubljana

ljubljana-ages

Here’s the number of structures built by years. I was able to identify causes for some spikes in building activity, but not all:

  • 1899: four years after the big earthquake,
  • 1919: rebuilding after WW1? I’m not sure there was much destruction here,
  • 1929: more building – in 1929 Ljublaana became the capital of Dravska banovina,
  • 1949: rebuilding after WW2,
  • 1959, 1969, 1979, 1989: might be effects of Yugoslav loans, but I suspect it’s more of an effect of administrative laziness, resulting in entering new buildings into evidence at the end of each decade,
  • 2004: the last surge of prosperity in independent Slovenia.

Generally, it’s been going downhill from 1969 on. The best spots were probably taken by then.

ages-chart

Here’s a animation of the whole thing. It shows city evolution between years 1500 and 2013, since there’s not much happening before that.

City of Ljubljana – growth between years 1500 – 2013 from Marko O’Hara on Vimeo.

Map was made with TileMill, animation in Processing.

See also the real estate prices map.

 

My heart rate during the latest episode of Game Of Thrones

Share Button

See also: My brainwaves while watching the final episode of Breaking Bad and other TV visualization, Number of faces per second in episodes of House of Cards.

I admit that I like Game Of Thrones. I like the story and the way the TV show is done. So when I read on Monday about the supposedly heartbreaking episode “Rains Of Castamere”, in which Robb Stark, his wife and his mother get killed along many others, I thought I’d test my affection by recording my pulse while watching, then plotting it to see how it correlated to the visual narration.
Here it is:

gameOfThrones

To do that, I used my old Arduino and PulseSensor to record pulse, then graphed the averages on a picture along with corresponding frames using Processing. There are some mistakes when the sensor grabbed double pulse when I jumped or moved, but it averaged decently.

If you saw the episode, you’ll probably relate.

Update: if anyone wants the code (Processing / Arduino) send me an email. My address is on About page.

Noise pollution caused by church towers in Ljubljana

Share Button

One Sunday I woke up to incessant and very loud tolling of nearby church bell. It was 9 o’clock in the morning. It didn’t seem fair that an institution can cause so much noise so early. As I work hard during the week, run almost every day, and write software, sometimes until late, I would very much prefer to sleep. The clergy would probably say that honest Christians are already awake at that time, so I’m no good anyway.

I then decided to research the matter. A number of facts surfaced, the most startling of which is a state decree, which states that church bells are not categorized as noise. If an inspector came to my house, measured sound levels while this was going on, and found out that they exceed proscribed levels, he would not be able to fine the aforementioned institution. He would probably bill me for the expenses of his time. But I digress.

Action was taken: city geometry was imported into computer along with church bell coordinates. Aggregate sound pressure for each building was calculated, then ranged so it could be visualized. Additionally, a point where there is least such noise was calculated. You can see results below. The point with least noise is on the green marker in the lower left corner. Lucky owner of that house.

Note: please notify me before embedding this map in your page.

I have to admit that the calculation is naive. It doesn’t take into account the elevation model, neither it accounts for building heights. Sound reflection is also ignored. But my curiosity was satisfied. I do live in the red zone.

Here are same maps on different scales. One is for entire country of Slovenia.

Edit: after this post went viral and other media (Dnevnik.si) published their own versions linking to me, I feel compelled to clarify my position about church bells. Personally, that is, as a person, and not a member of any organization, I’m bothered by long intervals of loud tolling on Sunday mornings. I’m told by other people they don’t like that either, and some other people point out that any attempt at playing music at this volume at similar hour of day would not end well.

I do somewhat like single chimes announcing hours of day, even at night. It’s a part of urban environment, and I’d probably subconsciously miss it should they quit. I’m not against Catholicism, the Church, or faith of any denomination.

When you toll so loudly next time please consider:

  • do unto others as you would have them do to you,
  • would Christ approve of that?

Thanks.