Computer Vision & More: DH Clinics 2017 (Workshop Report)

The goals of the DH Clinics program include teaching the basics of digital humanities (DH), reinforcing the relationships with the academic DH community, and building a knowledge network (read more). The innovative nature of the program is underlined by the fact that it has already won the LIBER Innovation Award, as well as the Victorine Initiatiefprijs.

In November 2017, we attended the fifth and final day of the clinics, focusing on audiovisual materials in a library and DH context. This workshop took place in the "Eemhuis" in Amersfoort, the Netherlands.

Geographical Information Systems & audiovisual content

In the morning session, Ben Companjen (UB Leiden) introduced CARTO, a geographical information system built on open-source components, followed by an interactive discussion of the value of Geographical Information Systems (GIS) for libraries. Subsequently, Berber Hagedoorn (University of Groningen) introduced audiovisual platforms such as EUScreen. She also discussed workflows of scholars in Media Studies, tool criticism, and how media scholars' needs can be met in the best way.

Computer Vision: understanding images and... solving sudoku

The afternoon program focused on Computer Vision, which "is concerned with the automatic extraction, analysis and understanding of useful information from a single image or a sequence of images." (source (people.cs.bris.ac.uk/~majid/vision/vision.html)). Along with the broader term Artificial Intelligence, computer vision has been one of the buzzwords of recent years. Outside of research and commercial contexts, possibilities of computer vision can also be harnessed by libraries and archives.

Martijn Kleppe (National Library of the Netherlands) introduced the topic of computer vision. He showed some applications of computer vision, for instance used in online tools such as Google Photos (performing face recognition), or even providing instant solutions for Scrabble or Sudoku! Martijn also explained how the developments in computer vision have accelerated in the past years, partially due to annotated image databases such as ImageNet. For more information about these developments, the speaker recommended to watch Fei Fei Li's TED Talk (How we teach computers to understand pictures).

19th Century Railway Crashes

Next up was Thomas Smits, a researcher-in-residence at the Dutch National Library. His research focuses on the large digital newspapers collection of the Dutch KB, and to what extent computer vision can find illustrations of nineteenth-century railway crashes. Thomas described the workflow and data processing pipeline of his project, and provided some insights into the workings of computer vision. Also, he introduced a novel tool developed at the KB to search for images in the newspaper collection. This tool allows you to use custom filters to find images from newspapers, including cartoons, buildings, and so forth.

Visually exploring advertisements

The final speaker, Melvin Wevers, introduced his PhD research related to the portrayal of american cultural values in Dutch newspapers. In his talk, he discussed the role of neural networks in image recognition, and the issues which may occur when applying this to a set of historic newspapers.

Like Thomas, Melvin was a researcher-in-residence at the National Library of the Netherlands. In this research project, he used neural networks to examine newspaper advertisements in the Dutch digitized newspaper collection (which contains around 20 million advertisements!). The project led to a tool, SIAMESE, which can be used to do exploratory visual searching in newspaper advertisements. It is based on a so-called Convolutional Neural Network to identify similar visual trends in advertisements.

Hands-on experience with the tools

After these presentations, we were put to work ourselves. We tried out the SIAMESE tool by entering the ID of an advertisement from a digitized newspaper, immediately retrieving other advertisements with visual similarities. We also could try out the novel Image search query builder tool discussed in Thomas Smits' presentation. This tool makes it possible to do filtered searches for photos and/or drawings.

Finally, we discussed the experiences in using the tools in a group setting. Participants enjoyed the inspiring nature of that these kinds of tools, but also found some improvement points. An interesting question was about the nature of computer vision -- in a way, these tools allows us to "see what the computer sees", which does not always correspond with our intuition of image contents. However, various participants agreed that the possibilities of computer vision would be extremely valuable for libraries' large image collections, for which manual annotation is infeasible.