Rotate your phone to navigate through the site properly

Multimodal Deep Learning in Digital Visual Studies
DARIAH-CH STUDY DAY
Abstract Eva Cetinić & Darío Negueruela

DARIAH-CH STUDY DAY

We are participating in the upcoming Study Day organized by DARIAH-CH, the Swiss node of DARIAH ERIC, with the purpose of bringing together and strengthening the Swiss Digital Humanities community. The gathering will take place in person on 20 October 2022 at the Università della Svizzera Italiana (USI), in Mendrisio.

Abstract Eva Cetinić & Darío Negueruela

Multimodality is inherent to almost all aspects of human perception, communication, and
production of information. However, as a phenomenon, multimodality is particularly important for the epistemological, interpretative and creative processes within art and art history. The notion of multimodal transformation became recently computationally operationalized on a meaningful and very convincing level. The field of multimodal deep learning significantly
advanced in recent years with the introduction of large pretrained visionlanguage models.
Those models made it possible to computationally generate semantically aligned textual descriptions of images, or vice versa, to render images corresponding to textual inputs. Such models are usually not easy to analyze or explain. They comprise huge parametric spaces trained on immensely large datasets, composed of data sampled from the Internet, and therefore often integrate and propagate various biases and dominant worldviews. By encoding numerous associations which exist between data items collected at a certain point in time, those models represent synchronic snapshots of collective data traces, embedded in specific technological, historical and cultural paradigms. Despite seemingly reproducing the known problem of cultural framing, their directive biases are inextricable of their intentionality as cultural objects, and can provide useful perspectival shifts and new hybrid possibilities in research and artistic production. The focus of this research project is to analyze how such models can be employed in the context of studying visual art and culture. Research enthusiasm in multimodal deep learning is rapidly expanding. However, its application in the domain of digital humanities is still scarce at the moment. Therefore, this project might be of interest to the DARIAHCH community, as it addresses questions and methods which are at the forefront of contemporary DH research.

Authors:, Eva Cetinić & Darío Negueruela

0,