The European Digital Treasures team continues with the presentations of the experts who participated in the workshop “New Digital Exponential Technologies Towards The Generation Of Business Models” on 2nd and 3rd of September, 2021 at the Provincial Historical Archive of Alicante (Spain).
The second speech was held by Artem Reshetnikov who is a deep learning researcher at Barcelona Supercomputing Center. Working in several companies and research centers, he received big experience in Computer Vision and Natural Language Processing and applying it to the tasks of different domains. For a long time, he was thinking about how to combine his two main passions: machine learning and art. The solution is the project where he works now: Saint George on a Bike is a project about the enrichment of metadata of paintings using Deep Learning and NLP approaches.
Abstract.“Saint George on a Bike” project proposes several novel approaches to enrichment of metadata (captions, tags, relationships between objects, iconographic description) for the Cultural Heritage domain, which relies on combining Deep Learning and semantic metadata about paintings. Working with cultural heritage presents challenges not existent for every-day images. Models for objects detection or caption generation are usually trained with datasets that contain correct descriptions of current images or labels for objects, which were generated manually. Apart from this conceptual problem, the paintings are limited in number and represent the same concept in potentially very different styles. Finally, the metadata associated with the images is often poor or inexistent, which makes it hard to properly generate quality metadata. Our approach can assist in generation of metadata for different tasks. By taking into account an exiting metadata of Cultural heritage objects and additional techniques, we can generate tags, relationships between objects or descriptive text which is likely to be directly related to the scene depicted in an image.
You can watch the whole session on YouTube here and read the manuscript paper here!
Written by Artem Reshetnikov & the European Digital Treasures Team.
