MISTI: Metadata-Informed Scientific Text and Image Representation through Contrastive Learning

Can publication metadata improve scientific image and text retrieval?

MISTI learns joint representations of scientific figures, captions, and publication metadata. The study tests whether contextual information such as titles, sections, and concepts improves retrieval beyond the image-caption pair alone.

What the study found

• Including metadata improved retrieval performance over the reported baselines.
• The model reached 30% Recall@1 in a caption-retrieval task with 70,000 candidates.

How the study works

The model uses contrastive learning on figures, captions, and metadata from the PubMed Open Access Subset, and is evaluated on information-retrieval tasks.

Scope and limitations

• The reported results depend on the tasks, candidate set, and publication collection used for evaluation.
• Retrieving a relevant figure or caption is different from establishing that a scientific claim is correct.

Abstract

In scientific publications, automatic representations of figures and their captions can be used in NLP, computer vision, and information retrieval tasks. Contrastive learning has proven effective for creating such joint representations for natural scenes, but its application to scientific imagery and descriptions remains under-explored. Recent open-access publication datasets provide an opportunity to understand the effectiveness of this technique as well as evaluate the usefulness of additional metadata, which are available only in the scientific context. Here, we introduce MISTI, a novel model that uses contrastive learning to simultaneously learn the representation of figures, captions, and metadata, such as a paper’s title, sections, and curated concepts from the PubMed Open Access Subset. We evaluate our model on multiple information retrieval tasks, showing substantial improvements over baseline models. Notably, incorporating metadata doubled retrieval performance, achieving a Recall@1 of 30% on a 70K-item caption retrieval task. We qualitatively explore how metadata can be used to strategically retrieve distinctive representations of the same concept but for different sections, such as introduction and results. Additionally, we show that our model seamlessly handles out-of-domain tasks related to image segmentation. We share our dataset and methods (https://github.com/Khempawin/scientific-image-caption-pair/tree/section-attr) and outline future research directions.

Abstract from the original work, reproduced under its Creative Commons license. The overview above summarizes the study.

Cite this work

Pawin Taechoyotin, Daniel Acuna (2024). MISTI: Metadata-Informed Scientific Text and Image Representation through Contrastive Learning. Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024). https://doi.org/10.18653/v1/2024.sdp-1.15

@inproceedings{taechoyotin2024misti,
  title = {MISTI: Metadata-Informed Scientific Text and Image Representation through Contrastive Learning},
  author = {Taechoyotin, Pawin and Acuna, Daniel},
  year = {2024},
  publication_date = {2024},
  booktitle = {Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024)},
  pages = {155--164},
  doi = {10.18653/v1/2024.sdp-1.15},
  url = {https://aclanthology.org/2024.sdp-1.15/}
}