Yaya Shi

EMScore: Evaluating Video Captioning via Coarse-Grained and Fine-Grained Embedding Matching
Yaya Shi, Xu Yang, Haiyang Xu, Chunfeng Yuan, Bing Li, Weiming Hu, Zheng-Jun Zha
CVPR 2022
To avoid the dependency on reference text in the video captioning evaluation, we proposed a reference-free metric EMScore, which measures consistency between candidate captions with video contents in both coarse-grained and fine-grained embedding matching levels. Moreover, two datasets are collected to study the metrics’ correlation with human judgments and sensitivity in the “hallucinating” case.

Learning Video-Text Aligned Representations for Video Captioning
Yaya Shi, Haiyang Xu, Chunfeng Yuan, Bing Li, Weiming Hu, Zheng-Jun Zha
ACM Trans TOMM
To reduce the semantic gap between vision and language in the captioning task, we proposed a retrieval-augmented encoder, which includes a retrieval unit to select semantically similar sentences for a given video as the semantic anchor and an alignment unit to learn video-text-aligned representations.

Object Relational Graph with Teacher-Recommended Learning for Video Captioning
Ziqi Zhang, Yaya Shi*, Chunfeng Yuan, Bing Li, Peijing Wang, Weiming Hu, Zheng-Jun Zha
CVPR 2020
To generate a more detailed and diverse description for a video, we proposed an object relational graph based encoder, which captures more detailed interaction features to enrich visual representation.