Sunday, June 26, 2016

Rich Image Captioning In The Wild

In this CVPR 2016 - Deep Vision Workshop paper, we present an image caption system that addresses new challenges of automatically describing images in the wild. The challenges include high quality caption quality with respect to human judgments, out-of-domain data handling, and low latency required in many applications. Built on top of a state-of-the-art framework, we developed a deep vision model that detects a broad range of visual concepts, an entity recognition model that identifies celebrities and landmarks, and a confidence model for the caption output. Experimental results show that our caption engine outperforms previous state-of-the-art systems significantly on both in-domain dataset (i.e. MS COCO) and out-of-domain datasets.

I'll present this work at the CVPR Deep Vision workshop. Hope to see many friends at CVPR this year.