KG_VCR: A Visual Commonsense Reasoning Model Using Knowledge Graph


KIPS Transactions on Software and Data Engineering, Vol. 9, No. 3, pp. 91-100, Mar. 2020
https://doi.org/10.3745/KTSDE.2020.9.3.91,   PDF Download:
Keywords: Visual Commonsense Reasoning, deep neural network, Graph Convolutional Network, Knowledge Graph Embedding
Abstract

Unlike the existing Visual Question Answering(VQA) problems, the new Visual Commonsense Reasoning(VCR) problems require deep common sense reasoning for answering questions: recognizing specific relationship between two objects in the image, presenting the rationale of the answer. In this paper, we propose a novel deep neural network model, KG_VCR, for VCR problems. In addition to make use of visual relations and contextual information between objects extracted from input data (images, natural language questions, and response lists), the KG_VCR also utilizes commonsense knowledge embedding extracted from an external knowledge base called ConceptNet. Specifically the proposed model employs a Graph Convolutional Neural Network(GCN) module to obtain commonsense knowledge embedding from the retrieved ConceptNet knowledge graph. By conducting a series of experiments with the VCR benchmark dataset, we show that the proposed KG_VCR model outperforms both the state of the art(SOTA) VQA model and the R2C VCR model.


Statistics
Show / Hide Statistics

Statistics (Cumulative Counts from September 1st, 2017)
Multiple requests among the same browser session are counted as one view.
If you mouse over a chart, the values of data points will be shown.


Cite this article
[IEEE Style]
J. Lee and I. Kim, "KG_VCR: A Visual Commonsense Reasoning Model Using Knowledge Graph," KIPS Transactions on Software and Data Engineering, vol. 9, no. 3, pp. 91-100, 2020. DOI: https://doi.org/10.3745/KTSDE.2020.9.3.91.

[ACM Style]
JaeYun Lee and Incheol Kim. 2020. KG_VCR: A Visual Commonsense Reasoning Model Using Knowledge Graph. KIPS Transactions on Software and Data Engineering, 9, 3, (2020), 91-100. DOI: https://doi.org/10.3745/KTSDE.2020.9.3.91.