Scene graph generation (SGG) is built on top of detected objects to predict object pairwise visual relations for describing the image content abstraction. Existing works have revealed that if the links between objects are given as prior knowledge, the performance of SGG is significantly improved. Inspired by this observation, in this article, we propose a relation regularized network (R2-Net), which can predict whether there is a relationship between two objects and encode this relation into object feature refinement and better SGG. Specifically, we first construct an affinity matrix among detected objects to represent the probability of a relationship between two objects. Graph convolution networks (GCNs) over this relation affinity matrix are then used as object encoders, producing relation-regularized representations of objects. With these relation-regularized features, our R2-Net can effectively refine object labels and generate scene graphs. Extensive experiments are conducted on the visual genome dataset for three SGG tasks (i.e., predicate classification, scene graph classification, and scene graph detection), demonstrating the effectiveness of our proposed method. Ablation studies also verify the key roles of our proposed components in performance improvement.

Relation Regularized Scene Graph Generation / Guo, Y.; Gao, L.; Song, J.; Wang, P.; Sebe, N.; Shen, H. T.; Li, X.. - In: IEEE TRANSACTIONS ON CYBERNETICS. - ISSN 2168-2267. - 52:7(2022), pp. 5961-5972. [10.1109/TCYB.2021.3052522]

Relation Regularized Scene Graph Generation

Song J.;Sebe N.;
2022-01-01

Abstract

Scene graph generation (SGG) is built on top of detected objects to predict object pairwise visual relations for describing the image content abstraction. Existing works have revealed that if the links between objects are given as prior knowledge, the performance of SGG is significantly improved. Inspired by this observation, in this article, we propose a relation regularized network (R2-Net), which can predict whether there is a relationship between two objects and encode this relation into object feature refinement and better SGG. Specifically, we first construct an affinity matrix among detected objects to represent the probability of a relationship between two objects. Graph convolution networks (GCNs) over this relation affinity matrix are then used as object encoders, producing relation-regularized representations of objects. With these relation-regularized features, our R2-Net can effectively refine object labels and generate scene graphs. Extensive experiments are conducted on the visual genome dataset for three SGG tasks (i.e., predicate classification, scene graph classification, and scene graph detection), demonstrating the effectiveness of our proposed method. Ablation studies also verify the key roles of our proposed components in performance improvement.
2022
7
Guo, Y.; Gao, L.; Song, J.; Wang, P.; Sebe, N.; Shen, H. T.; Li, X.
Relation Regularized Scene Graph Generation / Guo, Y.; Gao, L.; Song, J.; Wang, P.; Sebe, N.; Shen, H. T.; Li, X.. - In: IEEE TRANSACTIONS ON CYBERNETICS. - ISSN 2168-2267. - 52:7(2022), pp. 5961-5972. [10.1109/TCYB.2021.3052522]
File in questo prodotto:
File Dimensione Formato  
TCYB21.pdf

accesso aperto

Tipologia: Post-print referato (Refereed author’s manuscript)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 5.04 MB
Formato Adobe PDF
5.04 MB Adobe PDF Visualizza/Apri
Relation_Regularized_Scene_Graph_Generation.pdf

Solo gestori archivio

Tipologia: Versione editoriale (Publisher’s layout)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 4.71 MB
Formato Adobe PDF
4.71 MB Adobe PDF   Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11572/361264
Citazioni
  • ???jsp.display-item.citation.pmc??? 0
  • Scopus 4
  • ???jsp.display-item.citation.isi??? 3
social impact