Position Embeddings (PEs), an arguably indispensable component in Vision Transformers (ViTs), have been shown to improve the performance of ViTs on many vision tasks. However, PEs have a potentially high risk of privacy leakage since the spatial information of the input patches is exposed. This caveat naturally raises a series of interesting questions about the impact of PEs on accuracy, privacy, prediction consistency, etc. To tackle these issues, we propose a Masked Jigsaw Puzzle (MJP) position embedding method. In particular, MJP first shuffles the selected patches via our block-wise random jigsaw puzzle shuffle algorithm, and their corresponding PEs are occluded. Meanwhile, for the non-occluded patches, the PEs remain the original ones but their spatial relation is strengthened via our dense absolute localization regressor. The experimental results reveal that 1) PEs explicitly encode the 2D spatial relationship and lead to severe privacy leakage problems under gradient inversion attack; 2) Training ViTs with the naively shuffled patches can alleviate the problem, but it harms the accuracy; 3) Under a certain shuffle ratio, the proposed MJP not only boosts the performance and robustness on large-scale datasets (i.e., ImageNet-1K and ImageNet-C, -A/O) but also improves the privacy preservation ability under typical gradient attacks by a large margin. The source code and trained models are available at https://github.com/yhlleo/MJP.

Masked Jigsaw Puzzle: A Versatile Position Embedding for Vision Transformers / Ren, B.; Liu, Y.; Song, Y.; Bi, W.; Cucchiara, R.; Sebe, N.; Wang, W.. - 2023-:(2023), pp. 20382-20391. (Intervento presentato al convegno 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023 tenutosi a Vancouver, BC, Canada nel 17-24 June 2023) [10.1109/CVPR52729.2023.01952].

Masked Jigsaw Puzzle: A Versatile Position Embedding for Vision Transformers

Ren, B.;Liu, Y.;Song Y.;Sebe, N.;Wang, W.
2023-01-01

Abstract

Position Embeddings (PEs), an arguably indispensable component in Vision Transformers (ViTs), have been shown to improve the performance of ViTs on many vision tasks. However, PEs have a potentially high risk of privacy leakage since the spatial information of the input patches is exposed. This caveat naturally raises a series of interesting questions about the impact of PEs on accuracy, privacy, prediction consistency, etc. To tackle these issues, we propose a Masked Jigsaw Puzzle (MJP) position embedding method. In particular, MJP first shuffles the selected patches via our block-wise random jigsaw puzzle shuffle algorithm, and their corresponding PEs are occluded. Meanwhile, for the non-occluded patches, the PEs remain the original ones but their spatial relation is strengthened via our dense absolute localization regressor. The experimental results reveal that 1) PEs explicitly encode the 2D spatial relationship and lead to severe privacy leakage problems under gradient inversion attack; 2) Training ViTs with the naively shuffled patches can alleviate the problem, but it harms the accuracy; 3) Under a certain shuffle ratio, the proposed MJP not only boosts the performance and robustness on large-scale datasets (i.e., ImageNet-1K and ImageNet-C, -A/O) but also improves the privacy preservation ability under typical gradient attacks by a large margin. The source code and trained models are available at https://github.com/yhlleo/MJP.
2023
Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
Piscataway, NJ USA
IEEE Computer Society
979-8-3503-0129-8
Ren, B.; Liu, Y.; Song, Y.; Bi, W.; Cucchiara, R.; Sebe, N.; Wang, W.
Masked Jigsaw Puzzle: A Versatile Position Embedding for Vision Transformers / Ren, B.; Liu, Y.; Song, Y.; Bi, W.; Cucchiara, R.; Sebe, N.; Wang, W.. - 2023-:(2023), pp. 20382-20391. (Intervento presentato al convegno 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023 tenutosi a Vancouver, BC, Canada nel 17-24 June 2023) [10.1109/CVPR52729.2023.01952].
File in questo prodotto:
File Dimensione Formato  
Ren_Masked_Jigsaw_Puzzle_A_Versatile_Position_Embedding_for_Vision_Transformers_CVPR_2023_paper.pdf

accesso aperto

Descrizione: This CVPR paper is the Open Access version, provided by the Computer Vision Foundation. Except for this watermark, it is identical to the accepted version
Tipologia: Versione editoriale (Publisher’s layout)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 5.18 MB
Formato Adobe PDF
5.18 MB Adobe PDF Visualizza/Apri
Masked_Jigsaw_Puzzle_A_Versatile_Position_Embedding_for_Vision_Transformers.pdf

Solo gestori archivio

Tipologia: Versione editoriale (Publisher’s layout)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 1.18 MB
Formato Adobe PDF
1.18 MB Adobe PDF   Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11572/395069
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 3
  • ???jsp.display-item.citation.isi??? 2
social impact