Transformer is a powerful tool for capturing long-range dependencies and has shown impressive performance in hyperspectral image (HSI) classification. However, such power comes with a heavy memory footprint and huge computation burden. In this article, we propose two types of lightweight self-attention modules (a channel lightweight multihead self-attention (CLMSA) module and a position lightweight multihead self-attention (PLMSA) module) to reduce both memory and computation while associating each pixel or channel with global information. Moreover, we discover that transformers are ineffective in explicitly extracting local and multiscale features due to the fixed input size and tend to overfit when dealing with a small number of training samples. Therefore, a lightweight transformer (LiT) network, built with the proposed lightweight self-attention modules, is presented. LiT adopts convolutional blocks to explicitly extract local information in early layers and employs transformers to capture long-range dependencies in deep layers. Furthermore, we design a controlled multiclass stratified (CMS) sampling strategy to generate appropriately sized input data, ensure balanced sampling, and reduce the overlap of feature extraction regions between training and test samples. With appropriate training data, convolutional tokenization, and LiTs, LiT mitigates overfitting and enjoys both high computational efficiency and good performance. Experimental results on several HSI datasets verify the effectiveness of our design.

A Lightweight Transformer Network for Hyperspectral Image Classification / Zhang, Xuming; Su, Yuanchao; Gao, Lianru; Bruzzone, Lorenzo; Gu, Xingfa; Tian, Qingjiu. - In: IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING. - ISSN 0196-2892. - 61:(2023), pp. 551761701-551761717. [10.1109/TGRS.2023.3297858]

A Lightweight Transformer Network for Hyperspectral Image Classification

Bruzzone, Lorenzo;
2023-01-01

Abstract

Transformer is a powerful tool for capturing long-range dependencies and has shown impressive performance in hyperspectral image (HSI) classification. However, such power comes with a heavy memory footprint and huge computation burden. In this article, we propose two types of lightweight self-attention modules (a channel lightweight multihead self-attention (CLMSA) module and a position lightweight multihead self-attention (PLMSA) module) to reduce both memory and computation while associating each pixel or channel with global information. Moreover, we discover that transformers are ineffective in explicitly extracting local and multiscale features due to the fixed input size and tend to overfit when dealing with a small number of training samples. Therefore, a lightweight transformer (LiT) network, built with the proposed lightweight self-attention modules, is presented. LiT adopts convolutional blocks to explicitly extract local information in early layers and employs transformers to capture long-range dependencies in deep layers. Furthermore, we design a controlled multiclass stratified (CMS) sampling strategy to generate appropriately sized input data, ensure balanced sampling, and reduce the overlap of feature extraction regions between training and test samples. With appropriate training data, convolutional tokenization, and LiTs, LiT mitigates overfitting and enjoys both high computational efficiency and good performance. Experimental results on several HSI datasets verify the effectiveness of our design.
2023
Zhang, Xuming; Su, Yuanchao; Gao, Lianru; Bruzzone, Lorenzo; Gu, Xingfa; Tian, Qingjiu
A Lightweight Transformer Network for Hyperspectral Image Classification / Zhang, Xuming; Su, Yuanchao; Gao, Lianru; Bruzzone, Lorenzo; Gu, Xingfa; Tian, Qingjiu. - In: IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING. - ISSN 0196-2892. - 61:(2023), pp. 551761701-551761717. [10.1109/TGRS.2023.3297858]
File in questo prodotto:
File Dimensione Formato  
A_Lightweight_Transformer_Network_for_Hyperspectral_Image_Classification.pdf

Solo gestori archivio

Tipologia: Versione editoriale (Publisher’s layout)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 7.5 MB
Formato Adobe PDF
7.5 MB Adobe PDF   Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11572/401470
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 6
  • ???jsp.display-item.citation.isi??? 3
social impact