Grad-SAM: Explaining Transformers via Gradient Self-Attention Maps

Oren Barkan; Edan Hauon; Avi Caciularu; Ori Katz; Itzik Malkiel; Omri Armstrong; Noam Koenigstein

doi:10.1145/3459637.3482126

Grad-SAM: Explaining Transformers via Gradient Self-Attention Maps

Oren Barkan, Edan Hauon, Avi Caciularu, Ori Katz, Itzik Malkiel, Omri Armstrong, Noam Koenigstein

Computer Science discipline

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Transformer-based language models significantly advanced the state-of-the-art in many linguistic tasks. As this revolution continues, the ability to explain model predictions has become a major area of interest for the NLP community. In this work, we present Gradient Self-Attention Maps (Grad-SAM) - a novel gradient-based method that analyzes self-attention units and identifies the input elements that explain the model's prediction the best. Extensive evaluations on various benchmarks show that Grad-SAM obtains significant improvements over state-of-the-art alternatives.

Original language	English
Title of host publication	CIKM 2021 - Proceedings of the 30th ACM International Conference on Information and Knowledge Management
Publisher	Association for Computing Machinery
Pages	2882-2887
Number of pages	6
ISBN (Electronic)	9781450384469
DOIs	https://doi.org/10.1145/3459637.3482126
State	Published - 26 Oct 2021
Event	30th ACM International Conference on Information and Knowledge Management, CIKM 2021 - Virtual, Online, Australia Duration: 1 Nov 2021 → 5 Nov 2021

Publication series

Name	International Conference on Information and Knowledge Management, Proceedings

Conference

Conference	30th ACM International Conference on Information and Knowledge Management, CIKM 2021
Country/Territory	Australia
City	Virtual, Online
Period	1/11/21 → 5/11/21

Bibliographical note

Publisher Copyright:
© 2021 ACM.

Keywords

bert
deep learning
explainable & interpretable ai
nlp
self-attention
transformers
transparent machine learning

Access to Document

10.1145/3459637.3482126

Cite this

Barkan, O., Hauon, E., Caciularu, A., Katz, O., Malkiel, I., Armstrong, O., & Koenigstein, N. (2021). Grad-SAM: Explaining Transformers via Gradient Self-Attention Maps. In CIKM 2021 - Proceedings of the 30th ACM International Conference on Information and Knowledge Management (pp. 2882-2887). (International Conference on Information and Knowledge Management, Proceedings). Association for Computing Machinery. https://doi.org/10.1145/3459637.3482126

@inproceedings{e5eef7ffd0c040758ff498d0e7ebe8ae,

title = "Grad-SAM: Explaining Transformers via Gradient Self-Attention Maps",

abstract = "Transformer-based language models significantly advanced the state-of-the-art in many linguistic tasks. As this revolution continues, the ability to explain model predictions has become a major area of interest for the NLP community. In this work, we present Gradient Self-Attention Maps (Grad-SAM) - a novel gradient-based method that analyzes self-attention units and identifies the input elements that explain the model's prediction the best. Extensive evaluations on various benchmarks show that Grad-SAM obtains significant improvements over state-of-the-art alternatives.",

keywords = "bert, deep learning, explainable & interpretable ai, nlp, self-attention, transformers, transparent machine learning",

author = "Oren Barkan and Edan Hauon and Avi Caciularu and Ori Katz and Itzik Malkiel and Omri Armstrong and Noam Koenigstein",

note = "Publisher Copyright: {\textcopyright} 2021 ACM.; 30th ACM International Conference on Information and Knowledge Management, CIKM 2021 ; Conference date: 01-11-2021 Through 05-11-2021",

year = "2021",

month = oct,

day = "26",

doi = "10.1145/3459637.3482126",

language = "אנגלית",

series = "International Conference on Information and Knowledge Management, Proceedings",

publisher = "Association for Computing Machinery",

pages = "2882--2887",

booktitle = "CIKM 2021 - Proceedings of the 30th ACM International Conference on Information and Knowledge Management",

}

Barkan, O, Hauon, E, Caciularu, A, Katz, O, Malkiel, I, Armstrong, O & Koenigstein, N 2021, Grad-SAM: Explaining Transformers via Gradient Self-Attention Maps. in CIKM 2021 - Proceedings of the 30th ACM International Conference on Information and Knowledge Management. International Conference on Information and Knowledge Management, Proceedings, Association for Computing Machinery, pp. 2882-2887, 30th ACM International Conference on Information and Knowledge Management, CIKM 2021, Virtual, Online, Australia, 1/11/21. https://doi.org/10.1145/3459637.3482126

Grad-SAM: Explaining Transformers via Gradient Self-Attention Maps. / Barkan, Oren; Hauon, Edan; Caciularu, Avi et al.
CIKM 2021 - Proceedings of the 30th ACM International Conference on Information and Knowledge Management. Association for Computing Machinery, 2021. p. 2882-2887 (International Conference on Information and Knowledge Management, Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Grad-SAM

T2 - 30th ACM International Conference on Information and Knowledge Management, CIKM 2021

AU - Barkan, Oren

AU - Hauon, Edan

AU - Caciularu, Avi

AU - Katz, Ori

AU - Malkiel, Itzik

AU - Armstrong, Omri

AU - Koenigstein, Noam

PY - 2021/10/26

Y1 - 2021/10/26

N2 - Transformer-based language models significantly advanced the state-of-the-art in many linguistic tasks. As this revolution continues, the ability to explain model predictions has become a major area of interest for the NLP community. In this work, we present Gradient Self-Attention Maps (Grad-SAM) - a novel gradient-based method that analyzes self-attention units and identifies the input elements that explain the model's prediction the best. Extensive evaluations on various benchmarks show that Grad-SAM obtains significant improvements over state-of-the-art alternatives.

AB - Transformer-based language models significantly advanced the state-of-the-art in many linguistic tasks. As this revolution continues, the ability to explain model predictions has become a major area of interest for the NLP community. In this work, we present Gradient Self-Attention Maps (Grad-SAM) - a novel gradient-based method that analyzes self-attention units and identifies the input elements that explain the model's prediction the best. Extensive evaluations on various benchmarks show that Grad-SAM obtains significant improvements over state-of-the-art alternatives.

KW - bert

KW - deep learning

KW - explainable & interpretable ai

KW - nlp

KW - self-attention

KW - transformers

KW - transparent machine learning

UR - http://www.scopus.com/inward/record.url?scp=85119180055&partnerID=8YFLogxK

U2 - 10.1145/3459637.3482126

DO - 10.1145/3459637.3482126

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85119180055

T3 - International Conference on Information and Knowledge Management, Proceedings

SP - 2882

EP - 2887

BT - CIKM 2021 - Proceedings of the 30th ACM International Conference on Information and Knowledge Management

PB - Association for Computing Machinery

Y2 - 1 November 2021 through 5 November 2021

ER -

Barkan O, Hauon E, Caciularu A, Katz O, Malkiel I, Armstrong O et al. Grad-SAM: Explaining Transformers via Gradient Self-Attention Maps. In CIKM 2021 - Proceedings of the 30th ACM International Conference on Information and Knowledge Management. Association for Computing Machinery. 2021. p. 2882-2887. (International Conference on Information and Knowledge Management, Proceedings). doi: 10.1145/3459637.3482126

Grad-SAM: Explaining Transformers via Gradient Self-Attention Maps

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this