Image reconstruction from neuromorphic event cameras using laplacian-prediction and poisson integration with spiking and artificial neural networks

Hadar Cohen Duwek; Albert Shalumov; Elishai Ezra Tsur

doi:10.1109/CVPRW53098.2021.00147

Image reconstruction from neuromorphic event cameras using laplacian-prediction and poisson integration with spiking and artificial neural networks

Hadar Cohen Duwek, Albert Shalumov, Elishai Ezra Tsur

Computer Science discipline

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Event cameras are robust neuromorphic visual sensors, which communicate transients in luminance as events. Current paradigm for image reconstruction from event data relies on direct optimization of artificial Convolutional Neural Networks (CNNs). Here we proposed a two-phase neural network, which comprises a CNN, optimized for Laplacian prediction followed by a Spiking Neural Network (SNN) optimized for Poisson integration. By introducing Laplacian prediction into the pipeline, we provide image reconstruction with a network comprising only 200 parameters. We converted the CNN to SNN, providing a full neuromorphic implementation. We further optimized the network with Mish activation and a novel convoluted CNN design, proposing a hybrid of spiking and artificial neural network with < 100 parameters. Models were evaluated on both N-MNIST and N-Caltech101 datasets.

Original language	English
Title of host publication	Proceedings - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021
Publisher	IEEE Computer Society
Pages	1333-1341
Number of pages	9
ISBN (Electronic)	9781665448994
DOIs	https://doi.org/10.1109/CVPRW53098.2021.00147
State	Published - Jun 2021
Event	2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021 - Virtual, Online, United States Duration: 19 Jun 2021 → 25 Jun 2021

Publication series

Name	IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops
ISSN (Print)	2160-7508
ISSN (Electronic)	2160-7516

Conference

Conference	2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021
Country/Territory	United States
City	Virtual, Online
Period	19/06/21 → 25/06/21

Bibliographical note

Publisher Copyright:
© 2021 IEEE.

Access to Document

10.1109/CVPRW53098.2021.00147

Cite this

Duwek, H. C., Shalumov, A., & Tsur, E. E. (2021). Image reconstruction from neuromorphic event cameras using laplacian-prediction and poisson integration with spiking and artificial neural networks. In Proceedings - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021 (pp. 1333-1341). (IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops). IEEE Computer Society. https://doi.org/10.1109/CVPRW53098.2021.00147

Duwek, Hadar Cohen ; Shalumov, Albert ; Tsur, Elishai Ezra. / Image reconstruction from neuromorphic event cameras using laplacian-prediction and poisson integration with spiking and artificial neural networks. Proceedings - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021. IEEE Computer Society, 2021. pp. 1333-1341 (IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops).

@inproceedings{464fdbc284134418b2990e02045f9605,

title = "Image reconstruction from neuromorphic event cameras using laplacian-prediction and poisson integration with spiking and artificial neural networks",

abstract = "Event cameras are robust neuromorphic visual sensors, which communicate transients in luminance as events. Current paradigm for image reconstruction from event data relies on direct optimization of artificial Convolutional Neural Networks (CNNs). Here we proposed a two-phase neural network, which comprises a CNN, optimized for Laplacian prediction followed by a Spiking Neural Network (SNN) optimized for Poisson integration. By introducing Laplacian prediction into the pipeline, we provide image reconstruction with a network comprising only 200 parameters. We converted the CNN to SNN, providing a full neuromorphic implementation. We further optimized the network with Mish activation and a novel convoluted CNN design, proposing a hybrid of spiking and artificial neural network with < 100 parameters. Models were evaluated on both N-MNIST and N-Caltech101 datasets.",

author = "Duwek, {Hadar Cohen} and Albert Shalumov and Tsur, {Elishai Ezra}",

note = "Publisher Copyright: {\textcopyright} 2021 IEEE.; 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021 ; Conference date: 19-06-2021 Through 25-06-2021",

year = "2021",

month = jun,

doi = "10.1109/CVPRW53098.2021.00147",

language = "אנגלית",

series = "IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops",

publisher = "IEEE Computer Society",

pages = "1333--1341",

booktitle = "Proceedings - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021",

address = "ארצות הברית",

}

Duwek, HC, Shalumov, A & Tsur, EE 2021, Image reconstruction from neuromorphic event cameras using laplacian-prediction and poisson integration with spiking and artificial neural networks. in Proceedings - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, IEEE Computer Society, pp. 1333-1341, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021, Virtual, Online, United States, 19/06/21. https://doi.org/10.1109/CVPRW53098.2021.00147

Image reconstruction from neuromorphic event cameras using laplacian-prediction and poisson integration with spiking and artificial neural networks. / Duwek, Hadar Cohen; Shalumov, Albert; Tsur, Elishai Ezra.
Proceedings - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021. IEEE Computer Society, 2021. p. 1333-1341 (IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Image reconstruction from neuromorphic event cameras using laplacian-prediction and poisson integration with spiking and artificial neural networks

AU - Duwek, Hadar Cohen

AU - Shalumov, Albert

AU - Tsur, Elishai Ezra

PY - 2021/6

Y1 - 2021/6

N2 - Event cameras are robust neuromorphic visual sensors, which communicate transients in luminance as events. Current paradigm for image reconstruction from event data relies on direct optimization of artificial Convolutional Neural Networks (CNNs). Here we proposed a two-phase neural network, which comprises a CNN, optimized for Laplacian prediction followed by a Spiking Neural Network (SNN) optimized for Poisson integration. By introducing Laplacian prediction into the pipeline, we provide image reconstruction with a network comprising only 200 parameters. We converted the CNN to SNN, providing a full neuromorphic implementation. We further optimized the network with Mish activation and a novel convoluted CNN design, proposing a hybrid of spiking and artificial neural network with < 100 parameters. Models were evaluated on both N-MNIST and N-Caltech101 datasets.

AB - Event cameras are robust neuromorphic visual sensors, which communicate transients in luminance as events. Current paradigm for image reconstruction from event data relies on direct optimization of artificial Convolutional Neural Networks (CNNs). Here we proposed a two-phase neural network, which comprises a CNN, optimized for Laplacian prediction followed by a Spiking Neural Network (SNN) optimized for Poisson integration. By introducing Laplacian prediction into the pipeline, we provide image reconstruction with a network comprising only 200 parameters. We converted the CNN to SNN, providing a full neuromorphic implementation. We further optimized the network with Mish activation and a novel convoluted CNN design, proposing a hybrid of spiking and artificial neural network with < 100 parameters. Models were evaluated on both N-MNIST and N-Caltech101 datasets.

UR - http://www.scopus.com/inward/record.url?scp=85116016005&partnerID=8YFLogxK

U2 - 10.1109/CVPRW53098.2021.00147

DO - 10.1109/CVPRW53098.2021.00147

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85116016005

T3 - IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops

SP - 1333

EP - 1341

BT - Proceedings - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021

PB - IEEE Computer Society

T2 - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021

Y2 - 19 June 2021 through 25 June 2021

ER -

Duwek HC, Shalumov A, Tsur EE. Image reconstruction from neuromorphic event cameras using laplacian-prediction and poisson integration with spiking and artificial neural networks. In Proceedings - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021. IEEE Computer Society. 2021. p. 1333-1341. (IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops). doi: 10.1109/CVPRW53098.2021.00147

Image reconstruction from neuromorphic event cameras using laplacian-prediction and poisson integration with spiking and artificial neural networks

Abstract

Publication series

Conference

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this