Pay Attention to MLPs

Oct 23, 2021 · Bethesda, United States of America

This meeting is open to anyone interested in Artificial Intelligence or Machine Learning, regardless of experience or background. The point is to meet others interested in the area and potentially form collaborations or study groups. For those new to AI/ML, please identify yourself so you can connect with others with similar interests.

We'll continue the format as the previous meetups - introductions, couple of short presentations, open discussions on topics of interest.

As always, suggestions are most welcome!

=============
Join us for a discussion on "Attention to MLPs" followed by a meet and greet.

Paper: Pay Attention to MLPs (https://arxiv.org/pdf/2105.08050.pdf)
Review by : David Patton and Marc Pickett

Transformers have become one of the most important architectural innovations in deep learning and have enabled many breakthroughs over the past few years. Here we propose a simple network architecture, gMLP, based on MLPs with gating, and show that it can perform as well as Transformers in key language and vision applications. Our comparisons show that self-attention is not critical for Vision Transformers, as gMLP can achieve the same accuracy. For BERT, our model achieves parity with Transformers on pretraining perplexity and is better on some downstream NLP tasks. On finetuning tasks where gMLP performs worse, making the gMLP model substantially larger can close the gap with Transformers. In general, our experiments show that gMLP can scale as well as Transformers over increased data and compute

Event organizers
  • Bethesda Artificial Intelligence Meetup

    This is for people in interested in AI and Machine Learning. All levels of expertise are welcome. The meetup is for anything from getting up to speed with state-of-the-art techniques to contributing to the field.

    Recent Events
    More

Are you organizing Pay Attention to MLPs?

Claim the event and start manage its content.

I am the organizer
Social
Rating

based on 0 reviews

Featured Events