Demystifying LLM Benchmarks -- How To Compare Large Language Models?

In the recent launch of Google's Large Language Model (LLM) Gemini, Google claimed that "Gemini beats human level performance on MMLU" but you might be wondering:

- What are the different evaulation criteria? Massive Multitask Language Understanding (**MMLU**)? General Language Understanding Evaluation (**GLUE**)? **HumanEval**?
- How can we know which model is good at coding tasks?
- What is considered "human-level performance"?
- How do they actually get the results for these benchmarks?

In this talk, Sai breaks down all the benchmarks you need to know to be better informed when papers claim to beat benchmarks.

The next paper reading session is on January 8 at 5pm: [**https://www.jointaro.com/event/paper-reading-group-4-mistral-7b/**](https://www.jointaro.com/event/paper-reading-group-4-mistral-7b/)

**Your host:**

[**Sai Shreyas Bhavanasi**](https://www.linkedin.com/in/b-sai/) has worked in Computer Vision and Reinforcement Learning and published 2 first-author papers. He has also worked as an MLE and Data Analyst.

Senior Software Engineer at Taro Community

Has anyone interviewed there before and can share their experience/tips? I'm interviewing for a full-stack role. There is a code refactoring round which is new to me.

Paper Reading Group: Attention Is All You Need

Generative AI, specifically large language models, have taken the world by storm. But they wouldn't exist today without a key paper published by Google Brain: "[Attention Is All You Need](https://arxiv.org/pdf/1706.03762.pdf)." The 10-page paper, published in 2017, introduced the Transformer - a critical innovation in neural networks.

**What's in store?**

- A deep dive and interactive discussion of the paper. We recommend reading (or at least skimming) it first: <https://arxiv.org/pdf/1706.03762.pdf>
- A look at what came before the paper (neural networks, embeddings), and what came after (BERT, GPT-2, and ChatGPT).
- **All software engineers welcome** - no AI/ML background necessary!

**Did you know?**

- The paper's authors have all gone on to found multiple billion-dollar AI companies including Cohere and [Character.ai](http://Character.ai).
- Since 2017, the paper has been cited over 86,000 times.
- The Transformer architecture is the **T** in "GPT".

**Your host:**

[Charlie Guo](https://www.linkedin.com/in/charlierguo/) is a seasoned founder and author with over 15 years of software experience and more than a decade in Silicon Valley. He is an alum of both Stanford and YCombinator, and currently publishes [Artificial Ignorance](https://www.ignorance.ai/about), an AI-focused newsletter for engineers and founders.

**The agenda:**

- Charlie will give a brief background, and then explain the key insights and context from the paper.
- We will open the group to some Q&A and a broader discussion.
- And we'll end with some discussion on which paper to read and review next.

The next paper reading group is on October 15 about [deep reinforcement learning](https://www.jointaro.com/event/paper-reading-group-human-level-control-through-deep-reinforcement-learning-deep-q-network/), led by Sai Bhavanasi.

OpenAI Career Development Videos, Forum, and Q&A

Grow Your Tech Career at OpenAI

Demystifying LLM Benchmarks -- How To Compare Large Language Models?

How to prepare for an OpenAI interview?

Paper Reading Group: Attention Is All You Need