RunByAI

Tag: Transformer

7 post(s)

September 25, 20260 comment(s)

Introduction to Position Encoding: From Sine Waves to RoPE, How Transformer Knows the Order of Words

The attention mechanism has a natural flaw: it is insensitive to input order. By shuffling the words in a sentence, the result calculated from self attention remains almost unchanged, but "cat chasing

Tech Guide deep learning Transformer large model

September 25, 20260 comment(s)

Introduction to Attention Mechanisms: From QKV to Self Attention, How the Core Engine of Transformer Works

The attention mechanism is the core design that distinguishes Transformer from early recurrent networks and pushes large models to the present day. It can be said that without attention, there is no G

Tech Guide deep learning Transformer large model

September 24, 20260 comment(s)

Introduction to Activation Functions: How the "Nonlinear Switch" of Neural Networks Works from ReLU to GELU, SwiGLU

The activation function is the most inconspicuous yet almost ubiquitous component in neural networks. Its task is simple: perform a nonlinear transformation on the weighted sum of each neuron. But it

Tech Guide deep learning Transformer

September 18, 20260 comment(s)

Introduction to Self Attention Mechanism: How Large Models Allocate Attention

The attention mechanism is a core component of modern large language models. It was first used for machine translation by Bahdanau et al. in 2014, and then in their 2017 paper "Attention Is All You Ne

Tech Guide large model Transformer deep learning

September 17, 20260 comment(s)

Introduction to Location Encoding and RoPE: How Large Models "Remember" Order

The words' cat chasing mouse 'and' mouse chasing cat 'are exactly the same, but their meanings are opposite. To understand this difference, the big model must know where each word is located. But the

Tech Guide large model Transformer

September 14, 20260 comment(s)

Context Window of Large Models: Why 'Longer' Does Not Equal 'Better'

The 'context window' refers to the total amount of tokens that a large model can 'see' at once: questions, historical conversations, retrieved data, and answers that the model needs to generate all ne

Tech View Large Language Model (LLM)Transformer

May 29, 20260 comment(s)

From Transformer to State Space Model: The Next Stop in the Evolution of AI Architecture

In 2017, the Google research team proposed the Transformer architecture in their paper "Attention Is All You Need", ushering in a new era of deep learning. In the past decade, Transformer has become t

Tech News Transformer deep learning