Why do we remember stories better when they have a clear beginning, middle, and end?
Image: N509FZ, CC BY-SA 4.0, via Wikimedia Commons
to use an RNN/LSTM: for sequential data where order matters (mostly replaced by transformers)
Why do we remember stories better when they have a clear beginning, middle, and end?
Imagine you're listening to a friend recount a story. The first sentence sets the scene; the middle sentences build up the plot; the last sentence wraps everything up. If the story jumps around, it's confusing and hard to follow.
Sequential data, like stories, need order to be understood. RNNs remember past information to make sense of new data, just like remembering the beginning of a story helps you grasp the ending.
Example
Your friend starts a story with "Once upon a time," then "There was a dragon," and finally "The hero saved the day." RNNs are like remembering "Once upon a time" when you hear "There was a dragon."
Remember this
RNNs are good for tasks where remembering past events helps understand new ones.
Text adapted from Wikipedia, licensed under CC BY-SA 4.0.
Related concepts
Pre-LN
Why is normalizing data like tuning instruments before a concert?
weight tying does in language models: shares embedding and output projection matrices
Ever wonder how machines understand the sequence of words in a sentence?
384-dim all-MiniLM-L6-v2 optimizes: fast sentence similarity with 6 layers
All-MiniLM-L6-v2 optimizes fast sentence similarity with 6 layers
transformers use LayerNorm not BatchNorm
LayerNorm normalizes across all features, accommodating variable-length sequences unlike BatchNorm, which relies on fixed-size batches
the vocabulary size matters: larger vocab = shorter sequences but more parameters
Larger vocab leads to shorter sequences but more parameters
to use a CNN: for data with spatial structure like images or time series
Why can't we just feed all data into one big neural net?
Swipe through 100 ML concepts daily
Open Pocket Polymath