Chapter 56 — Evolutions
The Curve of Time, Chapter 56 —— Evolutions , in which Saskia talks about the time travel of evolution.
Rufus muses on some fun mathematical references.
Listen to full episode :
— 56 —
Evolutions
The doctor hadn’t found anything that worried him, so he’d given Saskia some heartburn medication and sent her home. Mica had come back with her, but she was skeptical that there was nothing to worry about. “Maybe he should have had one of those AI X-ray readers look at your chest?”
Saskia shrugged. She was less sure; if her affliction were something novel that her slipping in time had triggered, then she wasn’t convinced that any ML X-ray program would have been well prepped with its training data. She voiced her general qualms to Mica, who switched topics and zeroed in on a more primitive question.
“How does the machine classify what it’s seen anyway? Actually, how do the LLMs understand words? What they mean? Like, how do they know that some words are similar to others?”
This time, Saskia smiled. Mica’s question was a welcome distraction. “Do you remember how when I described how LLMs work, back in the hospital in Dallas?”
Mica didn’t look entirely confident about the memory.
“I was probably kind of vague,” Saskia admitted. “But imagine that we had a big long list of boxes, with one box corresponding to each word——so the first box might be for ‘a’, and the second box might be for ‘aardvark’, and so on. ‘Aardvark’ would be represented by the vector: zero, one, zero, zero, zero, etc. in the rest of the boxes. Kind of like we’ve added a new dimension for each new word in our language.”
Mica bobbed her head: “OK.”
“The thing is, that’s a lot of dimensions. Way more than we need. I mean the whole world around us sits in three dimensions. Four, if we want to account for how it changes over time. And suddenly we need 50,000 dimensions to capture day to day conversations? Also, it means that any given word is kind of just as close to every other word as any other word.”
Mica frowned.
So Saskia elaborated: “‘A’ sits on the x-axis, ‘aardvark’ sits on the y-axis, and so on. They’re each pointing off in their own direction.”
This time Mica nodded.
“And they’re each √2 apart——one unit back to the origin and one unit out in a new direction; makes a right angled triangle with sides: 1, 1, and √2.” Mica’s eyes boggled. “Pythagorus? Never mind, the thing is: imagine we squish and deform our 50,000 dimensional space so that we get rid of the apple direction. And instead, we think of ‘apple’ as a bit in the ‘pear’ direction, a bit in the ‘red’ dimension, and so on, but, like, nothing in the ‘titanium’ direction.”
Mica’s eyes looked up, and flicked side-to-side trying to visualize what Saskia was saying.
“In traditional computing, we’d keep doing this; carefully replacing words——think paring away dimensions——with a collection of components in the remaining dimensions. And eventually, every word would be kind of made up of some components——think coordinates——of each base direction that is left. Sort of like the way you can express every color as some amount of the three primary colors, red, blue and yellow.”
“Do you know what the primary color words are?” Mica asked. “Wait, how many of them are there?”
“Actually, that’s kind of all just a mental model,” Saskia hedged. “And remember we’re not doing traditional computing anymore. Instead, we just pick a few random directions——for our imaginary ‘primary words’. The big LLMs typically use a few thousand dimensions. And for every word we might use, we assign random components to each of our ‘primary word’ dimensions.” Saskia saw Mica’s eyes dilate. “No, don’t panic.”
“You’re telling me, instead of defining orange as a bit of red and a bit of yellow and no blue, you call orange a bit of each?”
Saskia nodded.
Mica squinted her eyes. “How does that help?”
“You’re forgetting the power of neural nets! The random components are just our initial settings. We have to set our dials somehow in order to start running data through the model. They’re the starting point. We then use backprop to train the model’s weights.”
“Backprop?”
“Backpropagation. That’s the fancy name for the way we tweaked our dials and knobs to improve our model——our coordinates are our new dials and knobs.” Saskia beamed at Mica. “The LLM learns its own embeddings. It’s an evolution of sorts. And it turns out, they’re better than anything we could come up with ourselves.”
Mica looked bewildered, but there was a flicker of comprehension.
“Oh”——Saskia realized she hadn’t answered Mica’s original question——“and the locations of the embeddings have semantic meaning. So similar words are embedded near each other.”
“What if a word has two meanings?” Mica asked. “Like ‘model’ could mean your computer model or,” she struck a pose, “a fashion model.”
Saskia was about to answer when her front door chime interrupted them. She glanced at the app on her phone. “Oh.”
It was Wassily. Saskia had completely forgotten that she’d texted him while back at the hospital, waiting for Mica to return from the bathroom. And now, here he was standing on her doorstep with a bouquet of flowers.
Mica peeked over Saskia’s shoulder and couldn’t suppress a frown.
Saskia excused herself and went to let Wassily in.
When she returned with him, Wassily gave Mica an oblivious wave. “Oh, hey Mica, I didn’t realize you were going to be here.”
Mica pulled a tight smile. “Beautiful flowers.”
“Thanks.” Wassliy glowed with pride, completely failing to catch Mica’s undertones. “I was just telling Saskia,” he barreled on, “her ability might be the next step in evolution.”
“Is evolution really that great?” Mica asked. “I mean, it didn’t give us wheels.”
Saskia closed her eyes and bent her head forward at a slight angle. She knew this wasn’t the time or place to contradict Mica, but a little part of her also felt that Mica would appreciate the insight that Saskia had first heard from Geoff Hinton, the man sometimes referred to as the godfather of AI. There was also the possibility that it’d break the tension. So, Saskia described how, actually, evolution did give humans wheels.
“Our wheels are six foot in diameter, since if they were smaller we’d have trouble with rough ground.”
Mica and Wassily glanced at each other and then back to Saskia. “We don’t have wheels,” Mica objected again.
Saskia pointed at her feet. “The thing is we don’t want to have to lug around six foot wheels, so we just keep two small pieces of the rim, and alternate them.” She waggled her feet.
Wassily squinted as he reckoned with what Saskia was saying. Mica, on the other hand, merely frowned.
“Our model even has two advantages over a traditional wheel. Firstly, by putting a bend-point in the center of our spokes we don’t need a fat tire at the end; our knees are shock absorbers. Secondly, getting nutrition to the rim is much easier since (a) we’ve only kept two small bits of the rim, and (b) we have direct lines to those bits. It’s much better for pumping blood.”
Within the moment’s silence that descended on the trio, Saskia thought she might have deflected Mica’s annoyance.
Mica, however, closed her eyes, took a breath, opened her eyes again, and turned to Wassily. “So, are you here to share another of insight about topological invariances?”
“Hey,” Saskia intervened gently, and turned to Wassily. “I liked your idea.” His suggestion that space-time was locally malleable, up to a point, nicely encapsulated the idea that the universe naturally prioritized global topological properties. It amounted to suggesting that the consistency of the overall embedding structure was more important than the details of the moment to moment; better to introduce little nudges to the fabric of space-time—when revisiting an event—than permit a time-travel paradox. Though she still wondered where that left doppelgangers.
Wassily nodded earnestly at Saskia’s conclusions. He turned to Mica while indicating Saskia. “Now she’s evolved, even without being able to slip in time.”
Saskia tipped her head to acknowledge Wassily’s kind words. “Even so,” she deflected, “I think LLMs, far more than me, are time travel impacting evolution.”
Mica scowled again, but Wassily bit: “Do tell.”
“At the very least, the LLMs are accelerators of evolution.” In Saskia’s view, it started with geological evolution, which was super slow. Then there was biological evolution, which was slow; unless you were a fruit fly, the rate of change was still measured in thousands of years. Cultural evolution was faster. It allowed whole societies to change in decades. “Then, there are LLMs, they accelerate everything. They take humanity’s knowledge and give it to us where and when we want it. They make web searches look inefficient, even if Google was already a leap ahead of a library, which in turn was a massive amplification over our oral traditions. They’re all forms of time travel, each a little more efficient than the last. Each putting us in dialogue with our ancestors, and people off in the future.”
Wassily tapped his nose in enthusiastic agreement. “I love it. Though it’s not what I was going to say.”
“Oh?” Saskia was surprised.
“No.” Wassily shook his head. “I wanted to explain why everything feels so heavy when you’re whizzing through time. And, more importantly, how that might open the door to another superpower that you didn’t realize you have.”
Well, Friends, that was chapter 56. I hope you enjoyed it!
I hope you’re also enjoying this new format with the commentary second. It occurred to me the other day that this new sequencing is kind of an unlock of a trapdoor. By this, I mean it allows bingers to smoothly follow the story without commentary interruption (as was the intention), but also without sacrificing the commentary; it’s easy enough to go back later in the podcast apps and pick up each episode where you ducked out. Moreover, while you could have previously sort of done that by listening to all the commentaries first, they don’t have the trigger that makes you want to skip to the next episode.
And anyway, I get it: it’s more fun to eat your cake first.
Why, you might ask, is a trapdoor the analogy that sprung to my mind? Well, trapdoor algorithms occasionally crop up in mathematics; they’re kind of a way of putting a name on a macro structure. The canonical `for instance’ is probably their use in modern cryptography. In essence, what underpins the safety of every financial transaction you make online is that it is much easier to multiply two numbers together than find a factorization. Trapdoors have an orientation for which way is easier to go through.
While on the topic of cool mathematics, and in case you’re not a subscriber to Grant Sanderson’s wonderful Youtube channel 3Blue1Brown, let me make you aware of his most recent video, since it is very on point for Wassily’s area of expertise. The video is a wonderful reflection on the subject of topology, and why mathematicians actually care about Mobius strips and Klein bottles. It’s also a superb example of mathematical taste. For the true layman it might take some concentrating, but if I were watching it in person with a fourth grader, I’m pretty confident I could explain most of it. Certainly, they would get the gist. I watched it with my youngest (who is admittedly taking a multivariable calculus class right now) and she thoroughly enjoyed it.
Lastly, before I go, one more fun thing: the idea, in today’s chapter, that humans have evolved wheels was something I pinched from a Q&A interview with Geoffrey Hinton a while back. In it, Hinton also talks about how we are actually now designing aircraft with feathers. It’s all well worth a listen (if you’re impatient, the wheels and feathers piece is about 31 min in if you’re scanning for it).
Until next week, be kind to someone and keep an eye out for the ripples of joy you’ve seeded.
Cheerio
Rufus
PS. If you think of someone who might enjoy joining us on this experiment, please forward them this email. And if you are one of those someone’s and you’d like to read more
PPS. Just a friendly reminder: If you’re enjoying this——still following along 56 weeks after we started this journey——I’d like to ask you one little favor: please make a personal recommendation to a friend you think might also enjoy this story. I’ve been reliably told that such recommendations are by far the best way to grow an audience, and what could be more fun than having someone else to talk about the story with! I’d, personally, consider this the best (ever so slightly belated) birthday present of all time. And, if you’re listening to this long after it’s original post: it is a time travel story after all, so I’ll still happily consider it a birthday present :)
