2 Comments
User's avatar
⭠ Return to thread
Keith Lowery's avatar

Haha! This is a great response. To be honest, the answer to "why bother?" was that I was quarantined in a hotel room, 1500 miles from home, waiting to finish Covid so I could fly home without being a slug and exposing everyone around me. I was bored and wanted to develop a better sense for what the most recent language models are capable of and these are some questions I wanted answered. I didn't doubt that some kind of propagandizing was going on, but now I have a better idea who and how and how much.

All of the examples of ideas you would like to flag are really stellar things to assess. Excellent really. While you seem to be self-assured that you already have a comprehensive grasp of the limitations of all these models and everything they will ever be capable of, the pace of innovation and constant algorithmic enhancements make me more inclined toward gathering empirical data rather than simply trusting my own untested opinions. Part of the purpose of this exercise was to test the actual capabilities and not presume I already had all the answers. Also, by building the automation, I'm able to test the limitations of new models as they emerge without going to so much trouble the next time.

Expand full comment
Ryan Davidson's avatar

Thanks! All of that's fair enough. Indeed, "Because I was stuck, isolated, and bored" strikes me as as an entirely sufficient reason to try something like this. You could certainly do a lot worse. Like, say, actually watching any of those shows, eh?

Though I the limitation I'm trying to get at is one of kind, not degree, I think. The human mind perceives, expresses, experiences, and generates meaning in ways I believe are fundamentally irreducible to propositions. Human consciousness isn't algorithmic, and meaning is a function of consciousness.

This is, for example, why there can never be a truly definitive, single "correct" meaning of, say, any decent poem (though there can certainly be definitively incorrect ones!). Or why the same poem can mean different things to the same person at different times.

The same is true of all narratives, to a greater or lesser extent. And the better and more effectively these layers of meaning are constructed, the less effective algorithmic analysis becomes.

So if you are going to subject a large n of material to algorithmic analysis, kids shows are probably your best bet. They're typically a lot simpler and less multilayered than material targeted at adults, or even teenagers. Young children don't have enough experience to be able to comprehend subtlety to any helpful degree.

Expand full comment