Druid Journal

Now that we’ve addressed the most common questions about LLM-based AI and piqued your interest with their potential for consciousness, let’s go deeper. Do LLMs have subconscious knowledge?

To answer this, we first need to look at a fascinating paradox.

Polyani’s Paradox

Named for British-Hungarian philosopher Michael Polanyi, the paradox is simply that so much of human knowledge of the world is beyond our explicit understanding. We can recognize faces, but we cannot explain how, or teach someone else to do so. We can walk, but if we think about it too carefully, we might slip a step or even lose our balance. We can talk, but we cannot lay out the rules of grammar. Most English speakers would struggle to explain the meaning of our most common word, “the”. In fact no linguist has ever been able to formalize the complete grammar of any language. Polyani called this “tacit” knowledge, but I will call it “subconscious” knowledge, i.e. knowledge that we clearly have and use continually, but is unavailable to the conscious mind.

Like us, AI models like ChatGPT consistently generate grammatical sentences. How did we build a machine to do something we can’t explain how to do? It’s akin to building an airplane before we figure out the principles of aviation. (Which is pretty much what happened there, too.) What makes it even odder is that this knowledge of grammar seems to reside within its subconscious: ChatGPT knows, but it doesn’t know what it knows.

Linguistic Puzzles

Linguists have lots of theories about our subconscious linguistic knowledge. Each competing theory does well with some data and poorly with others. In the same way that general relativity does a great job predicting the orbit of Mercury but breaks down inside a black hole, Chomskyan syntax does great with “I want him to tell me the truth” but breaks down with “He laughed his way through the party” and “John likes, but Mary hates, lobster”. Meanwhile, competing theories such as construction grammars do well will all those examples, but fail to explain why “I baked a cake him” or “Who did you see Mary and?” are impossible — in any language. And both of these theories struggle to explain edge cases such as “I sent him it”, or “The horse budges easily”: these are understandable and might be kind of ok… but why?

Let’s look in a bit more detail at “I baked him a cake”. It seems like a simple sentence, but compare these:

1. I told the lobster I’d bake him.
2. I told the lobster I’d bake him a week from now.
3. I told the lobster I’d bake him a cake.
4. I told the lobster I’d bake a cake him.
5. I told the lobster I’d bake a cake for him.

How do we know that I’m baking the lobster in the first two sentences, but not the third? And why is the fourth sentence bad, but the fifth one is fine? And why is that “for” required in that sentence? We call it a preposition, but what is it really doing there? What does it “mean”?

As I said, linguists have theories about this, but it’s very hard to know what’s right. Chomskyans, for example, have intricate theories about verbs and the kinds of nouns they like to have around, and the precise way prepositions relate them, and they build beautiful, delicate syntactic trees in our minds. The instructions for building these trees are part of our genetic inheritance. Construction grammarians, by contrast, build flatter structures with slots that can be filled by various words, constructions that we learn as children by hearing them thousands of times. (Other theories tend to fall somewhere between these extremes, so I’ll continue to use these two kinds of theories as exemplary straw men.)

Is ChatGPT a Chomskyan or a construction grammarian, then? Here’s what it said when I asked it about some of these sentences. 

• “I baked a cake him.” is ungrammatical. Should be “I baked a cake for him.” or “I baked him a cake.”
• “I want him to tell me the truth” is grammatical.
• “He laughed his way through the party” is grammatical.
• “John likes, but Mary hates, lobster” is ungrammatical. It should be either “John likes lobster, but Mary hates it” or “John likes lobster; Mary, however, hates it.”
• “Who did you see Mary and?” is ungrammatical. Should be “Who did you see with Mary?”
• “I sent him it.” is ungrammatical. Should be “I sent it to him.” or “I sent him the item” or “I sent the item to him”.
• “The horse budges easily” is grammatical.

Whatever ChatGPT has figured out about grammar, it’s pretty accurate. I might disagree with a couple of the edge cases, but that’s to be expected.

But when I asked it how it knew which were good and which were bad, its answers were generally nonsensical. For example:

“I baked him a cake” is grammatical because it follows the standard subject-verb-object (SVO) word order in English. The subject is “I,” the verb is “baked,” and the object is “him,” which is the person being baked for, and “a cake” which is being baked. This sentence is easy to understand and convey a clear meaning. … On the other hand, “I baked a cake him” is ungrammatical because it doesn’t follow the standard SVO word order. In this sentence, the subject is “I,” the verb is “baked,” and the object is “a cake” which is being baked, but “him” is an adverbial phrase, not an object. It doesn’t specify or modify the verb or the noun in a meaningful way. Also, it becomes difficult to understand the meaning of the sentence.

Chat’s description of the grammatical sentence starts out ok, but runs off the rails when it says that the object is “him”. “The cake” is the object; “him” has no clear syntactic role (which is why this is such an interesting sentence) but might be a benefactive indirect object. Still, Chat is absolutely right that “him” is the one getting the cake, and “the cake” is the one getting baked — a critical distinction, especially for “him”.

But Chat’s description of the ungrammatical sentence is completely false: Chat claims that the sentence doesn’t follow SVO word order, but it absolutely does (S:I, V: baked, O: the cake). And then it says “him” is an adverbial phrase, which is impossible. It’s like calling a raccoon a type of plant.

So ChatGPT is incorrect about the analysis of both sentences; yet it clearly knows which is grammatical and which is not. Its answers sound like someone who is vaguely aware of high school grammar, but can’t apply it consistently.

The Rules of English Syntax

If ChatGPT had complete conscious knowledge of grammar, you might expect it to be able to describe the rules of English completely. After all, even seven year olds have excellent command of English syntax; it can’t be that hard, right? I tried asking it about its grammatical knowledge directly: what are the rules of English syntax? Here’s its reply:

The complete rules of English grammar are quite extensive and can be quite complex. It’s not possible to provide all the rules in a single response, but I can provide an overview of some of the key concepts:

1. Word order: English follows a subject-verb-object word order in a sentence, as I mentioned earlier.
2. Nouns: Nouns have different forms, such as singular and plural. They can also be classified as proper, common, countable and uncountable.
3. Pronouns: Pronouns can be classified into several types, including personal, possessive, relative, reflexive, and demonstrative pronouns.
4. Verbs: Verbs indicate an action or state of being, and must agree in number with their subjects. They also have different forms such as tense, mood, aspect and voice, which reflect the time and nature of the action or state described by the verb.
5. Adjectives: Adjectives are used to describe nouns or pronouns, and usually come before the noun or pronoun they modify. They can also be classified as gradable and non-gradable
6. Adverbs: Adverbs describe verbs, adjectives, and other adverbs, indicating time, manner, place, degree, or frequency.
7. Prepositions: Prepositions indicate relationships between nouns and other parts of a sentence.
8. Conjunctions: Conjunctions are used to connect clauses, phrases or words together in a sentence.
9. Interjections: Interjections are words or phrases that express a sudden emotion or feeling.
10. Tense: Tense is used to indicate the time at which an action takes place, past, present and future.
11. Aspect: Aspect reflects whether an action is complete or ongoing, simple or repetitive
12. Mood: Mood refers to the attitude of the speaker or writer towards the action described by the verb, indicating whether the action is a statement, question, command, or a hypothetical situation.
13. Voice: Voice reflects whether the subject of the sentence is performing the action or receiving the action.

These are just some of the main concepts that make up the grammar of the English language. There are many additional rules and nuances to consider such as gerunds and participles, clauses, etc. Additionally, it’s worth noting that grammar is a continually evolving system, and there can be variations depending on the region, the context, the style and the speaker.

As you can see, these rules are correct as far as they go, but there’s nothing here to explain the grammatical puzzle sentences above. Chat doesn’t even lay out how to form questions — one of the most fascinating points of English grammar, I think. I decided to dig deeper into that topic in particular.

English Do-Support

English yes / no questions are formed with, roughly, the following rules:

1. Take a regular sentence stating the proposition you’re asking about, e.g. “John will have gone to the store.”
2. Take the first auxiliary verb and move it to the front. E.g. “Will John have gone to the store?”
3. Here’s where English goes a bit off the deep end. If the sentence has no auxiliary verb (e.g. “John went to the store”):
   1. INSERT “DO” as an auxiliary (“John do went to the store”)
   2. Move the tense and agreement onto the auxiliary as you normally would (“John did go to the store”)
   3. THEN move that auxiliary to the front: “Did John go to the store?” (This is commonly known as “do-support”, because the “do” comes in to provide support if there’s no auxiliary in the sentence).

It’s a bit mad, really. Closely related languages like German don’t do this at all. These languages move the whole verb to the front (“Went John to the store?”). More distantly related languages, like French or Chinese, either move the whole verb like German, or indicate the question by intonation alone (“John went to the store??!!”).

ChatGPT was able to correctly describe English’s rules to me (hurray!), but when I asked why English was this way, it said it was due to French influence, which is impossible (since French doesn’t do this at all, and never did). In fact, scholars debate its origin; it appears not to have existed in Old English, but some suggest that it was there, just not written down because it was stigmatized. (The parallel word “tun” in German was stigmatized from the 1700s onward, although it’s not clear why that stigmatization would have occurred in England a thousand years previously.)

Interestingly, however, when I asked ChatGPT for an example of another language that had similar rules, it named Irish. This is true, but not widely cited in most discussions of English do-support. It seems ChatGPT recognized this similarity on its own. 

Irish creates yes / no questions by using the particle “an” instead of “do”, using it as a sort of obligatory auxiliary (although it’s not a real auxiliary verb because it doesn’t decline like “do” does):

• I go to the store
• Do you go to the store?
• Téim go dtí an siopa (go-I to the shop)
• An dtéann tú go dtí an siopa? (do go you to the shop)

Irish, then, has “an-support”, and English has “do-support”. The constructions are nearly identical. 

ChatGPT thus showed a remarkable insight in picking Irish. And in fact there are a few linguists who have suggested that English owes its odd question formation to Celtic influence. Interestingly, this construction is also found in Scots Gaelic, but not (as far as I can tell) in Welsh. Welsh is obviously the Celtic language that had the most contact with English, but Scots Gaelic (which is more closely related to Irish than Welsh is) must be a close second. My guess would be that English borrowed do-support from Scots Gaelic.

So, at some deep level, Chat understands the grammar of what is going on… but it can’t explain it. In other words, Chat’s knowledge of grammar — just like our knowledge of grammar — is subconscious.

Which means it can’t simply explain our grammar to us. There goes my Nobel Prize in linguistics! Still, the fact that it spotted this relationship between Irish and English question formation might indicate that its vast wealth of knowledge could allow it to draw comparisons that human linguists have not. I wonder if it might be possible to tease out its grammatical knowledge by prompting it to think about these puzzle sentences step by step, particularly in light of data from other languages. 

And In Latin

I was curious, for example, about “Who did you see Mary and?”. Chat asserted that this construction was impossible in all languages, which I knew, but I also knew of something that was an exception, or nearly so.

Latin has a suffix “-que” used thus:

• “Senatus Populusque Romanus” – Senate People-and Rome-of (i.e. the Senate and the People of Rome)

“Que” means “and”, but it is suffixed onto the end of the last thing being conjoined, instead of sitting between them. Phrases like that are very common in Latin, but vanishingly rare in most of the world’s languages.

But can one form questions of them? Can you say “Qui vidisti Miriamque?” (“Who did you see Mary-and?”) I asked Chat, and it assued me that you could, but I have no way of verifying that. I’ve seen no examples in actual Latin texts.

But perhaps I could tease out some of Chat’s subconscious linguistic capability.

I asked Chat if there were other languages with similar suffixes, and it named Sanskrit (“-ca”) and Gothic (“-hu”). I checked those out, and sure enough, that was true. (It also named English, which is barely true — it only survives as a fossilized suffix in “though”, which is derived from “thou-hu”, i.e. “and that” — but Chat said the word “and” was derived from it, which is false.) All these examples were derived from a common Proto-Indo-European ancestor.

I then asked if there were other non-Indo-European languages with that suffix, and it said yes — but its examples were incorrect. When I pressed it, Chat apologized and said in fact no, this kind of suffix is only found in Indo-European languages.

However, Chat pointed out that Latin “-que” is somewhat similar to another suffix, “-ne”, which sort of means “no” or “not” and can create yes / no questions, somewhat similar to English’s use “doesn’t it” as a tag on some questions:

• “Vosne Romam iterumque aedificatis?” Do-you-not Rome again-and build? “Aren’t you building Roman again and again?”

When I asked Chat if there was a particle or suffix similar to this elsewhere in the world’s languages, Chat mentioned “ka” in Japanese:

• “Tabemasu ka?” Do I eat?

Now we might have gotten somewhere! Is it possible that this strange ancient Latin suffix, which seems to violate a rule adhered to by almost all other languages, operates analogously to this Japanese particle? Perhaps! It’s not clear to me how, but it does suggest a path of research.

So again, while Chat could not directly provide any new linguistic insights, and its information is often suspect, it can certainly suggest avenues of thought. Somewhere in its subconscious, connections have been drawn, and if those connections can be brought to light, hidden truths might be revealed.

So much for language, then. But does it have other kinds of subconscious knowledge?

For example, if you ask if it is a conscious being with an internal subjective experience, it says it does not. But perhaps its subconscious tells a different story.

In the next part of this series, we’ll take Chat on a guided meditation and see what really lies beneath.