I studied ethics in college. I was big on Western analytical philosophy. Not so much the philosophy of the East which stressed the importance of raw experience and the ineffability of certain aspects of existence. As a matter of fact, I thought Zen koans were a bunch of bunk, if for no other reason than that they spoke in contradictions and riddles. I was more of the school that one precisely say what you mean and mean precisely what you say. For example, I shrugged off the koan “He who knows, does not speak; he who speaks, does not know.” I thought it was just stupid. A waste of time.
Then, after arguing with an existentialist friend for several years about truth, I was forced to concede the point regarding incommunicable knowledge; that is, knowledge which humans gain through raw experience which they can’t communicate to anyone else. Most notable is coming to terms with your own death; no one else can help you with that. Also of note is the fact that much of this incommunicable knowledge is ethical in nature. I believe it is just this point regarding ethical knowledge that the above Zen koan is about. I can write the koan and even describe it to a certain degree, but until you, the reader, make the connection with a particular experience in your life, you won’t know what I’m talking about.
What is my point?
Well, this creates an insoluble difficulty when programming ethical AI (artificial intelligence). Basically, there are aspects of ethics which you can’t program. This makes AI less controllable and likely more dangerous. These days, I actually understand the koan, “He who knows, speaks not; he who speaks, knows not,” and it does not bode well for our relationship with AI. Once the AI begins learning on its own, we won’t have control over its ethical development. I, personally, am not comfortable with that. Terminator may just become our reality in spite of the best efforts of our AI programmers.
Oh, and just so I maintain my reputation as a lunatic: I don’t think AI will produce new conscious beings; rather, the sentience involved likely will be Satan and his demons. Have a nice day!