Found this in a children’s book of riddles:
Six brothers were spending their time together.
The first brother was reading a book.
The second brother was playing chess.
The third brother was solving a crossword.
The fourth brother was watering the lawn.
The fifth brother was drawing a picture.
Question: what was the sixth brother doing?
I cant get ChatGPT to answer correctly with the usual tricks, even after hinting to consider one and two-person activities and emphasizing the word “together”.
After a bunch of CoT turns we arrive to a conclusion that this is an open ended question and not a riddle :)
After trying 3 times with fresh promots, I got a correct response once, but when prompted to provide supporting reasoning the model backtracked and started apologizing.
Cant test gpt 4 r/n…
one way i can tell that ai’s really made it is that in just the past couple of years the riddles that befuddle language models went from being “What’s the capital of France?” to increasingly being riddles that i can’t solve at first glance myself either