I’m personally skeptical that LLMs provide enough benefit to outweigh their potential harm, and so I deeply appreciate those who are attempting to make them work without relying on pixie dust.
Simon Willison provides some spot-on guidance regarding both the promise and limitations of these technologies.
I think the message we should be emphasizing is this:
These are incredibly powerful tools. They are far harder to use effectively than they first appear. Invest the effort, but approach with caution: we accidentally invented computers that can lie to us and we can’t figure out how to make them stop.
There’s a time for linguistics, and there’s a time for grabbing the general public by the shoulders and shouting “It lies! The computer lies to you! Don’t trust anything it says!”
— Simon Willison, We need to tell people ChatGPT will lie to them, not debate linguistics