Depends on the product. From an original AI research point of view this is what you want, a model that can realize it is missing information and deviates from giving a result. But once profit became involved, marketing requires a fully confident output to get everyone to buy in. So we get what we get and not something that’s more reliable.
It’s not just that, it’s also the fact they scored the responses based on user feedback, and users tend to give better feedback for more confident, even if wrong, responses.
Depends on the product. From an original AI research point of view this is what you want, a model that can realize it is missing information and deviates from giving a result. But once profit became involved, marketing requires a fully confident output to get everyone to buy in. So we get what we get and not something that’s more reliable.
It’s not just that, it’s also the fact they scored the responses based on user feedback, and users tend to give better feedback for more confident, even if wrong, responses.