My goal was to get a sense of whether the test-to-image AI-Generation tools are able to develop coherent results. Based on the results from ChatBots, my impression is the answer is not yet. This is a conservative position. My uncertainty lies first in the question of whether a tool whose answers are non-deterministic can be coherent. Yet in favour of text-to-image, while a conversation with a ChatBot takes place over a series of questions and answers, being coherent in a picture is only required within one “answer”. This may be easier to accomplish.
The second root of uncertainty is the discrepancy in capabilities between the tools I started this test with (ChatGPT, You.com and Chatsonic) and Microsoft Bing, a more recent addition. The difference highlights the limitations, and the lack of coherence in the former tools. Bing demonstrates material progress. So my assessment is that if coherence is not there now, it is certainly likely the technology will get there. This means in the interim vigilance is required to assess the veracity of replies.
So while this study was intended to look at text-to-image, I drew some observations specific to ChatBots. The first point to make is one can be easily fooled into misplacing too much trust in these tools based on the confidence expressed in each answer. This is especially important at this early phase in the development of the technology. Extreme caution needs to be taken in accepting the veracity of an answer.
However, there are benefits*. Even if a ChatBot’s answer was incomplete, which many were, at least it provides a starting point for further research. When references to source materials are cited, a path is provided from that starting point. This is very useful, offering significant advantage over regular search engines by avoiding spending hours following the rabbit holes prevalent in the 10 million replies offered in the typical Google Search.
In the end, we might think of this as some sort of Turing Test. Are the flawed — inconsistent, partial, sometimes repetitive, and maybe wrong — answers from the ChatBot that much different than those we might get from a human? It is not unheard of that when talking with a fellow human, sometimes you have to ask several questions to get an answer. It does matter how you phrase the question, including the level of detail, and in certain cases asking the same question several different times can lead to different answers or repetitions. Should we expect an AI-based Chatbot to be better, worse, or about the same as a human?
* I’m aware of the ethical and other problematic issues reported elsewhere with these tools by other reviewers.
Leave a Reply