Test the first line
##Test Title
-The test column asks AI a question that it does not understand, and it often does not say "I don't know", but smoothly compiles a reasonable sounding answer. This phenomenon is known as' hallucination 'in the industry. It is neither a simple program defect nor a deliberate deception of the model, but an inherent characteristic brought about by the working principle of current large language models.
1、 What is illusion
Illusion usually refers to the model generating smooth and confident content, but in fact it is incorrect or unverifiable. It can be roughly divided into two categories: one is factual illusion, such as fabricating non-existent book titles, papers, legal provisions, or historical events; Another type is fidelity illusion, where the output of a model contradicts the input material it receives, such as when summarizing a document and drawing conclusions that are not present in the original text.
2、 Where do hallucinations come from
1. Predict the next word instead of verifying the facts. The core goal of the big language model is to predict the next most likely word element based on the previous context. It learns the statistical rules of language, rather than a retrievable fact database. When knowledge is missing, the model will still write down towards the direction of 'most similar answer'.
2. The training data itself has noise. The Internet corpus is mixed with outdated information, false statements and fictional content, and the model will be learned together.
3. Knowledge has a deadline. The knowledge of the model stays at the time point of training data, and it is not aware of what happens afterwards, but it may still provide old answers.
4. Randomness brought by sampling. In order to make the answers more natural, there is usually a certain degree of randomness in the generation process (such as temperature, Top-p sampling), which can amplify the possibility of deviation.
5. Side effects of alignment process. Human preferences often lean towards "giving an answer" rather than "I don't know", and reinforcement learning can make models more willing to confidently answer to a certain extent.
3、 How to alleviate
There is currently no method that can completely eliminate illusions, but there are several widely adopted paths in engineering:
·Retrieval Enhanced Generative (RAG): First, retrieve credible information, and then have the model answer based on the information and attach the source, turning "relying on memory" into "looking at materials".
·Tool invocation: Delegate tasks such as arithmetic, real-time queries, and database retrieval to external tools, and the model is only responsible for organizing the language.
·Make the model acknowledge uncertainty: Encourage it to explicitly state 'unable to confirm' when there is insufficient evidence through training and prompts.
·Multiple sampling and cross validation: Sampling the same question multiple times and comparing the answers to see if they are consistent. Inconsistencies often indicate risks.
·Reduce sampling randomness: Use lower temperatures when rigorous conclusions are needed to minimize divergence.
·Manual review and citation verification: treat model output as a draft rather than a final draft, especially in high-risk scenarios such as healthcare, legal, and finance.
4、 One sentence summary
Illusion is a phenomenon that accompanies the underlying mechanism of the big language model, which generates language through statistical means. It can be significantly reduced through retrieval, tool and process design, but it is difficult to completely eliminate. Only by understanding this can we not blindly believe in every word of AI in practical use, nor give up eating for fear of choking.
【 Reference source 】 Comprehensive compilation of publicly available machine learning research and industry technical materials. entry