A recent study has highlighted the effectiveness of natural language feedback in improving the performance of language models. A team of researchers from KAIST has introduced a new SelFee model designed explicitly for self-feedback and self-revision generation. Unlike previous approaches, SelFee does not require external, significant language or task-specific models to generate high-quality responses.
SelFee is a fine-tuned LLaMA-based instruction-following model that continuously revises its answers until it achieves a high-quality response within a single inference. Based on the given instruction, the model generates an initial solution and self-feedback sequences. By analyzing the content of the generated feedback, the model determines if a revision is needed. If so, it generates a revised answer based on the feedback. This iterative revision process is completed within a single inference, resulting in improved solutions compared to existing LLaMA-based models.
The researchers collected diverse instruction data from various sources, such as ShareGPT, Alpaca, Math, Code, and Flan Collection. To address the scarcity of feedback and revision data, they augmented the dataset using a distillation process from a teacher model called ChatGPT. This approach allowed them to generate more instances of feedback and revision at a more affordable cost.
To train the model, the researchers utilized data augmentation techniques using OpenAI API calls. They collected instructions from multiple sources and input them into ChatGPT to generate corresponding answers. They then obtained feedback on the generated answers by querying ChatGPT again. If a revision was deemed necessary, ChatGPT revised the answer based on self-generated feedback. This process was repeated until no further modifications were required.
SelFee was trained using the FastChat framework. Based on the instruction, the model was fine-tuned to generate the answer and feedback chain, including revisions. The researchers observed that increasing the minimum required revisions during the inference process improved answer quality. They found that a minimum of three revisions yielded the best performance, and even a 7B SelFee model that generated at least three revisions outperformed a 13B SelFee model that did not require modifications.
In terms of evaluation, the researchers adopted the Vicuna evaluation setting, which involved 80 diverse queries. Instead of conducting a human evaluation, they performed a pilot evaluation using GPT-4 as the evaluator. The relative scores compared to ChatGPT were reported, considering the positional bias of GPT-4.
While SelFee demonstrated comparable performance to ChatGPT in the Vicuna evaluation setting, it was found to lack knowledge in areas such as math, reasoning, factuality, and coding compared to ChatGPT.
Overall, SelFee introduces a novel approach to self-feedback and self-revision generation in language models. By fine-tuning the model to revise its answers continuously, SelFee achieves improved performance compared to existing models. The research findings highlight the importance of iterative revision in enhancing the quality of language model responses and suggest that increasing the inference computation of a model may be more effective than simply increasing its size.
n