A recent study has highlighted the effectiveness of natural language feedback in improving the performance of language models. A team of researchers from KAIST has introduced a new SelFee model designed explicitly for self-feedback and self-revision generation. Unlike previous approaches, SelFee does not require external, significant language or task-specific models to generate high-quality responses.
SelFee is a refined LLaMA-based instruction-following model that iteratively revises its solutions until it generates an accurate response in only one inference. The model develops an initial solution and self-feedback sequences based on the provided instruction. The model evaluates if a revision is required by examining the generated feedback’s content. If so, it produces a new response based on the comments. The iterative revision process is finished in a single inference, producing better answers than those produced by the current LLaMA-based models.A variety of instruction data was gathered by the researchers from multiple databases, including ShareGPT, Alpaca, Math, Code, and Flan Collection. Using a ChatGPT instructor model distillation method, they expanded the dataset. Using this strategy, they were able to produce more instances of feedback and revision at a more affordable cost.
To train the model, the researchers utilized data augmentation techniques using OpenAI API calls. They collected instructions from multiple sources and input them into ChatGPT to generate corresponding answers. They then obtained feedback on the generated answers by querying ChatGPT again. If a revision was deemed necessary, ChatGPT revised the answer based on self-generated feedback. This process was repeated until no further modifications were required.
SelFee was trained using the FastChat framework. Based on the instruction, the model was fine-tuned to generate the answer and feedback chain, including revisions. The researchers observed that increasing the minimum required revisions during the inference process improved answer quality. They found that a minimum of three revisions yielded the best performance, and even a 7B SelFee model that generated at least three revisions outperformed a 13B SelFee model that did not require modifications.
In terms of evaluation, the researchers adopted the Vicuna evaluation setting, which involved 80 diverse queries. Instead of conducting a human evaluation, they performed a pilot evaluation using GPT-4 as the evaluator. The relative scores compared to ChatGPT were reported, considering the positional bias of GPT-4.
While SelFee demonstrated comparable performance to ChatGPT in the Vicuna evaluation setting, it was found to lack knowledge in areas such as math, reasoning, factuality, and coding compared to ChatGPT.
Overall, SelFee introduces a novel approach to self-feedback and self-revision generation in language models. By fine-tuning the model to revise its answers continuously, SelFee achieves improved performance compared to existing models. The research findings highlight the importance of iterative revision in enhancing the quality of language model responses and suggest that increasing the inference computation of a model may be more effective than simply increasing its size.