Vicuna

Vicuna is an open-source chatbot that has made significant waves in the field of natural language processing. Trained by fine-tuning LLaMA on user-shared conversations from ShareGPT, Vicuna demonstrates competitive performance. It achieves more than 90% quality of OpenAI ChatGPT and Google Bard, outperforming other models like LLaMA and Stanford Alpaca in many cases. The training of Vicuna is a result of careful data collection and processing. Around 70K conversations were gathered from ShareGPT.com, and the training scripts were enhanced to handle multi-turn conversations and long sequences. The training was conducted with PyTorch FSDP on 8 A100 GPUs in a single day. To ensure the quality of the data, HTML was converted back to markdown, and inappropriate or low-quality samples were filtered out. Additionally, lengthy conversations were divided into smaller segments to fit the model's maximum context length. Vicuna's performance is evaluated using an innovative framework based on GPT-4. This framework involves creating diverse questions across eight categories and having GPT-4 rate the responses of different chatbots. The results show that Vicuna outperforms state-of-the-art open-source models in more than 90% of the questions and achieves competitive performance against proprietary models. However, like other large language models, Vicuna has its limitations. It struggles with tasks involving reasoning or mathematics and may have issues with self-identification and factual accuracy. Despite these limitations, Vicuna serves as an important open starting point for future research to address these challenges.