AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |
Back to Blog
![]() Add support for CodeLlama by in #25740.Remove SDPA for falcon to support earlier versions of PyTorch (Add proper Falcon docs and conversion script by in #25954. ![]() ![]() Falcon port #24523 by Falcon: Add RoPE scaling by in #25878.Both ‘base’ models trained only as causal language models as well as ‘instruct’ models that have received further fine-tuning are available. They are made available under the Apache 2.0 license.įalcon’s architecture is modern and optimized for inference, with multi-query attention and support for efficient attention variants like FlashAttention. The largest Falcon checkpoints have been trained on >=1T tokens of text, with a particular emphasis on the RefinedWeb corpus. Falcon is a class of causal decoder-only models built by TII.
0 Comments
Read More
Leave a Reply. |