{"id":2453,"date":"2024-05-14T18:07:51","date_gmt":"2024-05-14T10:07:51","guid":{"rendered":"https:\/\/infernews.com\/?p=2453"},"modified":"2024-05-20T02:29:00","modified_gmt":"2024-05-19T18:29:00","slug":"hiyouga-llama-factory-unify-efficient-fine-tuning-of-100-llms","status":"publish","type":"post","link":"https:\/\/infernews.com\/blog\/hiyouga-llama-factory-unify-efficient-fine-tuning-of-100-llms\/","title":{"rendered":"LLaMA-Factory: \u9ad8\u6548\u5fae\u8abf LLM"},"content":{"rendered":"<div class=\"vlp-link-container vlp-layout-spotlight-clone wp-block-visual-link-preview-link\"><a href=\"https:\/\/github.com\/hiyouga\/LLaMA-Factory\/blob\/main\/README_zh.md\" class=\"vlp-link\" title=\"\u76f8\u6bd4ChatGLM\uff0cLLaMA Factory \u7684 LoRA \u5fae\u8abf\u63d0\u4f9b\u4e86 3.7 \u500d\u7684\u52a0\u901f\" rel=\"nofollow\" target=\"_blank\"><\/a><div class=\"vlp-layout-zone-main\"><span class=\"vlp-block-0 vlp-link-title\">\u76f8\u6bd4ChatGLM\uff0cLLaMA Factory \u7684 LoRA \u5fae\u8abf\u63d0\u4f9b\u4e86 3.7 \u500d\u7684\u52a0\u901f<\/span><div class=\"vlp-block-1 vlp-link-summary\">\u652f\u6301\u591a\u7a2e\u6a21\u578b\uff1aLLaMA\u3001LLaVA\u3001Mistral\u3001Mixtral-MoE\u3001Qwen\u3001Yi\u3001Gemma\u3001Baichuan\u3001ChatGLM\u3001Phi <\/div><div class=\"vlp-block-2 vlp-link-image\"><img decoding=\"async\" src=\"https:\/\/repository-images.githubusercontent.com\/646410686\/ccb14d87-7454-4e82-a85c-f1f63a3c1ccb\" style=\"max-width: 1024px; max-height: 1024px\" \/><\/div><\/div><\/div>\n\n\n<p>\u4f60\u53ef\u9078\u7528\u672c\u5730\u90e8\u8655\u6216\u4f7f\u7528 <a href=\"https:\/\/colab.research.google.com\/drive\/1d5KQtbemerlSDSxZIfAaWXhKr30QypiK?usp=sharing\" target=\"_blank\" rel=\"noreferrer noopener\">Colab Notbook<\/a><\/p>\n\n\n\n<p><strong>\u9805\u76ee\u7279\u8272<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>\u591a<\/strong>\u7a2e<strong>\u6a21\u578b<\/strong>\uff1aLLaMA\u3001LLaVA\u3001Mistral\u3001Mixtral-MoE\u3001Qwen\u3001Yi\u3001Gemma\u3001Baichuan\u3001ChatGLM\u3001Phi \u7b49\u7b49\u3002<\/li>\n\n\n\n<li><strong>\u96c6\u6210\u65b9\u6cd5<\/strong>\uff1a\uff08\u589e\u91cf\uff09\u9810\u8a13\u7df4\u3001\uff08\u591a\u6a21\u614b\uff09\u6307\u4ee4\u76e3\u7763\u5fae\u8abf\u3001\u734e\u52f5\u6a21\u578b\u8a13\u7df4\u3001PPO \u8a13\u7df4\u3001DPO \u8a13\u7df4\u3001KTO \u8a13\u7df4\u548c ORPO \u8a13\u7df4\u3002<\/li>\n\n\n\n<li><strong>\u591a\u7a2e\u7cbe\u5ea6<\/strong>\uff1a32 \u6bd4\u7279\u5168\u53c3\u6578\u5fae\u8abf\u300116 \u6bd4\u7279\u51cd\u7d50\u5fae\u8abf\u300116 \u6bd4\u7279 LoRA \u5fae\u8abf\u548c\u57fa\u65bc AQLM\/AWQ\/GPTQ\/LLM.int8 \u7684 2\/4\/8 \u6bd4\u7279 QLoRA \u5fae\u8abf\u3002<\/li>\n\n\n\n<li><strong>\u5148\u9032\u7b97\u6cd5<\/strong>\uff1aGaLore\u3001BAdam\u3001DoRA\u3001LongLoRA\u3001LLaMA Pro\u3001Mixture-of-Depths\u3001LoRA+\u3001LoftQ \u548c Agent \u5fae\u8abf\u3002<\/li>\n\n\n\n<li><strong>\u5be6\u7528\u6280\u5de7<\/strong>\uff1aFlashAttention-2\u3001Unsloth\u3001RoPE scaling\u3001NEFTune \u548c rsLoRA\u3002<\/li>\n\n\n\n<li><strong>\u5be6\u9a57\u76e3\u63a7<\/strong>\uff1aLlamaBoard\u3001TensorBoard\u3001Wandb\u3001MLflow \u7b49\u7b49\u3002<\/li>\n\n\n\n<li><strong>\u6975\u901f\u63a8\u7406<\/strong>\uff1a\u57fa\u65bc vLLM \u7684 OpenAI \u98a8\u683c API\u3001\u700f\u89bd\u5668\u754c\u9762\u548c\u547d\u4ee4\u884c\u63a5\u53e3\u3002<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>\u4f60\u53ef\u9078\u7528\u672c\u5730\u90e8\u8655\u6216\u4f7f\u7528 Colab Notbook \u9805\u76ee\u7279\u8272<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"googlesitekit_rrm_CAowvqSiDA:productID":"","footnotes":""},"categories":[23],"tags":[],"class_list":["post-2453","post","type-post","status-publish","format-standard","hentry","category-news"],"_links":{"self":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/posts\/2453","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/comments?post=2453"}],"version-history":[{"count":0,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/posts\/2453\/revisions"}],"wp:attachment":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/media?parent=2453"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/categories?post=2453"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/tags?post=2453"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}