\u5927\u578b\u8a9e\u8a00\u6a21\u578b Llama \u7684\u547d\u540d\u65b9\u6848\u3002\u4e3b\u8981\u89e3\u91cb\u6a21\u578b\u540d\u7a31\u4e2d\u4e0d\u540c\u90e8\u5206\u7684\u542b\u7fa9\uff0c\u4f8b\u5982 Q \u4ee3\u8868\u91cf\u5316\u3001K \u4ee3\u8868\u4e00\u7a2e\u91cf\u5316\u65b9\u6cd5\u3001S\u3001M\u3001L \u4ee3\u8868\u5927\u5c0f\uff0c\u4ee5\u53ca FP16 \u7b49\u8868\u793a\u6d6e\u9ede\u6578\u7cbe\u5ea6\u7684\u683c\u5f0f\u3002\u6211\u5011\u4e26\u5206\u4eab\u4e0d\u540c\u91cf\u5316\u65b9\u6cd5\u5c0d\u6a21\u578b\u5927\u5c0f\u548c\u6548\u80fd\u7684\u5f71\u97ff\uff0c\u4e26\u5efa\u8b70\u9078\u64c7\u80fd\u517c\u9867\u6548\u80fd\u548c\u8a18\u61b6\u9ad4\u4f7f\u7528\u7684\u91cf\u5316\u7b49\u7d1a\u3002<\/p>\n\n\n\n
\u4ee5\u4e0b\u8aaa\u660e\u4e0d\u540c\u5927\u5c0f\u7684 LLaMA \u6a21\u578b\u5728\u91cf\u5316\u5f8c\u7684\u6548\u80fd\u8868\u73fe\u6bd4\u8f03\uff1a<\/p>\n\n\n\n
\u91cf\u5316 (Quantization) \u7684\u57fa\u672c\u6982\u5ff5<\/strong><\/p>\n\n\n\n \u91cf\u5316\u662f\u4e00\u7a2e\u58d3\u7e2e\u6280\u8853\uff0c\u65e8\u5728\u6e1b\u5c11\u6a21\u578b\u7684\u5927\u5c0f\uff0c\u4f7f\u5176\u66f4\u6613\u65bc\u7ba1\u7406\u3002 \u91cf\u5316\u683c\u5f0f<\/strong>\uff1aQ<\u6578\u5b57><0 \u6216 1> <\/p>\n\n\n\n \u820a\u5f0f\u91cf\u5316\u65b9\u6cd5\uff0c\u73fe\u5728\u4e0d\u5efa\u8b70\u4f7f\u7528\uff1aQ<\u6578\u5b57>_K<\u5b57\u6bcd>: <\/p>\n\n\n\n \u73fe\u4ee3 K \u91cf\u5316\u65b9\u6cd5\uff1a\u6578\u5b57\u8868\u793a\u6bcf\u500b\u6b0a\u91cd\u7684\u8fd1\u4f3c\u4f4d\u6578 (bits-per-weight, bpw)\u3002 iMatrix<\/strong><\/p>\n\n\n\n iMatrix \u5728\u91cf\u5316\u6a21\u578b\u4e4b\u524d\u6703\u9032\u884c\u6e2c\u8a66\uff0c\u4ee5\u78ba\u5b9a\u54ea\u4e9b\u90e8\u5206\u61c9\u8a72\u5177\u6709\u66f4\u9ad8\u7684\u7cbe\u5ea6\uff0c\u54ea\u4e9b\u90e8\u5206\u53ef\u4ee5\u964d\u4f4e\u7cbe\u5ea6\u3002 \u4e0d\u540c\u6a21\u578b\u5927\u5c0f\u7684\u6bd4\u8f03<\/strong><\/p>\n\n\n\n \u4e00\u822c\u539f\u5247\uff1a\u5728\u76f8\u540c\u7684\u91cf\u5316\u65b9\u6848\u4e0b\uff0c\u66f4\u591a\u4f4d\u6578\u901a\u5e38\u4ee3\u8868\u66f4\u597d\u7684\u6548\u80fd\u3002\u4f8b\u5982\uff0cQ4_K_S \u901a\u5e38\u6703\u6bd4 Q3_K_L \u8868\u73fe\u66f4\u597d\u3002 \u5176\u4ed6\u8003\u91cf<\/strong><\/p>\n\n\n\n \u786c\u9ad4\u9650\u5236\uff1a\u60a8\u7684\u786c\u9ad4\uff08\u4f8b\u5982 VRAM \u6216 RAM\uff09\u6703\u9650\u5236\u60a8\u53ef\u4ee5\u4f7f\u7528\u54ea\u7a2e\u91cf\u5316\u3002 \u7d50\u8ad6 \u4e0d\u540c\u5927\u5c0f\u7684 LLaMA \u6a21\u578b\u5728\u91cf\u5316\u5f8c\u7684\u6548\u80fd\u8868\u73fe\u6703\u53d7\u5230\u591a\u7a2e\u56e0\u7d20\u5f71\u97ff\uff0c\u5305\u62ec\u91cf\u5316\u65b9\u6cd5\u3001\u6a21\u578b\u5927\u5c0f\u3001\u5177\u9ad4\u61c9\u7528\u548c\u786c\u9ad4\u9650\u5236\u3002\u5728\u9078\u64c7\u91cf\u5316\u65b9\u6cd5\u6642\uff0c\u61c9\u7d9c\u5408\u8003\u91cf\u4ee5\u4e0a\u5404\u9805\u56e0\u7d20\u3002<\/p>\n\n\n\n Allowed quantization types: \u5927\u578b\u8a9e\u8a00\u6a21\u578b Llama \u7684\u547d\u540d\u65b9\u6848\u3002\u4e3b\u8981\u89e3\u91cb\u6a21\u578b\u540d\u7a31\u4e2d\u4e0d\u540c\u90e8\u5206\u7684\u542b\u7fa9\uff0c\u4f8b\u5982 Q \u4ee3\u8868\u91cf\u5316\u3001K \u4ee3\u8868\u4e00\u7a2e\u91cf\u5316\u65b9\u6cd5\u3001S\u3001M\u3001L \u4ee3\u8868\u5927\u5c0f\uff0c\u4ee5\u53ca FP16 \u7b49\u8868\u793a\u6d6e\u9ede\u6578\u7cbe\u5ea6\u7684\u683c\u5f0f\u3002\u6211\u5011\u4e26\u5206\u4eab\u4e0d\u540c\u91cf\u5316\u65b9\u6cd5\u5c0d\u6a21\u578b\u5927\u5c0f\u548c\u6548\u80fd\u7684\u5f71\u97ff\uff0c\u4e26\u5efa\u8b70\u9078\u64c7\u80fd\u517c\u9867\u6548\u80fd\u548c\u8a18\u61b6\u9ad4\u4f7f\u7528\u7684\u91cf\u5316\u7b49\u7d1a\u3002 \u4ee5\u4e0b\u8aaa\u660e\u4e0d\u540c\u5927\u5c0f\u7684 LLaMA \u6a21\u578b\u5728\u91cf\u5316\u5f8c\u7684\u6548\u80fd\u8868\u73fe\u6bd4\u8f03\uff1a \u91cf\u5316 (Quantization) \u7684\u57fa\u672c\u6982\u5ff5 \u91cf\u5316\u662f\u4e00\u7a2e\u58d3\u7e2e\u6280\u8853\uff0c\u65e8\u5728\u6e1b\u5c11\u6a21\u578b\u7684\u5927\u5c0f\uff0c\u4f7f\u5176\u66f4\u6613\u65bc\u7ba1\u7406\u3002\u91cf\u5316\u901a\u904e\u964d\u4f4e\u6b0a\u91cd\u7684\u7cbe\u5ea6\u4f86\u5be6\u73fe\u9019\u4e00\u9ede\u3002\u4e0d\u540c\u7684\u91cf\u5316\u65b9\u6cd5\u6703\u5c0d\u6a21\u578b\u6548\u80fd\u7522\u751f\u4e0d\u540c\u7684\u5f71\u97ff.GGUF \u662f\u4e00\u7a2e\u5e38\u898b\u7684\u91cf\u5316\u547d\u540d\u65b9\u6848\uff0c\u4e0d\u7279\u5b9a\u65bc\u67d0\u500b\u6a21\u578b\u3002 \u91cf\u5316\u683c\u5f0f\uff1aQ<\u6578\u5b57><0 \u6216 1> \u820a\u5f0f\u91cf\u5316\u65b9\u6cd5\uff0c\u73fe\u5728\u4e0d\u5efa\u8b70\u4f7f\u7528\uff1aQ<\u6578\u5b57>_K<\u5b57\u6bcd>: \u73fe\u4ee3 K \u91cf\u5316\u65b9\u6cd5\uff1a\u6578\u5b57\u8868\u793a\u6bcf\u500b\u6b0a\u91cd\u7684\u8fd1\u4f3c\u4f4d\u6578 (bits-per-weight, bpw)\u3002\u5b57\u6bcd\u8868\u793a\u5927\u5c0f\uff1aS (\u5c0f), M (\u4e2d), L (\u5927)\u3002\u4f8b\u5982\uff0cQ4_K_S \u5927\u7d04\u662f 4.5 bpw\uff0c\u800c Q4_K_M \u5927\u7d04\u662f 4.8 bpw\u3002\u5982\u679c\u53ea\u6709 Q_K \u800c\u6c92\u6709\u5c3e\u96a8\u5b57\u6bcd\uff0c\u901a\u5e38\u8868\u793a M (\u4e2d)\u3002iQ<\u6578\u5b57>_<\u5b57\u6bcd>: \u53e6\u4e00\u7a2e\u91cf\u5316\u683c\u5f0f\uff0c\u4ee5\u901f\u5ea6\u63db\u53d6\u6548\u7387\u3002\u5728\u76f8\u540c\u7684 bpw \u4e0b\uff0ciQ \u91cf\u5316\u6703\u66f4\u300c\u8070\u660e\u300d\uff0c\u4f46\u8a08\u7b97\u6642\u9593\u7a0d\u9577\u3002\u6578\u5b57\u540c\u6a23\u8868\u793a\u6bcf\u500b\u6b0a\u91cd\u7684\u8fd1\u4f3c\u4f4d\u6578\u3002\u5b57\u6bcd\u8868\u793a\u8a72\u985e\u5225\u5167\u7684\u5927\u5c0f\uff0c\u4f8b\u5982 iQ2 \u6709 XXS\u3001XS\u3001S \u548c M \u7b49\u7d1a\u3002FP16, BF16, FP32: \u539f\u59cb\u7684\u672a\u91cf\u5316\u6b0a\u91cd\uff0c\u5206\u5225\u70ba […]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"googlesitekit_rrm_CAowvqSiDA:productID":"","footnotes":""},"class_list":["post-4413","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/pages\/4413","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/comments?post=4413"}],"version-history":[{"count":0,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/pages\/4413\/revisions"}],"wp:attachment":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/media?parent=4413"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\u91cf\u5316\u901a\u904e\u964d\u4f4e\u6b0a\u91cd\u7684\u7cbe\u5ea6\u4f86\u5be6\u73fe\u9019\u4e00\u9ede\u3002
\u4e0d\u540c\u7684\u91cf\u5316\u65b9\u6cd5\u6703\u5c0d\u6a21\u578b\u6548\u80fd\u7522\u751f\u4e0d\u540c\u7684\u5f71\u97ff.
GGUF \u662f\u4e00\u7a2e\u5e38\u898b\u7684\u91cf\u5316\u547d\u540d\u65b9\u6848\uff0c\u4e0d\u7279\u5b9a\u65bc\u67d0\u500b\u6a21\u578b\u3002<\/p>\n\n\n\n
\u5b57\u6bcd\u8868\u793a\u5927\u5c0f\uff1aS (\u5c0f), M (\u4e2d), L (\u5927)\u3002\u4f8b\u5982\uff0cQ4_K_S \u5927\u7d04\u662f 4.5 bpw\uff0c\u800c Q4_K_M \u5927\u7d04\u662f 4.8 bpw\u3002
\u5982\u679c\u53ea\u6709 Q_K \u800c\u6c92\u6709\u5c3e\u96a8\u5b57\u6bcd\uff0c\u901a\u5e38\u8868\u793a M (\u4e2d)\u3002
iQ<\u6578\u5b57>_<\u5b57\u6bcd>: \u53e6\u4e00\u7a2e\u91cf\u5316\u683c\u5f0f\uff0c\u4ee5\u901f\u5ea6\u63db\u53d6\u6548\u7387\u3002\u5728\u76f8\u540c\u7684 bpw \u4e0b\uff0ciQ \u91cf\u5316\u6703\u66f4\u300c\u8070\u660e\u300d\uff0c\u4f46\u8a08\u7b97\u6642\u9593\u7a0d\u9577\u3002
\u6578\u5b57\u540c\u6a23\u8868\u793a\u6bcf\u500b\u6b0a\u91cd\u7684\u8fd1\u4f3c\u4f4d\u6578\u3002
\u5b57\u6bcd\u8868\u793a\u8a72\u985e\u5225\u5167\u7684\u5927\u5c0f\uff0c\u4f8b\u5982 iQ2 \u6709 XXS\u3001XS\u3001S \u548c M \u7b49\u7d1a\u3002
FP16, BF16, FP32: \u539f\u59cb\u7684\u672a\u91cf\u5316\u6b0a\u91cd\uff0c\u5206\u5225\u70ba 16 \u4f4d\u6d6e\u9ede\u6578\u300116 \u4f4d Brain Float \u548c 32 \u4f4d\u6d6e\u9ede\u6578\u683c\u5f0f\u3002\u9664\u975e\u60a8\u6253\u7b97\u81ea\u5df1\u91cd\u65b0\u91cf\u5316\uff0c\u5426\u5247\u4e0d\u5efa\u8b70\u4f7f\u7528\u3002<\/p>\n\n\n\n
\u5e73\u5747\u4f4d\u6578\u76f8\u540c\uff0c\u4f46\u4f4d\u6578\u7684\u5206\u914d\u66f4\u6709\u6548\u7387\uff0c\u7d50\u679c\u662f\u66f4\u8070\u660e\u7684\u91cf\u5316\u6a21\u578b\u3002
\u5728\u9078\u64c7\u91cf\u5316\u65b9\u5f0f\u6642\uff0ciMatrix \u901a\u5e38\u662f\u66f4\u597d\u7684\u9078\u64c7\u3002<\/p>\n\n\n\n
\u4f46\u9019\u4e26\u975e\u7dda\u6027\u95dc\u4fc2\uff0c\u964d\u4f4e\u7cbe\u78ba\u5ea6\u6642\u7684\u6548\u80fd\u640d\u5931\u4e26\u975e\u7dda\u6027\u4e0b\u964d\u3002
\u8f03\u5927\u578b\u865f\u8207\u8f03\u5c0f\u578b\u865f\u7684\u6bd4\u8f03\uff1a
\u8f03\u5c0f\u7684\u91cf\u5316 (\u4f8b\u5982 Q2) \u53ef\u80fd\u9069\u7528\u65bc\u8f03\u5927\u7684\u6a21\u578b (\u4f8b\u5982 70B)\uff0c\u4f46\u5728\u8f03\u5c0f\u7684\u6a21\u578b\u4e0a\u53ef\u80fd\u5c0e\u81f4\u6548\u80fd\u4e0b\u964d\u3002
\u5373\u4f7f\u662f 70B \u6a21\u578b\u7684\u8f03\u5c0f\u91cf\u5316\u7248\u672c\u4e5f\u53ef\u80fd\u6bd4 8B \u6a21\u578b\u7684\u8f03\u5927\u91cf\u5316\u7248\u672c\u66f4\u597d\u3002
\u4f4e\u65bc 4 bpw \u7684\u91cf\u5316\u662f\u76f8\u7576\u503c\u5f97\u61f7\u7591\u7684\u3002
70B \u7b49\u8f03\u5927\u4e14\u5bc6\u96c6\u7684\u6a21\u578b\u5728\u91cd\u5ea6\u58d3\u7e2e\u4e0b\u7684\u6548\u80fd\u6bd4\u5c0f\u578b\u6216\u7a00\u758f\u6a21\u578b\u66f4\u597d\u3002
8B \u548c 70B \u6a21\u578b\u5728\u6578\u64da\u96c6\u548c\u8a13\u7df4\u6280\u8853\u4e0a\u96d6\u6709\u91cd\u758a\uff0c\u4f46\u5b83\u5011\u662f\u5177\u6709\u4e0d\u540c\u80fd\u529b\u7684\u5b8c\u5168\u4e0d\u540c\u6a21\u578b\u3002
\u5177\u9ad4\u61c9\u7528\uff1a
\u5c0d\u65bc\u9700\u8981\u56b4\u683c\u9075\u5faa\u683c\u5f0f\u7684\u4efb\u52d9 (\u4f8b\u5982\u7a0b\u5f0f\u8a2d\u8a08)\uff0c\u8f03\u4f4e\u7684 bpw \u91cf\u5316\u6703\u5c0e\u81f4\u6548\u80fd\u4e0d\u4f73\u3002
\u5c0d\u65bc\u5275\u9020\u6027\u5beb\u4f5c\u7b49\u6c92\u6709\u300c\u932f\u8aa4\u300d\u7b54\u6848\u7684\u4efb\u52d9\uff0c\u8f03\u4f4e\u7684 bpw \u91cf\u5316\u53ef\u80fd\u53ef\u4ee5\u63a5\u53d7\u3002
\u6839\u64da\u500b\u4eba\u7d93\u9a57\uff0cQ4_K \u5c0d\u65bc\u5927\u578b\u6a21\u578b\u662f\u4e00\u500b\u57fa\u672c\u8981\u6c42\uff0c\u800c Q6_K \u5c0d\u65bc\u5c0f\u578b\u6a21\u578b\u5247\u662f\u4e00\u500b\u57fa\u672c\u8981\u6c42\u3002<\/p>\n\n\n\n
\u8a18\u61b6\u9ad4\u4f7f\u7528\uff1a \u8f03\u9ad8\u7684\u91cf\u5316\u7b49\u7d1a\u6703\u9700\u8981\u66f4\u591a\u8a18\u61b6\u9ad4\uff0c\u56e0\u6b64\u9700\u8981\u8003\u616e\u60a8\u7684\u786c\u9ad4\u9650\u5236.
Perplexity (PPL):
PPL \u662f\u4e00\u7a2e\u7528\u65bc\u8861\u91cf\u91cf\u5316\u6a21\u578b\u6548\u80fd\u7684\u6307\u6a19\u3002
PPL \u503c\u8d8a\u4f4e\uff0c\u8868\u793a\u6a21\u578b\u300c\u8d8a\u8070\u660e\u300d\u3002
\u7576\u6a21\u578b\u5f9e\u539f\u59cb\u7248\u672c\u91cf\u5316\u5f8c\uff0cPPL \u7684\u589e\u9577\u8868\u793a\u91cf\u5316\u5c0d\u6a21\u578b\u7684\u640d\u5bb3\u7a0b\u5ea6\u3002
\u4f8b\u5982\uff0cQ2_K_S \u91cf\u5316\u5c0e\u81f4 Llama-3-8B \u6a21\u578b\u589e\u52a0\u4e86 3.18 \u7684 PPL \u503c\u3002<\/p>\n\n\n\n
\u6c92\u6709\u7d55\u5c0d\u7684\u597d\u58de\u3002 \u54ea\u7a2e\u91cf\u5316\u6700\u9069\u5408\u60a8\u53d6\u6c7a\u65bc\u60a8\u7684\u9700\u6c42\u3001\u61c9\u7528\u7a0b\u5f0f\u548c\u786c\u9ad4\u9650\u5236\u3002
\u5efa\u8b70\uff1a\u76e1\u53ef\u80fd\u4f7f\u7528\u60a8\u53ef\u4ee5\u627f\u53d7\u7684\u6700\u9ad8 bpw\uff0c\u4ee5\u7372\u5f97\u6700\u4f73\u6548\u80fd\u3002<\/p>\n\n\n\n
2 or Q4_0 : 4.34G, +0.4685 ppl @ Llama-3-8B
3 or Q4_1 : 4.78G, +0.4511 ppl @ Llama-3-8B
8 or Q5_0 : 5.21G, +0.1316 ppl @ Llama-3-8B
9 or Q5_1 : 5.65G, +0.1062 ppl @ Llama-3-8B
19 or IQ2_XXS : 2.06 bpw quantization
20 or IQ2_XS : 2.31 bpw quantization
28 or IQ2_S : 2.5 bpw quantization
29 or IQ2_M : 2.7 bpw quantization
24 or IQ1_S : 1.56 bpw quantization
31 or IQ1_M : 1.75 bpw quantization
10 or Q2_K : 2.96G, +3.5199 ppl @ Llama-3-8B
21 or Q2_K_S : 2.96G, +3.1836 ppl @ Llama-3-8B
23 or IQ3_XXS : 3.06 bpw quantization
26 or IQ3_S : 3.44 bpw quantization
27 or IQ3_M : 3.66 bpw quantization mix
12 or Q3_K : alias for Q3_K_M
22 or IQ3_XS : 3.3 bpw quantization
11 or Q3_K_S : 3.41G, +1.6321 ppl @ Llama-3-8B
12 or Q3_K_M : 3.74G, +0.6569 ppl @ Llama-3-8B
13 or Q3_K_L : 4.03G, +0.5562 ppl @ Llama-3-8B
25 or IQ4_NL : 4.50 bpw non-linear quantization
30 or IQ4_XS : 4.25 bpw non-linear quantization
15 or Q4_K : alias for Q4_K_M
14 or Q4_K_S : 4.37G, +0.2689 ppl @ Llama-3-8B
15 or Q4_K_M : 4.58G, +0.1754 ppl @ Llama-3-8B
17 or Q5_K : alias for Q5_K_M
16 or Q5_K_S : 5.21G, +0.1049 ppl @ Llama-3-8B
17 or Q5_K_M : 5.33G, +0.0569 ppl @ Llama-3-8B
18 or Q6_K : 6.14G, +0.0217 ppl @ Llama-3-8B
7 or Q8_0 : 7.96G, +0.0026 ppl @ Llama-3-8B
33 or Q4_0_4_4 : 4.34G, +0.4685 ppl @ Llama-3-8B
34 or Q4_0_4_8 : 4.34G, +0.4685 ppl @ Llama-3-8B
35 or Q4_0_8_8 : 4.34G, +0.4685 ppl @ Llama-3-8B
1 or F16 : 14.00G, +0.0020 ppl @ Mistral-7B
32 or BF16 : 14.00G, -0.0050 ppl @ Mistral-7B
0 or F32 : 26.00G @ 7B<\/p>\n","protected":false},"excerpt":{"rendered":"