{"id":6286,"date":"2025-08-29T15:31:17","date_gmt":"2025-08-29T07:31:17","guid":{"rendered":"https:\/\/infernews.com\/?page_id=6286"},"modified":"2025-08-29T15:31:18","modified_gmt":"2025-08-29T07:31:18","slug":"local-inference-hack","status":"publish","type":"page","link":"https:\/\/infernews.com\/blog\/local-inference-hack\/","title":{"rendered":"Local Inference hack"},"content":{"rendered":"<figure class=\"wp-block-embed-youtube wp-block-embed is-type-video is-provider-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"lyte-wrapper\" title=\"2500\u5143\u4e0d\u5230\u88c5\u4e00\u53f0AI\u670d\u52a1\u5668\uff01\u53ccRadeon VII\u89e3\u9501vLLM\u5f20\u91cf\u5e76\u884c\uff0c\u6027\u80fd\u66b4\u6da86\u500d\u78be\u538bOllama\uff01\" style=\"width:853px;max-width:100%;margin:5px auto;\"><div class=\"lyMe\" id=\"WYL_OOwS_HHOWfs\" itemprop=\"video\" itemscope itemtype=\"https:\/\/schema.org\/VideoObject\"><div><meta itemprop=\"thumbnailUrl\" content=\"https:\/\/infernews.com\/blog\/wp-content\/plugins\/wp-youtube-lyte\/lyteCache.php?origThumbUrl=https%3A%2F%2Fi.ytimg.com%2Fvi%2FOOwS_HHOWfs%2Fhqdefault.jpg\" \/><meta itemprop=\"embedURL\" content=\"https:\/\/www.youtube.com\/embed\/OOwS_HHOWfs\" \/><meta itemprop=\"duration\" content=\"PT8M\" \/><meta itemprop=\"uploadDate\" content=\"2025-08-27T16:00:48Z\" \/><\/div><div id=\"lyte_OOwS_HHOWfs\" data-src=\"https:\/\/infernews.com\/blog\/wp-content\/plugins\/wp-youtube-lyte\/lyteCache.php?origThumbUrl=https%3A%2F%2Fi.ytimg.com%2Fvi%2FOOwS_HHOWfs%2Fhqdefault.jpg\" class=\"pL\"><div class=\"tC\"><div class=\"tT\" itemprop=\"name\">2500\u5143\u4e0d\u5230\u88c5\u4e00\u53f0AI\u670d\u52a1\u5668\uff01\u53ccRadeon VII\u89e3\u9501vLLM\u5f20\u91cf\u5e76\u884c\uff0c\u6027\u80fd\u66b4\u6da86\u500d\u78be\u538bOllama\uff01<\/div><\/div><div class=\"play\"><\/div><div class=\"ctrl\"><div class=\"Lctrl\"><\/div><div class=\"Rctrl\"><\/div><\/div><\/div><noscript><a href=\"https:\/\/youtu.be\/OOwS_HHOWfs\" rel=\"nofollow\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/infernews.com\/blog\/wp-content\/plugins\/wp-youtube-lyte\/lyteCache.php?origThumbUrl=https%3A%2F%2Fi.ytimg.com%2Fvi%2FOOwS_HHOWfs%2F0.jpg\" alt=\"2500\u5143\u4e0d\u5230\u88c5\u4e00\u53f0AI\u670d\u52a1\u5668\uff01\u53ccRadeon VII\u89e3\u9501vLLM\u5f20\u91cf\u5e76\u884c\uff0c\u6027\u80fd\u66b4\u6da86\u500d\u78be\u538bOllama\uff01\" width=\"853\" height=\"460\" \/><br \/>Watch this video on YouTube<\/a><\/noscript><meta itemprop=\"description\" content=\"\ud83d\udca52400\u5143\u9884\u7b97\u6311\u6218AI\u7b97\u529b\u5929\u82b1\u677f\uff01\u672c\u671f\u89c6\u9891\uff0c\u6211\u4eec\u89e3\u51b3\u4e86Radeon VII \/ MI50 \u7b49 gfx906 \u67b6\u6784\u663e\u5361\u957f\u671f\u4ee5\u6765\u65e0\u6cd5\u4f7f\u7528 vLLM \u5f20\u91cf\u5e76\u884c\u7684\u75db\u70b9\uff01 \u6211\u4eec\u5c06\u5168\u7a0b\u5c55\u793a\u5982\u4f55\u7528\u4e24\u5f20\u201c\u8fc7\u6c14\u201dRadeon VII\u663e\u5361\uff0c\u642d\u914dX99\u201c\u6d0b\u5783\u573e\u201d\u5e73\u53f0\uff0c\u7ec4\u88c5\u4e00\u53f0\u603b\u4ef7\u4ec52397\u5143\u7684AI\u7b97\u529b\u670d\u52a1\u5668\u3002\u901a\u8fc7\u793e\u533a\u5927\u795e nlzy \u63d0\u4f9b\u7684\u7279\u5236Docker\u5bb9\u5668\uff0c\u6211\u4eec\u6210\u529f\u89e3\u9501\u4e86vLLM\u7684\u5f20\u91cf\u5e76\u884c\u529f\u80fd\uff0c\u5728Qwen3 32B\u91cf\u5316\u6a21\u578b\u4e0a\u5b9e\u73b0\u4e8698 tokens\/s\u7684\u5e76\u53d1\u63a8\u7406\u901f\u5ea6\uff0c\u6027\u80fd\u662fOllama\u76846\u500d\u4ee5\u4e0a\uff01 \u89c6\u9891\u6700\u540e\uff0c\u6211\u4eec\u8fd8\u5c06\u901a\u8fc7Open WebUI\uff0c\u5b9e\u6218\u6f14\u793a8\u4e2a\u7528\u6237\u540c\u65f6\u8bbf\u95ee\u8fd9\u53f0\u670d\u52a1\u5668\u7684\u6d41\u7545\u4f53\u9a8c\uff01 \u672c\u671f\u786c\u4ef6\u914d\u7f6e\u6e05\u5355 (\u603b\u8ba1: ~2397\u5143) \u663e\u5361 (GPU): AMD Radeon VII 16GB x 2 (\u4e8c\u624b ~1600\u5143) \u4e3b\u677f (Motherboard): \u7cbe\u7ca4 X99M-PLUS (~300\u5143) \u5904\u7406\u5668 (CPU): Intel Xeon E5-2680 v4 (56\u5143) \u5185\u5b58 (RAM): \u6d77\u529b\u58eb ECC REG DDR4 8GB x 4 (176\u5143) \u786c\u76d8 (Storage): 512GB NVMe SSD (~150\u5143) \u7535\u6e90 (PSU): \u989d\u5b9a850W \u91d1\u724c\u8ba4\u8bc1\u7535\u6e90 (\u4e8c\u624b 115\u5143) \u3010\u672c\u671f\u65b9\u6848\u8d85\u8be6\u7ec6\u6587\u5b57\u7248\u6559\u7a0b\u3011 \u4e3a\u4e86\u65b9\u4fbf\u5927\u5bb6\u590d\u73b0\uff0c\u6211\u4eec\u5df2\u7ecf\u628a\u4eca\u5929\u6240\u6709\u7528\u5230\u7684\u547d\u4ee4\u3001\u94fe\u63a5\uff0c\u4ee5\u53ca\u4e00\u4efd\u8d85\u8be6\u7ec6\u7684\u6587\u5b57\u7248\u90e8\u7f72\u6559\u7a0b\uff0c\u5168\u90e8\u6574\u7406\u597d\uff0c\u653e\u5230\u4e86UP\u4e3b\u7684Gitee\u4e3b\u9875\u4e0a\uff01 \u76f4\u8fbe\u94fe\u63a5\uff1ahttps:\/\/gitee.com\/spoto\/R7vllm \u5982\u679c\u89c9\u5f97\u672c\u671f\u89c6\u9891\u5bf9\u4f60\u6709\u5e2e\u52a9\uff0c\u8bf7\u52a1-\u52a1-\u52a1\u5fc5 \u70b9\u8d5e\u3001\u6295\u5e01\u3001\u6536\u85cf \u4e09\u8fde\u652f\u6301\u4e00\u4e0bUP\u4e3b\uff01\u4f60\u4eec\u7684\u652f\u6301\u662f\u6211\u7206\u809d\u66f4\u65b0\u7684\u6700\u5927\u52a8\u529b\uff01\u611f\u8c22\u5927\u5bb6\uff01\"><\/div><\/div><div class=\"lL\" style=\"max-width:100%;width:853px;margin:5px auto;\"><\/div><figcaption><\/figcaption><\/figure>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"googlesitekit_rrm_CAowvqSiDA:productID":"","footnotes":""},"class_list":["post-6286","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/pages\/6286","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/comments?post=6286"}],"version-history":[{"count":0,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/pages\/6286\/revisions"}],"wp:attachment":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/media?parent=6286"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}