{"id":8805,"date":"2026-06-04T17:17:39","date_gmt":"2026-06-04T09:17:39","guid":{"rendered":"https:\/\/infernews.com\/blog\/self-distilled-policy-gradient\/"},"modified":"2026-06-04T17:19:20","modified_gmt":"2026-06-04T09:19:20","slug":"self-distilled-policy-gradient","status":"publish","type":"post","link":"https:\/\/infernews.com\/blog\/self-distilled-policy-gradient\/","title":{"rendered":"SDPG\uff1a\u81ea\u6211\u84b8\u993e\u53ca\u734e\u52f5\u7684\u8a13\u7df4\u6f14\u7b97\u6cd5"},"content":{"rendered":"\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/infernews.com\/blog\/wp-content\/uploads\/2026\/06\/pasted-5e1c5a70464b.jpg\" alt=\"Repository image for lauyikfung\/SDPG\"\/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">\u5728\u5927\u578b\u8a9e\u8a00\u6a21\u578b\u7684\u5f8c\u8a13\u7df4\u968e\u6bb5\uff0c\u5f37\u5316\u5b78\u7fd2\u53ef\u9a57\u8b49\u734e\u52f5\uff08RLVR\uff09\u5df2\u662f\u6578\u5b78\u8207\u7a0b\u5f0f\u63a8\u7406\u7684\u5e38\u7528\u914d\u65b9\u3002\u7136\u800c\u7576\u734e\u52f5\u53ea\u7d66\u5230\u5e8f\u5217\u5c64\u7d1a\uff0c\u6a21\u578b\u5728\u8a13\u7df4\u521d\u671f\u5bb9\u6613\u9047\u5230\u8a0a\u865f\u7a00\u758f\u3001\u512a\u52e2\u503c\u70ba\u8ca0\u6642\u4e0d\u7a69\u5b9a\u7684\u554f\u984c\u3002SDPG\uff08Self-Distilled Policy Gradient\uff09\u6b63\u662f\u91dd\u5c0d\u9019\u5169\u500b\u75db\u9ede\u800c\u8a2d\u8a08\u7684\u958b\u6e90\u9805\u76ee\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u9019\u500b\u9805\u76ee\u5c07 GRPO \u64f4\u5c55\u70ba\u4e00\u7a2e\u81ea\u6211\u84b8\u993e\u5f0f\u7684\u7b56\u7565\u68af\u5ea6\u65b9\u6cd5\uff1a\u5728\u540c\u4e00\u500b\u6a21\u578b\u4e2d\uff0c\u5b78\u751f\u53ea\u63a5\u6536\u554f\u984c\uff0c\u800c\u6559\u5e2b\u984d\u5916\u63a5\u6536\u7279\u6b0a\u8108\u7d61 c\u3002\u5169\u8005\u4e4b\u9593\u4ee5 full-vocabulary \u7684 token-level KL \u6563\u5ea6\u5373\u6642\u8a08\u7b97\u84b8\u993e\u8a0a\u865f\uff0c\u70ba\u8a13\u7df4\u63d0\u4f9b\u66f4\u5bc6\u96c6\u7684\u76e3\u7763\uff1b\u540c\u6642\u7d50\u5408\u6a19\u6e96\u5dee\u6b78\u4e00\u5316\u8207\u53ef\u5207\u63db\u7684 \u03b1 \u53c3\u8003\u6b63\u5247\u5316\uff0c\u4ee5\u63d0\u5347\u8a13\u7df4\u7a69\u5b9a\u6027\u3002\u7531\u65bc\u5b78\u751f\u8207\u6559\u5e2b\u5171\u4eab\u540c\u4e00\u7d44\u53c3\u6578\uff0c\u6574\u9ad4\u8a2d\u8a08\u4e5f\u907f\u514d\u4e86\u984d\u5916\u90e8\u7f72\u5927\u578b\u6559\u5e2b\u6a21\u578b\u6240\u5e36\u4f86\u7684\u8a18\u61b6\u9ad4\u8ca0\u64d4\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u74b0\u5883\u9700\u8981 8 \u5f35 A100\u3001H100 \u6216 H200\uff0c\u4ee5\u53ca\u672c\u5730 Ray \u53e2\u96c6\uff1b\u9810\u8a2d\u6a21\u578b\u70ba Qwen\/Qwen3-4B\uff0c\u4ea6\u53ef\u6307\u5411\u672c\u5730\u6b0a\u91cd\u3002\u8cc7\u6599\u683c\u5f0f\u63a1\u7528\u7279\u6b8a token \u5206\u9694\u6f14\u54e1\u984c\u76ee\u8207\u6559\u5e2b\u8108\u7d61\uff0c\u76f8\u95dc\u8173\u672c\u7686\u5df2\u9644\u4e0a\u3002\u5c0d\u6b63\u5728\u7814\u7a76 RLHF \u6216\u60f3\u628a\u63a8\u7406\u6a21\u578b\u84b8\u993e\u5f97\u66f4\u7a69\u5b9a\u7684\u5718\u968a\u800c\u8a00\uff0c\u9019\u662f\u4e00\u500b\u9580\u6abb\u660e\u78ba\u3001\u53ef\u91cd\u73fe\u7684\u5be6\u4f5c\u85cd\u672c\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>\u91cd\u9ede\u6458\u8981<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>\u5728 GRPO \u4e4b\u4e0a\u52a0\u5165 exact per-token forward KL \u81ea\u6211\u84b8\u993e\uff0c\u7de9\u89e3\u7a00\u758f\u734e\u52f5\u554f\u984c<\/li>\n\n\n\n<li>\u5b78\u751f\u8207\u6559\u5e2b\u5171\u7528\u540c\u4e00\u6a21\u578b\uff0c\u514d\u53bb\u5927\u578b\u6559\u5e2b\u7684\u984d\u5916\u8a18\u61b6\u6210\u672c<\/li>\n\n\n\n<li>\u5167\u5efa\u56db\u7a2e \u03b1 \u6b63\u5247\u6a21\u5f0f\uff08fkl\u3001rkl\u3001ufkl\u3001urkl\uff09\uff0c\u65b9\u4fbf\u6d88\u878d\u5be6\u9a57<\/li>\n\n\n\n<li>\u9810\u8a2d\u652f\u63f4 Qwen\/Qwen3-4B\uff0c\u53ef\u5728 verl RLHF \u6846\u67b6\u4e0a\u76f4\u63a5\u904b\u884c<\/li>\n\n\n\n<li>\u786c\u9ad4\u9580\u6abb\u70ba 8 \u5f35 A100\/H100\/H200\uff0c\u642d\u914d\u672c\u5730 Ray \u53e2\u96c6\u5373\u53ef\u555f\u52d5<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>GitHub\uff1a<\/strong> <a href=\"https:\/\/github.com\/lauyikfung\/SDPG\" rel=\"noopener noreferrer\">https:\/\/github.com\/lauyikfung\/SDPG<\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Paper\uff1a<\/strong> <a href=\"https:\/\/arxiv.org\/pdf\/2606.04036\" rel=\"noopener noreferrer\">https:\/\/arxiv.org\/pdf\/2606.04036<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>SDPG \u900f\u904e\u300c\u7279\u6b0a\u8108\u7d61\u300d\u8b93\u6a21\u578b\u81ea\u6211\u7763\u5c0e\uff0c\u5728\u7a00\u758f\u734e\u52f5\u5834\u666f\u63d0\u5347\u5f37\u5316\u5b78\u7fd2\u7684\u7a69\u5b9a\u6027\u8207\u8868\u73fe\u3002<\/p>\n","protected":false},"author":8,"featured_media":8804,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ai_generated_summary":"","wpai_meta_description":"","footnotes":""},"categories":[133,127],"tags":[],"class_list":["post-8805","post","type-post","status-publish","format-standard","hentry","category-133","category-127"],"_links":{"self":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/posts\/8805","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/comments?post=8805"}],"version-history":[{"count":1,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/posts\/8805\/revisions"}],"predecessor-version":[{"id":8808,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/posts\/8805\/revisions\/8808"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/media\/8804"}],"wp:attachment":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/media?parent=8805"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/categories?post=8805"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/tags?post=8805"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}