{"id":4453,"date":"2025-02-05T20:00:49","date_gmt":"2025-02-05T12:00:49","guid":{"rendered":"https:\/\/infernews.com\/?p=4453"},"modified":"2025-03-17T01:16:53","modified_gmt":"2025-03-16T17:16:53","slug":"deepseek-r1-%e6%a8%a1%e5%9e%8b%e7%9a%84%e8%a8%93%e7%b7%b4%e9%81%8e%e7%a8%8b","status":"publish","type":"post","link":"https:\/\/infernews.com\/blog\/deepseek-r1-%e6%a8%a1%e5%9e%8b%e7%9a%84%e8%a8%93%e7%b7%b4%e9%81%8e%e7%a8%8b\/","title":{"rendered":"DeepSeek R1 \u6a21\u578b\u7684\u8a13\u7df4\u904e\u7a0b"},"content":{"rendered":"\n<p>\u77ed\u7247\u95e1\u8ff0 DeepSeek R1 \u6a21\u578b\u7684\u8a13\u7df4\u904e\u7a0b\uff0c\u6838\u5fc3\u662f\u57fa\u65bc\u4eba\u985e\u56de\u994b\u7684\u5f37\u5316\u5b78\u7fd2\u3002\u9996\u5148\uff0c\u77ed\u7247\u89e3\u91cb\u5982\u4f55\u5229\u7528\u4eba\u985e\u504f\u597d\u8a13\u7df4\u734e\u52f5\u6a21\u578b (reward model)\uff1a\u6536\u96c6\u4eba\u985e\u5c0d\u4e0d\u540c\u6a21\u578b\u8f38\u51fa\u7684\u8a55\u5206\uff0c\u900f\u904e\u4f8b\u5982 Softmax \u51fd\u6578\u548c\u68af\u968e\u964d\u6cd5\uff0c\u8abf\u6574\u734e\u52f5\u6a21\u578b\uff0c\u4f7f\u5176\u7d66\u4e88\u4eba\u985e\u504f\u597d\u7684\u8f38\u51fa\u66f4\u9ad8\u5206\u6578\u3002\u77ed\u7247\u4ea6\u8aaa\u660e\u5982\u4f55\u4f7f\u7528\u8fd1\u7aef\u7b56\u7565\u6700\u4f73\u5316 (PPO) \u6f14\u7b97\u6cd5\uff0c\u7d50\u5408\u734e\u52f5\u6a21\u578b\u548c\u50f9\u503c\u6a21\u578b (value model) \u4f86\u5fae\u8abf\u8a9e\u8a00\u6a21\u578b (policy network)\uff1a\u6839\u64da\u734e\u52f5\u6a21\u578b\u7d66\u51fa\u7684\u734e\u52f5\uff0c\u4ee5\u53ca\u8a55\u50f9\u6a21\u578b\u9810\u6e2c\u7684\u734e\u52f5\u8207\u9810\u671f\u5dee\u7570 (advantage)\uff0c\u8abf\u6574\u7b56\u7565\u7db2\u7d61\uff0c\u4f7f\u5176\u66f4\u50be\u5411\u7522\u751f\u9ad8\u734e\u52f5\u7684\u8f38\u51fa\u3002\u6700\u5f8c\uff0c\u77ed\u7247\u7279\u5225\u4ecb\u7d39 DeepSeek R1 \u4f7f\u7528\u7684\u7fa4\u9ad4\u76f8\u5c0d\u7b56\u7565\u6700\u4f73\u5316 (group relative policy optimization)\uff0c\u9019\u662f\u4e00\u7a2e\u6539\u826f\u7684 PPO \u65b9\u6cd5\uff0c\u5c07\u734e\u52f5\u8207\u7fa4\u9ad4\u5167\u5176\u4ed6\u8f38\u51fa\u7684\u5e73\u5747\u734e\u52f5\u76f8\u6bd4\uff0c\u9f13\u52f5\u7522\u751f\u512a\u65bc\u5e73\u5747\u6c34\u6e96\u7684\u8f38\u51fa\uff0c\u89e3\u6c7a\u4e86\u50b3\u7d71\u734e\u52f5\u6a21\u578b\u53ef\u80fd\u51fa\u73fe\u7684\u300c\u734e\u52f5\u4f5c\u5f0a\u300d\u554f\u984c\u3002<\/p>\n\n\n<figure class=\"wp-block-embed-youtube wp-block-embed is-type-video is-provider-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"lyte-wrapper\" title=\"Reinforcement Learning in DeepSeek-R1 | Visually Explained\" style=\"width:853px;max-width:100%;margin:5px auto;\"><div class=\"lyMe\" id=\"WYL_H20Hd6Xb7Qo\" itemprop=\"video\" itemscope itemtype=\"https:\/\/schema.org\/VideoObject\"><div><meta itemprop=\"thumbnailUrl\" content=\"https:\/\/infernews.com\/blog\/wp-content\/plugins\/wp-youtube-lyte\/lyteCache.php?origThumbUrl=https%3A%2F%2Fi.ytimg.com%2Fvi%2FH20Hd6Xb7Qo%2Fhqdefault.jpg\" \/><meta itemprop=\"embedURL\" content=\"https:\/\/www.youtube.com\/embed\/H20Hd6Xb7Qo\" \/><meta itemprop=\"duration\" content=\"PT11M31S\" \/><meta itemprop=\"uploadDate\" content=\"2025-02-01T19:38:42Z\" \/><\/div><div id=\"lyte_H20Hd6Xb7Qo\" data-src=\"https:\/\/infernews.com\/blog\/wp-content\/plugins\/wp-youtube-lyte\/lyteCache.php?origThumbUrl=https%3A%2F%2Fi.ytimg.com%2Fvi%2FH20Hd6Xb7Qo%2Fhqdefault.jpg\" class=\"pL\"><div class=\"tC\"><div class=\"tT\" itemprop=\"name\">Reinforcement Learning in DeepSeek-R1 | Visually Explained<\/div><\/div><div class=\"play\"><\/div><div class=\"ctrl\"><div class=\"Lctrl\"><\/div><div class=\"Rctrl\"><\/div><\/div><\/div><noscript><a href=\"https:\/\/youtu.be\/H20Hd6Xb7Qo\" rel=\"nofollow\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/infernews.com\/blog\/wp-content\/plugins\/wp-youtube-lyte\/lyteCache.php?origThumbUrl=https%3A%2F%2Fi.ytimg.com%2Fvi%2FH20Hd6Xb7Qo%2F0.jpg\" alt=\"Reinforcement Learning in DeepSeek-R1 | Visually Explained\" width=\"853\" height=\"460\" \/><br \/>Watch this video on YouTube<\/a><\/noscript><meta itemprop=\"description\" content=\"Reinforcement Learning in DeepSeek-R1 | Visually Explained\"><\/div><\/div><div class=\"lL\" style=\"max-width:100%;width:853px;margin:5px auto;\"><\/div><figcaption><\/figcaption><\/figure>","protected":false},"excerpt":{"rendered":"<p>\u77ed\u7247\u95e1\u8ff0 DeepSeek R1 \u6a21\u578b\u7684\u8a13\u7df4\u904e\u7a0b\uff0c\u6838\u5fc3\u662f\u57fa\u65bc\u4eba\u985e\u56de\u994b\u7684\u5f37\u5316\u5b78\u7fd2\u3002\u9996\u5148\uff0c\u77ed\u7247\u89e3\u91cb\u5982\u4f55\u5229\u7528\u4eba\u985e\u504f\u597d\u8a13\u7df4\u734e\u52f5\u6a21\u578b (reward model)\uff1a\u6536\u96c6\u4eba\u985e\u5c0d\u4e0d\u540c\u6a21\u578b\u8f38\u51fa\u7684\u8a55\u5206\uff0c\u900f\u904e\u4f8b\u5982 Softmax \u51fd\u6578\u548c\u68af\u968e\u964d\u6cd5\uff0c\u8abf\u6574\u734e\u52f5\u6a21\u578b\uff0c\u4f7f\u5176\u7d66\u4e88\u4eba\u985e\u504f\u597d\u7684\u8f38\u51fa\u66f4\u9ad8\u5206\u6578\u3002\u77ed\u7247\u4ea6\u8aaa\u660e\u5982\u4f55\u4f7f\u7528\u8fd1\u7aef\u7b56\u7565\u6700\u4f73\u5316 (PPO) \u6f14\u7b97\u6cd5\uff0c\u7d50\u5408\u734e\u52f5\u6a21\u578b\u548c\u50f9\u503c\u6a21\u578b (value model) \u4f86\u5fae\u8abf\u8a9e\u8a00\u6a21\u578b (policy network)\uff1a\u6839\u64da\u734e\u52f5\u6a21\u578b\u7d66\u51fa\u7684\u734e\u52f5\uff0c\u4ee5\u53ca\u8a55\u50f9\u6a21\u578b\u9810\u6e2c\u7684\u734e\u52f5\u8207\u9810\u671f\u5dee\u7570 (advantage)\uff0c\u8abf\u6574\u7b56\u7565\u7db2\u7d61\uff0c\u4f7f\u5176\u66f4\u50be\u5411\u7522\u751f\u9ad8\u734e\u52f5\u7684\u8f38\u51fa\u3002\u6700\u5f8c\uff0c\u77ed\u7247\u7279\u5225\u4ecb\u7d39 DeepSeek R1 \u4f7f\u7528\u7684\u7fa4\u9ad4\u76f8\u5c0d\u7b56\u7565\u6700\u4f73\u5316 (group relative policy optimization)\uff0c\u9019\u662f\u4e00\u7a2e\u6539\u826f\u7684 PPO \u65b9\u6cd5\uff0c\u5c07\u734e\u52f5\u8207\u7fa4\u9ad4\u5167\u5176\u4ed6\u8f38\u51fa\u7684\u5e73\u5747\u734e\u52f5\u76f8\u6bd4\uff0c\u9f13\u52f5\u7522\u751f\u512a\u65bc\u5e73\u5747\u6c34\u6e96\u7684\u8f38\u51fa\uff0c\u89e3\u6c7a\u4e86\u50b3\u7d71\u734e\u52f5\u6a21\u578b\u53ef\u80fd\u51fa\u73fe\u7684\u300c\u734e\u52f5\u4f5c\u5f0a\u300d\u554f\u984c\u3002<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"googlesitekit_rrm_CAowvqSiDA:productID":"","footnotes":""},"categories":[76,127],"tags":[],"class_list":["post-4453","post","type-post","status-publish","format-standard","hentry","category-76","category-127"],"_links":{"self":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/posts\/4453","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/comments?post=4453"}],"version-history":[{"count":0,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/posts\/4453\/revisions"}],"wp:attachment":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/media?parent=4453"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/categories?post=4453"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/tags?post=4453"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}