{"id":5262,"date":"2025-04-02T17:56:20","date_gmt":"2025-04-02T09:56:20","guid":{"rendered":"https:\/\/infernews.com\/?page_id=5262"},"modified":"2025-04-02T18:32:54","modified_gmt":"2025-04-02T10:32:54","slug":"%e6%b3%a8%e6%84%8f%e5%8a%9b%e6%a9%9f%e5%88%b6%e5%88%86%e6%95%b8%ef%bc%88attention-scores%ef%bc%89","status":"publish","type":"page","link":"https:\/\/infernews.com\/blog\/%e6%b3%a8%e6%84%8f%e5%8a%9b%e6%a9%9f%e5%88%b6%e5%88%86%e6%95%b8%ef%bc%88attention-scores%ef%bc%89\/","title":{"rendered":"\u6ce8\u610f\u529b\u6a5f\u5236\u5206\u6578\uff08attention scores\uff09"},"content":{"rendered":"\n<p>\u6ce8\u610f\u529b\u6a5f\u5236\u7684\u5206\u6578\u901a\u5e38\u662f\u900f\u904e\u8a08\u7b97\u67e5\u8a62\uff08Query\uff09\u8207\u9375\uff08Key\uff09\u4e4b\u9593\u7684\u76f8\u4f3c\u5ea6\u4f86\u6c7a\u5b9a\u7684\u3002\u9019\u500b\u5206\u6578\u53cd\u6620\u4e86\u67e5\u8a62\u8207\u6bcf\u500b\u9375\u7684\u76f8\u95dc\u6027\uff0c\u4e26\u7528\u65bc\u8a08\u7b97\u6ce8\u610f\u529b\u6b0a\u91cd\uff0c\u9032\u800c\u6c7a\u5b9a\u6bcf\u500b\u503c\uff08Value\uff09\u5728\u6700\u7d42\u8f38\u51fa\u4e2d\u7684\u8ca2\u737b\u7a0b\u5ea6\u3002\u4ee5\u4e0b\u662f\u5e38\u898b\u7684\u5e7e\u7a2e\u8a08\u7b97\u65b9\u5f0f\uff1a<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">1. <strong>\u9ede\u7a4d\u6ce8\u610f\u529b\uff08Dot-Product Attention\uff09<\/strong><\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>\u8a08\u7b97\u65b9\u5f0f<\/strong>\uff1a( \\text{score} = Q \\cdot K )<\/li>\n\n\n\n<li><strong>\u8aaa\u660e<\/strong>\uff1a\u9019\u662f\u6700\u7c21\u55ae\u4e14\u5e38\u898b\u7684\u65b9\u5f0f\uff0c\u901a\u904e\u8a08\u7b97\u67e5\u8a62\u5411\u91cf ( Q ) \u548c\u9375\u5411\u91cf ( K ) \u7684\u9ede\u7a4d\u4f86\u8861\u91cf\u5b83\u5011\u7684\u76f8\u4f3c\u5ea6\u3002\u9ede\u7a4d\u8d8a\u5927\uff0c\u8868\u793a\u67e5\u8a62\u8207\u9375\u8d8a\u76f8\u95dc\u3002<\/li>\n\n\n\n<li><strong>\u512a\u9ede<\/strong>\uff1a\u8a08\u7b97\u7c21\u55ae\uff0c\u7279\u5225\u9069\u5408\u9ad8\u7dad\u6578\u64da\u3002<\/li>\n\n\n\n<li><strong>\u61c9\u7528<\/strong>\uff1a\u5ee3\u6cdb\u7528\u65bcTransformer\u6a21\u578b\u4e2d\u3002<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">2. <strong>\u7e2e\u653e\u9ede\u7a4d\u6ce8\u610f\u529b\uff08Scaled Dot-Product Attention\uff09<\/strong><\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>\u8a08\u7b97\u65b9\u5f0f<\/strong>\uff1a( \\text{score} = \\frac{Q \\cdot K}{\\sqrt{d_k}} )<\/li>\n\n\n\n<li><strong>\u8aaa\u660e<\/strong>\uff1a\u9019\u662f\u9ede\u7a4d\u6ce8\u610f\u529b\u7684\u6539\u9032\u7248\u672c\uff0c\u901a\u904e\u9664\u4ee5\u9375\u5411\u91cf\u7dad\u5ea6 ( d_k ) \u7684\u5e73\u65b9\u6839\u4f86\u7e2e\u653e\u5206\u6578\u3002\u9019\u6a23\u53ef\u4ee5\u907f\u514d\u9ad8\u7dad\u60c5\u6cc1\u4e0b\u9ede\u7a4d\u503c\u904e\u5927\uff0c\u5c0e\u81f4softmax\u51fd\u6578\u68af\u5ea6\u904e\u5c0f\u7684\u554f\u984c\u3002<\/li>\n\n\n\n<li><strong>\u512a\u9ede<\/strong>\uff1a\u5728\u9ad8\u7dad\u6578\u64da\u4e2d\u66f4\u7a69\u5b9a\u3002<\/li>\n\n\n\n<li><strong>\u61c9\u7528<\/strong>\uff1aTransformer\u6a21\u578b\u4e2d\u7684\u6a19\u6e96\u6ce8\u610f\u529b\u6a5f\u5236\u3002<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">3. <strong>\u52a0\u6027\u6ce8\u610f\u529b\uff08Additive Attention\uff09<\/strong><\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>\u8a08\u7b97\u65b9\u5f0f<\/strong>\uff1a( \\text{score} = V^T \\tanh(W_1 Q + W_2 K) )<\/li>\n\n\n\n<li><strong>\u8aaa\u660e<\/strong>\uff1a\u4f7f\u7528\u4e00\u500b\u795e\u7d93\u7db2\u7d61\u5c64\u8a08\u7b97\u5206\u6578\uff0c\u5176\u4e2d ( V )\u3001( W_1 )\u3001( W_2 ) \u662f\u53ef\u5b78\u7fd2\u7684\u53c3\u6578\u3002\u9019\u7a2e\u65b9\u5f0f\u901a\u904e\u975e\u7dda\u6027\u8b8a\u63db\u6355\u6349\u67e5\u8a62\u548c\u9375\u4e4b\u9593\u7684\u8907\u96dc\u95dc\u4fc2\u3002<\/li>\n\n\n\n<li><strong>\u512a\u9ede<\/strong>\uff1a\u66f4\u9748\u6d3b\uff0c\u80fd\u6355\u6349\u8907\u96dc\u7684\u76f8\u4f3c\u6027\u3002<\/li>\n\n\n\n<li><strong>\u61c9\u7528<\/strong>\uff1a\u5e38\u898b\u65bcRNN-based\u6a21\u578b\uff0c\u4f8b\u5982Bahdanau\u6ce8\u610f\u529b\u6a5f\u5236\u3002<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">4. <strong>\u9918\u5f26\u76f8\u4f3c\u5ea6\uff08Cosine Similarity\uff09<\/strong><\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>\u8a08\u7b97\u65b9\u5f0f<\/strong>\uff1a( \\text{score} = \\frac{Q \\cdot K}{|Q| |K|} )<\/li>\n\n\n\n<li><strong>\u8aaa\u660e<\/strong>\uff1a\u8a08\u7b97\u67e5\u8a62\u548c\u9375\u7684\u9918\u5f26\u76f8\u4f3c\u5ea6\uff0c\u95dc\u6ce8\u5b83\u5011\u7684\u65b9\u5411\u76f8\u4f3c\u6027\uff0c\u5206\u6578\u7bc4\u570d\u5728 [-1, 1] \u4e4b\u9593\u3002<\/li>\n\n\n\n<li><strong>\u512a\u9ede<\/strong>\uff1a\u5c0d\u5411\u91cf\u5927\u5c0f\u4e0d\u654f\u611f\uff0c\u9069\u5408\u9700\u8981\u65b9\u5411\u4e00\u81f4\u6027\u7684\u5834\u666f\u3002<\/li>\n\n\n\n<li><strong>\u61c9\u7528<\/strong>\uff1a\u5728\u67d0\u4e9b\u6587\u672c\u76f8\u4f3c\u5ea6\u8a08\u7b97\u4e2d\u8f03\u70ba\u5e38\u898b\u3002<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">5. <strong>\u96d9\u7dda\u6027\u6ce8\u610f\u529b\uff08Bilinear Attention\uff09<\/strong><\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>\u8a08\u7b97\u65b9\u5f0f<\/strong>\uff1a( \\text{score} = Q^T W K )<\/li>\n\n\n\n<li><strong>\u8aaa\u660e<\/strong>\uff1a\u5f15\u5165\u4e00\u500b\u53ef\u5b78\u7fd2\u7684\u6b0a\u91cd\u77e9\u9663 ( W )\uff0c\u5141\u8a31\u67e5\u8a62\u548c\u9375\u4e4b\u9593\u9032\u884c\u66f4\u8907\u96dc\u7684\u4ea4\u4e92\u3002<\/li>\n\n\n\n<li><strong>\u512a\u9ede<\/strong>\uff1a\u80fd\u5b78\u7fd2\u66f4\u8c50\u5bcc\u7684\u76f8\u4f3c\u6027\u6a21\u5f0f\u3002<\/li>\n\n\n\n<li><strong>\u61c9\u7528<\/strong>\uff1a\u5728\u67d0\u4e9b\u6ce8\u610f\u529b\u6a5f\u5236\u4e2d\u4f5c\u70ba\u66ff\u4ee3\u65b9\u6848\u3002<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">6. <strong>\u591a\u982d\u6ce8\u610f\u529b\uff08Multi-Head Attention\uff09<\/strong><\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>\u8a08\u7b97\u65b9\u5f0f<\/strong>\uff1a\u5728Transformer\u4e2d\uff0c\u591a\u982d\u6ce8\u610f\u529b\u4e26\u884c\u8a08\u7b97\u591a\u7d44\u5206\u6578\uff0c\u6bcf\u500b\u982d\u4f7f\u7528\u4e0d\u540c\u7684\u7dda\u6027\u8b8a\u63db\u751f\u6210\u67e5\u8a62\u3001\u9375\u548c\u503c\uff0c\u7136\u5f8c\u7368\u7acb\u8a08\u7b97\u5206\u6578\uff0c\u6700\u5f8c\u5c07\u7d50\u679c\u62fc\u63a5\u6216\u52a0\u6b0a\u6c42\u548c\u3002<\/li>\n\n\n\n<li><strong>\u8aaa\u660e<\/strong>\uff1a\u9019\u7a2e\u65b9\u5f0f\u5141\u8a31\u6a21\u578b\u5f9e\u4e0d\u540c\u89d2\u5ea6\u6355\u6349\u67e5\u8a62\u548c\u9375\u7684\u95dc\u4fc2\uff0c\u589e\u5f37\u8868\u9054\u80fd\u529b\u3002<\/li>\n\n\n\n<li><strong>\u61c9\u7528<\/strong>\uff1aTransformer\u6a21\u578b\u7684\u6838\u5fc3\u7d44\u6210\u90e8\u5206\u3002<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">\u7e3d\u7d50<\/h3>\n\n\n\n<p>\u6ce8\u610f\u529b\u6a5f\u5236\u7684\u5206\u6578\u662f\u57fa\u65bc\u67e5\u8a62\u8207\u9375\u7684\u76f8\u4f3c\u5ea6\u8a08\u7b97\u5f97\u51fa\u7684\uff0c\u5e38\u898b\u65b9\u6cd5\u5305\u62ec\u9ede\u7a4d\u6ce8\u610f\u529b\u3001\u7e2e\u653e\u9ede\u7a4d\u6ce8\u610f\u529b\u3001\u52a0\u6027\u6ce8\u610f\u529b\u3001\u9918\u5f26\u76f8\u4f3c\u5ea6\u548c\u96d9\u7dda\u6027\u6ce8\u610f\u529b\u7b49\u3002\u9019\u4e9b\u5206\u6578\u901a\u5e38\u6703\u7d93\u904esoftmax\u6b78\u4e00\u5316\uff0c\u8f49\u5316\u70ba\u6ce8\u610f\u529b\u6b0a\u91cd\uff0c\u518d\u7528\u65bc\u52a0\u6b0a\u6c42\u548c\u503c\u5411\u91cf\uff0c\u751f\u6210\u6700\u7d42\u7684\u4e0a\u4e0b\u6587\u8868\u793a\u3002\u9078\u64c7\u54ea\u7a2e\u65b9\u5f0f\u53d6\u6c7a\u65bc\u5177\u9ad4\u61c9\u7528\u5834\u666f\uff0c\u5176\u4e2d\u7e2e\u653e\u9ede\u7a4d\u6ce8\u610f\u529b\u56e0\u5176\u9ad8\u6548\u6027\u548c\u7a69\u5b9a\u6027\uff0c\u5728Transformer\u6a21\u578b\u4e2d\u61c9\u7528\u6700\u70ba\u5ee3\u6cdb\u3002<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p class=\"has-large-font-size\">\u89e3\u91cb<\/p>\n\n\n\n<p>\u300c\u6ce8\u610f\u529b\u5206\u6578\u300d\u662f\u5982\u4f55\u5f71\u97ff\u5230\u300c\u4e0b\u4e00\u500b token\u300d\u7684\u3002<\/p>\n\n\n\n<p>\u9019\u88e1\u7684\u95dc\u9375\u5728\u65bc\u7406\u89e3\u300c\u6ce8\u610f\u529b\u5206\u6578\u300d\uff08\u6216\u8005\u66f4\u7cbe\u78ba\u5730\u8aaa\uff0c\u662f\u7531\u5206\u6578\u7d93\u904e Softmax \u8f49\u63db\u5f8c\u7684\u300c\u6ce8\u610f\u529b\u6b0a\u91cd\u300d\uff09\u662f\u5982\u4f55\u88ab\u7528\u4f86<strong>\u66f4\u65b0\u67e5\u8a62 token (Query token) \u81ea\u8eab\u7684\u8868\u793a (representation)<\/strong>&nbsp;\u7684\u3002<\/p>\n\n\n\n<p>\u4ee5\u4e0b\u662f\u8a73\u7d30\u6b65\u9a5f\uff0c\u8457\u91cd\u65bc\u5206\u6578\u5e36\u4f86\u7684\u5f71\u97ff\uff1a<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>\u56de\u9867\uff1a\u5f97\u5230\u5206\u6578 (Scores)<\/strong>\n<ul class=\"wp-block-list\">\n<li>\u5c0d\u65bc\u4e00\u500b\u7279\u5b9a\u7684 token \u4f5c\u70ba\u300c\u67e5\u8a62 (Query)\u300d\uff0c\u6211\u5011\u6703\u8a08\u7b97\u5b83\u8207\u5e8f\u5217\u4e2d\u6240\u6709\u5176\u4ed6 token\uff08\u5305\u62ec\u5b83\u81ea\u5df1\uff09\u4f5c\u70ba\u300c\u9375 (Key)\u300d\u7684\u76f8\u4f3c\u5ea6\uff08\u539f\u59cb\u5206\u6578\uff09\u3002<\/li>\n\n\n\n<li>\u9019\u4e9b\u539f\u59cb\u5206\u6578\u63a5\u8457\u6703\u901a\u904e\u4e00\u500b\u00a0softmax\u00a0\u51fd\u6578\u3002\u9019\u500b\u6b65\u9a5f\u5c07\u539f\u59cb\u5206\u6578\u8f49\u63db\u6210<strong>\u6ce8\u610f\u529b\u6b0a\u91cd (Attention Weights)<\/strong>\uff08\u4e5f\u5c31\u662f\u6211\u5011\u5728\u6a21\u64ec\u4e2d\u8996\u89ba\u5316\u4e26\u6a19\u793a\u70ba &#8220;Score&#8221; \u7684\u90a3\u4e9b\u503c\uff09\u3002<\/li>\n\n\n\n<li>\u6700\u91cd\u8981\u7684\u662f\uff0c\u9019\u4e9b\u6ce8\u610f\u529b\u6b0a\u91cd\u662f<strong>\u6a5f\u7387\u5206\u4f48<\/strong>\uff1a\u5b83\u5011\u90fd\u662f\u6b63\u6578\uff0c\u4e26\u4e14\u5c0d\u65bc\u6bcf\u4e00\u500b Query \u4f86\u8aaa\uff0c\u6240\u6709\u6b0a\u91cd\u52a0\u7e3d\u8d77\u4f86\u7b49\u65bc 1\u3002<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>\u5f15\u5165\u300c\u503c (Value)\u300d\u5411\u91cf<\/strong>\n<ul class=\"wp-block-list\">\n<li>\u5728\u5f9e\u521d\u59cb\u7684 token embedding\uff08\u8a5e\u5d4c\u5165\uff09\u7522\u751f\u67e5\u8a62 (Q) \u548c\u9375 (K) \u5411\u91cf\u7684\u540c\u6642\uff0cTransformer \u4e5f\u6703\u4f7f\u7528\u53e6\u4e00\u500b\u5b78\u7fd2\u5230\u7684\u6b0a\u91cd\u77e9\u9663 (Wv) \u4f86\u7522\u751f\u7b2c\u4e09\u7d44\u5411\u91cf\uff0c\u7a31\u70ba**\u300c\u503c (V)\u300d\u5411\u91cf**\u3002<\/li>\n\n\n\n<li>\u4f60\u53ef\u4ee5\u628a\u4e00\u500b token \u7684 Value \u5411\u91cf\u60f3\u50cf\u6210\u5b83\u6240\u6301\u6709\u3001\u4e26\u4e14\u6e96\u5099\u597d\u8981\u5206\u4eab\uff08\u5982\u679c\u5b83\u88ab\u95dc\u6ce8\u5230\u7684\u8a71\uff09\u7684<strong>\u5be6\u969b\u300c\u8cc7\u8a0a\u300d\u6216\u300c\u5167\u5bb9\u300d<\/strong>\u3002Key \u662f\u7528\u4f86<em>\u5224\u65b7\u76f8\u95dc\u6027<\/em>\u7684\uff0c\u800c Value \u5247\u662f<em>\u5be6\u969b\u88ab\u50b3\u905e\u7684\u5167\u5bb9<\/em>\u3002<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>\u52a0\u6b0a\u7e3d\u548c (Weighted Sum) &#8211; \u5206\u6578\u767c\u63ee\u4f5c\u7528\u4e4b\u8655<\/strong>\n<ul class=\"wp-block-list\">\n<li>\u6838\u5fc3\u7684\u64cd\u4f5c\u662f\uff1a<strong>\u67e5\u8a62 token \u7684\u65b0\u8868\u793a<\/strong>\u00a0\u662f\u900f\u904e\u8a08\u7b97\u5e8f\u5217\u4e2d<strong>\u6240\u6709 Value \u5411\u91cf\u7684\u52a0\u6b0a\u7e3d\u548c<\/strong>\u5f97\u5230\u7684\u3002<\/li>\n\n\n\n<li>\u800c\u9019\u500b\u52a0\u6b0a\u7e3d\u548c\u6240\u4f7f\u7528\u7684<strong>\u6b0a\u91cd<\/strong>\uff0c<strong>\u6b63\u662f<\/strong>\u8a72 Query \u5c0d\u6bcf\u500b Key \u8a08\u7b97\u51fa\u4f86\u7684<strong>\u6ce8\u610f\u529b\u6b0a\u91cd (Attention Weights\uff0c\u5373\u6211\u5011\u7684 &#8220;Scores&#8221;)<\/strong>\u3002<\/li>\n\n\n\n<li><strong>\u516c\u5f0f\u6982\u5ff5\uff1a<\/strong>\u00a0\u67e5\u8a62 token i \u7684\u8f38\u51fa = sum( \u6ce8\u610f\u529b\u6b0a\u91cd_i_\u5230_j * \u503c\u5411\u91cf_j )\u00a0\uff08\u5c0d\u5e8f\u5217\u4e2d\u6240\u6709\u7684 j \u9032\u884c\u52a0\u7e3d\uff09<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>\u5206\u6578\u5982\u4f55\u5f71\u97ff\u8f38\u51fa (Output)<\/strong>\n<ul class=\"wp-block-list\">\n<li><strong>\u9ad8\u5206 (\u9ad8\u6ce8\u610f\u529b\u6b0a\u91cd)\uff1a<\/strong>\u00a0\u5982\u679c Query \u7d66\u4e88 Key\u00a0j\u00a0\u5f88\u9ad8\u7684\u6ce8\u610f\u529b\u5206\u6578\/\u6b0a\u91cd\uff0c\u90a3\u9ebc\u5c0d\u61c9\u7684\u00a0Value_j\u00a0\u5411\u91cf\u5728\u52a0\u6b0a\u7e3d\u548c\u4e2d\u5c31\u6703\u4e58\u4ee5\u4e00\u500b\u8f03\u5927\u7684\u6b0a\u91cd\u3002\u9019\u610f\u5473\u8457<strong>\u5305\u542b\u5728\u00a0Value_j\u00a0\u4e2d\u7684\u8cc7\u8a0a<\/strong>\u5c07\u5c0d Query token \u7684\u65b0\u8868\u793a\u7522\u751f\u5f37\u70c8\u7684\u8ca2\u737b\u3002Query \u5f9e token\u00a0j\u300c\u5438\u53d6\u300d\u4e86\u5927\u91cf\u8cc7\u8a0a\u3002<\/li>\n\n\n\n<li><strong>\u4f4e\u5206 (\u4f4e\u6ce8\u610f\u529b\u6b0a\u91cd)\uff1a<\/strong>\u00a0\u5982\u679c Query \u7d66\u4e88 Key\u00a0k\u00a0\u5f88\u4f4e\u7684\u6ce8\u610f\u529b\u5206\u6578\/\u6b0a\u91cd\uff0c\u90a3\u9ebc\u00a0Value_k\u00a0\u5c31\u6703\u4e58\u4ee5\u4e00\u500b\u975e\u5e38\u5c0f\u7684\u6b0a\u91cd\u3002\u4f86\u81ea token\u00a0k\u00a0\u7684\u8cc7\u8a0a\u5c0d Query \u66f4\u65b0\u5f8c\u7684\u72c0\u614b\u5f71\u97ff\u5c31\u5fae\u4e4e\u5176\u5fae\u3002<\/li>\n\n\n\n<li><strong>\u7d50\u679c\uff1a<\/strong>\u00a0Query token \u7684\u8f38\u51fa\u8868\u793a\u8b8a\u6210\u4e86\u4e00\u7a2e\u8cc7\u8a0a\u7684<strong>\u878d\u5408 (blend)<\/strong>\uff0c\u9019\u4e9b\u8cc7\u8a0a\u4f86\u81ea\u5e8f\u5217\u4e2d\u7684\u6240\u6709 token\uff0c\u4f46\u878d\u5408\u7684\u6bd4\u4f8b\u662f\u6839\u64da\u6bcf\u500b token \u88ab\u5224\u65b7\u7684\u76f8\u95dc\u6027\uff08\u7531 Q-K \u76f8\u4f3c\u5ea6\u5206\u6578\u6c7a\u5b9a\uff09\u4f86\u8abf\u914d\u7684\u3002\u9019\u662f\u4e00\u500b\u5177\u5099<strong>\u4e0a\u4e0b\u6587\u611f\u77e5\u80fd\u529b (context-aware)<\/strong>\u00a0\u7684\u8868\u793a\u3002<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<p><strong>\u985e\u6bd4\uff1a<\/strong>&nbsp;\u60f3\u50cf\u4f60\u5728\u505a\u4e00\u676f\u6c34\u679c<strong>\u679c\u6614<\/strong>\uff08\u4ee3\u8868 Query \u7684\u65b0\u8868\u793a\uff09\u3002<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>\u6bcf\u7a2e\u53ef\u7528\u7684\u6c34\u679c\u90fd\u662f\u4e00\u500b token\u3002<\/li>\n\n\n\n<li>\u4f60\u6839\u64da<strong>\u7576\u524d\u7684\u559c\u597d\u7a0b\u5ea6<\/strong>\uff08Query-Key \u76f8\u4f3c\u5ea6\uff09\u5c0d\u6bcf\u7a2e\u6c34\u679c\u6253\u7684\u5206\u6578\uff0c\u6c7a\u5b9a\u4e86<strong>\u6ce8\u610f\u529b\u5206\u6578\/\u6b0a\u91cd<\/strong>\u3002<\/li>\n\n\n\n<li>\u6c34\u679c\u672c\u8eab\uff08\u5b83\u7684\u5473\u9053\u3001\u71df\u990a\u6210\u5206\uff09\u5c31\u662f\u00a0<strong>Value \u5411\u91cf<\/strong>\u3002<\/li>\n\n\n\n<li>\u4f60\u6700\u7d42\u88fd\u4f5c\u679c\u6614\u7684\u65b9\u5f0f\uff0c\u662f\u6839\u64da\u4f60\u5c0d\u6bcf\u7a2e\u6c34\u679c\u7684\u559c\u597d\u7a0b\u5ea6\uff08\u6ce8\u610f\u529b\u6b0a\u91cd\uff09\uff0c\u52a0\u5165<strong>\u4e0d\u540c\u91cf\u7684\u6c34\u679c<\/strong>\uff08Value\uff09\u3002\u4f60\u6703\u52a0\u5165\u5f88\u591a\u4f60\u559c\u6b61\u7684\u6c34\u679c\uff08\u9ad8\u5206\uff09\uff0c\u800c\u53ea\u52a0\u4e00\u9ede\u9ede\u6216\u5b8c\u5168\u4e0d\u52a0\u4f60\u4e0d\u559c\u6b61\u7684\u6c34\u679c\uff08\u4f4e\u5206\uff09\u3002<\/li>\n<\/ul>\n\n\n\n<p><strong>\u95dc\u65bc\u300c\u4e0b\u4e00\u500b token\u300d\uff1a<\/strong><\/p>\n\n\n\n<p>\u6ce8\u610f\u529b\u5206\u6578\u4e26<strong>\u4e0d\u76f4\u63a5<\/strong>\u6c7a\u5b9a\u5e8f\u5217\u4e2d<strong>\u7269\u7406\u4f4d\u7f6e\u76f8\u9130\u7684\u4e0b\u4e00\u500b token<\/strong>&nbsp;\u7684\u5c6c\u6027\u3002\u76f8\u53cd\u5730\uff0c\u5b83\u5011\u6c7a\u5b9a\u7684\u662f<strong>\u67e5\u8a62 token \u672c\u8eab\u7684\u3001\u66f4\u65b0\u5f8c\u7684\u72c0\u614b<\/strong>\u3002\u9019\u500b\u7d93\u904e\u66f4\u65b0\u3001\u7406\u89e3\u4e86\u4e0a\u4e0b\u6587\u7684 Query token \u8868\u793a\uff0c\u901a\u5e38\u6703\u63a5\u8457\uff1a<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>\u5728\u540c\u4e00\u500b Transformer \u5340\u584a (block) \u4e2d\uff0c\u88ab\u50b3\u905e\u5230\u4e00\u500b\u524d\u994b\u795e\u7d93\u7db2\u8def (Feed-Forward Network)\u3002<\/li>\n\n\n\n<li>\u6216\u8005\uff0c\u88ab\u7576\u4f5c\u8f38\u5165\u50b3\u905e\u5230<strong>\u4e0b\u4e00\u500b Transformer \u5c64<\/strong>\uff08\u5982\u679c\u6a21\u578b\u6709\u591a\u5c64\u7684\u8a71\uff09\u3002<\/li>\n<\/ul>\n\n\n\n<p>\u6240\u4ee5\uff0c\u5206\u6578\u5f71\u97ff\u7684\u662f Query token \u5c0d\u5176\u4e0a\u4e0b\u6587\u7684\u7406\u89e3\uff0c\u800c<strong>\u9019\u7a2e\u66f4\u4f73\u7684\u7406\u89e3<\/strong>\u6703\u5f71\u97ff\u5f8c\u7e8c\u7684\u8655\u7406\u6b65\u9a5f\uff0c\u53ef\u80fd\u5f71\u97ff\u5230\u5176\u4ed6 token \u5728\u4e4b\u5f8c\u5982\u4f55\u88ab\u8655\u7406\uff0c\u6216\u8005\u6700\u7d42\u5982\u4f55\u751f\u6210\u6a21\u578b\u7684\u8f38\u51fa\u3002\u5b83\u5f71\u97ff\u7684\u662f<strong>\u8cc7\u8a0a\u6d41\u548c\u8868\u793a\u7684\u66f4\u65b0<\/strong>\uff0c\u800c\u4e0d\u662f\u76f4\u63a5\u6539\u8b8a\u5e8f\u5217\u4e2d\u7684\u4e0b\u4e00\u500b\u5143\u7d20\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u6ce8\u610f\u529b\u6a5f\u5236\u7684\u5206\u6578\u901a\u5e38\u662f\u900f\u904e\u8a08\u7b97\u67e5\u8a62\uff08Query\uff09\u8207\u9375\uff08Key\uff09\u4e4b\u9593\u7684\u76f8\u4f3c\u5ea6\u4f86\u6c7a\u5b9a\u7684\u3002\u9019\u500b\u5206\u6578\u53cd\u6620\u4e86\u67e5\u8a62\u8207\u6bcf\u500b\u9375\u7684\u76f8\u95dc\u6027\uff0c\u4e26\u7528\u65bc\u8a08\u7b97\u6ce8\u610f\u529b\u6b0a\u91cd\uff0c\u9032\u800c\u6c7a\u5b9a\u6bcf\u500b\u503c\uff08Value\uff09\u5728\u6700\u7d42\u8f38\u51fa\u4e2d\u7684\u8ca2\u737b\u7a0b\u5ea6\u3002\u4ee5\u4e0b\u662f\u5e38\u898b\u7684\u5e7e\u7a2e\u8a08\u7b97\u65b9\u5f0f\uff1a 1. \u9ede\u7a4d\u6ce8\u610f\u529b\uff08Dot-Product Attention\uff09 2. \u7e2e\u653e\u9ede\u7a4d\u6ce8\u610f\u529b\uff08Scaled Dot-Product Attention\uff09 3. \u52a0\u6027\u6ce8\u610f\u529b\uff08Additive Attention\uff09 4. \u9918\u5f26\u76f8\u4f3c\u5ea6\uff08Cosine Similarity\uff09 5. \u96d9\u7dda\u6027\u6ce8\u610f\u529b\uff08Bilinear Attention\uff09 6. \u591a\u982d\u6ce8\u610f\u529b\uff08Multi-Head Attention\uff09 \u7e3d\u7d50 \u6ce8\u610f\u529b\u6a5f\u5236\u7684\u5206\u6578\u662f\u57fa\u65bc\u67e5\u8a62\u8207\u9375\u7684\u76f8\u4f3c\u5ea6\u8a08\u7b97\u5f97\u51fa\u7684\uff0c\u5e38\u898b\u65b9\u6cd5\u5305\u62ec\u9ede\u7a4d\u6ce8\u610f\u529b\u3001\u7e2e\u653e\u9ede\u7a4d\u6ce8\u610f\u529b\u3001\u52a0\u6027\u6ce8\u610f\u529b\u3001\u9918\u5f26\u76f8\u4f3c\u5ea6\u548c\u96d9\u7dda\u6027\u6ce8\u610f\u529b\u7b49\u3002\u9019\u4e9b\u5206\u6578\u901a\u5e38\u6703\u7d93\u904esoftmax\u6b78\u4e00\u5316\uff0c\u8f49\u5316\u70ba\u6ce8\u610f\u529b\u6b0a\u91cd\uff0c\u518d\u7528\u65bc\u52a0\u6b0a\u6c42\u548c\u503c\u5411\u91cf\uff0c\u751f\u6210\u6700\u7d42\u7684\u4e0a\u4e0b\u6587\u8868\u793a\u3002\u9078\u64c7\u54ea\u7a2e\u65b9\u5f0f\u53d6\u6c7a\u65bc\u5177\u9ad4\u61c9\u7528\u5834\u666f\uff0c\u5176\u4e2d\u7e2e\u653e\u9ede\u7a4d\u6ce8\u610f\u529b\u56e0\u5176\u9ad8\u6548\u6027\u548c\u7a69\u5b9a\u6027\uff0c\u5728Transformer\u6a21\u578b\u4e2d\u61c9\u7528\u6700\u70ba\u5ee3\u6cdb\u3002 \u89e3\u91cb \u300c\u6ce8\u610f\u529b\u5206\u6578\u300d\u662f\u5982\u4f55\u5f71\u97ff\u5230\u300c\u4e0b\u4e00\u500b token\u300d\u7684\u3002 \u9019\u88e1\u7684\u95dc\u9375\u5728\u65bc\u7406\u89e3\u300c\u6ce8\u610f\u529b\u5206\u6578\u300d\uff08\u6216\u8005\u66f4\u7cbe\u78ba\u5730\u8aaa\uff0c\u662f\u7531\u5206\u6578\u7d93\u904e Softmax \u8f49\u63db\u5f8c\u7684\u300c\u6ce8\u610f\u529b\u6b0a\u91cd\u300d\uff09\u662f\u5982\u4f55\u88ab\u7528\u4f86\u66f4\u65b0\u67e5\u8a62 token (Query token) \u81ea\u8eab\u7684\u8868\u793a (representation)&nbsp;\u7684\u3002 \u4ee5\u4e0b\u662f\u8a73\u7d30\u6b65\u9a5f\uff0c\u8457\u91cd\u65bc\u5206\u6578\u5e36\u4f86\u7684\u5f71\u97ff\uff1a \u985e\u6bd4\uff1a&nbsp;\u60f3\u50cf\u4f60\u5728\u505a\u4e00\u676f\u6c34\u679c\u679c\u6614\uff08\u4ee3\u8868 Query \u7684\u65b0\u8868\u793a\uff09\u3002 \u95dc\u65bc\u300c\u4e0b\u4e00\u500b token\u300d\uff1a \u6ce8\u610f\u529b\u5206\u6578\u4e26\u4e0d\u76f4\u63a5\u6c7a\u5b9a\u5e8f\u5217\u4e2d\u7269\u7406\u4f4d\u7f6e\u76f8\u9130\u7684\u4e0b\u4e00\u500b token&nbsp;\u7684\u5c6c\u6027\u3002\u76f8\u53cd\u5730\uff0c\u5b83\u5011\u6c7a\u5b9a\u7684\u662f\u67e5\u8a62 token \u672c\u8eab\u7684\u3001\u66f4\u65b0\u5f8c\u7684\u72c0\u614b\u3002\u9019\u500b\u7d93\u904e\u66f4\u65b0\u3001\u7406\u89e3\u4e86\u4e0a\u4e0b\u6587\u7684 Query token \u8868\u793a\uff0c\u901a\u5e38\u6703\u63a5\u8457\uff1a \u6240\u4ee5\uff0c\u5206\u6578\u5f71\u97ff\u7684\u662f Query token \u5c0d\u5176\u4e0a\u4e0b\u6587\u7684\u7406\u89e3\uff0c\u800c\u9019\u7a2e\u66f4\u4f73\u7684\u7406\u89e3\u6703\u5f71\u97ff\u5f8c\u7e8c\u7684\u8655\u7406\u6b65\u9a5f\uff0c\u53ef\u80fd\u5f71\u97ff\u5230\u5176\u4ed6 token \u5728\u4e4b\u5f8c\u5982\u4f55\u88ab\u8655\u7406\uff0c\u6216\u8005\u6700\u7d42\u5982\u4f55\u751f\u6210\u6a21\u578b\u7684\u8f38\u51fa\u3002\u5b83\u5f71\u97ff\u7684\u662f\u8cc7\u8a0a\u6d41\u548c\u8868\u793a\u7684\u66f4\u65b0\uff0c\u800c\u4e0d\u662f\u76f4\u63a5\u6539\u8b8a\u5e8f\u5217\u4e2d\u7684\u4e0b\u4e00\u500b\u5143\u7d20\u3002<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"googlesitekit_rrm_CAowvqSiDA:productID":"","footnotes":""},"class_list":["post-5262","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/pages\/5262","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/comments?post=5262"}],"version-history":[{"count":0,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/pages\/5262\/revisions"}],"wp:attachment":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/media?parent=5262"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}