{"id":8296,"date":"2026-05-17T22:09:18","date_gmt":"2026-05-17T14:09:18","guid":{"rendered":"https:\/\/infernews.com\/blog\/long-context-pre-training-with-lighthouse-attention\/"},"modified":"2026-05-17T22:09:18","modified_gmt":"2026-05-17T14:09:18","slug":"long-context-pre-training-with-lighthouse-attention","status":"publish","type":"post","link":"https:\/\/infernews.com\/blog\/long-context-pre-training-with-lighthouse-attention\/","title":{"rendered":"Lighthouse Attention\uff1a\u9577\u4e0a\u4e0b\u6587\u8a13\u7df4\u65b0\u601d\u8def"},"content":{"rendered":"<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/infernews.com\/blog\/wp-content\/uploads\/2026\/05\/pasted-bc1391335b0c.jpg\" alt=\"Repository image for ighoshsubho\/lighthouse-attention\"><\/figure>\n<p>\u5982\u679c\u4f60\u6709\u7559\u610f\u5927\u578b\u8a9e\u8a00\u6a21\u578b\uff0c\u61c9\u8a72\u77e5\u9053\u6587\u5b57\u6108\u9577\uff0c\u904b\u7b97\u6210\u672c\u5f80\u5f80\u5347\u5f97\u6108\u5feb\u3002Lighthouse Attention \u91dd\u5c0d\u7684\u6b63\u6b63\u662f\u9019\u500b\u75db\u9ede\uff1a\u5728\u6975\u9577\u4e0a\u4e0b\u6587\u8a13\u7df4\u6642\uff0c\u7528\u5206\u5c64\u6311\u9078\u7684\u65b9\u6cd5\uff0c\u5148\u7e2e\u7d30\u9700\u8981\u91cd\u9ede\u8655\u7406\u7684\u5167\u5bb9\uff0c\u518d\u4ea4\u56de\u73fe\u6210\u7684\u9ad8\u6548\u6ce8\u610f\u529b\u6d41\u7a0b\u8655\u7406\u3002<\/p>\n<p>\u9019\u500b\u5132\u5b58\u5eab\u4e0d\u662f\u4e00\u500b\u5373\u958b\u5373\u7528\u7684\u804a\u5929\u7a0b\u5f0f\uff0c\u800c\u662f\u5efa\u57fa\u65bc PyTorch \u7684 torchtitan \u8a13\u7df4\u6846\u67b6\u4e4b\u4e0a\uff0c\u4ee5\u88dc\u4e01\u5f62\u5f0f\u6574\u5408\u3002\u63db\u53e5\u8a71\u8aaa\uff0c\u8f03\u9069\u5408\u672c\u8eab\u5df2\u7d93\u505a\u6a21\u578b\u8a13\u7df4\u3001\u60f3\u6bd4\u8f03\u4e0d\u540c\u6ce8\u610f\u529b\u6a5f\u5236\u7684\u4eba\uff1b\u4e00\u822c\u7528\u5bb6\u672a\u5fc5\u6703\u76f4\u63a5\u8dd1\u8d77\uff0c\u4f46\u8b80\u5b83\u7684\u8a2d\u8a08\u4ecd\u5f88\u6709\u53c3\u8003\u50f9\u503c\u3002<\/p>\n<p>\u5b83\u8f03\u7279\u5225\u7684\u5730\u65b9\uff0c\u5728\u65bc\u4e0d\u662f\u628a\u7a00\u758f\u6a5f\u5236\u786c\u585e\u9032\u81ea\u8a02\u6838\u5fc3\uff0c\u800c\u662f\u5148\u505a\u9078\u64c7\uff0c\u518d\u6cbf\u7528\u73fe\u6709\u7684 FlashAttention \u5bc6\u96c6\u8a08\u7b97\u8def\u7dda\u3002\u9019\u6a23\u7684\u597d\u8655\u662f\u8f03\u6613\u53d7\u60e0\u65bc\u4e0a\u6e38\u512a\u5316\uff0c\u4e5f\u6e1b\u5c11\u70ba\u65b0\u65b9\u6cd5\u91cd\u5beb\u6574\u5957\u5e95\u5c64\u6838\u5fc3\u7684\u8ca0\u64d4\u3002\u8cc7\u6599\u986f\u793a\uff0c\u5b83\u63d0\u4f9b <code>norm<\/code>\u3001<code>dilated<\/code> \u548c <code>gla<\/code> \u4e09\u7a2e\u8a55\u5206\u8b8a\u9ad4\uff0c\u4ea6\u652f\u63f4\u53ef\u9078\u7684 context parallel \u8def\u5f91\u3002<\/p>\n<p>\u5982\u679c\u4f60\u60f3\u4e0a\u624b\uff0c\u8f03\u5be6\u969b\u7684\u505a\u6cd5\u662f\u5148\u628a\u5b83\u7576\u6210\u7814\u7a76\u539f\u578b\uff1a\u6309\u7248\u672c\u8981\u6c42\u6e96\u5099\u597d torchtitan\u3001\u5c0d\u61c9\u63d0\u4ea4\u7248\u672c\u3001\u5169\u500b\u984d\u5916\u539f\u59cb\u6a94\u53ca\u88dc\u4e01\uff0c\u518d\u7528 <code>configs<\/code> \u5167\u4e0d\u540c\u8a2d\u5b9a\u6bd4\u8f03 top-K\u3001pool\u3001\u5927\u7d30\u5c64\u6578\u8207 scorer \u5dee\u7570\u3002\u5b98\u65b9\u8cc7\u8a0a\u4ea6\u63d0\u5230\uff0c\u5b83\u66fe\u5728 530M Llama-3 \u898f\u6a21\u3001\u4ee5\u53ca\u6700\u9ad8\u767e\u842c token \u8a13\u7df4\u60c5\u5883\u4e0b\u9a57\u8b49\u3002<\/p>\n<p>\u91cd\u9ede\u53ef\u4ee5\u9019\u6a23\u770b\uff1a<br \/>\n&#8211; \u4e3b\u8981\u7528\u9014\u662f\u964d\u4f4e\u8d85\u9577\u4e0a\u4e0b\u6587\u8a13\u7df4\u6642\u7684\u6ce8\u610f\u529b\u6210\u672c<br \/>\n&#8211; \u6838\u5fc3\u505a\u6cd5\u662f\u5206\u5c64\u6311\u9078\u91cd\u8981\u7247\u6bb5\uff0c\u518d\u4ea4\u7531\u5bc6\u96c6\u6ce8\u610f\u529b\u8a08\u7b97<br \/>\n&#8211; \u5df2\u5217\u51fa\u591a\u7d44\u53ef\u6bd4\u8f03\u8a2d\u5b9a\uff1atop-K\u3001pool\u3001levels\u3001scorer\u3001CP<br \/>\n&#8211; \u76f8\u95dc\u8a55\u5206\u6216\u8def\u7dda\u5305\u62ec <strong>norm<\/strong>\u3001<strong>dilated<\/strong>\u3001<strong>gla<\/strong><br \/>\n&#8211; \u8f03\u9069\u5408\u6a21\u578b\u7814\u7a76\u3001\u8a13\u7df4\u57fa\u5efa\u958b\u767c\u53ca\u9577\u6587\u672c\u5be6\u9a57\u5834\u666f<\/p>\n<p>\u53ea\u8981\u662f\u6a19\u6e96\u7684 decoder-only Transformer \/ causal LM\uff0c\u57fa\u672c\u90fd\u53ef\u4ee5\u66ff\u63db\u5b83\u7684 Q\/K\/V self-attention \u5c64\u3002<\/p>\n<p>\u5224\u65b7\u6a19\u6e96<br \/>\n\u53ea\u8981\u6a21\u578b\u6eff\u8db3\u4e0b\u9762\u5e7e\u9ede\uff0c\u5c31\u901a\u5e38\u80fd\u505a\u9019\u7a2e\u66ff\u63db\uff1a<br \/>\n\u6709 self-attention \u5c64\uff0c\u800c\u4e0d\u662f\u4f9d\u8cf4\u8907\u96dc\u7684\u5916\u90e8\u7de8\u78bc\u5668\u3002<br \/>\n\u5c64\u88e1\u80fd\u6e05\u695a\u627e\u5230 q_proj \/ k_proj \/ v_proj \u6216\u7b49\u50f9\u5be6\u73fe\u3002<br \/>\n\u662f decoder-only \u67b6\u69cb\uff0c\u4f7f\u7528 causal mask\u3002<br \/>\n\u6c92\u6709\u628a attention \u908f\u8f2f\u5beb\u6b7b\u6210\u7279\u5225\u96e3\u62c6\u7684\u81ea\u5b9a\u7fa9\u6a21\u584a\u3002<\/p>\n<p>\u6700\u9069\u5408\u7684\u6a21\u578b\u985e\u578b<br \/>\nLlama \u7cfb\u5217\uff1a\u6700\u5e38\u898b\uff0c\u7d50\u69cb\u6a19\u6e96\uff0cQ\/K\/V \u5206\u660e\uff0c\u6700\u5bb9\u6613\u6539\u3002<br \/>\nQwen \u7cfb\u5217\uff1a\u4e5f\u662f\u6a19\u6e96 decoder-only \u8def\u7dda\uff0c\u901a\u5e38\u540c\u6a23\u9069\u5408\u505a attention \u66ff\u63db\u3002<br \/>\nMistral \u7cfb\u5217\uff1a\u540c\u6a23\u5c6c\u65bc decoder-only LLM\uff0c\u7406\u8ad6\u4e0a\u4e5f\u9069\u5408\u3002<br \/>\nGPT-style \/ LLaMA-style \u81ea\u56de\u6b78\u6a21\u578b\uff1a\u53ea\u8981\u662f\u55ae\u5411 causal attention\uff0c\u4e00\u822c\u90fd\u80fd\u6539\u3002<\/p>\n<p>\u6574\u9ad4\u4f86\u8aaa\uff0cLighthouse Attention \u6700\u5438\u5f15\u4e4b\u8655\uff0c\u4e0d\u53ea\u662f\u8ffd\u6c42\u66f4\u5feb\uff0c\u800c\u662f\u5617\u8a66\u5728\u8a13\u7df4\u671f\u4fdd\u7559\u8207\u73fe\u6709\u751f\u614b\u7684\u76f8\u5bb9\u6027\u3002\u5c0d\u9700\u8981\u63a2\u7d22 98K\u3001512K \u751a\u81f3\u66f4\u9577\u4e0a\u4e0b\u6587\u8a13\u7df4\u7684\u4eba\uff0c\u5b83\u662f\u4e00\u500b\u503c\u5f97\u7d30\u770b\u3001\u4f46\u660e\u986f\u504f\u7814\u7a76\u8207\u5de5\u7a0b\u7528\u9014\u7684\u5c08\u6848\u3002<\/p>\n<p><strong>\u7db2\u5740\uff1a<\/strong> <a href=\"https:\/\/github.com\/ighoshsubho\/lighthouse-attention\" rel=\"noopener noreferrer\">https:\/\/github.com\/ighoshsubho\/lighthouse-attention<\/a><\/p>\n<p><strong>\u7db2\u5740\uff1a<\/strong> <a href=\"https:\/\/nousresearch.com\/lighthouse-attention\" rel=\"noopener noreferrer\">https:\/\/nousresearch.com\/lighthouse-attention<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u9019\u500b\u5c08\u6848\u805a\u7126\u8d85\u9577\u6587\u672c\u8a13\u7df4\uff0c\u76ee\u6a19\u662f\u6e1b\u5c11\u6ce8\u610f\u529b\u8a08\u7b97\u6210\u672c\u3002\u5c0d\u7814\u7a76\u9577\u4e0a\u4e0b\u6587\u6a21\u578b\u7684\u4eba\u4f86\u8aaa\uff0c\u65b9\u5411\u76f8\u7576\u6e05\u6670\u3002<\/p>\n","protected":false},"author":8,"featured_media":8295,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[133,127,160],"tags":[],"class_list":["post-8296","post","type-post","status-publish","format-standard","hentry","category-133","category-127","category-160"],"_links":{"self":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/posts\/8296","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/comments?post=8296"}],"version-history":[{"count":0,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/posts\/8296\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/media\/8295"}],"wp:attachment":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/media?parent=8296"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/categories?post=8296"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/tags?post=8296"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}