{"id":6607,"date":"2025-10-16T18:49:38","date_gmt":"2025-10-16T10:49:38","guid":{"rendered":"https:\/\/infernews.com\/?p=6607"},"modified":"2025-10-16T18:49:47","modified_gmt":"2025-10-16T10:49:47","slug":"%e8%aa%8d%e8%ad%98-token","status":"publish","type":"post","link":"https:\/\/infernews.com\/blog\/%e8%aa%8d%e8%ad%98-token\/","title":{"rendered":"\u8a8d\u8b58 Token"},"content":{"rendered":"\n<p><strong>Token \u662f\u4ec0\u9ebc\uff1f<\/strong><br>Token \u662f\u8a9e\u8a00\u6a21\u578b\u904b\u7b97\u8207\u8a08\u8cbb\u7684\u57fa\u672c\u55ae\u4f4d\u3002\u6a21\u578b\u6703\u5c07\u8f38\u5165\u53e5\u5b50\u5207\u5206\u70ba token\uff0c\u9019\u4e9b token \u53ef\u80fd\u662f\u55ae\u5b57\u3001\u5b50\u8a5e\u3001\u751a\u81f3\u55ae\u4e00\u5b57\u5143\uff0c\u6bcf\u500b token \u5728\u6a21\u578b\u5167\u90fd\u6709\u4e00\u500b\u6578\u5b57\u7de8\u865f\uff0c\u6a21\u578b\u5be6\u969b\u904b\u7b97\u90fd\u662f\u5728\u9019\u4e9b\u6578\u503c\u4e0a\u9032\u884c\u3002\u200b<\/p>\n\n\n\n<p><strong>\u4e0d\u540c\u6a21\u578b\u70ba\u4ec0\u9ebc token \u8a08\u7b97\u4e0d\u4e00\u6a23\uff1f<\/strong><br>\u6bcf\u500b\u6a21\u578b\u7684 tokenizer\uff08\u5206\u8a5e\u5668\uff09\u90fd\u6709\u81ea\u5df1\u7684 vocab\uff08\u8a5e\u5f59\u8868\uff09\u8207\u5207\u5206\u898f\u5247\u3002\u4f8b\u5982\u540c\u4e00\u53e5\u300cHello world\u300d\u7d93 OpenAI \u7684 tokenizer \u6703\u7522\u751f 3 \u500b token\uff0c\u4f46\u7528 Google \u6216 Anthropic \u670d\u52d9\u5247\u53ef\u80fd\u662f 4 \u500b\u6216\u66f4\u591a\/\u66f4\u5c11\uff0c\u9019\u53d6\u6c7a\u65bc\u5404\u5bb6\u8a5e\u5f59\u8868\u8a2d\u8a08\u8207\u5207\u5206\u7b56\u7565\u3002\u200b<\/p>\n\n\n\n<p><strong>Token \u5982\u4f55\u7522\u751f\uff1f<\/strong><br>Tokenizer \u9996\u5148\u6703\u5f9e\u8a9e\u6599\u8a13\u7df4\u51fa\u4e00\u5957\u8a5e\u5f59\u8868\u3002\u7c21\u6613\u7684\u505a\u6cd5\u5982\u300c\u5b57\u7b26\u7d1a\u300d\u5207\u5206\uff0c\u6bcf\u4e00\u5b57\u5143\u90fd\u662f\u4e00\u500b token\uff0c\u9019\u6703\u9020\u6210 token \u6578\u66b4\u589e\u3002\u9032\u5316\u7684\u65b9\u6cd5\u662f\u5c07\u9ad8\u983b\u51fa\u73fe\u7684\u8a5e\u7d44\u5408\u6210\u8f03\u9577\u7684\u5b50\u8a5e\uff08subword\uff09\uff0c\u6e1b\u5c11 token \u6578\uff0c\u9032\u800c\u63d0\u5347\u6548\u7387\u3002\u200b<\/p>\n\n\n\n<p><strong>\u7f55\u898b\u8a5e\u8207\u7279\u6b8a\u8a9e\u8a00\u600e\u9ebc\u5206\u8a5e\uff1f<\/strong><br>\u50cf\u4eba\u540d\u3001\u4e82\u78bc\u3001\u7a00\u6709\u55ae\u8a5e\u3001\u51b7\u9580\u8a9e\u8a00\u7b49\u82e5\u5728\u8a9e\u6599\u4e2d\u8f03\u5c11\u898b\uff0c\u6703\u88ab\u5207\u6210\u66f4\u591a\u66f4\u5c0f\u7684 token\u3002\u9019\u4ee3\u8868\u4f7f\u7528\u6a21\u578b\u8655\u7406\u4e2d\u6587\u3001\u7cb5\u8a9e\u3001\u5c0f\u8a9e\u7a2e\u6216\u975e\u4e3b\u6d41\u7a0b\u5f0f\u8a9e\u8a00\u6642\uff0ctoken \u6578\u53ef\u80fd\u986f\u8457\u589e\u52a0\uff0c\u5c0e\u81f4 API \u6210\u672c\u4e0a\u5347\u3002\u200b<\/p>\n\n\n<figure class=\"wp-block-embed-youtube wp-block-embed is-type-video is-provider-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"lyte-wrapper\" title=\"Most devs don&amp;#039;t understand how LLM tokens work\" style=\"width:853px;max-width:100%;margin:5px auto;\"><div class=\"lyMe\" id=\"WYL_nKSk_TiR8YA\" itemprop=\"video\" itemscope itemtype=\"https:\/\/schema.org\/VideoObject\"><div><meta itemprop=\"thumbnailUrl\" content=\"https:\/\/infernews.com\/blog\/wp-content\/plugins\/wp-youtube-lyte\/lyteCache.php?origThumbUrl=https%3A%2F%2Fi.ytimg.com%2Fvi%2FnKSk_TiR8YA%2Fhqdefault.jpg\" \/><meta itemprop=\"embedURL\" content=\"https:\/\/www.youtube.com\/embed\/nKSk_TiR8YA\" \/><meta itemprop=\"duration\" content=\"PT10M58S\" \/><meta itemprop=\"uploadDate\" content=\"2025-09-19T11:31:59Z\" \/><\/div><div id=\"lyte_nKSk_TiR8YA\" data-src=\"https:\/\/infernews.com\/blog\/wp-content\/plugins\/wp-youtube-lyte\/lyteCache.php?origThumbUrl=https%3A%2F%2Fi.ytimg.com%2Fvi%2FnKSk_TiR8YA%2Fhqdefault.jpg\" class=\"pL\"><div class=\"tC\"><div class=\"tT\" itemprop=\"name\">Most devs don&#039;t understand how LLM tokens work<\/div><\/div><div class=\"play\"><\/div><div class=\"ctrl\"><div class=\"Lctrl\"><\/div><div class=\"Rctrl\"><\/div><\/div><\/div><noscript><a href=\"https:\/\/youtu.be\/nKSk_TiR8YA\" rel=\"nofollow\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/infernews.com\/blog\/wp-content\/plugins\/wp-youtube-lyte\/lyteCache.php?origThumbUrl=https%3A%2F%2Fi.ytimg.com%2Fvi%2FnKSk_TiR8YA%2F0.jpg\" alt=\"Most devs don&amp;#039;t understand how LLM tokens work\" width=\"853\" height=\"460\" \/><br \/>Watch this video on YouTube<\/a><\/noscript><meta itemprop=\"description\" content=\"Most devs are using LLMs daily but don&#039;t have a clue about some of the fundamentals. Understanding tokens is crucial because you need to know how you&#039;re being billed, and why billing is different per provider. Join my newsletter on AI Hero: https:\/\/www.aihero.dev\/s\/y-newsletter Code: https:\/\/github.com\/mattpocock\/ai-sdk-tips\/tree\/main\/exercises\/03-tokens 00:00 Intro 00:33 Input &amp; Output Tokens 01:11 Monitoring Token Usage 02:16 What are tokens? 02:50 Tiktoken 03:47 Full LLM Process 04:49 Building Token Vocabularies 05:25 Character-Level Tokenizer 06:37 Vocabulary Size 07:20 Subword-Level Tokenizer 08:33 Building Longer Subwords 08:58 Unusual Words 09:45 Summary\"><\/div><\/div><div class=\"lL\" style=\"max-width:100%;width:853px;margin:5px auto;\"><\/div><figcaption><\/figcaption><\/figure>\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Token \u662f\u4ec0\u9ebc\uff1fToken \u662f\u8a9e\u8a00\u6a21\u578b\u904b\u7b97\u8207\u8a08\u8cbb\u7684\u57fa\u672c\u55ae\u4f4d\u3002\u6a21\u578b\u6703\u5c07\u8f38\u5165\u53e5\u5b50\u5207\u5206\u70ba token\uff0c\u9019\u4e9b token \u53ef\u80fd\u662f\u55ae\u5b57\u3001\u5b50\u8a5e\u3001\u751a\u81f3\u55ae\u4e00\u5b57\u5143\uff0c\u6bcf\u500b token \u5728\u6a21\u578b\u5167\u90fd\u6709\u4e00\u500b\u6578\u5b57\u7de8\u865f\uff0c\u6a21\u578b\u5be6\u969b\u904b\u7b97\u90fd\u662f\u5728\u9019\u4e9b\u6578\u503c\u4e0a\u9032\u884c\u3002\u200b \u4e0d\u540c\u6a21\u578b\u70ba\u4ec0\u9ebc token \u8a08\u7b97\u4e0d\u4e00\u6a23\uff1f\u6bcf\u500b\u6a21\u578b\u7684 tokenizer\uff08\u5206\u8a5e\u5668\uff09\u90fd\u6709\u81ea\u5df1\u7684 vocab\uff08\u8a5e\u5f59\u8868\uff09\u8207\u5207\u5206\u898f\u5247\u3002\u4f8b\u5982\u540c\u4e00\u53e5\u300cHello world\u300d\u7d93 OpenAI \u7684 tokenizer \u6703\u7522\u751f 3 \u500b token\uff0c\u4f46\u7528 Google \u6216 Anthropic \u670d\u52d9\u5247\u53ef\u80fd\u662f 4 \u500b\u6216\u66f4\u591a\/\u66f4\u5c11\uff0c\u9019\u53d6\u6c7a\u65bc\u5404\u5bb6\u8a5e\u5f59\u8868\u8a2d\u8a08\u8207\u5207\u5206\u7b56\u7565\u3002\u200b Token \u5982\u4f55\u7522\u751f\uff1fTokenizer \u9996\u5148\u6703\u5f9e\u8a9e\u6599\u8a13\u7df4\u51fa\u4e00\u5957\u8a5e\u5f59\u8868\u3002\u7c21\u6613\u7684\u505a\u6cd5\u5982\u300c\u5b57\u7b26\u7d1a\u300d\u5207\u5206\uff0c\u6bcf\u4e00\u5b57\u5143\u90fd\u662f\u4e00\u500b token\uff0c\u9019\u6703\u9020\u6210 token \u6578\u66b4\u589e\u3002\u9032\u5316\u7684\u65b9\u6cd5\u662f\u5c07\u9ad8\u983b\u51fa\u73fe\u7684\u8a5e\u7d44\u5408\u6210\u8f03\u9577\u7684\u5b50\u8a5e\uff08subword\uff09\uff0c\u6e1b\u5c11 token \u6578\uff0c\u9032\u800c\u63d0\u5347\u6548\u7387\u3002\u200b \u7f55\u898b\u8a5e\u8207\u7279\u6b8a\u8a9e\u8a00\u600e\u9ebc\u5206\u8a5e\uff1f\u50cf\u4eba\u540d\u3001\u4e82\u78bc\u3001\u7a00\u6709\u55ae\u8a5e\u3001\u51b7\u9580\u8a9e\u8a00\u7b49\u82e5\u5728\u8a9e\u6599\u4e2d\u8f03\u5c11\u898b\uff0c\u6703\u88ab\u5207\u6210\u66f4\u591a\u66f4\u5c0f\u7684 token\u3002\u9019\u4ee3\u8868\u4f7f\u7528\u6a21\u578b\u8655\u7406\u4e2d\u6587\u3001\u7cb5\u8a9e\u3001\u5c0f\u8a9e\u7a2e\u6216\u975e\u4e3b\u6d41\u7a0b\u5f0f\u8a9e\u8a00\u6642\uff0ctoken \u6578\u53ef\u80fd\u986f\u8457\u589e\u52a0\uff0c\u5c0e\u81f4 API \u6210\u672c\u4e0a\u5347\u3002\u200b<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"googlesitekit_rrm_CAowvqSiDA:productID":"","footnotes":""},"categories":[27],"tags":[],"class_list":["post-6607","post","type-post","status-publish","format-standard","hentry","category-paper"],"_links":{"self":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/posts\/6607","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/comments?post=6607"}],"version-history":[{"count":0,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/posts\/6607\/revisions"}],"wp:attachment":[{"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/media?parent=6607"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/categories?post=6607"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/infernews.com\/blog\/wp-json\/wp\/v2\/tags?post=6607"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}