市场营销策划方案格式模板_济南的网站制作公司_建站优化推广_cilimao磁力猫最新版地址

转载自：| 03_language_model/03_Bert完形填空.ipynb | 基于transformers使用Bert模型做完形填空 |Open In Colab |

完形填空

利用语言模型，可以完成完形填空（fill mask），预测缺失的单词。
当前，效果最好的语言模型是Bert系列的预训练语言模型。

!pip install transformers

import osfrom transformers import pipelineos.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE"
model_name = "hfl/chinese-macbert-base"nlp = pipeline("fill-mask",model=model_name,tokenizer=model_name,device=-1,  # gpu device id)
from pprint import pprintpprint(nlp(f"明天天{nlp.tokenizer.mask_token}很好?"))
print("*" * 42)
pprint(nlp(f"明天心{nlp.tokenizer.mask_token}很好?"))
print("*" * 42)
pprint(nlp(f"张亮在哪里任{nlp.tokenizer.mask_token}?"))
print("*" * 42)
pprint(nlp(f"少先队员{nlp.tokenizer.mask_token}该为老人让座位。"))

模型默认保存在：~/.cache/huggingface/transformers

不通过pipeline，可以自己写预测逻辑：

from transformers import AutoModelWithLMHead, AutoTokenizer
import torch# tokenizer = AutoTokenizer.from_pretrained("distilbert-base-cased")
# model = AutoModelWithLMHead.from_pretrained("distilbert-base-cased")
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelWithLMHead.from_pretrained(model_name)sequence = f"明天天{nlp.tokenizer.mask_token}很好."
input = tokenizer.encode(sequence, return_tensors="pt")
mask_token_index = torch.where(input == tokenizer.mask_token_id)[1]
token_logits = model(input).logits
mask_token_logits = token_logits[0, mask_token_index, :]
top_5_tokens = torch.topk(mask_token_logits, 5, dim=1).indices[0].tolist()
for token in top_5_tokens:print(sequence.replace(tokenizer.mask_token, tokenizer.decode([token])))