代码拉取完成,页面将自动刷新
随着深度学习的发展,NLP领域涌现了一大批高质量的Transformer类预训练模型,多次刷新了不同NLP任务的SOTA(State of the Art),极大地推动了自然语言处理的进展。
PaddleNLP为用户提供了常用的预训练模型及其相应权重,如 BERT
、ERNIE
、ALBERT
、RoBERTa
、XLNet
等,采用统一的API进行加载、训练和调用,
让开发者能够方便快捷地应用各种Transformer类预训练模型及其下游任务,且相应预训练模型权重下载速度快、稳定。
PaddleNLP Transformer API在提供丰富预训练模型的同时,也降低了用户的使用门槛。 使用Auto模块,可以加载不同网络结构的预训练模型,无需查找模型对应的类别。只需十几行代码,用户即可完成模型加载和下游任务Fine-tuning。
from functools import partial
import numpy as np
import paddle
from paddlenlp.datasets import load_dataset
from paddlenlp.transformers import AutoModelForSequenceClassification, AutoTokenizer
train_ds = load_dataset("chnsenticorp", splits=["train"])
model = AutoModelForSequenceClassification.from_pretrained("bert-wwm-chinese", num_classes=len(train_ds.label_list))
tokenizer = AutoTokenizer.from_pretrained("bert-wwm-chinese")
def convert_example(example, tokenizer):
encoded_inputs = tokenizer(text=example["text"], max_seq_len=512, pad_to_max_seq_len=True)
return tuple([np.array(x, dtype="int64") for x in [
encoded_inputs["input_ids"], encoded_inputs["token_type_ids"], [example["label"]]]])
train_ds = train_ds.map(partial(convert_example, tokenizer=tokenizer))
batch_sampler = paddle.io.BatchSampler(dataset=train_ds, batch_size=8, shuffle=True)
train_data_loader = paddle.io.DataLoader(dataset=train_ds, batch_sampler=batch_sampler, return_list=True)
optimizer = paddle.optimizer.AdamW(learning_rate=0.001, parameters=model.parameters())
criterion = paddle.nn.loss.CrossEntropyLoss()
for input_ids, token_type_ids, labels in train_data_loader():
logits = model(input_ids, token_type_ids)
loss = criterion(logits, labels)
loss.backward()
optimizer.step()
optimizer.clear_grad()
上面的代码给出使用预训练模型的简要示例,更完整详细的示例代码, 可以参考:使用预训练模型Fine-tune完成中文文本分类任务
from_pretrained()
方法加载。
Auto模块(包括AutoModel, AutoTokenizer, 及各种下游任务类)提供了方便易用的接口,
无需指定类别,即可调用不同网络结构的预训练模型。
第一个参数是汇总表中对应的 Pretrained Weight
,可加载对应的预训练权重。
AutoModelForSequenceClassification
初始化 __init__
所需的其他参数,如 num_classes
等,
也是通过 from_pretrained()
传入。Tokenizer
使用同样的 from_pretrained
方法加载。Dataset
的 map
函数,使用 tokenizer
将 dataset
从原始文本处理成模型的输入。BatchSampler
和 DataLoader
,shuffle数据、组合Batch。PaddleNLP的Transformer预训练模型包含从 huggingface.co 直接转换的模型权重和百度自研模型权重,方便社区用户直接迁移使用。 目前共包含了40多个主流预训练模型,500多个模型权重。
.. toctree:: :maxdepth: 3 ALBERT <transformers/ALBERT/contents> BART <transformers/BART/contents> BERT <transformers/BERT/contents> BigBird <transformers/BigBird/contents> Blenderbot <transformers/Blenderbot/contents> Blenderbot-Small <transformers/Blenderbot-Small/contents> ChineseBert <transformers/ChineseBert/contents> ConvBert <transformers/ConvBert/contents> CTRL <transformers/CTRL/contents> Deberta <transformers/Deberta/contents> DebertaV2 <transformers/DebertaV2/contents> DistilBert <transformers/DistilBert/contents> ELECTRA <transformers/ELECTRA/contents> ERNIE <transformers/ERNIE/contents> ERNIE-CTM <transformers/ERNIE-CTM/contents> ERNIE-DOC <transformers/ERNIE-DOC/contents> ERNIE-GEN <transformers/ERNIE-GEN/contents> ERNIE-GRAM <transformers/ERNIE-GRAM/contents> ERNIE-M <transformers/ERNIE-M/contents> FNet <transformers/FNet/contents> Funnel <transformers/Funnel/contents> GPT <transformers/GPT/contents> LayoutLM <transformers/LayoutLM/contents> LayoutLMV2 <transformers/LayoutLMV2/contents> LayoutXLM <transformers/LayoutXLM/contents> Luke <transformers/Luke/contents> MBart <transformers/MBart/contents> MegatronBert <transformers/MegatronBert/contents> MobileBert <transformers/MobileBert/contents> MPNet <transformers/MPNet/contents> NeZha <transformers/NeZha/contents> PPMiniLM <transformers/PPMiniLM/contents> ProphetNet <transformers/ProphetNet/contents> Reformer <transformers/Reformer/contents> RemBert <transformers/RemBert/contents> RoBERTa <transformers/RoBERTa/contents> RoFormer <transformers/RoFormer/contents> SKEP <transformers/SKEP/contents> SqueezeBert <transformers/SqueezeBert/contents> T5 <transformers/T5/contents> TinyBert <transformers/TinyBert/contents> UnifiedTransformer <transformers/UnifiedTransformer/contents> UNIMO <transformers/UNIMO/contents> XLNet <transformers/XLNet/contents>
Model | Sequence Classification | Token Classification | Question Answering | Text Generation | Multiple Choice |
---|---|---|---|---|---|
ALBERT | ✅ | ✅ | ✅ | ❌ | ✅ |
BART | ✅ | ✅ | ✅ | ✅ | ❌ |
BERT | ✅ | ✅ | ✅ | ❌ | ✅ |
BigBird | ✅ | ✅ | ✅ | ❌ | ✅ |
Blenderbot | ❌ | ❌ | ❌ | ✅ | ❌ |
Blenderbot-Small | ❌ | ❌ | ❌ | ✅ | ❌ |
ChineseBert | ✅ | ✅ | ✅ | ❌ | ❌ |
ConvBert | ✅ | ✅ | ✅ | ❌ | ✅ |
CTRL | ✅ | ❌ | ❌ | ❌ | ❌ |
Deberta | ✅ | ✅ | ✅ | ❌ | ✅ |
DebertaV2 | ✅ | ✅ | ✅ | ❌ | ✅ |
DistilBert | ✅ | ✅ | ✅ | ❌ | ❌ |
ELECTRA | ✅ | ✅ | ✅ | ❌ | ✅ |
ERNIE | ✅ | ✅ | ✅ | ❌ | ✅ |
ERNIE-CTM | ❌ | ✅ | ❌ | ❌ | ❌ |
ERNIE-DOC | ✅ | ✅ | ✅ | ❌ | ❌ |
ERNIE-GEN | ❌ | ❌ | ❌ | ✅ | ❌ |
ERNIE-GRAM | ✅ | ✅ | ✅ | ❌ | ❌ |
ERNIE-M | ✅ | ✅ | ✅ | ❌ | ❌ |
FNet | ✅ | ✅ | ✅ | ❌ | ✅ |
Funnel | ✅ | ✅ | ✅ | ❌ | ❌ |
GPT | ✅ | ✅ | ❌ | ✅ | ❌ |
LayoutLM | ✅ | ✅ | ❌ | ❌ | ❌ |
LayoutLMV2 | ❌ | ✅ | ❌ | ❌ | ❌ |
LayoutXLM | ❌ | ✅ | ❌ | ❌ | ❌ |
Luke | ❌ | ✅ | ✅ | ❌ | ❌ |
MBart | ✅ | ❌ | ✅ | ❌ | ✅ |
MegatronBert | ✅ | ✅ | ✅ | ❌ | ✅ |
MobileBert | ✅ | ❌ | ✅ | ❌ | ❌ |
MPNet | ✅ | ✅ | ✅ | ❌ | ✅ |
NeZha | ✅ | ✅ | ✅ | ❌ | ✅ |
PPMiniLM | ✅ | ❌ | ❌ | ❌ | ❌ |
ProphetNet | ❌ | ❌ | ❌ | ✅ | ❌ |
Reformer | ✅ | ❌ | ✅ | ❌ | ❌ |
RemBert | ✅ | ✅ | ✅ | ❌ | ✅ |
RoBERTa | ✅ | ✅ | ✅ | ❌ | ✅ |
RoFormer | ✅ | ✅ | ✅ | ❌ | ❌ |
SKEP | ✅ | ✅ | ❌ | ❌ | ❌ |
SqueezeBert | ✅ | ✅ | ✅ | ❌ | ❌ |
T5 | ❌ | ❌ | ❌ | ✅ | ❌ |
TinyBert | ✅ | ❌ | ❌ | ❌ | ❌ |
UnifiedTransformer | ❌ | ❌ | ❌ | ✅ | ❌ |
XLNet | ✅ | ✅ | ✅ | ❌ | ✅ |
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。