1 Star 0 Fork 2

abo/nlpOption

forked from zck/nlpOption 
加入 Gitee
与超过 1200万 开发者一起发现、参与优秀开源项目,私有仓库也完全免费 :)
免费加入
文件
克隆/下载
data_cls.py 1.21 KB
一键复制 编辑 原始数据 按行查看 历史
zck 提交于 2022-04-28 19:31 . init
import json
from tqdm import tqdm
def is_target_first(text, target, word):
return text.find(target) <= text.find(word)
def load_dict(dict_path):
with open(dict_path, "r", encoding="utf-8") as f:
words = [word.strip() for word in f.readlines()]
word2id = dict(zip(words, range(len(words))))
id2word = dict((v, k) for k, v in word2id.items())
return word2id, id2word
def read(data_path):
with open(data_path, "r", encoding="utf-8") as f:
for line in f.readlines():
items = line.strip().split("\t")
assert len(items) == 3
example = {"label": int(items[0]), "target_text": items[1], "text": items[2]}
yield example
def convert_example_to_feature(example, tokenizer, label2id, max_seq_len=512, is_test=False):
encoded_inputs = tokenizer(example["target_text"], text_pair=example["text"], max_seq_len=max_seq_len,
return_length=True)
if not is_test:
label = example["label"]
return encoded_inputs["input_ids"], encoded_inputs["token_type_ids"], encoded_inputs["seq_len"], label
return encoded_inputs["input_ids"], encoded_inputs["token_type_ids"], encoded_inputs["seq_len"]
Loading...
马建仓 AI 助手
尝试更多
代码解读
代码找茬
代码优化
1
https://gitee.com/heshao996/nlp-option.git
[email protected]:heshao996/nlp-option.git
heshao996
nlp-option
nlpOption
master

搜索帮助