paddlespeech.text.models.ernie_linear.dataset module

class paddlespeech.text.models.ernie_linear.dataset.PuncDataset(train_path, vocab_path, punc_path, seq_len=100)[source]

Bases: Dataset

Methods

load_vocab
preprocess

load_vocab(vocab_path, extra_word_list=[], encoding='utf-8')[source]

preprocess(txt_seqs: list)[source]

class paddlespeech.text.models.ernie_linear.dataset.PuncDatasetFromErnieTokenizer(train_path, punc_path, pretrained_token='ernie-1.0', seq_len=100)[source]

Bases: Dataset

Methods

load_vocab
preprocess

load_vocab(vocab_path, extra_word_list=[], encoding='utf-8')[source]

preprocess(txt_seqs: list)[source]