Hi, this warning means that the sequence you have encoded is longer than the maximum sequence length the model can handle. It isn't related to the tokens you have added.

RoBERTa can only handle sequences of a maximum of 512 tokens, so you should make sure you only pass sequences of a max length of 512 or else it will crash. You can truncate your sequence so that it fits, or you can use another model that can accept longer sequences.

Hi, this warning means that the sequence you have encoded is longer than the maximum sequence length the model can handle. It isn't related to the tokens you have added.

RoBERTa can only handle sequences of a maximum of 512 tokens, so you should make sure you only pass sequences of a max length of 512 or else it will crash. You can truncate your sequence so that it fits, or you can use another model that can accept longer sequences.

Thanks! My bad, completely misunderstood the warning, all fixed. However, my problem now seems to be that the function tokenizer.encode_plus() used in glue_convert_examples_to_features() gets exponentially slower the more words I add to the tokenizer's vocabulary.

For example, starting with a tokenizer vocabulary size of 50265, the tokenizer.encode_plus() takes ~0.00048 sec per call. If I add 1200 more words, giving me a tokenizer vocabulary size of 51465, the tokenizer.encode_plus() now takes ~0.05729 sec per call, which is ~120x slower. It gets even worse the more words I add, causing me waiting times up to 1h just to pre-process the dataset. What causes this exponential (or extreme linear) growth to happen? Is it possible to optimize it?

Hi, this warning means that the sequence you have encoded is longer than the maximum sequence length the model can handle. It isn't related to the tokens you have added.
RoBERTa can only handle sequences of a maximum of 512 tokens, so you should make sure you only pass sequences of a max length of 512 or else it will crash. You can truncate your sequence so that it fits, or you can use another model that can accept longer sequences.

Thanks! My bad, completely misunderstood the warning, all fixed. However, my problem now seems to be that the function tokenizer.encode_plus() used in glue_convert_examples_to_features() gets exponentially slower the more words I add to the tokenizer's vocabulary.

For example, starting with a tokenizer vocabulary size of 50265, the tokenizer.encode_plus() takes ~0.00048 sec per call. If I add 1200 more words, giving me a tokenizer vocabulary size of 51465, the tokenizer.encode_plus() now takes ~0.05729 sec per call, which is ~120x slower. It gets even worse the more words I add, causing me waiting times up to 1h just to pre-process the dataset. What causes this exponential (or extreme linear) growth to happen? Is it possible to optimize it?

Please see #1830 , #1621 and #1881

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.