The "extra quality" configuration yields a noticeable jump in tasks that require nuance—sentiment analysis on imbalanced datasets, legal document classification, and medical NER.
Related search suggestions provided.
One of the hidden gems of WALS is the ability to add new tokens post-hoc. With extra quality, the least squares solver for new token embeddings runs until the residual drops below 1e-7, meaning the new token integrates seamlessly into the semantic space as if it had been pretrained from the beginning. wals roberta sets extra quality
# Replace with reconstructed weights (lossless compression) new_embedding = torch.nn.Embedding.from_pretrained(torch.tensor(reconstructed_embeddings)) model.set_input_embeddings(new_embedding) The "extra quality" configuration yields a noticeable jump