weird criterion to decide if needed to adjust the padding size

[hyusterr]

transformers/src/transformers/trainer_pt_utils.py

Line 93 in 241c04d

if len(tensor1.shape) == 1 or tensor1.shape[1] == tensor2.shape[1]:

When working with Trainer and model output with TokenClassiferOutput, the Trainer collect outputs batch by batch in evaluate_loop with nested_concat.
The criterion to decide whether it is needed to adjust the shape is by tensor.shape[1].
However, if user does not exclude attentions with ignore_key beforehand, the shape[1] of the attentions tensor is always the same. Leading to error occurred with torch.cat.
I think this design is kind of weird, especially because this behavior is not emphasized in the token classification tutorial: https://huggingface.co/docs/transformers/tasks/token_classification

[Rocketknight1]

cc @SunMarc @muellerzr