Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Excessive Redundant Bounding Boxes in PaliGemma2 Fine-Tuning for Detection Tasks #358

Open
1 of 2 tasks
David-19940718 opened this issue Jan 23, 2025 · 1 comment
Open
1 of 2 tasks

Excessive Redundant Bounding Boxes in PaliGemma2 Fine-Tuning for Detection Tasks #358

David-19940718 opened this issue Jan 23, 2025 · 1 comment
Labels
bug Something isn't working

Comments

@David-19940718
Copy link

David-19940718 commented Jan 23, 2025 •
edited
Loading

Search before asking

  • I have searched the Roboflow Notebooks issues and found no similar bug report.

Notebook name

https://github.com/roboflow/notebooks/blob/main/notebooks/how-to-finetune-paligemma2-on-detection-dataset.ipynb

Bug

Thank you for your excellent work. While attempting to fine-tune for a downstream detection task based on PaliGemma2, I noticed that the final training results produced many redundant bounding boxes. Based on the predictions, the model is capable of detecting the targets, but it continuously outputs additional bounding boxes until it reaches the set max-new-tokens limit. Could you provide any insights or suggestions on this issue?

<loc0400><loc0516><loc0652><loc0712> 7 of clubs ; <loc0292><loc0300><loc0584><loc0512> 8 of clubs ; <loc0406><loc0724><loc0708><loc1007> 5 of clubs ; <loc0216><loc0084><loc0528><loc0316> 6 of clubs ; <loc0400><loc0516><loc0648><loc0708> 6 of clubs ; <loc0292><loc0295><loc0580><loc0512> 8 of clubs ; <loc0412><loc0732><loc0701><loc1007> 4 of clubs ; <loc0208><loc0080><loc0528><loc0316> 5 of clubs ; <loc0756><loc0136><loc1023><loc0316> 10 of clubs ; <loc0000><loc0000><loc1023><loc1016> 9 of clubs ; <loc0000><loc0000><loc0580><loc0540> 10 of clubs ; <loc0416><loc0540><loc0644><loc0708> 8 of clubs ; <loc0756><loc0144><loc0880><loc0292> 5 of clubs ; <loc0756><loc0144><loc1023><loc0322> 2 of clubs ; <loc0756><loc0144><loc1023><loc0316> 9 of clubs ; <loc0756><loc0144><loc1023><loc0316> 5 of clubs ; <loc0756><loc0144><loc1023><loc0305> 5 of clubs ; <loc0756><loc0144><loc1023><loc0305> 5 of clubs ; <loc0756><loc0144><loc1023><loc0295> 5 of clubs ; <loc0756><loc0232><loc1023><loc0322> 5 of clubs ; <loc0738><loc0000><loc1023><loc0136> 5 of clubs ; <loc0756><loc0000><loc1023><loc0136> 5 of clubs ; <loc0756><loc0000><loc1023><loc0136> 5 of clubs ; <loc0000><loc0000><loc0580><loc0372> 10 of clubs ; <loc0738><loc0000><loc1023><loc0136> 5 of clubs ; <loc0738><loc0000><loc1023>

Environment

  • Local
  • OS: Ubuntu 20.04
  • Python: 3.10.6
  • Transformers: 4.47.0

Minimal Reproducible Example

No response

Additional

Additionally, here is the terminal log output:

ubuntu@ubuntu:/ssd2/workspace/mllm/fine-tune-paligemma/Google-PaliGemma2-Finetune$ CUDA_VISIBLE_DEVICES=2,3 python train.py --lora --epochs 8
hyperparameters: remove_unused_columns=False, gradient_accumulation_steps=16, warmup_steps=2, weight_decay=1e-06, adam_beta2=0.999, logging_steps=50, optim=adamw_hf, save_strategy=steps, save_steps=200, save_total_limit=1, bf16=True, report_to=['tensorboard'], dataloader_pin_memory=False
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:05<00:00,  3.00s/it]
trainable params: 11,876,352 || all params: 3,045,003,504 || trainable%: 0.3900
freezing vision model layers
freezing multi-modal projector
Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
{'loss': 2.6005, 'grad_norm': 23.956716537475586, 'learning_rate': 1.7587939698492464e-05, 'epoch': 0.99}                                                                                                                   
{'loss': 1.9132, 'grad_norm': 23.055450439453125, 'learning_rate': 1.5075376884422112e-05, 'epoch': 1.97}                                                                                                                   
{'loss': 1.6768, 'grad_norm': 33.97260284423828, 'learning_rate': 1.256281407035176e-05, 'epoch': 2.95}                                                                                                                     
{'loss': 1.5855, 'grad_norm': 26.143875122070312, 'learning_rate': 1.0050251256281408e-05, 'epoch': 3.93}                                                                                                                   
{'loss': 1.5406, 'grad_norm': 24.072601318359375, 'learning_rate': 7.537688442211056e-06, 'epoch': 4.91}                                                                                                                    
{'loss': 1.515, 'grad_norm': 34.959720611572266, 'learning_rate': 5.025125628140704e-06, 'epoch': 5.89}                                                                                                                     
{'loss': 1.5009, 'grad_norm': 29.38210105895996, 'learning_rate': 2.512562814070352e-06, 'epoch': 6.87}                                                                                                                     
{'loss': 1.4799, 'grad_norm': 39.997161865234375, 'learning_rate': 0.0, 'epoch': 7.85}                                                                                                                                      
{'train_runtime': 5094.8047, 'train_samples_per_second': 1.273, 'train_steps_per_second': 0.079, 'train_loss': 1.726562728881836, 'epoch': 7.85}                                                                            
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 400/400 [1:24:54<00:00, 12.74s/it]
  0%|                                                                                                                                                                                                | 0/44 [00:00<?, ?it/s]The 'batch_size' attribute of HybridCache is deprecated and will be removed in v4.49. Use the more precisely named 'self.max_batch_size' attribute instead.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 44/44 [32:50<00:00, 44.77s/it]
map_result: MeanAveragePrecisionResult:
Metric target: MetricTarget.BOXES
Class agnostic: False
mAP @ 50:95: 0.4415
mAP @ 50:    0.4892
mAP @ 75:    0.4738
mAP scores: [0.48919323 0.48853667 0.48907614 0.47919466 0.47776663 0.47379989
 0.47379989 0.4553695  0.38925978 0.19944912]
IoU thresh: [0.5  0.55 0.6  0.65 0.7  0.75 0.8  0.85 0.9  0.95]
AP per class:
  0: [0.03280268 0.03280268 0.03280268 0.03280268 0.03280268 0.03280268
 0.03280268 0.03280268 0.03280268 0.02328249]
  1: [0.07497781 0.07497781 0.07497781 0.07497781 0.07497781 0.07497781
 0.07497781 0.07497781 0.07497781 0.02398884]
  2: [0.05993939 0.05993939 0.05993939 0.05993939 0.05993939 0.05993939
 0.05993939 0.05993939 0.04830372 0.03577281]
  3: [0.07545013 0.07545013 0.07545013 0.07545013 0.07545013 0.07545013
 0.07545013 0.07545013 0.07545013 0.04084158]
  4: [0.23377338 0.23377338 0.23377338 0.23377338 0.23377338 0.23377338
 0.23377338 0.23377338 0.23377338 0.12376238]
  5: [0.1980198 0.1980198 0.1980198 0.1980198 0.1980198 0.1980198 0.1980198
 0.1980198 0.1980198 0.1980198]
  6: [0.24752475 0.24752475 0.24752475 0.24752475 0.24752475 0.24752475
 0.24752475 0.24752475 0.24752475 0.24752475]
  7: [0.23883888 0.23883888 0.23883888 0.23883888 0.23883888 0.23883888
 0.23883888 0.09207921 0.09207921 0.        ]
  8: [0.330033   0.330033   0.330033   0.330033   0.330033   0.330033
 0.330033   0.330033   0.330033   0.08250825]
  9: [0.330033 0.330033 0.330033 0.330033 0.330033 0.330033 0.330033 0.330033
 0.330033 0.330033]
  10: [0.4950495 0.4950495 0.4950495 0.4950495 0.4950495 0.4950495 0.4950495
 0.4950495 0.4950495 0.4950495]
  11: [0.32850071 0.32850071 0.32850071 0.32850071 0.32850071 0.32850071
 0.32850071 0.32850071 0.12835926 0.04084158]
  12: [0.7029703 0.7029703 0.7029703 0.7029703 0.7029703 0.7029703 0.7029703
 0.7029703 0.7029703 0.       ]
  13: [1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
  14: [0.4950495 0.4950495 0.4950495 0.4950495 0.4950495 0.4950495 0.4950495
 0.4950495 0.4950495 0.       ]
  15: [0.32850071 0.32850071 0.32850071 0.32850071 0.32850071 0.32850071
 0.32850071 0.32850071 0.20226308 0.0466761 ]
  16: [0.46153408 0.42739274 0.34818482 0.34818482 0.27392739 0.27392739
 0.27392739 0.27392739 0.27392739 0.27392739]
  17: [0.34771334 0.34771334 0.34771334 0.34771334 0.34771334 0.34771334
 0.34771334 0.34771334 0.34771334 0.16184311]
  18: [0.34470678 0.34470678 0.34470678 0.34470678 0.34470678 0.34470678
 0.34470678 0.34470678 0.19455264 0.04158416]
  19: [0.11261932 0.11261932 0.11261932 0.11261932 0.11261932 0.11261932
 0.11261932 0.07263803 0.0090009  0.        ]
  20: [0.76661952 0.76661952 0.76661952 0.76661952 0.76661952 0.76661952
 0.76661952 0.76661952 0.47100424 0.04950495]
  21: [0.62164074 0.62164074 0.62164074 0.62164074 0.62164074 0.62164074
 0.62164074 0.62164074 0.62164074 0.04950495]
  22: [0.83168317 0.83168317 0.83168317 0.83168317 0.83168317 0.83168317
 0.83168317 0.83168317 0.61110325 0.        ]
  23: [0.71047105 0.71047105 0.71047105 0.47794779 0.47794779 0.47794779
 0.47794779 0.47794779 0.08550855 0.04950495]
  24: [0.51815182 0.51815182 0.51815182 0.51815182 0.51815182 0.51815182
 0.51815182 0.51815182 0.51815182 0.43894389]
  25: [0.71239981 0.71239981 0.71239981 0.71239981 0.71239981 0.71239981
 0.71239981 0.71239981 0.71239981 0.5709571 ]
  26: [0.6165732  0.6165732  0.6165732  0.6165732  0.6165732  0.6165732
 0.6165732  0.6165732  0.52602183 0.20660066]
  27: [0.72811567 0.72811567 0.72811567 0.44680182 0.44680182 0.44680182
 0.44680182 0.15558699 0.07260726 0.        ]
  28: [0.74422442 0.74422442 0.74422442 0.74422442 0.74422442 0.5379538
 0.5379538  0.5379538  0.5379538  0.34818482]
  29: [0.77310231 0.77310231 0.77310231 0.77310231 0.77310231 0.77310231
 0.77310231 0.77310231 0.77310231 0.14438944]
  30: [0.74014555 0.74014555 0.74014555 0.74014555 0.74014555 0.74014555
 0.74014555 0.74014555 0.74014555 0.17161716]
  31: [0.55941981 0.55941981 0.55941981 0.55941981 0.55941981 0.55941981
 0.55941981 0.55941981 0.55941981 0.01414427]
  32: [0.47854785 0.47854785 0.47854785 0.47854785 0.47854785 0.47854785
 0.47854785 0.47854785 0.47854785 0.22277228]
  33: [0.42285479 0.42285479 0.42285479 0.42285479 0.42285479 0.42285479
 0.42285479 0.42285479 0.24752475 0.25636492]
  34: [0.2491377  0.2491377  0.2491377  0.2491377  0.2491377  0.2491377
 0.2491377  0.10245912 0.10245912 0.04479019]
  35: [0.08392268 0.08392268 0.08392268 0.08392268 0.08392268 0.08392268
 0.08392268 0.08392268 0.08392268 0.01815182]
  36: [0.44554455 0.44554455 0.44554455 0.44554455 0.44554455 0.44554455
 0.44554455 0.44554455 0.44554455 0.0990099 ]
  37: [0.4950495 0.4950495 0.4950495 0.4950495 0.4950495 0.4950495 0.4950495
 0.4950495 0.4950495 1.       ]
  38: [1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
  39: [0.44059406 0.44059406 0.44059406 0.44059406 0.44059406 0.44059406
 0.44059406 0.44059406 0.4950495  0.08168317]
  40: [1.         1.         1.         1.         1.         1.
 1.         1.         1.         0.69059406]
  41: [0.17030549 0.17030549 0.17030549 0.17030549 0.17030549 0.17030549
 0.17030549 0.17030549 0.13645595 0.00634679]
  42: [0.8019802 0.8019802 0.8019802 0.8019802 0.8019802 0.8019802 0.8019802
 0.8019802 0.8019802 0.0990099]
  43: [0.91584158 0.91584158 0.80693069 0.80693069 0.80693069 0.80693069
 0.80693069 0.80693069 0.08168317 0.        ]
  44: [0.81848185 0.81848185 0.81848185 0.81848185 0.81848185 0.81848185
 0.81848185 0.81848185 0.81848185 0.1320132 ]
  45: [0.25990099 0.25990099 0.47607261 0.47607261 0.47607261 0.47607261
 0.47607261 0.14232673 0.14232673 0.14232673]
  46: [0.82791136 0.82791136 0.82791136 0.82791136 0.82791136 0.82791136
 0.82791136 0.82791136 0.82791136 0.42479962]
  47: [0.7019802  0.7019802  0.7019802  0.7019802  0.7019802  0.7019802
 0.7019802  0.7019802  0.06534653 0.        ]
  48: [0.09806695 0.09806695 0.09806695 0.09806695 0.09806695 0.09806695
 0.09806695 0.09806695 0.09806695 0.00884017]
  49: [0.62871287 0.62871287 0.62871287 0.62871287 0.62871287 0.62871287
 0.62871287 0.62871287 0.62871287 0.62871287]
  50: [0.71287129 0.71287129 0.71287129 0.71287129 0.71287129 0.71287129
 0.71287129 0.71287129 0.42574257 0.30693069]
  51: [0.12575994 0.12575994 0.12575994 0.12575994 0.12575994 0.12575994
 0.12575994 0.12575994 0.12575994 0.        ]

Small objects:
  MeanAveragePrecisionResult:
  Metric target: MetricTarget.BOXES
  Class agnostic: False
  mAP @ 50:95: 0.0000
  mAP @ 50:    0.0000
  mAP @ 75:    0.0000
  mAP scores: [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
  IoU thresh: [0.5  0.55 0.6  0.65 0.7  0.75 0.8  0.85 0.9  0.95]
  AP per class:
    No results
  
Medium objects:
  MeanAveragePrecisionResult:
  Metric target: MetricTarget.BOXES
  Class agnostic: False
  mAP @ 50:95: 0.0000
  mAP @ 50:    0.0000
  mAP @ 75:    0.0000
  mAP scores: [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
  IoU thresh: [0.5  0.55 0.6  0.65 0.7  0.75 0.8  0.85 0.9  0.95]
  AP per class:
    No results
  
Large objects:
  MeanAveragePrecisionResult:
  Metric target: MetricTarget.BOXES
  Class agnostic: False
  mAP @ 50:95: 0.4800
  mAP @ 50:    0.5283
  mAP @ 75:    0.5193
  mAP scores: [0.52829363 0.52765895 0.53252578 0.52467807 0.52340872 0.51931504
   0.51931504 0.49847883 0.42658422 0.20021746]
  IoU thresh: [0.5  0.55 0.6  0.65 0.7  0.75 0.8  0.85 0.9  0.95]
  AP per class:
    0: [0.03619003 0.03619003 0.03619003 0.03619003 0.03619003 0.03619003
   0.03619003 0.03619003 0.03619003 0.02657742]
    1: [0.05798595 0.05798595 0.05798595 0.05798595 0.05798595 0.05798595
   0.05798595 0.05798595 0.05798595 0.0256342 ]
    2: [0.04018435 0.04018435 0.04018435 0.04018435 0.04018435 0.04018435
   0.04018435 0.04018435 0.0289021  0.01678454]
    3: [0.11543189 0.11543189 0.11543189 0.11543189 0.11543189 0.11543189
   0.11543189 0.11543189 0.11543189 0.08168317]
    4: [0.26520509 0.26520509 0.26520509 0.26520509 0.26520509 0.26520509
   0.26520509 0.26520509 0.26520509 0.12376238]
    5: [0.24752475 0.24752475 0.24752475 0.24752475 0.24752475 0.24752475
   0.24752475 0.24752475 0.24752475 0.24752475]
    6: [0.330033 0.330033 0.330033 0.330033 0.330033 0.330033 0.330033 0.330033
   0.330033 0.330033]
    7: [0.56831683 0.56831683 0.56831683 0.56831683 0.56831683 0.56831683
   0.56831683 0.4019802  0.4019802  0.        ]
    8: [0.330033   0.330033   0.330033   0.330033   0.330033   0.330033
   0.330033   0.330033   0.330033   0.08250825]
    9: [0.330033 0.330033 0.330033 0.330033 0.330033 0.330033 0.330033 0.330033
   0.330033 0.330033]
    10: [0.4950495 0.4950495 0.4950495 0.4950495 0.4950495 0.4950495 0.4950495
   0.4950495 0.4950495 0.4950495]
    11: [0.62623762 0.62623762 0.62623762 0.62623762 0.62623762 0.62623762
   0.62623762 0.62623762 0.41831683 0.33663366]
    12: [0.75247525 0.75247525 0.75247525 0.75247525 0.75247525 0.75247525
   0.75247525 0.75247525 0.75247525 0.        ]
    13: [1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
    14: [0.4950495 0.4950495 0.4950495 0.4950495 0.4950495 0.4950495 0.4950495
   0.4950495 0.4950495 0.       ]
    15: [0.57953795 0.57953795 0.57953795 0.57953795 0.57953795 0.57953795
   0.57953795 0.57953795 0.1379538  0.03630363]
    16: [0.44494449 0.41194119 0.33993399 0.33993399 0.27392739 0.27392739
   0.27392739 0.27392739 0.27392739 0.27392739]
    17: [0.37600189 0.37600189 0.37600189 0.37600189 0.37600189 0.37600189
   0.37600189 0.37600189 0.37600189 0.16184311]
    18: [0.33200051 0.33200051 0.33200051 0.33200051 0.33200051 0.33200051
   0.33200051 0.33200051 0.18223141 0.03272827]
    19: [0.34899919 0.34899919 0.34899919 0.34899919 0.34899919 0.34899919
   0.34899919 0.19309074 0.0330033  0.        ]
    20: [0.81188119 0.81188119 0.81188119 0.81188119 0.81188119 0.81188119
   0.81188119 0.81188119 0.5049505  0.04950495]
    21: [0.48797737 0.48797737 0.48797737 0.48797737 0.48797737 0.48797737
   0.48797737 0.48797737 0.48797737 0.04950495]
    22: [0.74257426 0.74257426 0.74257426 0.74257426 0.74257426 0.74257426
   0.74257426 0.74257426 0.5675389  0.        ]
    23: [0.60003143 0.60003143 0.60003143 0.39336791 0.39336791 0.39336791
   0.39336791 0.39336791 0.07150715 0.04950495]
    24: [0.57001414 0.57001414 0.57001414 0.57001414 0.57001414 0.57001414
   0.57001414 0.57001414 0.57001414 0.49080622]
    25: [0.71239981 0.71239981 0.71239981 0.71239981 0.71239981 0.71239981
   0.71239981 0.71239981 0.71239981 0.5709571 ]
    26: [0.60591059 0.60591059 0.60591059 0.60591059 0.60591059 0.60591059
   0.60591059 0.60591059 0.51749175 0.20660066]
    27: [0.59619491 0.59619491 0.59619491 0.39477771 0.39477771 0.39477771
   0.39477771 0.11745292 0.08958039 0.        ]
    28: [0.77062706 0.77062706 0.77062706 0.77062706 0.77062706 0.55775578
   0.55775578 0.55775578 0.55775578 0.36138614]
    29: [0.77310231 0.77310231 0.77310231 0.77310231 0.77310231 0.77310231
   0.77310231 0.77310231 0.77310231 0.14438944]
    30: [0.73443344 0.73443344 0.73443344 0.73443344 0.73443344 0.73443344
   0.73443344 0.73443344 0.73443344 0.16984006]
    31: [0.6039604 0.6039604 0.6039604 0.6039604 0.6039604 0.6039604 0.6039604
   0.6039604 0.6039604 0.0330033]
    32: [0.51980198 0.51980198 0.51980198 0.51980198 0.51980198 0.51980198
   0.51980198 0.51980198 0.51980198 0.24339934]
    33: [0.41136256 0.41136256 0.41136256 0.41136256 0.41136256 0.41136256
   0.41136256 0.41136256 0.25636492 0.24752475]
    34: [0.26440296 0.26440296 0.26440296 0.26440296 0.26440296 0.26440296
   0.26440296 0.11423612 0.11423612 0.04696624]
    35: [0.15470297 0.15470297 0.15470297 0.15470297 0.15470297 0.15470297
   0.15470297 0.15470297 0.15470297 0.04084158]
    36: [0.41254125 0.41254125 0.41254125 0.41254125 0.41254125 0.41254125
   0.41254125 0.41254125 0.41254125 0.12376238]
    37: [1.        1.        1.        1.        1.        1.        1.
   1.        1.        0.4950495]
    38: [1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
    39: [0.7019802  0.7019802  0.7019802  0.7019802  0.7019802  0.7019802
   0.7019802  0.7019802  0.75643564 0.06534653]
    40: [1.         1.         1.         1.         1.         1.
   1.         1.         1.         0.69059406]
    41: [0.17030549 0.17030549 0.17030549 0.17030549 0.17030549 0.17030549
   0.17030549 0.17030549 0.13645595 0.00634679]
    42: [0.75247525 0.75247525 0.75247525 0.75247525 0.75247525 0.75247525
   0.75247525 0.75247525 0.75247525 0.0990099 ]
    43: [0.80693069 0.80693069 0.91584158 0.91584158 0.91584158 0.91584158
   0.91584158 0.91584158 0.08168317 0.        ]
    44: [0.81848185 0.81848185 0.81848185 0.81848185 0.81848185 0.81848185
   0.81848185 0.81848185 0.81848185 0.1320132 ]
    45: [0.25990099 0.25990099 0.47607261 0.47607261 0.47607261 0.47607261
   0.47607261 0.14232673 0.14232673 0.14232673]
    46: [0.87741631 0.87741631 0.87741631 0.87741631 0.87741631 0.87741631
   0.87741631 0.87741631 0.87741631 0.39179632]
    47: [0.75643564 0.75643564 0.75643564 0.75643564 0.75643564 0.75643564
   0.75643564 0.75643564 0.06534653 0.        ]
    48: [0.09806695 0.09806695 0.09806695 0.09806695 0.09806695 0.09806695
   0.09806695 0.09806695 0.09806695 0.00884017]
    49: [0.61103253 0.61103253 0.61103253 0.61103253 0.61103253 0.61103253
   0.61103253 0.61103253 0.61103253 0.61103253]
    50: [0.64686469 0.64686469 0.64686469 0.64686469 0.64686469 0.64686469
   0.64686469 0.64686469 0.45874587 0.33993399]
    51: [0.42822549 0.42822549 0.42822549 0.42822549 0.42822549 0.42822549
   0.42822549 0.42822549 0.42822549 0.        ]

Image

Are you willing to submit a PR?

  • Yes I'd like to help by submitting a PR!
@David-19940718 David-19940718 added the bug Something isn't working label Jan 23, 2025
@David-19940718
Copy link
Author

By the way, I am using LoRA and have frozen the parameters for the vision_tower and multi_modal_projector layers. Could this be affecting the results?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@David-19940718