Can you quantize the IBM Granite 3.0 8B Instruct Accelerator model?

[PhilipAD]

Hey all! I’ve been playing around with the new IBM Granite 3.0 8B Instruct model and its accelerator, and I’m loving the speed boost it provides.

I’m curious if anyone has tried quantizing the accelerator model (ibm-granite/granite-3.0-8b-instruct-accelerator) to further reduce its size and potentially speed things up even more. I'm currently using an AWQ version of ibm-granite/granite-3.1-8b-instruct.

I know quantization works well for the main model, but I’m not sure if it’s possible or advisable for the accelerator. Has anyone given this a shot? If so, what methods did you use, and how did it affect performance? I’d love to hear about your experiences or any tips you might have!

Also, if quantizing the accelerator isn’t recommended, I’d be interested in understanding why. Thanks in advance!