self.calib_counter returning None type in modules/bmm.py

[lorsonblair]

In forward(self, m1, m2) (line 183), if self.calib_counter > 0: (line 195) throws an error (cannot compare None type to int) as self.calib_counter is somehow type None. self.calib_counter is of int type and has the value of qmodel_calibration_new after declaration and assignment on line 113 using self.register_buffer("calib_counter", ...). In the previous version (downloaded and installed Jan 2025), that line was: if self.calib_counter: ... , which I reverted to to avoid the error.

Any idea why that may be causing the error? I'm running pytorch 2.4.1 with Cuda on aarch64, python=3.10. Thanks for your response. Let me know if you need additional details.

[chichun-charlie-liu]

Thanks for identifying the issue.

Long story short, directly check if self.counter: will cause complaints from PyTorch starting from a certain version, because converting a tensor to bool is not ideal and may cause confusion. To address this issue, we simply use if tensor>0: in place of if tensor:.
At the same time, a minor change was made correspondingly. Previously we will delete this counter (together with the calibration quantizers) at the end of calibration process, which makes self.calib_counter None, hence, the error you observed. The correct behavior, as in Linear.py's forward, would simply be NOT to delete the counter. Unfortunately, it seems like we may have missed to update this line in bmm.py.

Please simply change self.quantize_calib_m1 = self.quantize_calib_m2 = self.calib_counter = None to self.quantize_calib_m1 = self.quantize_calib_m2 =None on your local code. We will open a PR to fix this bug shortly.

[lorsonblair]

I made the change and it worked. Thank you.