Repository navigation
(Ask) why InferInheritedType handle int8 to fp16 out? #844
Description
As you described in the docs,
InferInheritedType(info); // All Inputs inherit the type of their previous node’s outputs and sets outputs to the data type of the first input.
But in the code, it seems that you handle int8 for special case. Can you tell me why you do this?
https://github.com/openppl-public/ppl.nn/blob/252e7f27eec3976a3be48bb21f15c660cddec6af/src/ppl/nn/engines/cuda/optimizer/opt_kernel.h#L264
Activity
Si-XU commentedon Jul 20, 2023
Int8 type should be used combining with quant scale.
We strictly use function of UnifyToOutputQuant and CopyQuantType to deal with int8 inputs.
ppl.nn/src/ppl/nn/engines/cuda/optimizer/opt_kernel.h
This statement is just in case some operations forget to handle int8 inputs.
DeepTecher commentedon Jul 20, 2023
Okay. But why do we handle int8 input to fp16 input in the case that some operations forget to handle int8 inputs?
However, perhaps some ops may be not quant op, but support int8 input and int8 output. Like on Min/Max operation
Si-XU commentedon Jul 21, 2023
Yes, some ops support int8 input and int8 output directlly.
However, we suggest the user to specify which ops use int8 precision via "quant.json". The rest of the operators will not save quant information and use float16 by default.