[OpenVINO]: Updated documentation about weight compression #529
Conversation
|
@ljaljushkin, please take a look as well. |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Co-authored-by: Lyalyushkin Nikolay <nikolay.lyalyushkin@intel.com>
Co-authored-by: Lyalyushkin Nikolay <nikolay.lyalyushkin@intel.com>
Co-authored-by: Lyalyushkin Nikolay <nikolay.lyalyushkin@intel.com>
Co-authored-by: Lyalyushkin Nikolay <nikolay.lyalyushkin@intel.com>
Co-authored-by: Ella Charlaix <80481427+echarlaix@users.noreply.github.com>
Co-authored-by: Ella Charlaix <80481427+echarlaix@users.noreply.github.com>
|
|
||
| > **NOTE:** `load_in_8bit` is enabled by default for models larger than 1 billion parameters. | ||
|
|
||
| For the 4-bit weight quantization we recommend using the NNCF API like below: |
There was a problem hiding this comment.
I would much recommend optimum-cli over NNCF API for this. It's such a quick and easy method. And (unless that has been fixed very recently) NNCF fails on SPR/EMR with a BF16 error and it's not easy to know how to work around that.
There was a problem hiding this comment.
I guess we fixed this issue in the recent version of NNCF. @alexsu52, please confirm.
There was a problem hiding this comment.
Yes, you are right. Models with float16 and float32 weigths work on SPR/EMR.
|
I think we can merge this. @echarlaix |
No description provided.