Compile the model online in EP

[wujiangGitHub]

Hello, some of the backends of onnxruntime require online compilation of models, such as openvino. Some CPUs with poor online compilation of large models may take more than 30 minutes. Is there any solution for this? Or are there any requirements for supporting such backends?

[RyanUnderhill]

Is this an onnxruntime specific issue? Or is there a solution with onnxruntime that genai is not making accessible?

[wujiangGitHub]

Thank you for your answer. 
Because cuda and cpu are different, openvino is the backend engine that needs to compile the model online. The online compilation time of large models may be longer. I would like to ask, if jenAI supports EPs such as openvino and rknpu in the future, can the inference performance of large models meet the requirements?

[RyanUnderhill]

Ah yes, we do plan on supporting OpenVINO and compile based EPs. We will ensure the user experience is usable, as a long compilation time would not be usable. It will probably be through a separate optimized model for these providers.

[wujiangGitHub]

Thank you very much for your answer. Are these optimized models compiled and saved offline? Or can the backend provider optimize the model to improve the online compilation time to make it usable?