I am trying to run the inference on my Nvidia GPU. I have installed Microsoft.ML.OnnxRuntime.Gpu version 1.12.0 in Visual Studio 2019 using NuGet package manager.
Cuda Version 11.3.
I get exception: Exception thrown at 0x00007FFF94A23A48 (cudnn_cnn_infer64_8.dll) session.Run(runOptions, inputNames.data(), &inputTensor, 1, outputNames.data(), &outputTensor, 1);
When I run the inference on CPU it works fine.
Ort::SessionOptions ort_session_options; OrtCUDAProviderOptions options; options.device_id = 0; OrtSessionOptionsAppendExecutionProvider_CUDA(ort_session_options, options.device_id); . . . . session.Run(runOptions, inputNames.data(), &inputTensor, 1, outputNames.data(), &outputTensor, 1);
I am using:
ONNX Runtime Version 1.12
ONNX Runtime API C++
Architecture X64
Execution Provider CUDA
Execution Provider Library Version CUDA 11.3
I have tried with different versions of Microsoft.ML.OnnxRuntime.Gpu version and was not able to run the inference on GPU.
Related questions 2142 How do I profile C++ code running on Linux? 0 Why do my ONNXRuntime Inference crash on GPU without any log? 1878 When to use virtual destructors? Related questions 2142 How do I profile C++ code running on Linux? 0 Why do my ONNXRuntime Inference crash on GPU without any log? 1878 When to use virtual destructors? 5 onnxruntime inference is way slower than pytorch on GPU 0 ONNX converted TensorFlow saved model runs on CPU but not on GPU 3 RuntimeError: CUDA error: no kernel image is available for execution on the device (rastervision) 1 Trying to use onnxruntime with GPU. SessionOptionsAppendExecutionProvider_CUDA gives error Load 4 more related questions Show fewer related questions Reset to default
