Add basic TensorRT support on Windows #53864
Conversation
Would you please squash the two commits into one? I will approve it so that the merge process can test it.
|
Squashed commits and fixed pylint warning |
|
Seems like Windows build is failing when TRT is disabled, I did not test this scenario. |
|
After we get the servers pass the checking, we will need to get the input from NVIDIA @DEKHTIARJonathan for this. |
|
I fixed build problems and bad formatting |
|
@itmo153277 are you committing this on your personal time or for the account of your employer ? |
| @@ -945,7 +945,7 @@ def is_cuda_compatible(lib, cuda_ver, cudnn_ver): | |||
|
|
|||
| def set_tf_tensorrt_version(environ_cp): | |||
| """Set TF_TENSORRT_VERSION.""" | |||
| if not is_linux(): | |||
There was a problem hiding this comment.
This test can clearly be improved.
if not (is_linux() or is_windows()):
There was a problem hiding this comment.
Fixed
| @@ -12,11 +10,9 @@ py_library( | |||
| name = "compiler", | |||
| srcs = ["__init__.py"], | |||
| srcs_version = "PY3", | |||
| deps = if_windows( | |||
| ["//tensorflow/python/compiler/tensorrt:trt_convert_windows"], | |||
There was a problem hiding this comment.
@bixia1 are we good to remove that file ?
There was a problem hiding this comment.
If the PR works then we don't need it. Right?
|
@DEKHTIARJonathan On my personal time. I understand that full support might take too much resources. I thought just giving it "experimental" status for Windows would be ok even if it means that it might become unstable in the future, There seems to be no major problem if I just forcefully enable it like this, but without this everything is actively preventing even building it for Windows. This PR is just to give an option to build it for Windows anyway if they want to for their own risk, Otherwise, it sounds like TF-TRT just doesn't work on Windows, but it is not true |
| @@ -52,9 +51,6 @@ | |||
| from tensorflow.python.util.lazy_loader import LazyLoader | |||
| from tensorflow.python.util.tf_export import tf_export | |||
|
|
|||
| if platform.system() == "Windows": | |||
| raise RuntimeError("Windows platform is not supported") | |||
There was a problem hiding this comment.
let's replace the Error with the following:
logging.warn(
"Windows support is provided experimentally. No guarantee is made regarding "
"functionality or engineering support. Use at your own risk."
)
There was a problem hiding this comment.
@bixia1 shall we move the warning to C++ ? That way we make sure that the warning is printed regardless of the entry point (C++ or Python) ?
There was a problem hiding this comment.
Let's also make sure the same warning appears in the documentation.
There was a problem hiding this comment.
Since the module is included in standard_ops.py regardless of whether trt is enabled or not, this warning has to be lazy. Should I put it inside _check_trt_version_compatibility instead?
There was a problem hiding this comment.
@DEKHTIARJonathan I added the disclaimer to both python (_check_trt_version_compatibility) and C++ (op init) parts. This should cover most of the use cases. As for the documentation, I am not sure where exactly I should put it.
@bixia1 Should I create a new PR and add it here?
There was a problem hiding this comment.
The only mention of TF-TRT Windows support (or lack thereof) I could find is this: https://github.com/tensorflow/tensorflow/blob/r2.8/tensorflow/python/compiler/tensorrt/trt_convert.py#L910
It is displayed on the tensorflow website here: https://www.tensorflow.org/api_docs/python/tf/experimental/tensorrt/Converter
Should I put the disclaimer there?
There was a problem hiding this comment.
Updated the docs and created a PR here
| @@ -12,11 +10,9 @@ py_library( | |||
| name = "compiler", | |||
| srcs = ["__init__.py"], | |||
| srcs_version = "PY3", | |||
| deps = if_windows( | |||
| ["//tensorflow/python/compiler/tensorrt:trt_convert_windows"], | |||
There was a problem hiding this comment.
If the PR works then we don't need it. Right?
| @@ -52,9 +51,6 @@ | |||
| from tensorflow.python.util.lazy_loader import LazyLoader | |||
| from tensorflow.python.util.tf_export import tf_export | |||
|
|
|||
| if platform.system() == "Windows": | |||
| raise RuntimeError("Windows platform is not supported") | |||
tensorflow/tensorflow.bzl
Outdated
| ] + if_windows([clean_dep("//tensorflow/python:pywrap_tensorflow_import_lib")]) | ||
| ] | ||
|
|
||
| def tf_custom_op_library_additional_deps(): |
There was a problem hiding this comment.
Is there any other place that still use this? I have this question because I wonder why don't you just fix this definition and not adding X_internal.
There was a problem hiding this comment.
I think it is used in tf_custom_op_library macro (user-defined custom ops). There is also a Windows-specific custom logic with tf_custom_op_library_additional_deps_impl related to library exports. I looked into PR that added this, but couldn't fully understand why it was needed. So I was just afraid to break anything and thought that this was the least invasive solution.
There was a problem hiding this comment.
It looks to me that tf_custom_op_library_additional_deps_impl is not used anywhere.
tf_custom_op_library_additional_deps is used by tf_custom_op_library, but shouldn't we replace that with the new tf_custom_op_library_additional_deps_internal also? If not, why not?
There was a problem hiding this comment.
I see.
The sole reason why I changed it for TF-TRT is because it was forming a dependency cycle: TF-TRT -> pywrap_tensorflow_import_lib -> TF-TRT. Currently they are linked together anyway, so I removed pywrap_tensorflow_import_lib from TF-TRT dependencies. But tf_custom_op_library is not connected to TF-TRT so I wanted to leave it untouched to not break anything.
tf_custom_op_library_additional_deps_impl is used here. However, it does not seem to call tf_custom_op_library_additional_deps so I think it will not break if I change tf_custom_op_library_additional_deps.
How about this: instead of creating X_internal, I just move pywrap_tensorflow_import_lib inside tf_custom_op_library? In theory, nothing will break this way, and the circular dependency will be resolved, too.
There was a problem hiding this comment.
Updated with the method I wrote above. I also followed this tutorial and confirmed that everything works as intended
There was a problem hiding this comment.
Thanks!
gbaned
added
kokoro:force-run
ready to pull
and removed
kokoro:force-run
ready to pull
labels
|
This PR causes two tests in https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/delegates/xnnpack to fail: |
@akuegel Is there anywhere I can see the test logs? I can't see how any of the changes in this PR can cause these tests to fail |
Maybe you can run the tests yourself, hopefully it should be possible to reproduce the failure. In any case, here is one example failure: Conv2D.SparseWeights test: As far as I can tell, all tests fail due to precision issues. Maybe this is to be expected? In that case, there needs to be a discussion whether we are ok with increasing the error tolerances of these tests. |
|
This seems unrelated to this PR. I will try to reproduce it on my end and see what I can do. |
Yes, it is possible this is unrelated. Just wanted to let you know what is currently blocking the merge. If it turns out the tests pass again at a later time, the PR will be merged. |
|
@akuegel (cc @bixia1 ) The test cases that were failing are generated randomly (both op parameters and input data). Seeds for random generators are not logged. It is unlikely that I will be able to reproduce it. It is also possible that the failure is environment-dependent. The tests can certainly be improved (e.g. make them deterministic or at least log the seed so that a failure can be reproduced). However, I think this goes beyond the scope of this PR. If you want, I can open an issue for this. @bixia1 What should I do now? Can this PR be merged? |
|
The failure is likely not relevant and I am working on "manually merging" this PR. |
copybara-service
bot
merged commit 1f9f7b4
into
tensorflow:master
For some reason TF-TRT is disabled on Windows. But this feature would be really good to speed up inference (in my case, the difference was 5x). Since TensorRT is available on both Linux and Windows, it is very frustrating that TF-TRT only works in Linux.
This PR enables TRT ops on Windows.
Changes:
The text was updated successfully, but these errors were encountered: