How to disable tf.data's autotuning?

Issue

I am currently running a small model originated from here. I want to disable tf.data‘s autotuning.

The error message comes like this:

2022-09-14 02:57:26.254717: I tensorflow/core/framework/model.cc:1439] Starting optimization of tunable parameters with GradientDescent
2022-09-14 02:57:26.254980: I tensorflow/core/framework/model.cc:1494] Number of tunable parameters: 0
2022-09-14 02:57:26.255059: I tensorflow/core/kernels/data/model_dataset_op.cc:200] Waiting for 20480 ms.
2022-09-14 02:57:46.735341: I tensorflow/core/framework/model.cc:1439] Starting optimization of tunable parameters with GradientDescent
2022-09-14 02:57:46.735523: I tensorflow/core/framework/model.cc:1494] Number of tunable parameters: 0
2022-09-14 02:57:46.735562: I tensorflow/core/kernels/data/model_dataset_op.cc:200] Waiting for 40960 ms.
2022-09-14 02:58:27.695827: I tensorflow/core/framework/model.cc:1439] Starting optimization of tunable parameters with GradientDescent
2022-09-14 02:58:27.696006: I tensorflow/core/framework/model.cc:1494] Number of tunable parameters: 0
2022-09-14 02:58:27.696046: I tensorflow/core/kernels/data/model_dataset_op.cc:200] Waiting for 60000 ms.
2022-09-14 02:59:27.696275: I tensorflow/core/framework/model.cc:1439] Starting optimization of tunable parameters with GradientDescent
2022-09-14 02:59:27.696469: I tensorflow/core/framework/model.cc:1494] Number of tunable parameters: 0
2022-09-14 02:59:27.696507: I tensorflow/core/kernels/data/model_dataset_op.cc:200] Waiting for 60000 ms.
2022-09-14 03:00:27.696740: I tensorflow/core/framework/model.cc:1439] Starting optimization of tunable parameters with GradientDescent
2022-09-14 03:00:27.696911: I tensorflow/core/framework/model.cc:1494] Number of tunable parameters: 0
2022-09-14 03:00:27.696951: I tensorflow/core/kernels/data/model_dataset_op.cc:200] Waiting for 60000 ms.
2022-09-14 03:01:27.697178: I tensorflow/core/framework/model.cc:1439] Starting optimization of tunable parameters with GradientDescent
2022-09-14 03:01:27.697368: I tensorflow/core/framework/model.cc:1494] Number of tunable parameters: 0
2022-09-14 03:01:27.697407: I tensorflow/core/kernels/data/model_dataset_op.cc:200] Waiting for 60000 ms.
2022-09-14 03:02:27.697638: I tensorflow/core/framework/model.cc:1439] Starting optimization of tunable parameters with GradientDescent
2022-09-14 03:02:27.697811: I tensorflow/core/framework/model.cc:1494] Number of tunable parameters: 0
2022-09-14 03:02:27.697850: I tensorflow/core/kernels/data/model_dataset_op.cc:200] Waiting for 60000 ms.
2022-09-14 03:03:27.698078: I tensorflow/core/framework/model.cc:1439] Starting optimization of tunable parameters with GradientDescent
2022-09-14 03:03:27.698250: I tensorflow/core/framework/model.cc:1494] Number of tunable parameters: 0
2022-09-14 03:03:27.698309: I tensorflow/core/kernels/data/model_dataset_op.cc:200] Waiting for 60000 ms.
2022-09-14 03:04:27.698543: I tensorflow/core/framework/model.cc:1439] Starting optimization of tunable parameters with GradientDescent
2022-09-14 03:04:27.698724: I tensorflow/core/framework/model.cc:1494] Number of tunable parameters: 0
2022-09-14 03:04:27.698762: I tensorflow/core/kernels/data/model_dataset_op.cc:200] Waiting for 60000 ms.
2022-09-14 03:05:27.698990: I tensorflow/core/framework/model.cc:1439] Starting optimization of tunable parameters with GradientDescent
2022-09-14 03:05:27.699164: I tensorflow/core/framework/model.cc:1494] Number of tunable parameters: 0
2022-09-14 03:05:27.699202: I tensorflow/core/kernels/data/model_dataset_op.cc:200] Waiting for 60000 ms.
2022-09-14 03:06:27.699453: I tensorflow/core/framework/model.cc:1439] Starting optimization of tunable parameters with GradientDescent
2022-09-14 03:06:27.699627: I tensorflow/core/framework/model.cc:1494] Number of tunable parameters: 0
2022-09-14 03:06:27.699668: I tensorflow/core/kernels/data/model_dataset_op.cc:200] Waiting for 60000 ms.
2022-09-14 03:07:27.699893: I tensorflow/core/framework/model.cc:1439] Starting optimization of tunable parameters with GradientDescent
2022-09-14 03:07:27.700112: I tensorflow/core/framework/model.cc:1494] Number of tunable parameters: 0
2022-09-14 03:07:27.700175: I tensorflow/core/kernels/data/model_dataset_op.cc:200] Waiting for 60000 ms.

The execution is stuck in there, and I am suspecting tf.data‘s autotuning is not going well.

Is there any way to prevent autotuning?

I am using TensorFlow v2.4 with RTX 2080Ti GPU.

Solution

Adding the following code snippet to the dataset would disable autotuning.

autotune_options.autotune = False
options = tf.data.Options()
options.experimental_optimization = autotune_options
train_dataset = train_dataset.with_options(options)

By the way, the stuck problem was not the autotuning.

Answered By – Jueon Park

This Answer collected from stackoverflow, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0

Leave a Reply

(*) Required, Your email will not be published