create tensorflow dataset from list_files


I am trying to create tensroflow dataset :

path_imgs = ('./images/train/*.jpg')
path_masks =('./masks/train/*.jpg'

images =, shuffle=False)
masks =, shuffle=False)

dataset =,

and I am receiving:

Unbatching a tensor is only supported for rank >= 1


Try something like this:

import tensorflow as tf

path_imgs = ('/content/images/*.jpg')
path_masks = ('/content/masks/*.jpg')

images =, shuffle=False)
masks =, shuffle=False)

ds =, masks))

def load_data(image_path, mask_path):
  return tf.image.decode_image(, tf.image.decode_image(

ds =

for x, y in ds:
  print(x.shape, y.shape)
(100, 100, 3) (100, 100, 3)
(100, 100, 3) (100, 100, 3)

Note, however, what the docs state regarding

The file_pattern argument should be a small number of glob patterns. If your filenames have already been globbed, use Dataset.from_tensor_slices(filenames) instead, as re-globbing every filename with list_files may result in poor performance with remote storage systems.

Splitting also works:

train_ds, test_ds = tf.keras.utils.split_dataset(ds, left_size=0.5, right_size=0.5, shuffle=True, seed=123)

Here is the notebook to try it out.

Answered By – AloneTogether

This Answer collected from stackoverflow, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0

Leave a Reply

(*) Required, Your email will not be published