Data cardinality is ambiguous: Make sure all arrays contain the same number of samples Tensorflow

Issue

So I have been working on an age estimation model.

I have 2 axes, x_train and y_train.

x_train has images, and y_train has labels(age as integers)

On x_train, the shape of a single image array is (200, 200, 3), while the shape of the whole x_train is (10000, 200, 200, 3).

The shape of y_train is (10000,)

=================================================

Currently y_train looks like this:
[24 47 35 ... 75 3 33]

and x_train like this(when I print first image with x_train[0]:

[[[0.40392157 0.29411766 0.2784314 ]
  [0.4117647  0.3019608  0.28627452]
  [0.41568628 0.30588236 0.29411766]
  ...
  [0.8745098  0.5882353  0.6156863 ]
  [0.88235295 0.59607846 0.62352943]
  [0.8862745  0.6        0.627451  ]]

 [[0.41568628 0.30588236 0.2901961 ]
  [0.41960785 0.30980393 0.29411766]
  [0.41960785 0.30980393 0.29803923]
  ...
  [0.88235295 0.59607846 0.62352943]
  [0.8862745  0.6        0.627451  ]
  [0.8901961  0.6039216  0.6313726 ]]

 [[0.42745098 0.31764707 0.3019608 ]
  [0.42352942 0.3137255  0.3019608 ]
  [0.41568628 0.30588236 0.29411766]
  ...
  [0.8862745  0.6        0.627451  ]
  [0.8901961  0.6039216  0.6313726 ]
  [0.89411765 0.60784316 0.63529414]]

 ...

 [[0.92941177 0.80784315 0.7254902 ]
  [0.92941177 0.80784315 0.7254902 ]
  [0.9254902  0.8039216  0.72156864]
  ...
  [0.9098039  0.7058824  0.72156864]
  [0.9098039  0.7058824  0.72156864]
  [0.9098039  0.7058824  0.72156864]]

 [[0.93333334 0.8117647  0.7294118 ]
  [0.93333334 0.8117647  0.7294118 ]
  [0.92941177 0.80784315 0.7254902 ]
  ...
  [0.90588236 0.70980394 0.72156864]
  [0.90588236 0.70980394 0.72156864]
  [0.90588236 0.70980394 0.72156864]]]

This was my actual code for generating x_train and y_train:

head = '/content/crop_part1/' #data dir
l = os.listdir(head)

x_train = [] # images
y_train = [] # age


for i in l[:10000]:
    img= np.array(Image.open(f'{head}{i}'))
    img= np.resize(img,(img_height, img_width,3))
    img = img.astype('float32')
    img /= 255  
    x_train.append(img)

    y_train.append(int(i.split('_')[0]))

When I fit my model, I get this error:

Data cardinality is ambiguous:
  x sizes: 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200,  200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 2...
  y sizes: 10000
Make sure all arrays contain the same number of samples.

I assume that the shape of y_train should be like x_train. Previously I was using the nice tensorflow readymade input pipeline features, like padding. But how do I do it here?

Do we have to pad y_train with zeros, something like this:

[16, 0, 0]
[4, 0, 0]
[12, 0, 0]

Or do we have to reshape it? If yes then how?

Solution

model.fit makes different assumptions based on what data type it receives:

• If the data received is a list containing python types, such as a float or int, then the data is promoted to a Tensor.
• If the data is a list of tensors (or numpy’s array), then fit assumes that there is a 1 to 1 mapping between each tensor and each input of the model. It makes life easier when trying to train a model with multiple inputs.

In your case, x is a list of numpy’s array, while y is a list of integer.
fit is confused, because x seems to map to model with 10000 different inputs, where each inputs expect a 200 samples, while y maps to the output of the model that would expect 10000 samples, hence the ambiguous cardinality.

One possible fix is to convert your list of numpy 3D arrays to a one 4D array. You can use np.stack for that:

x_train = np.stack(x_train, axis=0)  

Then, the first dimension of your input and output is equal to 10000, and the cardinality is not ambiguous anymore.

Answered By – Lescurel

This Answer collected from stackoverflow, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0