Error writing TFRecords, Networks reads double the values (Input to reshape is a tensor with n*2 values, but the requested shape has n values)

Issue

I’m writing this question as a reminder for myself cause i already know that i will recreate this error again and i don’t want to spend again half an hour fixing it.

I’m currently working on a machine learning project, and i encountered an error during the execution of the network:
when i execute the neural network after writing the Tfrecords like this

def write_to_tfrec_spatial(training_directories, path, filename):
  record_file = filename
  n_samples = len(training_directories)
  print()
  print(n_samples)
  with tf.io.TFRecordWriter(record_file) as writer:

    print("writing", end=": ")
    for i in range(n_samples):
      if(i % 50) == 0:
        print()
      print(i, end=",")

      dir = path + training_directories[i]

      loaded = np.load(dir)
      ground = loaded["rad"]

      if normalization:
        ground = ground / max_norm_value
        print(np.amax(ground), end=",")

      padded_ground = np.pad(ground, [(3, 2), (0, 0)], mode='constant')
      inputs = data_augmentation(padded_ground)

      for input in inputs:
        tf_example = image_example_spatial(input=input, ground=padded_ground)
        writer.write(tf_example.SerializeToString())
  return record_file

I then executed the network like this:

model.fit(training_dataset, steps_per_epoch=steps, epochs=60, validation_data=validation_dataset, callbacks=my_callbacks)

But i get the following error:

2 root error(s) found.
  (0) INVALID_ARGUMENT:  Input to reshape is a tensor with 376832 values, but the requested shape has 188416
     [[{{node Reshape}}]]
     [[IteratorGetNext]]
     [[IteratorGetNext/_428]]
  (1) INVALID_ARGUMENT:  Input to reshape is a tensor with 376832 values, but the requested shape has 188416
     [[{{node Reshape}}]]
     [[IteratorGetNext]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_165085]

I can’t understand why i get exactly double the values, i checked multiple times the shapes and they were always correct, but the TFRecord keeps returning the wrong number of values

Solution

The error here was that by manipolating the image i indirectly changed the dtype of the numpy array from np.float32 to np.float64.

By doing this i was writing tf.float64 tensors and then reading them with this:

input_raw = tf.io.decode_raw(new_record['input'], out_type=tf.float32, little_endian=True, fixed_length=None, name=None)

and trying to decode them as tf.float32

To fix the issue i had to cast the numpy arrays to np.float32 again after the manipulation like this:

rad = np.float32(rad)

Answered By – BestDogeStackoverflow

This Answer collected from stackoverflow, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0

Leave a Reply

(*) Required, Your email will not be published