Tensorflow 1.12 different results than Tensorflow 2.4

Issue

I’m trying to upgrade my model to tensorflow 2.4 but the network achieves lower accuracy after upgrade. I noticed that loss function for a single batch is different even though:

• I use model = keras.models.load_model('path/to/model.h5') with the same path for both versions (this file was created using tf 1.12)
• I check that weights match
• I check that batch used is the same
• I replicated this problem on both proprietary dataset and keras.datasets.mnist.

I expect that if I manage to achieve the same loss on both versions I will also achieve the same accuracy after training.

Requirements tf 1.12 version

# python version == 3.6
tensorflow_gpu==1.12
keras==2.2.4
h5py==2.10.0
opencv-python==4.2.0.34

Requirements tf 2.4.1

# python version == 3.8
tensorflow==2.4.1
h5py==2.10.0
opencv-python==4.5.3.56

Model definition (this is the same in both versions):

def mobile_net(no_classes):
    base = MobileNetV2(weights='imagenet', include_top=False, input_shape=(224, 224, 3))
    for layer in base.layers:
        layer.trainable = False

    x = GlobalAveragePooling2D()(base.output)
    x = Dense(32, activation='relu')(x)
    x = Dense(128, activation='relu')(x)

    y = GlobalMaxPooling2D()(base.output)
    y = Dense(32, activation='relu')(y)
    y = Dense(128, activation='relu')(y)

    conc = Add()([x, y])
    conc = Dense(32, activation='relu')(conc)

    prediction = Dense(no_classes, activation='softmax')(conc)
    model = Model(inputs=base.input, outputs=prediction)
    optimizer = Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False)
    model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy'])

    return model

train method (almost the same in both versions):

keras.backend.set_image_dim_ordering('tf')  # only in tf 1.12

# load data
x_train, y_train = ...
x_train, y_train = x_train[:4], y_train[:4]  # select just one batch for testing purposes


model = keras.models.load_model('path/to/model.h5')  # in tf 1.12
model = tensorflow.keras.models.load_model('path/to/model.h5')  # in tf 2.4
print(f'check that the values are the same: {x_train.sum() + y_train.argmax(axis=1).sum()}')
weights = model.get_weights()
print(f'check that weights are the same: {[weight.sum() for weight in weights]}')
model.fit(x_train, y_train, batch_size=4, verbose=2)

tf 1.12 output:

check that the values are the same: 18266047
check that weights are the same: [-4.311309, 37.386337, 26.299068, …, -10.376889, 0.0, -13.127711, 0.0, 4.9316425, 0.0]
Epoch 1/1

• 18s – loss: 2.6805 – acc: 0.2500

tf 2.4 output:

check that the values are the same: 18266047
check that weights are the same: [-4.311309, 37.386337, 26.299068, …, -10.376889, 0.0, -13.127711, 0.0, 4.9316425, 0.0]
1/1 – 6s – loss: 2.8985 – accuracy: 0.2500

Where does this difference in loss come from?

Solution

This difference comes from the fact that MobileNet contains BatchNormalization layers. Their behaviour changed in Tensorflow 2.x. You can read more here. To recreate Tensorflow 1.x behaviour of BatchNormalization layers I added the following fragment to the model creation code.

def mobile_net(no_classes):
    base = MobileNetV2(weights='imagenet', include_top=False, input_shape=(224, 224, 3))
    ### changed in Tensorflow 2.4.1
    for layer in base.layers:
        if layer.__class__ == BatchNormalization:
            layer.trainable = True
        else:
            layer.trainable = False
    ### end of change

    x = GlobalAveragePooling2D()(base.output)
    x = Dense(32, activation='relu')(x)
    x = Dense(128, activation='relu')(x)

    y = GlobalMaxPooling2D()(base.output)
    y = Dense(32, activation='relu')(y)
    y = Dense(128, activation='relu')(y)

    conc = Add()([x, y])
    conc = Dense(32, activation='relu')(conc)

    prediction = Dense(no_classes, activation='softmax')(conc)
    model = Model(inputs=base.input, outputs=prediction)
    optimizer = Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False)
    model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy'])

    return model

The model now returns the same loss both on tf 1.12 and 2.4.1.

Answered By – YuseqYaseq

This Answer collected from stackoverflow, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0