Different 2D Convolution results between PyTorch and Keras

Question

I am trying to find the equivalent keras representation of the following PyTorch line:

conv1 = nn.Conv2d(int(filters*s), filters, kernel_size=(3, 1), stride=int(1/s), padding=(1, 0))

Here is the full code

import torch
from torch import nn

import torch
import tensorflow as tf
import numpy as np

from keras import initializers
from tensorflow.keras import layers

# constants between both libraries
filters = 2
s = 1

# Set the seed for reproducible set of numbers between keras and pytorch
torch.manual_seed(0)
np.random.seed(0)
tf.random.set_seed(0)

# Create a tensor in PyTorch of shape (128, 2, 1024, 1)
pytorch_tensor = torch.rand((128, 2, 1024, 1))

# Convert the PyTorch tensor to a NumPy array
numpy_array = pytorch_tensor.numpy()

# Reshape the NumPy array to the desired shape for Keras (128, 1024, 1, 2)
numpy_array = np.transpose(numpy_array, (0, 2, 3, 1))

# Create a tensor in Keras (TensorFlow) from the NumPy array
keras_tensor = tf.constant(numpy_array)

# PyTorch zeropad and convolution 
pytorch_conv1 = nn.Conv2d(int(filters*s), filters, kernel_size=(3, 1), stride=int(1/s), padding=(1, 0))(pytorch_tensor)

print(f"Pytorch values: {pytorch_conv1}")

# Keras zeropad and convolution 
keras_zeropad = layers.ZeroPadding2D(padding=((0,0),(1,0)))(keras_tensor)
keras_conv1 = layers.Conv2D(filters, kernel_size=(3, 1), strides=int(1/s), padding='valid', data_format='channels_last')(keras_zeropad)

print(f"Keras tensor values: {keras_conv1}")

I thought the issue was padding since PyTorch and Keras handle it differently. Namely, inside Keras’ Conv2D layer the padding can only be set to valid or same. Instead I am using the ZeroPadding2D layer before the Conv2D so asymmetric padding can be done in Keras. However, despite this I still cannot get the output data to match.

I wrote the following code which generates the same input tensor for PyTorch and Keras so I know the data is the same. I’ve also tried initializing the weights and bias's the same way but the data still doesn’t match.

I’ve seen posts about using the PyTorch weights in Keras to achieve the same results but I would like to avoid that for my application.

Update: Changing the Keras Conv2D layer to the following still produces vastly different results compared to PyTorch. The padding and shape are equal, however Keras produces new data every time. I’ve looked into kernel initializers but nothing has produced data that matches PyTorch.

keras_zeropad = layers.ZeroPadding2D(padding=(1,0))(keras_tensor)
keras_conv1 = layers.Conv2D(filters, kernel_size=(3, 1), strides=int(1/s), padding='valid', data_format='channels_last')(keras_zeropad)

print(f"Keras tensor values: {keras_conv1}")
print(f"Keras tensor shape: {keras_conv1.shape}")

For readability the input tensor shape was changed to (1,2,4,1). Here is the PyTorch and Keras data that should match.

Pytorch values: tensor([[[[-0.1842],
          [-0.2021],
          [-0.3707],
          [-0.2296]],

         [[ 0.2727],
          [ 0.0941],
          [ 0.0927],
          [ 0.1981]]]], grad_fn=<ConvolutionBackward0>)
PyTorch tensor shape: torch.Size([1, 2, 4, 1])
Keras tensor values: [[[[-0.57757664  0.40883008]]

  [[ 0.21837917  0.6590299 ]]

  [[ 0.28047863  0.3622663 ]]

  [[ 0.2800742   0.2882987 ]]]]
Keras tensor shape: (1, 4, 1, 2)

Answer 1

If you have a horizontal kernel size of 3 and want the same behavior as padding='same', the padding set manually must be symmetrical and equal to 1 on both ends:

pad = layers.ZeroPadding2D(padding=(1,0))

conv = layers.Conv2D(filters, 
                     kernel_size=(3, 1), 
                     strides=int(1/s), 
                     padding='valid', 
                     data_format='channels_last')

Then conv(pad(keras_tensor)) will have a shape of (128, 1024, 1, 2).