SpriteKit Inelastic Collision Reducing Velocity

Question

I'm building a pong/breaker game with a ball and non-static blocks. I'd like the ball to never stop moving, but whenever it hits a block it loses velocity.

I have the restitusion = 1 for all sprites involved, I've tried setting the mass equal to each other and the density and the friction = 0. But, the ball still loses velocity on a bounce.

When the ball hits a block I'm removing it in the didBegin(contact:) function. I've also tried delaying the removal and it didn't help.

I'd like for the ball to have a constant velocity, but still be able to interact with the blocks as later I'd like to add blocks that can be hit without immediately being broken. So, the blocks can't be static but the ball needs to have a constant velocity.

My code for creating the ball node:

    func ballNode(_ position: CGPoint?) -> SKSpriteNode {
        let node = SKSpriteNode()
        node.position = position == nil ? CGPoint(x: size.width/2, y: 100) : position!
        node.size = CGSize(width: 17, height: 17)

        //background
        let background = SKShapeNode(circleOfRadius: 8.5)
        background.fillColor = UIColor.white

        node.addChild(background)

        //physics
        node.physicsBody = SKPhysicsBody(circleOfRadius: 8.5)
        node.physicsBody?.allowsRotation = true
        node.physicsBody?.friction = 0
        node.physicsBody?.restitution = 1
        node.physicsBody?.linearDamping = 0
        node.physicsBody?.angularDamping = 0

        node.physicsBody?.categoryBitMask = BallCategory
        node.physicsBody?.contactTestBitMask = AddBlockBorderCategory | PaddleCategory
        node.physicsBody?.collisionBitMask = PaddleCategory | BlockCategory | BorderCategory

        return node
    }

My code for creating the block node:

    func createBlockNode() -> BlockNode {
        let width = (size.width-CGFloat(6*layout[0].count))/CGFloat(layout[0].count)
        let height = width*0.5
        let nodeSize = CGSize(width: width, height: height)
        let node = BlockNode(texture: nil, size: nodeSize)

        let background = SKShapeNode(rectOf: nodeSize)
        background.fillColor = .darkGray
        background.strokeColor = .lightGray

        //physics
        node.physicsBody = SKPhysicsBody(rectangleOf: nodeSize)
        node.physicsBody?.restitution = 1
        node.physicsBody?.allowsRotation = true
        node.physicsBody?.friction = 0
        node.physicsBody?.categoryBitMask = BlockCategory
        node.physicsBody?.contactTestBitMask = BallCategory

        node.addChild(background)
        return node
    }

And a screen recording: screen recording of the ball losing velocity

I'm starting the ball using this:

ball!.physicsBody?.applyForce(CGVector(dx: 0, dy: 50))