Maximum size to inverse a matrix?

Question

I want to inverse a 401x401 (upper triangular square) matrix in python but run into issues when the matrix size exceeds 153x153. For all sizes below this, everything works like a charm but then the product of the inverted matrix by the initial matrix is not the identity matrix anymore.

import numpy as np
import math
from scipy import special

N=1
check = np.identity(N)

def calculate_matrix(N):
    matrix = np.zeros( (N,N) ) 
    matrix[0]=[1/(2**(j))*special.binom(j,j/2) if j%2==0 else 0 for j in range(N)]
    for p in range(0,math.floor((N-2)/2)+1):
        i=2*p+1
        matrix[i]=[(-1)**(2*q-p)/(4**q)*special.binom(2*q+1,(q-p)) if (q-i>=0 and q%2==1) else 0 for q in range(N)]
    for p in range(1,math.floor((N-1)/2)+1):
        i=2*p
        matrix[i]=[(-1)**(2*q-p)/(2**(2*q-1))*special.binom(2*q,(q-p)) if (q-i>=0 and q%2==0) else 0 for q in range(N)]
    return matrix

while (np.allclose(check, np.identity(N))==True and N<=402):
    N=N+1
    matrix = calculate_matrix(N)
    inverse_matrix = np.linalg.inv(matrix)
    check=np.matmul(inverse_matrix, matrix)

print("Maximum size of the matrix that can be inverted:", N-1)

Any idea what goes wrong?

Addition: it appears (thx user@2357112) that the matrix is ill-defined, hence the issue. A workaround if it exists would be much appreciated as it appears that partial pivoting is not a solution here.