I have a requirement to authenticate ASP.NET Core App via Google Sign In, but a Web API between client app (i.e. ASP.NET Core app) and Google sign in... I know it sounds confusing, so let me explain it with diagrams.
Here is the typical way to include google sign-in button and get user authenticated via their google credentials, It works perfectly fine
Step 1: First, create a new app on google identity developer portal, generate ClientId, ClientSecret and specify redirect_url like : https://yoursite.com/signin-google.
Step 2: In the Startup.cs class of ASP.NET Core project, Use AddGoogle as authentication middleware and it works perfectly fine.
Including diagram below for your understanding:

And here is the proposed flow. With a REST API in between client and google sign in. Question is, how do I authenticate client ?

From what I understand, you want your Client Application to invoke the REST Service on behalf of the user. The REST service needs assurance that both the Client and the User are authenticated.
We have achieved this using the OpenID Connect Hybrid flow. The bad news is that we had to add another service to the solution to achieve this.
How this differs from your proposed solution is this:
We happened to use IdentityServer4 when implementing the Identity Server, but the protocol exists independently of any one implementation.