I am following this link to use Google OAuth for my server side app: https://developers.google.com/identity/sign-in/web/server-side-flow
Everything works fine. But every time when I login, the oauth consent screen pops up and ask me to allow those scopes of permissions. Is there a way to skip that screen? Once a user allowed my app to access some scopes of Google resources, that decision is stored in that user's profile, I think.
Yes, it is stored on the profile. You can check this by going to the Apps with access to your account page, as described here.
I think the problem is that you're not actually storing the access & refresh tokens that you get in step 7. If you have these, you don't need to do the whole OAuth flow again. Since you're not specifying which language you're actually using, I'm going to link to my favorite, which is python. This guide explains the whole OAuth flow from start to finish, and it includes actually storing the credentials for later use.
Even if you don't use python, this should point you to where your problem is...