How to have a webserver store files as securely as possible?

Question

I want to build a webserver which can be used to store files securely in such a way that even if an attacker gains access to the webserver, they cannot view the decrypted files.

So far, I have devised the following approach:

1. Upon registration of users, asymmetric keys are generated where the private key can only be used with a passphrase consisting of [user password] + [random salt]. The random salt is stored in plain text in the server. Of course, the user passwords need to be sufficiently long.
2. When a user uploads a file, a new symmetric encryption key is generated to encrypt the file. The encryption key is then encrypted itself using the user's public key.
3. When a user downloads a file, they need to provide their password, which is used to decrypt the symmetric encryption key using their private key. This decrypted symmetric key can now decrypt the file.
4. When a user wants to share a file with another user, they need to provide their password to decrypt the symmetric encryption key. The symmetric encryption key is then encrypted using the public key of the other user.

This way, even if an attacker gains access to the webserver, there is no way that they are capable of obtaining the decrypted files. However, the disadvantage is that when a user shares a file with a large group of users, the server needs to encrypt the symmetric key many times for each of the users that have access. The same holds for giving a single user access to many files.

Is there any other way to improve on this? Or is this system not even secure at all? I was thinking of using a key management system, but it is yet unclear to me what the added value would be.

Side note: I have already considered the case of a user losing their password. A counter measure could be to always encrypt the symmetric key using the public key of a super user, such that the super user will have the ability to give other users access. This of course introduces a weak spot, which is determined by the password strength of said super user. Another counter measure is that users always appear in groups (e.g. the organization they are registered with) and then users will be able to restore each other's access.