C++ Copy assignment with same object

Question

In the well-known C++ Primer book I found the following code snippet

Message& Message::operator=(const Message &rhs)
{
    // handle self-assignment by removing pointer before inserting them
    remove_from_Folders(); // updating existing Folders
    contents = rhs.contents; // copy message contents from rhs
    folders = rhs.folders; // copy Folder pointers from rhs
    add_to_Folders(rhs); // add this Message to those Folders
    return *this;
}

The Message class:

class Message {
    friend class Folder;
public: 
    // folders is implicity initialized to the empty set
    explicit Message(const std::string &str = ""):
        contents(str) { }
    // copy control to manage pointers to this Message
    Message(const Message&);           // copy constructor
    Message& operator=(const Message&) // copy assignment
    ~Message();                        // destructor
    // add/remove this Message from the specified Folder's set of messages
    void save(Folder&);
    void remove(Folder&);
private:
    std::string contents; // actual message text
    std::set<Folder*> folders; // Folder's that have this message
    // utility functions used by copy constructor, assignment, and destructor
    // add this Message to the Folders that point to the parameter
    void add_to_Folders(const Message&);
    // remove this Message from every Folder in folders
    void remove_from_Folders();
}

void Message::save(Folder &f)
{
    folders.insert(&f); // add the given Folder to our list of Folders
    f.addMsg(this);     // add this Message to f's set of Messages
}

void Message::remove(Folder &f)
{
    folders.erase(&f); // take the given Folder out of our list of Folders
    f.remMsg(this);    // remove this Message to f's set of Messages
}

// add this Message to Folders that point to m
void Message::add_to_Folders(const Message &m)
{
    for (auto f : m.folders) // for each Folder that holds m
        f->addMsg(this); // add a pointer to this Message to that Folder
}

void Message::remove_from_Folders()
{
    for (auto f : folders) // for each pointer in folders
        f->remMsg(this);   // remove this Message from that Folder
    folders.clear();       // no Folder points to this Message
}

Message::~Message()
{
    remove_from_Folders();
}

Now my question: We see that in remove_from_Folders the whole folder set gets cleared. We also see that the rhs parameter from the copy-assignment operator is a reference. So why does calling remove_from_Folders() in it and afterwards doing folders = rhs.folders; in it work here? Since rhs is a reference, wouldn't rhs.folders now be the empty set when we use the same object (i.e. doing a self-(copy)assignment)? Isn't it that how references work? Or is there a special rule the compiler fulfills when operating with assignment-operators?