The IV used in schemes such as CBC has to be random and unpredictable. But at the same time it does not have to be kept secret.
If the IV does not have to be secret, why does it have to be random then? I fail to make sense out of these seemingly contradicting requirements.
I have seen descriptions of attacks which exploit the non-randomness. So, I would understand why randomness is needed. However, things get confusing when the requirements specify that the IV does not have to be secret !This seems to defeat the whole purpose of randomness.
Somebody help clarify this please.
I think you are reversing the roles.
When a cryptographic protocol is designed, it is designed with certain assumptions in mind. The more assumptions you use, the less useful the protocol is, as you are less likely to find scenarios in which the assumptions hold.
In the case of CBC, the IV was designed to not need to be secret. You can keep it a secret, if you like. The algorithm is definitely not less secure this way. It is not, however, a requirement.
Having a non-random IV, on the other hand, causes the entire protocol to be unsuitable for certain applications. When choosing between adding a requirement to the protocol and adding a requirement to its data, the right choice is obvious.
In other words, the IV does not need to be secret, merely because it can be non-secret.