Refer to figure.
When the critical angle of attack is exceeded on a rotor blade, the boundary layer on the upper side of the blade can begin to separate. Here’s a brief explanation:

Angle of Attack (AoA): The angle between the chord line of a rotor blade and the direction of the relative airflow is called the angle of attack.

Critical Angle of Attack: This is the maximum angle of attack at which the rotor blade can operate efficiently. Beyond this point, the airflow over the blade can no longer smoothly follow the contour of the upper surface.

Boundary Layer Separation: When the angle of attack exceeds this critical value, the smooth airflow (boundary layer) over the blade starts to separate from the blade's surface. This separation disrupts the lift generated by the blade and leads to a significant drop in aerodynamic efficiency, often referred to as "stall."

In a helicopter rotor, if the angle of attack exceeds the critical value, this can lead to uneven lift distribution, vibrations, and potentially loss of control if not managed properly.