There's a contrast between an underlying fear of brainwashing in modern society, as seen in films and literature, and the apparent belief of many citizens who sit on juries that brainwashing is hogwash, leading to criminal convictions for people like Patty Hearst and Lee Malvo who based their defense on claims of brainwashing.
People seem to distinguish between brainwashing now and brainwashing in the future - it's the latter that's apparently more fearsome. The future of brainwashing, if Hollywood and conspiracy theorists are to be believed, involves more high-tech approaches that are a lot scarier than verbal or physical "assaults on identity." Combined with hypnosis techniques, a futuristic brain implant might be all that's needed to control a human's thoughts, actions and beliefs. But most scientists agree that the field of neurology is nowhere close to that level of understanding of the human brain.
Additionally, many psychologists believe large-scale brainwashing -- for instance, by way of the mass media and subliminal messages -- is impossible, because the thought-reform process requires isolation and absolute dependence of the subject. It's just not that easy to change a person's core personality and belief system.