McClellan

How come General McClellan wasn't relieved sooner than he was? He failed to act on numerous occasions when he should of attacked the Rebels. Also many believed him to be a Confederate sympathizer. Why wasn't he replaced sooner?