Grant in 1864

Grant showed potential with his success in the west, and with momentum and moral on their side going in to 1864, how did Grant suffer the heavy defeats he did and why was he not replaced as commander of the Army of the Potomac as was the case with past Union generals?