Both George Washington, first President of the United States, and Thomas Jefferson, author of the US Constitution and third President of the United States, recognised the dilemma of slavery.
“Washington conceded the system of human bondage that underpinned the economy of 18th Century Virginia was a ‘wicked, cruel and unnatural trade’.” And Jefferson “A bon vivant who lived in luxury at a palatial Virginia estate, Jefferson knew America’s original sin was a ‘depravity’, as he described it.” 
Yet both were slave owners. The article in the BBC Should Washington and Jefferson monuments come down? enumerates many of the contradictory actions and opinions of these two men.
“Jefferson embodies the inherent contradictions, a kind of self-deception, that co-exists in us, too,” says Professor Joseph Ellis, author of American Sphinx: The Character of Thomas Jefferson . And may be this is the point: these continue today.
“Should Americans therefore disavow these founding fathers as scoundrels and national embarrassments, or accept them as men of their time, demigods with feet of clay, who bore their imperfections even as they sought to steer their country beyond them?” 
Unlike confederates like Robert Lee “… neither of those two persons led the nation in treasonous insurrection to overthrow the government they had formed in order to preserve the institution of slavery. Period.” says Dr Clarence Jones – the African-American speechwriter who helped civil rights legend Dr Martin Luther King Jr craft his 1963 “I Have a Dream” address. 
Within my lifetime racial slurs and terms have fallen out of favour in open and public discourse. I’ll submit this represents a recognition of the moral issue such terminology reflects, although the underlying racism may remain. Is this progress? May be.
“Professor Ellis believes a plaque should be put up at the Jefferson Memorial to correct the record and admit some of the Declaration author’s less savoury statements.”  In doing so these monuments remind us of the attitudes prevalent in the past enabling us to measure the degree change; they become a benchmark.
No one is perfect as captured in the allusion to Jesus’ comment in John 8, 7 “Let he who is without sin cast the first stone”. It is difficult to assess past actions and opinions in a present context. But we can see them and determine if they are recognised as flaws and if so whether we have taken steps to improve upon them and if not then what steps need to be taken.