No, I am not wrong. You are partially right when you say they wanted to protect the the free exercised of religion without the interference of government, but they also didn't want religion to influence our government. A wall of separation to keep them pure. Jefferson even states that legislature should "make no law respecting an establishment of religion."
Keep in mind, many of the founding fathers were deists or non-believers and didn't believe in religion and made a point that our government was not established on Christianity in any way.
Also, "Under God" was added to the pledge in 1954 and should be taken off. "In God We Trust" was not added to money until around 1886 and I still think it goes against the establishment clause of the first amendment. The military should not employ military Chaplains, fund groups such as the "boys scouts of america" since they prohibit atheists and agnostics from membership yet vow a duty to God and country. If they want to prohibit Atheists, that is okay, just don't use GOV money.
Since the USA is still predominately Christian, the First amendment is not being enforced to its fullest due to religious influence.
I really don't mind people practicing religion on their own, but I want them to keep it out of my life and government. Like the original start of this thread, they should read into the constitution with more understanding of what our founding fathers meant.