A lot of people think that the SAT doesn't matter, but that's simply not true. Other than high school grades, the SAT is probably the most important thing that can affect whether a person gets into the college of their choice or has to settle for attending somewhere where the standards are not as strict.
This does not mean that these other schools are not good, but most people have an opinion about where they wish to attend school, and low SAT scores can keep them from that.
The SAT scores can also indicate the likelihood of getting grants and loans and scholarships. The SAT is more important than you think, but that doesn’t mean that you have to panic about taking it. There are a lot of test preparation options out there for you, and if you start preparing for it early you shouldn't have much of a problem scoring well on it.
Intelligence does matter, of course, but so does study and preparation, which can often be more significant to SAT scores and other standardized tests.