Edsger Dijkstra allegedly said "computer science is no more about computers than astronomy is about telescopes." Often people replace "computers" with "programming," for much the same idea.
An education in computer science will expose you to a wide variety of topics that have nothing to do with programming itself (although programming is often used as a vehicle for communicating and solving those problems) -- topics that have to do with modeling and manipulating information in interesting ways that tend to be very relevant in modern game development (even if it won't seem apparent while you're slogging through the classwork). For example I have leveraged the knowledge I acquired about graph theory in several instances to build automatic layout and diagramming tools, "visual programming" type editors, and unique-coloring algorithms to allow designers to have a holistic view of the the content they produced for a game, et cetera.
You don't need a computer science degree to be a game programmer. You could have another degree (or, if you're quite good, no degree at all, but that could be risky in today's job market at least) -- math or physics, for example. You could even get one of those "game school" degrees (but I would not recommend it). It is important that you study what you are interested in, because college will only give you benefits proportional to the effort that you put into it, and you're not likely to do that if you're not invested in your coursework in some fashion.
Certainly studying on your own, outside of school, is important, too -- because most colleges won't be teaching you how their coursework can be applied directly to game development. Plus, building games and little tech demos on your own will help you have a portfolio of interesting work to talk about in your future job interviews. Some interviewers, myself included, are greatly encouraged by a candidate who takes the time to pursue knowledge on his or her own.
As for where you should go... that depends on you as well. You should go to a school that interests you, where you think you'd fit in, where you like the environment, et cetera. You should visit schools if possible. You don't need to go to Brown, MIT or Carnegie Mellon or whatever in order to get a good, solid CS education. In fact, I'm personally a bit skeptical that there is any significant benefit at all to doing so (many such schools are actually more well known for their graduate programs, and their undergraduate material may be no better off than anywhere else).