I'd start with a good .22, an inexpensive bolt action or Ruger 10/22. Both good for target practice and putting food in a pot if needed. With Aguila Sub Sonic Snipers they are great for getting rid of the evil neighborhood squirrels too.
For the EOTWAWKI / SHTF rifle, can't beat an SKS or AK. I have a Romanian WASR GP 10/63 that I bought for less than $400 and it is my favorite rifle. Took a little work to clean up, but plenty accurate, small, (especially with a side folding stock) easy to handle and shoots a nice .30 caliber round that is good out to about 300 yards.
There are a lot inexpensive AK-74 style rifles these days too. 5.45 x 39 is pretty damn cheap and effective as well. The Afghan's nick named it the Devil's Dart.