You need some sort of reason to get the restriction lifted, such as carrying large amounts of money, living in a dangerous area, etc. My wife got her restrictions removed because she worked the night shift and had to walk in an unlighted parking lot.. May not work in your county.
Having said that, NY supreme court has ruled that a permit is legal to carry with or without restrictions. You do not violate penal code by carrying with restrictions. You may offend the issuing judge and he will pull your permit, but you have not done a criminal act.