AJ, I am curious what IWBs you have tried? The Crossbreed Supertuck and the Kholster are two very popular IWBs, both renowned for their comfort and concealment. I personally use the Kholster for my Glock (sub-compact, but with a fullsized mag) and would recommend it to anyone. BTW, I am about the same build as you, just on a slightly smaller scale.
However, if you are set on the OWB, the belt and shirt are key. I also use a Blackhawk SERPA (same gun, same mag) and it is invisible when I wear the right shirt. As mentioned earlier, a good sturdy belt pulled as tight as you can handle it will pull the gun closer to you. Pulling the butt in even a half inch can make a world of difference. Also mentioned, a thicker cloth does a wonderful job at resisting the print and dark colors don't show shadows as well, and therefore blend whatever slight print there may be. I have come to appreciate black polo shirts in the last couple years.
Of course, if trading away the XD is an option, there are plenty of .380s out there that are much more attuned to concealment.
The two enemies of the people are criminals and government, so let us tie the second down with the chains of the Constitution so the second will not become the legalized version of the first. - Thomas Jefferson