I know this is an old post, but I am successfully growing olives in Greensboro; my catch is they are potted, and I bring them in for the winter. Two of them have lived a couple years in Northern VA as well, and moved with me. If you're looking to plant them in the ground, you'll have to watch...