How Do You Know What Is Literal?

"That verse isn't supposed to be taken literally."

I'm sure that many of you have been in a situation where this was said. You are discussing a verse which is either claimed to be crazy, contradictory, silly, atrocious, or whatever. The theist in the conversation then claims that the verse is not supposed to be taken literally, it is a metaphor or a figure of speech. My reply to that is usually to ask "How do we determine what is literal and what is not?"

There are many things in the bible that seem impossible, it seems that most people believe some of them are literally true, miracles performed by God. For example, in my experience, most people think that Mary was literally a virgin when she gave birth to Jesus. On the other hand, there are other things in the bible which many people think are not to be taken literally. For example, many Christians think that the 6 day creation is not meant to be a literal account, but instead is poetic and was simply something the audience at the time could understand. There are of course many who believe the 6 day account is completely accurate (I wish I could think of an example that is nearly universally thought of as metaphor, but I doubt such a thing really exists.)

So my question becomes, how do you determine what box to put each miracle that is described in the bible? What method do we use to decide X is real and Y is metaphor? If 2 Christians disagree what is the appropriate way to determine who is right? When I was a Christian, I thought about this a lot. I tried to figure out what was real and what was not. Eventually, I put so many things in the "not real" box and I reevaluated the whole thing. Instead of trying to figure out what I should throw out of the 'real' box from the 'metaphor' box, I tried to figure out what deserved to go in the 'real' box. Once I had to justify things being real, rather than justifying things being metaphor, or poetic, I discovered that I had trouble justifying that anything was literally true.