Yeah, I think it's rooted in the idea that fiction = not real.

On the surface it seems like a made up story wouldn't be useful to know in the "real world", but it's hard to believe that when you read great fiction.

Also, the value is harder to quantify, it doesn't always appear where you expect it to, and it's hard to rationalize (to yourself or others) why you're spending hours reading about things that may have never even happened. I guess that's what keeps some people away from the genre entirely.