Looking at my list and considering the question, it's a highly subjective "feeling" most of the time. I think some games aim for it and achieve it (say, Myst and 7th Guest), but mostly its a subjective magic where everything came together for me, including the games I played or skipped before playing a particular one.
As I understand it, it's a dopamine thing, as in "The feel-good hormone dopamine improves long-term memory". If you managed to hit high levels of dopamine (mostly because you are having fun) when you play, you remember the game, and it has a magic glow of "atmosphere." Best game in the world the same week as an important funeral - probably no good memories because no "good brain chemicals"; unless it cheered you up, then it will be the best game for the rest of your life since it moved you from "negative brain chems" "good brain chems" state.
Also, I think it is kind of like movies - lots of movies "should" be good, but just lack something - two that I love are John Carter and Prince of Persia, misses as movies even though made by the same brains that made Pirates of the Caribbean, a terrific hit. Why the last was a hit and the first two weren't, I don't know, but it is obvious when watching them. Yet take it apart, and editing, music, plot, acting, and so forth are all equally good. (Though I'm mad - John Carter in part was destroyed by pandering to 3D effects, which killed the 2D editing. Grrr. All the slow pacing or weird stuff suddenly made sense when I saw it in 3D finally, but killed an otherwise fun movie.)
But I hear music in my head with certain games, linking certain albums to certain games. And when I hear the album, I often think of the game and want to go play. Castle Master, Ultima Underworld and more have an album I bought at the same time and listened to over and over linked to them - something outside the game designer's control.
My $.02 for free, late at night, and on painkillers, so, worth what you paid for it.

Other opinions?