This is adapted from the introduction to a game I'm writing.
Imagine you’re watching a group of mimes, performing for you. There’s three or four of them, and they’re doing some ordinary thing, like cooking a meal or cleaning the house, except that all the things they interact with are imaginary. They’re very skilled. When one of them picks up a bucket full of water, we see him strain against its weight, even though we don’t see the bucket itself. He sets it down. Another mime picks it up. We still don’t see the bucket, it’s imaginary, but when the mime goes to pick it up, she picks it up from exactly where the last mime put it down. The mimes pass each other things, open and close doors, trip over objects someone else left lying on the floor, and so on. Even though all the things they interact with are imaginary and invisible, as we watch them we feel we can almost see the bucket, or the doorway, or whatever it is. The rigourous discipline of the mimes, treating every object as if it were real, remembering where things are, their weight, their size, creates the illusion of reality. That’s what makes the mime show compelling (and creepy).
In roleplaying games, we’re doing the same thing as those mimes, except that instead of acting out a scene ourselves, we talk about a scene, describing imaginary characters and things. We each take responsibility for describing the actions of an imaginary person, and talking about how they interact with the world around them. Just like the mimes, when one person describes putting something down, the other players can describe their character picking that thing up. The characters pass each other things, open and close doors, trip over things, and they also explore dungeons and fight monsters.
Just like the mimes, the illusion of reality is important. We know the things we describe aren’t real, but we treat them as if they are, in the game. The sword your character wields has a particular weight and length, the weather is a certain way. Your character dodges like so when the strange subterranean horror lurches in her direction. We remember these things, and they form part of the ongoing narrative of play. They take on an illusion of reality.
In roleplaying games, that narrative, the things we’ve said, the weight of your sword, the slavering fangs of the monster, is called the “fiction”.
The rules of the game will sometimes refer you to the fiction of play. They’ll ask you “what is happening now?” and say “when this happens, do this”. You’ll know what rules to use by looking to the fiction of the game. You make a decision about whether the rules apply, but you make that decision based on the things people have said and that everyone has agreed to. It's not the same as looking at a number someone rolled on a die, or the card they drew, or the position of a miniature on a map. But it's not the same as someone making an arbitrary ruling either. You make your decision based on maintaining that illusion of reality. Just like a mime can't walk through the imaginary wall they've created without shattering the illusion, when you look to the fiction in a roleplaying game, you're making a judgement that's constrained by that illusion of reality, the fiction.