There are two types of challenges in the game: those that are challenges from the get go, and those that begin in the classic mode.
Challenge games: you can play against friends or random opponents. You won't be able to reject games against random opponents if you are the one who starts them. If a friend asks you to participate in a Challenge game, you will be able to reject the invitation BEFORE you start to play. If you exit the game once it has already started, you will not be able to reject it.
Challenges in classic games: If you both have characters during a classic game, you will be able to challenge your opponent the moment you are about to win another one. If this happens, you'll have to answer 6 questions each. If the challenger wins, he will "steal" the character from the user that's been challenged. If the user that's been challenged wins, he will retain the same number of characters and the challenger will lose one in return. It is NOT possible to quit in these type of challenges.