In an Optimistic Rollup, state commitments are published to Ethereum without any direct proof of the validity of these commitments. Instead, these commitments are considered pending for a period of time (called the "challenge window"). If a proposed state commitment goes unchallenged for the duration of the challenge window (currently set to 7 days), then it is considered final. Once a commitment is considered final, smart contracts on Ethereum can safely accept proofs about the state of Optimism based on that commitment.
When a state commitment is challenged, it can be invalidated through a "fault proof" (formerly known as a "fraud proof") process. If the commitment is successfully challenged, then it is removed from the
StateCommitmentChain to eventually be replaced by another proposed commitment. It's important to note that a successful challenge does not roll back Optimism itself, only the published commitments about the state of the chain. The ordering of transactions and the state of Optimism is unchanged by a fault proof challenge.
The fault proof process is currently undergoing major redevelopment as a side-effect of the November 11th EVM Equivalence update. You can read more about this process within the Protocol specs section of the developer documentation.