We consider isolated interacting quantum systems that are taken out of equilibrium instantaneously (quenched). We study numerically and analytically the probability of finding the initial state later on in time (the so-called fidelity or Loschmidt echo), the relaxation time of the system, and the evolution of few-body observables. The fidelity decays fastest for systems described by full random matrices, where simultaneous many-body interactions are implied. In the realm of realistic systems with two-body interactions, the dynamics is slower and dependent on the energy of the initial state. The fastest fidelity decay in this case is Gaussian and can persist until saturation. The fidelity also plays a central role in the short-time dynamics of few-body observables that commute with the system Hamiltonian before the quench. Our analyses are mainly developed for initial states that can be prepared in experiments with cold atoms in optical lattices.