Categorical quantum mechanics (CQM) uses symmetric monoidal categories to formalize quantum theory, in order to extract the key structures that yield protocols such as teleportation in an abstract way. This formalism admits a purely graphical calculus, but the causal structure of these diagrams, and the formalism in general, is unclear. We begin by considering the signaling abilities of probabilistic devices with inputs and outputs and we show how a non-signaling device can become a perfect signaling device under time-reversal. This conflicts with the causal structure of relativity, and suggests that an `asymmetry' is needed when formalizing causality in CQM. We then show how a fixed causal structure within CQM corresponds to topological connectedness in the graphical language, and that correlations, either classical or quantum, force terminality of the tensor unit. We also show that well-definedness of a global state forces the monoidal product to be only partially defined, which in turn results in a covariance theorem. These structural results lead to a mathematical entity which we call a `causal category'.