The essential ingredients of a quantum theory are usually a Hilbert space of states and an algebra of operators encoding observables. The mathematical operations available with these structures translate fairly well into physical operations (preparation, measurement etc.) in a non-relativistic world. This correspondence weakens in quantum field theory, where the direct operational meaning of the observable algebra structure (encoded usually through commutators) is lost. The situation becomes even worse when we want to give a more dynamical role to spacetime as for example in attempts to formulate a quantum theory of gravity. I argue that a revision of the structures that we think of as fundamental in a quantum theory is in order. I go on to outline a proposal in this direction, based on the so called 'general boundary formulation', emphasizing the operational meaning of the ingredients. If time permits I will also comment on the relation to the framework of algebraic quantum field theory.