Don't put me on hold!

If you need to interact with someone to get something done, then one of the worst things that can happen to you is to be put on hold. While you're waiting there's not much else you can do, and often you don't even know how long the wait will be. Believe it or not, this is the way in which scientists use supercomputers: one submits a "job" (a complete description of some calculation) to a queue, where it will sit for hours or days before it starts. This is called batch processing, and is a remnant straight from the dinosaurs of computing, from a time when you submitted punch cards to an operator and received a result the next day.

There are many reasons why batch processing makes sense. For example, if a job has a particular portion of a computer to itself for some time, it can use this portion in the most efficient way. A queue of jobs that are ready to run also ensures that a computer is not idle -- if one job aborts with an error, the next job can run right away, even on nights and weekends. And if a calculation runs for several weeks, then waiting for a few days until it starts isn't that bad.

However, this is not how everyday scientific computing looks like. Science consists of trying out new things, new ideas, of making things work that have never been done before. While there are some "production calculations" where one wants to run for a long time, the vast majority of calculations are exploratory: Does this work? What happens if I do this? Does that make things fail right away -- or is it a breakthrough? Each such idea may consist of a few minutes of planning and programming, and then... waiting for hours or days in a queue.

Batch processing in supercomputing is probably the singlemost worst issue preventing progress. It slows down seasoned experts, it slows down students as they learn, it deters newcomers from entering the field, and (and this is the worst) it destroys many ideas that are never implemented, never tested. Since each new idea may take days to try out, and since people's time is limited, only tried-and-true approaches are followed. Compared to a desktop running Mathematica or Python, this slows down progress by two orders of magnitude.

Yet for some strange reason, I do not hear the screams of complaint of the more senior HPC users. Maybe this has something to do with the lack of modern user interfaces -- ssh is the standard, graphical user interfaces, or IDEs are almost unheard of -- this is a laudable exception (but is far from being mainstream): HPC computing seems stuck in 1976, when the first Cray-1 was installed. Since then, efficiency and performance have improved enormously, but all user interface improvements since then have passed by without leaving so much as a scratch.

This is all done praying to the false god of More Performance. Yes, batch processing ensures that more cycles are used on HPC systems, and that "more calculation" is done. But it does not ensure that more science is done, or that HPC-supported science is used in more disciplines. I know that, if I had an interactive supercomputer at my hands, my programming productivity would increase tenfold, and this would immediately benefit also my collaborators, and would give us an enormous performance boost in terms of ideas that we can try, ideas that succeed, and papers that we publish.

Batch processing supercomputing must die. Don't put me on hold.