Speaker
Description
With the development and deployment of telescopes like LOFAR2.0, SKA and ngVLA, radio interferometry is entering the Big Data Era, in which a single observation will produce tens of TB of data. To manage this data deluge within an acceptable and sustainable costing envelope for computing hardware and energy consumption, at least an order of magnitude improvement in computational performance is needed. Data processing pipelines need to scale well across multiple nodes and use processing components optimised for energy efficiency, including optimisation by exploiting accelerators.
In this talk, I will introduce the magnitude of these challenges using experience from the International LOFAR telescope, showing why current practices are unsustainable. I will then discuss how these challenges are currently being addressed in the development of the self-calibration pipelines for LOFAR and SKA. The performance of this pipeline is benchmarked regularly to identify processing bottlenecks and define steps to remedy them. These tests indicate that visibility predictions, calibration solvers and imaging are primary targets for performance improvements. One of the key features of the pipeline architecture is modularity, which should enable us to replace individual processing components by newly developed ones, thereby making these pipelines open to new processing approaches explored by the community, of which I will give a number of examples and indicate steps taken to facilitate integration of such community contributions into these pipelines.
