Lessons from a blind study of simulated lenses: image reconstructions do not always reproduce true convergence

In the coming years, strong gravitational lens discoveries are expected to increase in frequency by two orders of magnitude. Lens-modelling techniques are being developed to prepare for the coming massive influx of new lens data, and blind tests of lens reconstruction with simulated data are needed for validation. In this paper, we present a systematic blind study of a sample of 15 simulated strong gravitational lenses from the EAGLE suite of hydrodynamic simulations. We model these lenses with a free-form technique and evaluate reconstructed mass distributions using criteria based on shape, orientation, and lensed image reconstruction. Especially useful is a lensing analogue of the Roche potential in binary star systems, which we call the lensing Roche potential. This we introduce in order to factor out the well-known problem of steepness or mass-sheet degeneracy. Einstein radii are on average well recovered with a relative error of ∼ 5 per cent for quads and ∼ 25 per cent for doubles; the position angle of ellipticity is on average also reproduced well up to ±10°, but the reconstructed mass maps tend to be too round and too shallow. It is also easy to reproduce the lensed images, but optimizing on this criterion does not guarantee better reconstruction of the mass distribution.