Large-scale cosmological simulations are an indispensable tool for modern cosmology. To enable model-space exploration, fast and accurate predictions are critical. In this paper, we show that the performance of such simulations can be further improved with time-stepping schemes that use input from cosmological perturbation theory. Specifically, we introduce a class of time-stepping schemes derived by matching the particle trajectories in a single leapfrog/Verlet drift-kick-drift step to those predicted by Lagrangian perturbation theory (LPT). As a corollary, these schemes exactly yield the analytic Zel'dovich solution in 1D in the pre-shell-crossing regime (i.e. before particle trajectories cross). One representative of this class is the popular ‘FASTPM’ scheme by Feng et al. 2016 [1], which we take as our baseline. We then construct more powerful LPT-inspired integrators and show that they outperform FASTPM and standard integrators in fast simulations in two and three dimensions with O(1−100) timesteps, requiring fewer steps to accurately reproduce the power spectrum and bispectrum of the density field. Furthermore, we demonstrate analytically and numerically that, for any integrator, convergence is limited in the post-shell-crossing regime (to order [Formula presented] for planar-wave collapse), owing to the lacking regularity of the acceleration field, which makes the use of high-order integrators in this regime futile. Also, we study the impact of the timestep spacing and of a decaying mode present in the initial conditions. Importantly, we find that symplecticity of the integrator plays a minor role for fast approximate simulations with a small number of timesteps.