Linnebo and Pettigrew (Philos Q 64:267–283, 2014) have recently developed a version of non-eliminative mathematical structuralism based on Fregean abstraction principles. They recognize that this version of structuralism is vulnerable to the well-known problem of non-rigid structures. This paper offers a solution to the problem for this version of structuralism. The solution involves expanding the languages used to describe mathematical structures. We then argue that this solution is philosophically acceptable to those who endorse mathematical structuralism based on Fregean abstraction principles.