Well, there's his Core-given power, erasing memories temporarily. Maybe Larry could trip and end up wiping a random asylum person's memory - possibly the memory of Joker being kept confined?

Then when Larry figures out what happened, he can go Puella Magi (he can transform into a magical guy, summon halberds, and create shields) to fix what he's done.