What is the broken plural (jam' taksir)?

Where the sound plurals bolt an ending onto an intact singular, taksir reaches inside and rearranges the letters' vowels — كِتَاب to كُتُب, رَجُل to رِجَال, مَسْجِد to مَسَاجِد. The patterns are many and learned by exposure, not by rule; the dictionary habit of noting every plural exists for this reason. Grammatically it declines with the ordinary three vowels — except the grand patterns like مَسَاجِد, which cross into muntaha al-jumu' and lose their tanwin.

“Broken” describes what happens to the word, not to the rule. The singular's shape is taken apart and recast: رَجُل becomes رِجَال, كِتَاب becomes كُتُب.

The consequence is the part that matters. Because nothing was suffixed, a broken plural takes ordinary vowel endings like any singular noun — رِجَالٌ / رِجَالًا / رِجَالٍ — while a sound plural takes letter endings. Same meaning, completely different i'rab.

Is there a rule for forming the broken plural, or must each one be memorised?

There are patterns, and a great many plurals fall into a limited set of them — but the pattern cannot be predicted from the singular with certainty. Learn the plural together with the word, the same way you learn its gender.