You can recurse into a subpattern using the following constructs (depending on the flavor), assuming n is a capturing group number, and name the name of a capturing group.
(?n)\g<n>\g'0'(?&name)\g<name>\g'name'(?P>name)The following pattern:
\[(?<angle><(?&angle)*+>)*\]
Will match text such as: [<<><>><>] - well balanced angle brackets within square brackets. Recursion is often used for balanced constructs matching.