If your goal is to check if an expression is valid (it also means its substring that contains only brackets forms a correct bracket sequence), then regexps won't help you.

Regular expressions can only handle so called "regular languages" (though JS regexps maybe somewhat more powerful than their theoretic counterparts, the price of such power is greater complexity) while language of correct bracket sequences isn't regular.

See those slides — they can give you a glimpse into why regular expressions cannot recognize correct bracket sequence.

Nevertheless, the problem isn't so hard. You should just maintain a stack and go over your string from the left to the right. Every time you meet an opening bracket, you push it to the stack. When you meet a closing bracket, you pop top element of the stack and check if its type matches your one (yes, this algorithm can handle brackets of multiple types). At the end you should just check if the stack is empty.