Yes the ReDoS analysis doesn't really understand back-references or lookaheads.

And yes, that's an FP.

But I also think your regexp is broken, and fixing the regexp would also remove the ReDoS FP.

I can get the regexp to match strings like "\\d", "\\w", and "[]".
But I can't get it to match any string that has something between the square brackets.
The reason for the that is the lazy repetition (*?), because that lazy repetition will always just trivially match the empty string.
The following backref will then match the empty string, which is only possible when the entire string is "[]".

From the name of the regexp it sounds like you are trying to match char escapes (which it does) and char classes (it only matches []).

Maybe try /^(?:\\.|\[[^\]]+\])$/ instead, that way your regexp matches anything enclosed by square brackets, and it doesn't match [] (which is a syntax error).

Some examples that show why I think your current regexp is broken:

> /^(?:\\.|\[(?=((?:\\.|.)*?))\1\])$/.test("\\d") // should match - and does - good
true
> /^(?:\\.|\[(?=((?:\\.|.)*?))\1\])$/.test("[]") // should not match - but does - bad
true
> /^(?:\\.|\[(?=((?:\\.|.)*?))\1\])$/.test("[a-z]") // should match - but doesn't - bad
false

Mmmh, indeed, well spotted thanks, there's a hole in my test suite, and another one in my understanding of look aheads :-)

Thankfully, the bug has no consequence on the functionality of the lib. This is part of the logic to determine if a non-capturing group is required before applying a quantifier. This bug means there will be useless non-capturing groups. It will soon be fixed.

Edit2: I went for the atomic version of your suggestion, with * as quantifier: /^(?:\\.|\[(?=((?:\\.|[^\]])*))\1\])$/

Btw, /[]+/ is not a syntax error in JS, even in u mode.

I may reject it at some point, because it makes little sense to quantify never, but for now, since it doesn't cause syntax errors I'll keep it.

Mmmh, indeed, well spotted thanks, there's a hole in my test suite, and another one in my understanding of look aheads :-)

No problem.
Regular expressions are hard. I was basically dreaming in automata when I made the ReDoS libraries, and even I get something wrong every once in a while.

Btw, /[]+/ is not a syntax error in JS

Hmm. I misremembered. It's a syntax errors in some languages, I just remembered wrong for JS.

Edit2: I went for the atomic version of your suggestion, with * as quantifier: /^(?:\\.|\[(?=((?:\\.|[^\]])*))\1\])$/

This is nitpicking, but I'm not fond of using features like backreferences when something can be expressed using a plain regular expression.

Looking at your regexp and mine, they don't match the same things (not just the [] thing).
You have a greedy atomic match that will try to match \\. before trying to match [\]], and that's why your regexp correctly matches a string such as "[a\\]]" (because the "\\" string is not eaten by the [\]] term).
Try to flip the order of \\. and [^\]] in your regexp, then the string "[a\\]]" will no longer be matched.
(My regexp would mistakenly not match "[a\\]]", and your regexp made me realize it).

The proper fix (I think), is to modify the regular expression to something like: /^(?:\\.|\[([^\]\\]|\\.)*\])$/.