I think it's pretty clear that the "
Terms of Service" are incompatible with the GPL, which states that no further restrictions may be added to software distributed under the GPL.
Fact: YaBB SE is GPL'ed.
Fact: Most Mods include YaBB SE source code. Therefore they're also GPL'ed.
If they don't include any YaBB SE source code, they're not GPL'ed, but as soon as they're applied to a YaBB SE source tree, they become GPL'ed and no additional restrictions can be applied to that combined work.
Disclaimer:
I am not a lawyer, and I hadn't heard of "YaBB SE mods" ~24h ago. The above is based on my understanding of the situation. I do not represent the FSF in any capacity.