This will need to be done in iptables. There are number of ways of restricting connections in iptables. These involve making use of the limit and iplimit matching extensions. The limit extension which comes bundled with iptables by default matches by looking at the number of connections created over a given period (sec, min, hr, day). "iplimit" actually limits the number of connections from a particular host or network, but it is not present in the default installation. I'm surprised you're using up so many connections. How many users have you got behind your IPCOP? Are you using peer to peer software? You can quite easily cut down on the number of long-term connections by using the built-in proxy server.
Eight users all using peer to peer software will easily consume more than 200 connections. I do think the 200 connection limit is a bit low. Why do you think the modem is the problem? How does the problem manifest itself? Is there an error message? How do you know the limit is 200 connections?
Some time internet just stop, after rebooting of the modem everything is OK. This is happen 3-4 times for a day some times more. My ICP say that is manufacture problem of this modem. This problem is not only with me. All customers who using peer to peer software is the same.