Omer,
Thanks for your question - it's certainly an interesting one.
The solution to your problem is quite simple actually. What you need is a Dual DMVPN network between the Hubs (Your Datacenter Routers) and spokes (Endpoints/remote offices).
Luckily DMVPN is a topic heavily covered here on Firewall.cx. You can start on the introduction which explains how the service works and also provides examples and links to other pages here that show how to configure it:
www.firewall.cx/cisco-technical-knowledg...sco-dmvpn-intro.html
In short, with DMVPN all spokes create an IPSec protected (optional) GRE tunnel with the HUB (HQ Router). The Hub works as a central database keeping track of every spoke's public IP address. This allows spokes to also create direct VPN tunnels with each other without passing traffic through the Hub.
The great part of DMVPN is that the spokes can use dynamic public IP addresses as there is no requirement for static public IPs except for the Hub router.
You'll also be happy to know that you can use GNS3 to fully setup this scenario - I've personally setup exactly what you're describing through GNS3 before implementing it at my customer. I used two HQ Hubs - each connected to the internet via a different ISP and 5 spokes which used an ADSL as a primary connection but also with ISDN Backup (more complex!).
Spokes would connect to the primary Hub and if that went down, they would fallback to the secondary Hub.
Read up on the DMVPN articles we've got here and let us know if you still have any questions.
Good luck!
Chris.
