I have just installed a new Pix535 in our core network. I wanted to seperate my different zones by a firewall. The Main zone i wanted to protect hosts all our servers. We will call this zone x.y.z.0/24. The Gateway address for everything to go out to the Internet is x.y.z.254. We have a Layer3 switch on x.y.z.210 which handles all the routing so all internal traffic doesn't get sent to the Internet. Everything worked ok with the Layer3 switch in place handling all the core routing.
So, I put in a Pix firewall to act as the Layer 3 switch so that all the different zones on our network go through the firewall. There are around 6 zones off the firewall which then have further routing into several different WANs.
I simulated the network to do some testing with the Pix in the first place (didn't simulate the internet traffic though). All worked fine and dandy.
Right, here is what i had done which was a mistake which i didn't actually notice. I put x.y.z.210 on my pix interface. Anyone spot why this is a mistake ? Well, stop at this point (draw a diagram if needed) to have a think as the next line will be why it was a mistake.
Right, the Firewall went in. All zones could talk to each other (using the No Nat-Control) and I was happy. Next day, we had reports that no external e-mail was getting out. The Mail Server on x.y.z.30 could not get out to the Internet. The Gateway was the Pix x.y.z.210 and the default route on the Pix was x.y.z.254. Yup, traffic coming into the Interface couldn't then go back out the same interface, doh.
So, my fix was. Move the x.y.z.210 address off the Pix and put it back on the switch and then point the switch to the Pix's new interface x.y.z.250 (routing on the switch only pointed internal traffic back into the pix, all other traffic went to x.y.z.254). Tried this and it all stopped working. For some reason, as soon as i moved the 210 address off the pix and put it onto the switch, the pix couldn't talk to the x.y.z.0 network. I could ping the pix's own interface but that was it. The odd thing, if i did a clear arp on the pix, i would get 4 replies back and then if i did another ping, i got nothing again, until clearing the arp. Also tried rebooting the machines i was pinging and made no difference.
Due to this actually being in the live network now, i had limited time to play around with this IP Change, once it didn't work for 30mins, i had to roll back to atleast get most of the network back up and running. I tried the IP change 3 times and each time it failed. Tried, rebooting the Switch, Pix, even other switches with TrunkVLANs in the x.y.z.0/24 network, machines. Even tried turning the proxyarp off the pix (suggestion from our support company).
Not got to the bottom of the cause and got cheesed off trying to find out the issue. In the end i just put a different address on the Pix x.y.z.251 which worked so i have no clue as to why the original x.y.z.250 address would not work on the Pix ? The strange thing is that the x.y.z.250 address worked fine until i put the x.y.z.210 address on the switch ?
Sorry if this isn't very easy to follow, if anyone is interested in seeing this problem better i will do a diagram and post it.