Load-balancing problem

[dever]

Hi there,
I'm having an issue with my load balanced SPS configuration. I'm using Cisco
LocalDirectors to handle the load balancing.
Here's my setup:
I have two virtual servers (one hosts applications and web services, one
hosts a portal, we'll call them VirtualServers 1 & 2). Both of these are
bound to two "real" (physical) servers (RealServer 1 & 2) on ports 80 and
443. ICMP is disabled on the local director and the servers each have one
NIC, which is configured with standard settings from DHCP. No special DNS or
gateway or anything.
Basically, the portals work (mostly) fine from a workstation. But when I go
into the console on any one of the real servers and try to browse or telnet
on port 80 to the virtual server, about 60% of the time it acts as if the
server can't be found.
Using nslookup, I can tell it's resolving to the correct IP address for the
virtual server. Again, from a workstation, this same operation works fine
consistently. I said mostly fine above because when you declare a datasource
for a web part in Sharepoint, the request for the data is generated from the
server. This obviates the problem since it causes the same scenario -- a
real server requesting data from a virtual server. This errors out, and
therefore is a bigger problem.

My hunch is that this has something to do with masquerading or reverse
masquerading. RealSvr1 makes a request to VirtualServer1, the localdirector
routes the request back to RealSvr1 or RealSvr2, and the RealSvr(1/2)
responds directly to the source requestor, RealSvr1, without going back
through the localdirector. So the response doesn't look like it's coming
from VirtualServer1 anymore. That's my best guess at this point anyway.

Has anyone seen this behavior before? Does anyone know a remedy to get this
working properly?

Thanks very much for your time,
Esteban

[Ed Horley]

Esteban,
Even if the real servers reply back directly the traffic would still have to
go through the LD assuming you have the network built out properly.
Basically the LD works by dividing the network at the layer 2 level. It
then performs layer 3 and 4 functions for hosts "behind" it by providing
virtual IP's on the network. It will not block any traffic directed to the
real servers and will pass through rARP and ARP as expected, it only
provides functions for the VIP (with exceptions here and there). Most
likely you are running into an issue where the client requests are going to
the VIP and since you don't have restrictions on the real servers those
servers are replying back directly to the host. Since the host will see the
reply back coming from the "real" ip address of the server and not the VIP
it will most likely reject the traffic. This might help you figure out a
way around the problem.
http://www.cisco.com/en/US/products/hw/contnetw/ps1894/prod_configuration_examples_list.html

I am a little rusty with them, we switched to the CSS11K series about 3
years ago so I haven't worked on them in awhile. Hard to believe Cisco sold
them as long as they did. They are now an end-of-life / end-of-sale
product. If you are doing a new web deployment I would really recommend you
upgrade to some newer technology. Redline, F5 BigIP, Cisco CSS, Foundry
ServerIron stuff would all be better options then what you have.

Regards,
Ed Horley
Microsoft MVP Server-Networking

Post by dever
Hi there,
I'm having an issue with my load balanced SPS configuration. I'm using Cisco
LocalDirectors to handle the load balancing.
I have two virtual servers (one hosts applications and web services, one
hosts a portal, we'll call them VirtualServers 1 & 2). Both of these are
bound to two "real" (physical) servers (RealServer 1 & 2) on ports 80 and
443. ICMP is disabled on the local director and the servers each have one
NIC, which is configured with standard settings from DHCP. No special DNS or
gateway or anything.
Basically, the portals work (mostly) fine from a workstation. But when I go
into the console on any one of the real servers and try to browse or telnet
on port 80 to the virtual server, about 60% of the time it acts as if the
server can't be found.
Using nslookup, I can tell it's resolving to the correct IP address for the
virtual server. Again, from a workstation, this same operation works fine
consistently. I said mostly fine above because when you declare a datasource
for a web part in Sharepoint, the request for the data is generated from the
server. This obviates the problem since it causes the same scenario -- a
real server requesting data from a virtual server. This errors out, and
therefore is a bigger problem.
My hunch is that this has something to do with masquerading or reverse
masquerading. RealSvr1 makes a request to VirtualServer1, the
localdirector
routes the request back to RealSvr1 or RealSvr2, and the RealSvr(1/2)
responds directly to the source requestor, RealSvr1, without going back
through the localdirector. So the response doesn't look like it's coming
from VirtualServer1 anymore. That's my best guess at this point anyway.
Has anyone seen this behavior before? Does anyone know a remedy to get this
working properly?
Thanks very much for your time,
Esteban

[dever]

I think I follow the theory of why I'm not getting responses, but when you
say "since you don't have restrictions on the real servers those servers are
replying back directly to the host," I'm wondering whay you mean by
restrictions. What kind of restrictions are those? Are they a configuration
point on the LD or on the real servers?
Thanks for your help

Post by Ed Horley
Esteban,
Even if the real servers reply back directly the traffic would still have
to go through the LD assuming you have the network built out properly.
Basically the LD works by dividing the network at the layer 2 level. It
then performs layer 3 and 4 functions for hosts "behind" it by providing
virtual IP's on the network. It will not block any traffic directed to
the real servers and will pass through rARP and ARP as expected, it only
provides functions for the VIP (with exceptions here and there). Most
likely you are running into an issue where the client requests are going
to the VIP and since you don't have restrictions on the real servers those
servers are replying back directly to the host. Since the host will see
the reply back coming from the "real" ip address of the server and not the
VIP it will most likely reject the traffic. This might help you figure
out a way around the problem.
http://www.cisco.com/en/US/products/hw/contnetw/ps1894/prod_configuration_examples_list.html
I am a little rusty with them, we switched to the CSS11K series about 3
years ago so I haven't worked on them in awhile. Hard to believe Cisco
sold them as long as they did. They are now an end-of-life / end-of-sale
product. If you are doing a new web deployment I would really recommend
you upgrade to some newer technology. Redline, F5 BigIP, Cisco CSS,
Foundry ServerIron stuff would all be better options then what you have.
Regards,
Ed Horley
Microsoft MVP Server-Networking

Post by dever
Hi there,
I'm having an issue with my load balanced SPS configuration. I'm using Cisco
LocalDirectors to handle the load balancing.
I have two virtual servers (one hosts applications and web services, one
hosts a portal, we'll call them VirtualServers 1 & 2). Both of these are
bound to two "real" (physical) servers (RealServer 1 & 2) on ports 80 and
443. ICMP is disabled on the local director and the servers each have one
NIC, which is configured with standard settings from DHCP. No special DNS or
gateway or anything.
Basically, the portals work (mostly) fine from a workstation. But when I go
into the console on any one of the real servers and try to browse or telnet
on port 80 to the virtual server, about 60% of the time it acts as if the
server can't be found.
Using nslookup, I can tell it's resolving to the correct IP address for the
virtual server. Again, from a workstation, this same operation works fine
consistently. I said mostly fine above because when you declare a datasource
for a web part in Sharepoint, the request for the data is generated from the
server. This obviates the problem since it causes the same scenario -- a
real server requesting data from a virtual server. This errors out, and
therefore is a bigger problem.
My hunch is that this has something to do with masquerading or reverse
masquerading. RealSvr1 makes a request to VirtualServer1, the
localdirector
routes the request back to RealSvr1 or RealSvr2, and the RealSvr(1/2)
responds directly to the source requestor, RealSvr1, without going back
through the localdirector. So the response doesn't look like it's coming
from VirtualServer1 anymore. That's my best guess at this point anyway.
Has anyone seen this behavior before? Does anyone know a remedy to get this
working properly?
Thanks very much for your time,
Esteban