Gangmax Blog

Solaris下的snoop

为了分析服务器上的http请求内容，从这里知道了Solaris下面的snoop工具，具体命令如下：

1
2
/usr/sbin/snoop -d nxge2 -v > b.txt
tar cvf - b.txt | gzip -c > b.tar.gz

找的的具体内容（部分）：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253

ETHER:  ----- Ether Header -----
ETHER:  
ETHER:  Packet 10244 arrived at 8:03:50.63821
ETHER:  Packet size = 501 bytes
ETHER:  Destination = 0:10:db:ff:20:a0, 
ETHER:  Source      = 0:14:4f:46:59:44, 
ETHER:  Ethertype = 0800 (IP)
ETHER:  
IP:   ----- IP Header -----
IP:   
IP:   Version = 4
IP:   Header length = 20 bytes
IP:   Type of service = 0x00
IP:         xxx. .... = 0 (precedence)
IP:         ...0 .... = normal delay
IP:         .... 0... = normal throughput
IP:         .... .0.. = normal reliability
IP:         .... ..0. = not ECN capable transport
IP:         .... ...0 = no ECN congestion experienced
IP:   Total length = 487 bytes
IP:   Identification = 8418
IP:   Flags = 0x4
IP:         .1.. .... = do not fragment
IP:         ..0. .... = last fragment
IP:   Fragment offset = 0 bytes
IP:   Time to live = 64 seconds/hops
IP:   Protocol = 6 (TCP)
IP:   Header checksum = 1cfe
IP:   Source address = 141.146.155.3, itsptis1
IP:   Destination address = 144.20.66.135, www-proxy-adc.us.oracle.com
IP:   No options
IP:   
TCP:  ----- TCP Header -----
TCP:  
TCP:  Source port = 40991
TCP:  Destination port = 80 (HTTP)
TCP:  Sequence number = 4173603062
TCP:  Acknowledgement number = 170360723
TCP:  Data offset = 20 bytes
TCP:  Flags = 0x18
TCP:        0... .... = No ECN congestion window reduced
TCP:        .0.. .... = No ECN echo
TCP:        ..0. .... = No urgent pointer
TCP:        ...1 .... = Acknowledgement
TCP:        .... 1... = Push
TCP:        .... .0.. = No reset
TCP:        .... ..0. = No Syn
TCP:        .... ...0 = No Fin
TCP:  Window = 50400
TCP:  Checksum = 0xfd0a
TCP:  Urgent pointer = 0
TCP:  No options
TCP:  
HTTP: ----- HyperText Transfer Protocol -----
HTTP: 
HTTP: POST http://167.117.132.1:9085/arsys/services/ARService?server=remedyqa.crdc.kp.org&webService=HPD_HelpDesk_Query_IR_PR HTTP/1.1
%.. = do not fragment
IP:         ..0. .... = last fragment
IP:   Fragment offset = 0 bytes
IP:   Time to live = 64 seconds/hops
IP:   Protocol = 6 (TCP)
IP:   Header checksum = 1cfe
IP:   Source address = 141.146.155.3, itsptis1
IP:   Destination address = 144.20.66.135, www-proxy-adc.us.oracle.com
IP:   No options
IP:   
TCP:  ----- TCP Header -----
TCP:  
TCP:  Source port = 40991
TCP:  Destination port = 80 (HTTP)
TCP:  Sequence number = 4173603062
TCP:  Acknowledgement number = 170360723
TCP:  Data offset = 20 bytes
TCP:  Flags = 0x18
TCP:        0... .... = No ECN congestion window reduced
TCP:        .0.. .... = No ECN echo
TCP:        ..0. .... = No urgent pointer
TCP:        ...1 .... = Acknowledgement
TCP:        .... 1... = Push
TCP:        .... .0.. = No reset
TCP:        .... ..0. = No Syn
TCP:        .... ...0 = No Fin
TCP:  Window = 50400
TCP:  Checksum = 0xfd0a
TCP:  Urgent pointer = 0
TCP:  No options
TCP:  
HTTP: ----- HyperText Transfer Protocol -----
HTTP: 
HTTP: POST http://167.117.132.1:9085/arsys/services/ARService?server=remedyqa.crdc.kp.org&webService=HPD_HelpDesk_Query_IR_PR HTTP/1.1
HTTP: Accept: text/xml, multipart/related
HTTP: Content-Type: text/xml; charset=utf-8
HTTP: SOAPAction: "urn:HPD_HelpDesk_Query_IR_PR/SearchIRByAssignedGroup"
HTTP: User-Agent: Metro/2.1 (branches/2.1-6728; 2011-02-03T14:14:58+0000) JAXWS-RI/2.2.3 JAXWS/2.2
HTTP: [...]
HTTP: 

ETHER:  ----- Ether Header -----
ETHER:  
ETHER:  Packet 10245 arrived at 8:03:50.63828
ETHER:  Packet size = 653 bytes
ETHER:  Destination = 0:10:db:ff:20:a0, 
ETHER:  Source      = 0:14:4f:46:59:44, 
ETHER:  Ethertype = 0800 (IP)
ETHER:  
IP:   ----- IP Header -----
IP:   
IP:   Version = 4
IP:   Header length = 20 bytes
IP:   Type of service = 0x00
IP:         xxx. .... = 0 (precedence)
IP:         ...0 .... = normal delay
IP:         .... 0... = normal throughput
IP:         .... .0.. = normal reliability
IP:         .... ..0. = not ECN capable transport
IP:         .... ...0 = no ECN congestion experienced
IP:   Total length = 639 bytes
IP:   Identification = 8419
IP:   Flags = 0x4
IP:         .1.. .... = do not fragment
IP:         ..0. .... = last fragment
IP:   Fragment offset = 0 bytes
IP:   Time to live = 64 seconds/hops
IP:   Protocol = 6 (TCP)
IP:   Header checksum = 1c65
IP:   Source address = 141.146.155.3, itsptis1
IP:   Destination address = 144.20.66.135, www-proxy-adc.us.oracle.com
IP:   No options
IP:   
TCP:  ----- TCP Header -----
TCP:  
TCP:  Source port = 40991
TCP:  Destination port = 80 (HTTP)
TCP:  Sequence number = 4173603509
TCP:  Acknowledgement number = 170360723
TCP:  Data offset = 20 bytes
TCP:  Flags = 0x18
TCP:        0... .... = No ECN congestion window reduced
TCP:        .0.. .... = No ECN echo
TCP:        ..0. .... = No urgent pointer
TCP:        ...1 .... = Acknowledgement
TCP:        .... 1... = Push
TCP:        .... .0.. = No reset
TCP:        .... ..0. = No Syn
TCP:        .... ...0 = No Fin
TCP:  Window = 50400
TCP:  Checksum = 0xfda2
TCP:  Urgent pointer = 0
TCP:  No options
TCP:  
HTTP: ----- HyperText Transfer Protocol -----
HTTP: 
HTTP: <?xml version='1.0' encoding='UTF-8'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Header><AuthenticationInfo xmlns="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:userName xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">WSWEBSUN</SOAP-ENV:userName><SOAP-ENV:password xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">Kaiser22</SOAP-ENV:password></AuthenticationInfo></S:Header><S:Body><SearchIRByAssignedGroup xmlns="urn:HPD_HelpDesk_Query_IR_PR"><Group>ORACLE EBONDING</Group><Status>Assigned</Status></SearchIRByAssignedGroup></S:Body></S:Envelope>
HTTP: 

ETHER:  ----- Ether Header -----
ETHER:  
ETHER:  Packet 10246 arrived at 8:03:50.63842
ETHER:  Packet size = 262 bytes
ETHER:  Destination = 0:14:4f:46:59:44, 
ETHER:  Source      = 0:10:db:ff:20:a0, 
ETHER:  Ethertype = 0800 (IP)
ETHER:  
IP:   ----- IP Header -----
IP:   
IP:   Version = 4
IP:   Header length = 20 bytes
IP:   Type of service = 0x00
IP:         xxx. .... = 0 (precedence)
IP:         ...0 .... = normal delay
IP:         .... 0... = normal throughput
IP:         .... .0.. = normal reliability
IP:         .... ..0. = not ECN capable transport
IP:         .... ...0 = no ECN congestion experienced
IP:   Total length = 248 bytes
IP:   Identification = 39156
IP:   Flags = 0x4
IP:         .1.. .... = do not fragment
IP:         ..0. .... = last fragment
IP:   Fragment offset = 0 bytes
IP:   Time to live = 254 seconds/hops
IP:   Protocol = 17 (UDP)
IP:   Header checksum = c3ed
IP:   Source address = 140.85.106.29, itspdispatcher4.oracle.com
IP:   Destination address = 141.146.155.13, itspportal1.oracle.com
IP:   No options
IP:   
UDP:  ----- UDP Header -----
UDP:  
UDP:  Source port = 8088
UDP:  Destination port = 8088 
UDP:  Length = 228 
UDP:  Checksum = C54C 
UDP:  

ETHER:  ----- Ether Header -----
ETHER:  
ETHER:  Packet 10247 arrived at 8:03:50.63883
ETHER:  Packet size = 60 bytes
ETHER:  Destination = 0:14:4f:46:59:44, 
ETHER:  Source      = 0:10:db:ff:20:a0, 
ETHER:  Ethertype = 0800 (IP)
ETHER:  
IP:   ----- IP Header -----
IP:   
IP:   Version = 4
IP:   Header length = 20 bytes
IP:   Type of service = 0x00
IP:         xxx. .... = 0 (precedence)
IP:         ...0 .... = normal delay
IP:         .... 0... = normal throughput
IP:         .... .0.. = normal reliability
IP:         .... ..0. = not ECN capable transport
IP:         .... ...0 = no ECN congestion experienced
IP:   Total length = 40 bytes
IP:   Identification = 42410
IP:   Flags = 0x4
IP:         .1.. .... = do not fragment
IP:         ..0. .... = last fragment
IP:   Fragment offset = 0 bytes
IP:   Time to live = 253 seconds/hops
IP:   Protocol = 6 (TCP)
IP:   Header checksum = dcf3
IP:   Source address = 144.20.66.135, www-proxy-adc.us.oracle.com
IP:   Destination address = 141.146.155.3, itsptis1
IP:   No options
IP:   
TCP:  ----- TCP Header -----
TCP:  
TCP:  Source port = 80
TCP:  Destination port = 40991 
TCP:  Sequence number = 170360723
TCP:  Acknowledgement number = 4173604108
TCP:  Data offset = 20 bytes
TCP:  Flags = 0x10
TCP:        0... .... = No ECN congestion window reduced
TCP:        .0.. .... = No ECN echo
TCP:        ..0. .... = No urgent pointer
TCP:        ...1 .... = Acknowledgement
TCP:        .... 0... = No push
TCP:        .... .0.. = No reset
TCP:        .... ..0. = No Syn
TCP:        .... ...0 = No Fin
TCP:  Window = 5246
TCP:  Checksum = 0x582a
TCP:  Urgent pointer = 0
TCP:  No options
TCP:  
HTTP:  ----- HTTP:   -----
HTTP:  
HTTP:  ""
HTTP:

这个问题的背景是这样的：在PROD(MCSP)->QA(Kaiser)的测试被激活之后，我们发现每当adatper试图访问对方的web service时，都会得到HTTP 500的错误。

首先是怀疑指向对方web service的URL不对。但是检查后发现并没有问题。

在我们的UAT(MCSP)->QA(Kaiser)测试中，是没有这个问题的。而且同一时刻使用SoapUI访问也没有问题，但在PROD环境上就有问题。所以问题一定出在PROD环境的配置上。可是怎么看也没有看出来有什么配置错误。

然后就陷入了僵局。

后来我想到了用”network package analysis Tool”查看包内容，于是就有了这篇文章。而且我根据这个思路确实找到了问题的原因。

从包分析工具的输出中可以看到：我们发向Kaiser的http请求，目的地址是“www-proxy-adc.us.oracle.com”，这是我们的代理服务器地址，而不是kaiser server在我们vpn中的ip地址。换句话说：本来不应该使用proxy来访问的ip地址，实际上还是使用了proxy去访问。这就导致proxy server在处理请求的时候，找不到去往kaiser server IP的路由，于是它返回了HTTP 500错误。而造成问题的根本原因，是我们设置的”http.nonProxyHosts”属性中设置的”kaiser server IP”没有生效。在这个”system property”中：一共包含有五六个”server addresses”，其中还有3个重复了两遍。我不知道问题是出在这个”system property”的长度过长还是有重复内容导致错误。总之，去掉了重复内容并把”kaiser server IP”的地址放到了前面（不同地址之间用”|”分隔），重启Glassfish，就解决了这个问题。

Comments