Please pardon the wide distribution of this message, but considering the timeliness of the issue at hand, I wanted to reach the right audience as quickly as possible to explore the issue while it's still in it's prime. This has to do with both some research issues as well as commercial use and provision of Internet services. I've been doing some research on network paths, delay and propagation on the Internet. As many of you know, today's Internet isn't the neat, simple thing that it was just 5 years ago. That Internet could be described as a mostly hierarchal system of networks, and used relatively simple routing protocols, such as RIP and EGP to maintain global connectivity. The connectivity on the Internet today is much more complex and considerably less easy to model. There is no "backbone" network which provides transit at the top level of any hierarchy. The world isn't just a simple tree. Today's network is a complex, world-wide "web" of connectivity between different providers of Internet connectivity and service. There are networks run by commercial concerns, non-profit entities, government agencies and other interested parties. I was attempting to characterize some of these connectivity paths using the popular and familiar "traceroute" tool. This is a handy tool for investigating the routing which packets take on their way to a specified destination. This is done by transmitting "probe" packets addressed to the destination, with varying TTL (Transit Time Limit) values, and awaiting the response of ICMP (IP Control Message Pings) responses. (ICMP is normally used by the "PING" tools to measure response times). In the process of doing this investigation, I have encountered what appears to be abnormal connectivity paths to destinations. These paths are *very* unusual in that the paths is highly dependent on the destination address; even minor changes in the destination address seem to have wildly different effects on the path. It is as if there is some effects on the path at the "quantum" level, where even minor differences in the host path seem to have path warping effects. My initial thought was that perhaps there was some experimental deployment of new routing protocols causing difficulties, such as S-DRIP, but further investigation reveals that the source of the traceroute pings have no effect once the paths crosses a point which seems to be a discontinuity in the routing fabric. Further, some of the probed paths seems to cross normal paths on well-known provider's networks. That is, a path which would normally proceed across (e.g.,) SprintLink would be intertwined with a path across (e.g.,) ANS. Consider this one: $ traceroute 144.202.90.1 traceroute to 144.202.90.1 (144.202.90.1), 30 hops max, 38 byte packets 1 en-0.ENSS136.t3.ANS.NET (192.41.177.253) 60 ms 29 ms 28 ms 2 t3-0.cnss58.Washington-DC.t3.ans.net (140.222.58.1) 116 ms 73 ms 93 ms 3 sl-mae-e-F0/0.sprintlink.net (192.41.177.241) 186 ms 164 ms 187 ms 4 sl-dc-8-H1/0-T3.sprintlink.net (144.228.10.41) 280 ms 257 ms 279 ms 5 t3-1.cnss72.Greensboro.t3.ans.net (140.222.72.2) 364 ms 351 ms 366 ms 6 sl-dc-7-F0.sprintlink.net (144.228.20.7) 455 ms 445 ms 460 ms 7 t3-0.cnss104.Atlanta.t3.ans.net (140.222.104.1) 561 ms 538 ms 553 ms 8 t3-1.cnss16.Los-Angeles.t3.ans.net (140.222.16.2) 656 ms 633 ms 647 ms 9 t3-2.cnss64.Houston.t3.ans.net (140.222.64.3) 735 ms 729 ms 742 ms 10 sl-starnet-1-S0-T1.sprintlink.net (144.228.27.6) 828 ms 808 ms 834 ms 11 t3-2.cnss8.San-Francisco.t3.ans.net (140.222.8.3) 937 ms 916 ms 931 ms 12 1.90.202.144.in-addr.arpa (144.202.90.1) 1030 ms 1006 ms 1021 ms Could this be the case of some mutual cooperation between ANS and SprintLink? This seems highly unlikely, in that core backbone links seem to be involved. Could this be a problem with the underlying transmission fabric? This, too, seems unlikely given that the Cisco routers on Sprintlink would likely have an incompatible link-level protocol with the RS6000 routers used by ANS. Besides, the TOPS20-like operating system in the Ciscos are incompatible with the AIX OS used on the IBM platforms. Thinking that this was simply "one of those weird-ass Internet things", I probed a different address: $ traceroute 144.202.90.2 traceroute to 144.202.90.2 (144.202.90.2), 30 hops max, 38 byte packets 1 Alternet-GW.TransSys.COM (144.202.42.1) 63 ms 28 ms 26 ms 2 Falls-Church1.VA.ALTER.NET (137.39.3.33) 93 ms 99 ms 93 ms 3 Falls-Church4.VA.ALTER.NET (137.39.8.1) 187 ms 165 ms 188 ms 4 IBMpc01.UU.NET (153.39.1.1) 280 ms 257 ms 272 ms 5 IBMpc02.UU.NET (153.39.1.2) 374 ms 352 ms 366 ms 6 ftp.UU.NET (192.48.96.9) 455 ms 444 ms 461 ms 7 ftp.microsoft.com (198.105.232.1) 561 ms 523 ms 554 ms 8 hubbub.cisco.com (198.92.30.32) 641 ms 635 ms 648 ms 9 149.20.64.4 (149.20.64.4) 749 ms 663 ms 740 ms 10 2.90.202.144.in-addr.arpa (144.202.90.2) 842 ms 818 ms 833 ms Now this was weird! The path to the destination seems to be diverted by "ruts" in the Information Superhighway! That is, the significant traffic to some high-volume FTP servers seems to "steer" the traffic off its normal "lane". Note FTP.UU.NET, FTP.MICROSOFT.COM, hubub.cisco.com which is also known as FTP.CISCO.COM. Well, I was on to something now. I tried to probe another address on this network to see if I could further understand the problem: $ traceroute 144.202.90.3 traceroute to 144.202.90.3 (144.202.90.3), 30 hops max, 38 byte packets 1 Alternet-GW.TransSys.COM (144.202.42.1) 60 ms 28 ms 27 ms 2 Falls-Church1.VA.ALTER.NET (137.39.3.33) 115 ms 70 ms 93 ms 3 Falls-Church4.VA.ALTER.NET (137.39.8.1) 187 ms 164 ms 189 ms 4 Vienna3.VA.ALTER.NET (137.39.11.4) 281 ms 259 ms 272 ms 5 134.222.84.1 (134.222.84.1) 366 ms 304 ms 367 ms 6 iij-gate.iij.net (192.244.176.209) 453 ms 444 ms 459 ms 7 whitehouse.gov (198.137.241.30) 548 ms 539 ms 554 ms 8 archie.iij.ad.jp (192.244.176.60) 641 ms 632 ms 647 ms 9 jatz.aarnet.edu.au (139.130.204.4) 734 ms 878 ms 687 ms 10 DOCKMASTER.NCSC.MIL (26.1.0.172) 829 ms 806 ms 835 ms 11 ns.kreonet.re.kr (134.75.30.1) 929 ms 883 ms 930 ms 12 ard.FBI.GOV (192.108.246.6) 1010 ms 1009 ms 1030 ms 13 oxygen.house.gov (137.18.128.6) 1129 ms 1093 ms 1117 ms 14 3.90.202.144.in-addr.arpa (144.202.90.3) 1218 ms 1183 ms 1209 ms Now, things are getting truly bizzare. What originally seemed to be a "local", US phenomenon seems to have broadened into something of truly world-wide scope. Here is also the first evidence of carriage of traffic by governmental agencies, which claim to be out of the business of subsidizing internet backbones. Further, we see some very weird physical behavior manifesting itself in the temporal domain. Notice how the measured delta delay times seems to be approximately 89 milliseconds per hop, regardless of the distance between the two points. It is as if there are hyper-spatial paths with constant propagation times independent of the distance which connect these sites. This is much the same as hyperlinks on HTML documents take readers from one document to another, anywhere on the planet. I made another probe to attempt to characterize this "tear" in the routing fabric of the Internet: $ traceroute 144.202.90.4 traceroute to 144.202.90.4 (144.202.90.4), 30 hops max, 38 byte packets 1 en-0.ENSS136.t3.ANS.NET (192.41.177.253) 59 ms 28 ms 28 ms 2 t3-0.cnss58.Washington-DC.t3.ans.net (140.222.58.1) 94 ms 71 ms 95 ms 3 sl-mae-e-F0/0.sprintlink.net (192.41.177.241) 187 ms 162 ms 187 ms 4 sl-dc-8-H1/0-T3.sprintlink.net (144.228.10.41) 281 ms 257 ms 272 ms 5 border2-hssi4-0.Washington.mci.net (204.70.57.9) 370 ms 274 ms 366 ms 6 t3-1.cnss72.Greensboro.t3.ans.net (140.222.72.2) 453 ms 447 ms 460 ms 7 sl-dc-7-F0.sprintlink.net (144.228.20.7) 562 ms 539 ms 553 ms 8 t3-0.cnss104.Atlanta.t3.ans.net (140.222.104.1) 656 ms 633 ms 647 ms 9 core-hssi-3.SanFrancisco.mci.net (204.70.1.38) 735 ms 648 ms 742 ms 10 t3-2.cnss64.Houston.t3.ans.net (140.222.64.3) 829 ms 820 ms 835 ms 11 pacnap-e0/0.net99.net (204.157.38.2) 923 ms 915 ms 929 ms 12 sl-starnet-1-S0-T1.sprintlink.net (144.228.27.6) 1019 ms 995 ms 1027 ms 13 core-hssi-2.Seattle.mci.net (204.70.1.49) 1112 ms 1028 ms 1119 ms 14 t3-2.cnss8.San-Francisco.t3.ans.net (140.222.8.3) 1206 ms 1196 ms 1210 ms 15 4.90.202.144.in-addr.arpa (144.202.90.4) 1300 ms 1287 ms 1303 ms Now, this is a very similar path to the first attempt. But notice how a slight change in the address, that of the last octet changing from 1 to 4 made?! Notice that the total number of bits in the address did not change, only the position of the last bit. Yet this slight change causes further anomalies in the path. It is as if the packet is caught in some wormhole-like discontinuity in the link-space-time continuum of the Internet; the packet is on a link, minding it's own business, when it suddenly experiences some warp effect and is transported on to another link on the Internet. These are the only addresses which I've discovered, to date, which experience these bizzare effects. Perhaps it's the pattern of the address bits, when converted into symbols for transmission over common transmission facilities which provokes this effect via some yet to be understood or determined mechanism. I invite you, the interested reader, to probe this stable phenomenon with traceroute in further attempts to understand what's going on. This is a strange time for this to be happening on the Internet, where we have better routing protocols and technologies than at any other time in the past. Louis Mamakos I thought you might enjoy this April Fool's Day joke from the IETF mailing list.