Telefon: +47 73 55 79 60, Fax: +47 73 55 79 01
Postadresse: UNINETT driftssenter, N-7465 Trondheim
Besøksadresse: Tempeveien 22, 4. etg., Trondheim"; $mal_temabilde="teknisk"; // obligatorisk, se liste på // www.uninett.no/info/info/bilder.html $mal_blurb=""; // valgfritt (tekst under bilde) $mal_kontekst=""; // valgfritt $mal_overskrift="E2E topology and measurement thoughts"; // $mal_modus=""; // valgfritt $mal_tittel=""; // valgfritt, normalt blir tittelen // tjener: kontekst - overskrift - modus include "uninetttopp.php3"; ?>

Draft version 0.2

Background

This document gives some thoughts regarding an end to end (E2E) measurement infrastructure. It has a special emphasis on topology, i.e. the process of retrieving the end to end topology. Retrieving topology can in effect serve multiple purposes: Note: This document does not address AA. We assume authentication and authorization is incorporated into the model in an appropriate way.

We mention/incorporate on-demand/scheduled meters, although our focus is on measurement points on the path (E2E topology) in question. In essence, our model assumes that regularly scheduled measurements of various types are done from each point along the path. We introduce the term path points to distinguish these measurements points from meters, that typically reside on the side of the path, but still monitor the path, or part of it.

Task at hand

Machine M1 in Barcelona, Spain is communicating with machine M2 in Trondheim, Norway.

The problem is inter-domain, it includes a set of providers and two customer networks (at each end of the path).

Terminology

As far as possible we seek to adopt terminology in use and established elsewhere:

Prerequisites

Our path finder model has some prerequisites. We recognize that a similar study has the same prerequisites! And a suggested solution as well! See "[1] Cross-Domain Monitoring - Path Discovery". for details. We refere to the document in the following.

1) RD-trace

We assume there is a way of retrieving the router hop path, i.e. the traceroute between M1 and M2. We further assume there is a mechanism to map each router hop to an autonomous system (AS) number. We assume that each provider has an AS number.

In our example the trace may look like this:

  M1, G1, G2, G3, G4, G5, G6, G7, M2

  where Gn is a router described by an ip address, typically the 
  ip address of the ingress interface. In other words, the 
  trace resembles the data given by the traceroute tool.
In our example three domains (AS) are involved. The trace with domain info is:
  [               D1            ] [     D2      ] [         D3          ]    

  (M1,D1) (G1,D1) (G2,D1) (G3,D1) (G4,D2) (G5,D2) (G6,D3) (G7,D3) (M2,D3)
  
We denote this the RD-trace (Router,Domain)-trace.

Solution: [1] suggests to use the University of Oregon's DNS interface to the global BGP mesh. Example:

$ host -t txt 1.46.39.128.asn.routeviews.org
1.46.39.128.asn.routeviews.org text "224" "128.39.0.0" "16"
This shows that the router oslo-gw1.uninett.no belongs in AS 224 (which is the domain UNINETT's AS number).

2) PF and DM lookup

We also need a directory service in order to locate the path finders (PFs) and domain managers (DMs) og each domain. [1] suggests either WHOIS or DNS, see the document for details. The DNS solution is based on using the TXT records and is a solution that requires little work.

Alternatively a model where the PF and DM announce themselves through a multicast group might be feasable.

The Path Finder (PF)

Traceroute itself does not have detailed enough information to know what the interesting measurement data are for the path in question. It lacks details on what router interfaces are involved in the trace, and it does not have any layer 2 information. Further it does not have a clue on location.

The path finder typically has access to a management system that continously maintains the topology of the provider network. The detail and quality of the topology information may vary, ideally it should be a model of the running network. In any circumstance, the task of the PF is to return a list of measurement points (MP), on a best effort basis. This includes both path points and meters.

We denote the function that the PF serve as getMP. Input will be the RD-trace (Either the complete trace or the relevant part for the domain in question. The relevant part must include the next hop on each side of the domain). In our example we do a getMP at the PF of domain D2:

   synopsis: getMP [RD-trace]

   example:

      %getMP (G3,D1) (G4,D2) (G5,D2) (G6,D3) 
    	 PP1, PP2 (M1), PP3, PP4, PP5 (M2), PP6
In return we get a list of MPs (measurement points) that describes the path. Connected meters are shown in parenthesis.

Note: This is just an abstract notation meant as an example, actual implementation, how the function is invoked, synopsis, etc, is outside our scope.

The task of deducing relevant meters is complex, we reduce the complexity by listing all directly connected meters. Meters that are not directly connected (on layer 3) to a router along the path are not included. We have no idea if the actual measurements the meters do or can do are relevant for the problem at hand.

Anyway, in turn we can use the MP-list to get measurement data from the domain manager (DM) for the domain.

Measurement Points (MPs)

As mentioned measurement points may be routers, router interfaces, switches or switch interfaces or meters. In our example the MP-list for P2 will typically be (ascii art for the time being):
      G4             G5	
   
  [>] [€] [<]    [>] [€] [<]
  PP1 PP2 PP3    PP4 PP5 PP6 
       |              |
       M1             M2

  synopsis:
     [>]  ingress interface of router X
     [€]  router X
     [<]  egress interface of router X
      |   directly connected meter 
In our example the two routers are directly connected. If there is an intermediate layer 2 topology involved we would have more observation points, i.e. switches and switch ports.

Typical attributes of a measurement point are (here a router interface):

   attribute		example
   ----------------------------------------------------
   name			trd-oslo
   domain_name		uninett
   ip address		128.39.46.2
   layer		3
   entity		port	
   ifindex		12
   contained_in		trd-gw
   location_utm		- (see contained in's location)
   location_descr	- (see contained_in's location)
   description		...
   ----------------------------------------------------

A chassis example:

   attribute		example
   ----------------------------------------------------
   name			trd-gw
   domain_name		uninett
   ip address		128.39.0.82
   layer		3
   entity		chassis	
   ifindex		-
   contained_in		-
   location_utm		63 24 50.807 N 10 24 21.387 E
   location_descr	U4-064,Math building, NTNU, ...
   description		...
   ----------------------------------------------------

The domain manager (DM)

The DM maintains an index of all measurement data for all the measurement points within the domain. The data can be categorized by type and set. Some examples are:

   type			set		relevant entity	
   ----------------------------------------------------
   interfCounters	octets		port
   interfCounters	packets		port
   interfCounters	errors		port
   interfCounters	drops		port
   netflow		IP source	port
   netflow		IP dest		port
   netflow		IP src-dest	port
   netflow		IP port source	port
   ...
   chassis		cpuload		chassis
   chassis		memory		chassis
   trip			roundtrip delay	port/chassis
   trip			roundtrip loss	port/chassis
   trip			delay variation port/chassis
   ...
   ----------------------------------------------------
The DM should answer to a function getDataSets. Which returns a table of datasets with there name and location:
   synopsis: getDataSets [MP]

   example
      %getDataSets trd-oslo@uninett

        set	location
	--------------------------------------------------------
	octet	http://drift.uninett.no/stat-q/plot-all/trd-oslo
	IP src	http://stager.uninett.no/...
	...
Note: This merely points out the idea, an URL is of course not sufficient. Web services, XML and SOAP may be appropriate. Or CORBA. In any circumstance, we need to know the location of the data. We probably also need a common way of interpreting (and thus processing) the data in the data sets. This could i.e. allow for combining data from different observation points from different domains.

Manager of Managers

Typically a provider of a domain does not manage the networks of its customers. The provider can only provide topology information to the customer edge router. Within the customer network there may be several router and switch hops involved before we reach the end machine in question.

If the customer is large he may have an independent AS number. In that case he should provide a "globally visable" PF and DM.

In many cases the customer is an integral part of the provider AS number. In this case we suggest a solution of manager of managers. The PF and DM for the provider answers to getMP and getDataSet for the whole AS, including the customer networks. When a given customer network is involved in the trace the provider PF asks the (locally visable) customer PF of topology details. Similarly for the DM. In turn he returns the complete picture to the inquirer.

As a consequence the provider PF and DM need to maintain a local list of all his customer's PFs and DMs. He also needs to query all the customer PFs if they are involved in the path in question. Alternatively, he may derive this from his own topology information, this depends on how "rich" topology information is maintained.

In the primer case, and anyway useful for other purposes, a boolean function manageIP should be supported by all the PFs:

   synopsis: manageIP [IP address]

   example (requested toward NTNU's PF):

      %manageIP 129.241.190.190
	true
Note that manegeIP and getMP may be usefull for the purpose of machine tracking within a provider network. If all custumers provide a PF it will be possible to effectively locate a given machine within the provider network at any time. The machine can be located down to physical location, UTM coordinates or more roughly. This depends on the quality of the given customers PF.

NAV - a possible PF

NAV maintains a model of the managed network and can serve as a PF. NAV can retrieve MPs on layer three and layer two. It has information on location for all routers and switches and, if maintained by the local NAV administrator, it has information on the actual location (i.e room number) of all connected machines in the network (the requirement being that the cross connects / patches from switch ports to jacks is maintained).

In NAV meters can be categorized as a server subcategory.

NAV can easily be expanded to serve the manager of manager model and thus fulfill the provider wide topology requirement.

NAV or Stager as DM / measurement contribitors

NAV provides an integral solution with Cricket and RRD for data collection. NAV has an RRD meta database containing meta information of all collected statistics. This could serve as a DM. Or it could be a provider for a more encompassing DM that uses NAV for some of the MP data.

Stager could complement NAV, or Stager could be a complete DM. Todays implementation of Stager supports netflow data. In the future "any" measurement data can be collected with Stager. Stager is designed around the concept of observation points, which suits our model well. An observation point is in effect the same as a measurement point.

faltin, 29/11-04