Truncation - Revision history

Nishal: fixed formatting for block diagrams by using

2024-01-18T13:07:11Z

fixed formatting for block diagrams by using <pre>

← Older revision		Revision as of 13:07, 18 January 2024
Line 17:		Line 17:

	A sample flow diagram showing the typical collection process might look like this:		A sample flow diagram showing the typical collection process might look like this:
			<pre>
	+-----------+ +-------------------+ +----------------+ +------------------+		+-----------+ +-------------------+ +----------------+ +------------------+
	\| IX Switch \| ----> \| Centralized \| ----> \| sflowtool \| ----> \| IIFQ \|		\| IX Switch \| ----> \| Centralized \| ----> \| sflowtool \| ----> \| IIFQ \|
Line 24:		Line 24:
	+-------------------+ +----------------+ +------------------+		+-------------------+ +----------------+ +------------------+

			</pre>
	In this diagram:		In this diagram:
	* IX Switch: The starting point where flow data is generated.		* IX Switch: The starting point where flow data is generated.

Nishal: added in additional text to show details of truncation.

2024-01-18T13:06:24Z

added in additional text to show details of truncation.

← Older revision		Revision as of 13:06, 18 January 2024
Line 1:		Line 1:
	Participating Internet Exchange Points (IXPs) may use open-source software which has been published by the South African ISP Association (operators of the Johannesburg and Cape Town IXPs) to truncate IPv4 addresses in flow exports to the nearest enclosing /24, and IPv6 addresses to the nearest enclosing /48. This has no effect on the resulting statistics, and ensures that no individual IP addresses (which may be categorized as Personally Identifiable Information, or PII, in some jurisdictions) are transmitted outside the perimeter of the IXP's control and responsibility.		Participating Internet Exchange Points (IXPs) may use open-source software which has been published by the South African ISP Association (operators of the Johannesburg and Cape Town IXPs) to truncate IPv4 addresses in flow exports to the nearest enclosing /24, and IPv6 addresses to the nearest enclosing /48. This has no effect on the resulting statistics, and ensures that no individual IP addresses (which may be categorized as Personally Identifiable Information, or PII, in some jurisdictions) are transmitted outside the perimeter of the IXP's control and responsibility.

			SFlow and NetFlow packet structures inherently exclude any packet payload data. Therefore, basic flow exports don't allow for the reconstruction of detailed IP transaction information. Nonetheless, for operators desiring enhanced privacy, the option exists to anonymise flow exports. This is achieved by converting the last 8 bits of IPv4 addresses and the last 80 bits of IPv6 addresses to zero. Such masking enables aggregation at the network level (up to /24 for IPv4 and /48 for IPv6), further safeguarding individual end-user IP addresses. This method of anonymisation works up to the generally permissible routing boundaries, observed by netwokr operators, and, importantly, does not impact the overall statistical analysis whilst ensuring that individual IP addresses, potentially classified as Personally Identifiable Information (PII) in certain jurisdictions, are never disseminated beyond the IXP's management domain.

			In a collaborative initiative, open-source developers from Cyberstorm.mu, in partnership with the community-run Internet exchange points in South Africa, have developed extensions for the widely-used flow export tool 'sflowtool'. By activating an optional switch (-a) for anonymisation, the tool obscures the trailing bits of IP addresses in the flow data.

			For instance, a flow initially recorded as
			198.51.100.123 <-> 203.0.113.112
			would be transformed to
			198.51.100.0 <-> 203.0.113.0.

			Similarly, an IPv6 flow like
			2001:db8:100:d351:b0b3:193a:9087:ea35 <-> 2001:db8:900:3193:3fb7:55a7:dcda:ccaf
			would be anonymised to
			2001:db8:100::0 <-> 2001:db8:900::0.


			A sample flow diagram showing the typical collection process might look like this:

			+-----------+ +-------------------+ +----------------+ +------------------+
			\| IX Switch \| ----> \| Centralized \| ----> \| sflowtool \| ----> \| IIFQ \|
			\| \| \| Collector \| \| Utility \| \| Collector \|
			+-----------+ \| (Data Aggregation)\| \| (Anonymization)\| \| (Data Analysis) \|
			+-------------------+ +----------------+ +------------------+

			In this diagram:
			* IX Switch: The starting point where flow data is generated.
			* Centralized Collector: Receives data from the IX switch. This is where initial data aggregation may occur by the IX for internal analysis..
			* sflowtool Utility: Processes the data from the Centralized Collector, applying anonymisation to the IP addresses.
			* External Collector: The final destination of the data, where it is analysed by the IIFQ collection software.

			The code for this extension is currently available at https://github.com/cyberstormdotmu/sflowtool and is pending integration into the upstream software.

Woody: Created page with "Participating Internet Exchange Points (IXPs) may use open-source software which has been published by the South African ISP Association (operators of the Johannesburg and Cape Town IXPs) to truncate IPv4 addresses in flow exports to the nearest enclosing /24, and IPv6 addresses to the nearest enclosing /48. This has no effect on the resulting statistics, and ensures that no individual IP addresses (which may be categorized as Personally Identifiable Information, or PII,..."

2023-11-20T09:48:09Z

Created page with "Participating Internet Exchange Points (IXPs) may use open-source software which has been published by the South African ISP Association (operators of the Johannesburg and Cape Town IXPs) to truncate IPv4 addresses in flow exports to the nearest enclosing /24, and IPv6 addresses to the nearest enclosing /48. This has no effect on the resulting statistics, and ensures that no individual IP addresses (which may be categorized as Personally Identifiable Information, or PII,..."

New page

Participating Internet Exchange Points (IXPs) may use open-source software which has been published by the South African ISP Association (operators of the Johannesburg and Cape Town IXPs) to truncate IPv4 addresses in flow exports to the nearest enclosing /24, and IPv6 addresses to the nearest enclosing /48. This has no effect on the resulting statistics, and ensures that no individual IP addresses (which may be categorized as Personally Identifiable Information, or PII, in some jurisdictions) are transmitted outside the perimeter of the IXP's control and responsibility.