Nick Mathewson 3e3b9219ec Merge remote-tracking branch 'karsten/geoip-manual-update-oct2013' 11 years ago
..
README.geoip 2bf195d0ce Add script to fix "A1" entries in geoip file. 12 years ago
deanonymind.py d93caf7f0a Update GeoIP file script and manual substitutions. 11 years ago
geoip 3b0265198f Update to the October 2013 GeoIP database. 11 years ago
geoip-manual f4ef7c76f3 Modify 1 and delete 1 geoip A1 substitution. 11 years ago
geoip6 96a7313475 Update geoip6. 12 years ago
include.am b17aa28e46 Actually distribute the geoip6 file. 12 years ago
torrc.sample.in 3b6c48328b Or maybe it is 2013. Who can say? 11 years ago

README.geoip

README.geoip -- information on the IP-to-country-code file shipped with tor
===========================================================================

The IP-to-country-code file in src/config/geoip is based on MaxMind's
GeoLite Country database with the following modifications:

- Those "A1" ("Anonymous Proxy") entries lying inbetween two entries with
the same country code are automatically changed to that country code.
These changes can be overriden by specifying a different country code
in src/config/geoip-manual.

- Other "A1" entries are replaced with country codes specified in
src/config/geoip-manual, or are left as is if there is no corresponding
entry in that file. Even non-"A1" entries can be modified by adding a
replacement entry to src/config/geoip-manual. Handle with care.


1. Updating the geoip file from a MaxMind database file
-------------------------------------------------------

Download the most recent MaxMind GeoLite Country database:
http://geolite.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip

Run `python deanonymind.py` in the local directory. Review the output to
learn about applied automatic/manual changes and watch out for any
warnings.

Possibly edit geoip-manual to make more/fewer/different manual changes and
re-run `python deanonymind.py`.

When done, prepend the new geoip file with a comment like this:

# Last updated based on $DATE Maxmind GeoLite Country
# See README.geoip for details on the conversion.


2. Verifying automatic and manual changes using diff
----------------------------------------------------

To unzip the original MaxMind file and look at the automatic changes, run:

unzip GeoIPCountryCSV.zip
diff -U1 GeoIPCountryWhois.csv AutomaticGeoIPCountryWhois.csv

To look at subsequent manual changes, run:

diff -U1 AutomaticGeoIPCountryWhois.csv ManualGeoIPCountryWhois.csv

To manually generate the geoip file and compare it to the automatically
created one, run:

cut -d, -f3-5 < ManualGeoIPCountryWhois.csv | sed 's/"//g' > mygeoip
diff -U1 geoip mygeoip


3. Verifying automatic and manual changes using blockfinder
-----------------------------------------------------------

Blockfinder is a powerful tool to handle multiple IP-to-country data
sources. Blockfinder has a function to specify a country code and compare
conflicting country code assignments in different data sources.

We can use blockfinder to compare A1 entries in the original MaxMind file
with the same or overlapping blocks in the file generated above and in the
RIR delegation files:

git clone https://github.com/ioerror/blockfinder
cd blockfinder/
python blockfinder -i
python blockfinder -r ../GeoIPCountryWhois.csv
python blockfinder -r ../ManualGeoIPCountryWhois.csv
python blockfinder -p A1 > A1-comparison.txt

The output marks conflicts between assignments using either '*' in case of
two different opinions or '#' for three or more different opinions about
the country code for a given block.

The '*' conflicts are most likely harmless, because there will always be
at least two opinions with the original MaxMind file saying A1 and the
other two sources saying something more meaningful.

However, watch out for '#' conflicts. In these cases, the original
MaxMind file ("A1"), the updated MaxMind file (hopefully the correct
country code), and the RIR delegation files (some other country code) all
disagree.

There are perfectly valid cases where the updated MaxMind file and the RIR
delegation files don't agree. But each of those cases must be verified
manually.