Add an offline mode to chutney, which disables DNS

Using --offline makes chutney much more reliable, and removes the
dependency on an external DNS resolver. Chutney automatically does this
when the default /etc/resolv.conf file is missing. To use offline mode
by default, set CHUTNEY_DNS_CONF="/dev/null".

This change makes chutney ignore tor's compile-time ServerDNSResolvConfFile
default. If you rely on a custom default for this option (which isn't
/etc/resolv.conf), use --dns-conf-default or CHUTNEY_DNS_CONF="".

To use a custom DNS file, use --dns-conf PATH or CHUTNEY_DNS_CONF="PATH".

Closes ticket 21903, and other related tickets.

README

@@ -48,9 +48,14 @@ Traffic Options:
   --rounds           CHUTNEY_ROUNDS=N
   --hs-multi-client  CHUTNEY_HS_MULTI_CLIENT=N
 
-Address Options:
+Address/DNS Options:
   --ipv4             CHUTNEY_LISTEN_ADDRESS
   --ipv6             CHUTNEY_LISTEN_ADDRESS_V6
+  # Chutney uses /etc/resolv.conf if none of these options are set
+  --dns-conf         CHUTNEY_DNS_CONF=PATH
+  --offline          CHUTNEY_DNS_CONF=/dev/null
+  # Use tor's compile-time default for ServerDNSResolvConfFile
+  --dns-conf-default CHUTNEY_DNS_CONF=""
 
 Warning Options:
   --all-warnings     CHUTNEY_WARNINGS_IGNORE_EXPECTED=false
@@ -149,6 +154,33 @@ Changing the network address:
    chutney verifies IPv6 client, bridge client (?), hidden service, and exit
    connections. It does not use IPv6 SOCKSPorts or HiddenServicePorts.
 
+Using DNS:
+
+   Chutney verify uses IP addresses by default. It does not need to look up
+   any hostnames. We recommend that chutney users disable DNS using --offline
+   or CHUTNEY_DNS_CONF=/dev/null , because any DNS failures causes tests to
+   fail. Chutney's DNS queries also produce external traffic in a predictable
+   pattern.
+
+   If you want to use a hostname with CHUTNEY_LISTEN_ADDRESS[_V6], or you want
+   to run tests that use DNS, set CHUTNEY_DNS_CONF to the path to a file in
+   resolv.conf format. Chutney's default of /etc/resolv.conf should be fine for
+   most UNIX-based operating systems. If your tor is compiled with a different
+   default, use --dns-resolv-conf-default or CHUTNEY_DNS_CONF="".
+
+   When the CHUTNEY_DNS_CONF file does not exist, or is a broken symlink,
+   chutney uses /dev/null instead. This is a workaround for bugs in tor's
+   use of eventdns. For example, macOS deletes the resolv.conf file when it
+   thinks the network is down: this can make tor exits reject all traffic,
+   even if a working DNS server is running on 127.0.0.1:53.
+
+   When tor has no working name servers (including --offline mode), it can
+   crash on SETCONF. (Chutney does not use SETCONF, but some external tor
+   controllers do.) To avoid this crash, set CHUTNEY_DNS_CONF to a file
+   containing a working name server address. For your convenience, chutney
+   provides a local resolv.conf file containing IPv4, IPv6, and "localhost".
+   Use --dns-conf resolv.conf (relative paths work).
+
 The configuration files:
   networks/basic holds the configuration for the network you're configuring
   above.  It refers to some torrc template files in torrc_templates/.

lib/chutney/TorNet.py

@@ -756,6 +756,10 @@ DEFAULTS = {
     'controlling_pid': (int(os.environ.get('CHUTNEY_CONTROLLING_PID', 0))
                         if 'CHUTNEY_CONTROLLING_PID' in os.environ
                         else None),
+    # a DNS config file (for ServerDNSResolvConfFile)
+    'dns_conf': (os.environ.get('CHUTNEY_DNS_CONF', '/etc/resolv.conf')
+                        if 'CHUTNEY_DNS_CONF' in os.environ
+                        else None),
 }
 
 
@@ -775,6 +779,10 @@ class TorEnviron(chutney.Templating.Environ):
           hs_hostname: the hostname of the key generated by a hidden service
           owning_controller_process: the __OwningControllerProcess torrc line,
              disabled if tor should continue after the script exits
+          server_dns_resolv_conf: the ServerDNSResolvConfFile torrc line,
+             disabled if tor should use the default DNS conf.
+             If the dns_conf file is missing, this option is also disabled:
+             otherwise, exits would not work due to tor bug #21900.
 
        Environment fields used:
           nodenum: chutney's internal node number for the node
@@ -792,6 +800,8 @@ class TorEnviron(chutney.Templating.Environ):
           hs-hostname (note hyphen): cached hidden service hostname value
           controlling_pid: the PID of the controlling process. After this
              process exits, the child tor processes will exit
+          dns_conf: the path to a DNS config file for Tor Exits. If this file
+             is empty or unreadable, Tor will try 127.0.0.1:53.
     """
 
     def __init__(self, parent=None, **kwargs):
@@ -867,6 +877,34 @@ class TorEnviron(chutney.Templating.Environ):
         else:
             return ocp_line
 
+    # the default resolv.conf path is set at compile time
+    # there's no easy way to get it out of tor, so we use the typical value
+    DEFAULT_DNS_RESOLV_CONF = "/etc/resolv.conf"
+    # if we can't find the specified file, use this one as a substitute
+    OFFLINE_DNS_RESOLV_CONF = "/dev/null"
+
+    def _get_server_dns_resolv_conf(self, my):
+        if my['dns_conf'] == "":
+            # if the user asked for tor's default
+            return "#ServerDNSResolvConfFile using tor's compile-time default"
+        elif my['dns_conf'] is None:
+            # if there is no DNS conf file set
+            print("CHUTNEY_DNS_CONF not specified, using '%s'."
+                  % (DEFAULT_DNS_RESOLV_CONF))
+            dns_conf = DEFAULT_DNS_RESOLV_CONF
+        else:
+            dns_conf = my['dns_conf']
+        dns_conf = os.path.abspath(my['dns_conf'])
+        # work around Tor bug #21900, where exits fail when the DNS conf
+        # file does not exist, or is a broken symlink
+        # (os.path.exists returns False for broken symbolic links)
+        if not os.path.exists(dns_conf):
+            # Issue a warning so the user notices
+            print("CHUTNEY_DNS_CONF '%s' does not exist, using '%s'."
+                  % (dns_conf, OFFLINE_DNS_RESOLV_CONF))
+            dns_conf = OFFLINE_DNS_RESOLV_CONF
+        return "ServerDNSResolvConfFile %s" % (dns_conf)
+
 
 class Network(object):
 

resolv.conf

@@ -0,0 +1,4 @@
+# Use localhost as the resolver
+127.0.0.1
+::1
+localhost

tools/ignore.warnings

@@ -19,6 +19,8 @@ Consensus with empty bandwidth
 Could not add queued signature to new consensus: Mismatched digest
 Could not add queued signature to new consensus: Valid-After times do not match
 Could not open.*sr-state.*No such file or directory
+# Chutney does not use DNS by default
+Couldn't set up any working nameservers. Network not up yet
 Currently, sandboxing is only implemented on Linux
 # We ignore consensus failure warnings
 Error publishing .* consensus
@@ -50,6 +52,8 @@ TestingTorNetwork is set
 # Older versions might need them, we should remove them at some point in 0.3.*
 The DirAuthority options 'hs' and 'no-hs' are obsolete
 This copy of Tor was compiled.*to run in a non-anonymous mode
+# Chutney does not use DNS by default
+Unable to parse '.*', or no nameservers in '.*'
 # Tor Bug 21525?
 Unable to store signatures posted by .* Mismatched digest
 Unable to store signatures posted by .* Valid-After times do not match

tools/test-network.sh

@@ -15,6 +15,10 @@ export CHUTNEY_WARNINGS_SUMMARY=${CHUTNEY_WARNINGS_SUMMARY:-true}
 # default to exiting when this script exits
 export CHUTNEY_CONTROLLING_PID=${CHUTNEY_CONTROLLING_PID:-$$}
 
+# default to no DNS: this is a safe, working default for most users
+# If a custom test expects DNS, it needs to set CHUTNEY_DNS_CONF
+export CHUTNEY_DNS_CONF=${CHUTNEY_DNS_CONF:-/dev/null}
+
 # what we say when we fail
 UPDATE_YOUR_CHUTNEY="Please update your chutney using 'git pull'."
 
@@ -111,6 +115,21 @@ do
       export CHUTNEY_LISTEN_ADDRESS_V6="$2"
       shift
       ;;
+    # The DNS server config for Tor Exits. Chutney's default is
+    # /etc/resolv.conf, even if tor's compile time default is different.
+    --dns-conf)
+      export CHUTNEY_DNS_CONF="$2"
+      shift
+      ;;
+    # Do not make any DNS queries. This is incompatible with external
+    # controllers that use SETCONF.
+    --offline)
+      export CHUTNEY_DNS_CONF="/dev/null"
+      ;;
+    # Use tor's compile-time default for ServerDNSResolvConfFile.
+    --dns-conf-default)
+      export CHUTNEY_DNS_CONF=""
+      ;;
     # Warning Options
     # we summarise unexpected warnings by default
     # this shows all warnings per-node
@@ -158,6 +177,12 @@ do
   shift
 done
 
+# If the DNS server doesn't work, tor exits may reject all exit traffic, and
+# chutney may fail
+if [ "$CHUTNEY_WARNINGS_ONLY" != true ]; then
+  $ECHO "$myname: using CHUTNEY_DNS_CONF '$CHUTNEY_DNS_CONF'"
+fi
+
 # optional: $TOR_DIR is the tor build directory
 # it's used to find the location of tor binaries
 # if it's not set:

torrc_templates/relay-non-dir.tmpl

@@ -15,3 +15,14 @@ ExitRelay 0
 # then half the minimum testing consensus interval
 TestingServerDownloadSchedule 0, 5
 TestingServerConsensusDownloadSchedule 0, 5
+
+# These options are set here so they apply to IPv4 and IPv6 Exits
+#
+# Tell Exits to avoid using DNS: otherwise, chutney will fail if DNS fails
+# (Chutney only accesses 127.0.0.1 and ::1, so it doesn't need DNS)
+ServerDNSDetectHijacking 0
+ServerDNSTestAddresses
+# If this option is /dev/null, or any other empty or unreadable file, tor exits
+# will not use DNS. Otherwise, DNS is enabled with this config.
+# (If the following line is commented out, tor uses /etc/resolv.conf.)
+${server_dns_resolv_conf}