r12257@catbus: nickm | 2007-03-20 23:17:52 -0400

 Add specification to proposal 104; settle on "extra info" approach rather than "long descriptor" approach.


svn:r9890

doc/spec/proposals/104-short-descriptors.txt

@@ -9,9 +9,7 @@ Status: Open
 Overview:
 
   This document proposes moving unused-by-clients information from regular
-  router descriptors into a special "long form" router descriptor.
-
-  It presents options; it is not yet a complete proposal.
+  router descriptors into a new "extra info" router descriptor.
 
 Proposal:
 
@@ -20,23 +18,21 @@ Proposal:
   "write-history" fields are used only by the authorities for monitoring the
   status of the network.  If we took them out, the size of a compressed list
   of all the routers would fall by about 60%.  (No other disposable field
-  would save more than 2%.)
-
-  One possible solution here is that routers should generate and upload a
-  short-form and long-form descriptor.  Only the short-form descriptor should
-  ever be used by anybody for routing.  The long-form descriptor should be
-  used only for analytics and other tools.  (If we allowed people to route
-  with long descriptors, we'd have to ensure that they stayed in sync with
-  the short ones somehow.  So let's not do that.)  We can ensure that the
-  short descriptors are used by only recommending those in the network
-  statuses.
-
-  Another possible solution would be to drop these fields from descriptors,
-  and have them uploaded as a part of a separate "bandwidth report" to the
-  authorities.  This could help prevent the mistake of using long descriptors
-  in the place of short ones. It could also be generalized later to be an
-  overall status report, to include sanitized GeoIP information and whatever
-  else comes up.
+  would save much more than 2%.)
+
+  We propose to remove these fields from descriptors, and and have them
+  uploaded as a part of a separate signed "extra info" to the authorities.
+  This document will be signed.  A hash of this document will be included in
+  the regular descriptors.
+
+  (We considered another design, where routers would generate and upload a
+  short-form and a long-form descriptor.  Only the short-form descriptor would
+  ever be used by anybody for routing.  The long-form descriptor would be
+  used only for analytics and other tools.   We decided against this because
+  well-behaved tools would need to download short-form descriptors too (as
+  these would be the only ones indexed), and hence get redundant info. Badly
+  behaved tools would download only long-form descriptors, and expose
+  themselves to partitioning attacks and that like.)
 
 Other disposable fields:
 
@@ -55,63 +51,109 @@ Other disposable fields:
    have much disk, or have really slow disk (e.g. USB). And we don't
    store caches compressed right now. -RD]
 
-Issues:
-
-  Indexing long descriptor or bandwidth reports presents an issue: right now
-  the way to make sure you have the same copy of a descriptor as everyone
-  else is to request the descriptor by its digest, and to make sure that
-  the digest you request is the one that the authorities like.
-
-  Authorities should presumably list the digests of short descriptors, since
-  that's what most everybody will be using.  Including a second digest for
-  long descriptors/bandwidth reports in the networkstatus would only bloat it
-  with information nobody wants.
-
-  Possible solutions are:
-   1) Drop the property that you can be sure of having the same long
-      descriptor as others.  This seems unoptimal, but if nobody caches
-      long descriptors so you have to go to the authority to get them,
-      maybe it's not so bad.
-   2) Have a separate extra-information-status that also gets generated by the
-      authorities; use it to tell which long descriptors others have.  Also a
-      pain.
-   3) Have short descriptors include a hash of the corresponding long
-      descriptor/extra-info.  This would keep the same order of magnitude
-      performance increase (~59.2% savings as opposed to 61% savings.)
-      This would require longdesc/extra-info downloaders to fetch
-      router data before they could know which longdescs/extra info to fetch.
-   4) Have each authority make a signed concatenated "extra info" document,
-      and hope we never need to reconcile them.
-   5) ????
+Specification:
+
+  1. Extra Info Format.
+
+    A "extra info" descriptor contains the following fields:
+
+    "extra-info" Nickname IP FINGERPRINT
+        Identifies what router this is an extra info descriptor for.
+        FINGERPRINT is encoded in hex, with no spaces.
+
+    "published"
+        As currently documented in dir-spec.txt
+
+    "read-history"
+    "write-history"
+        As currently documented in dir-spec.txt
+
+    "router-signature" NL Signature NL
+
+        A signature of the PKCS1-padded hash of the entire extra info
+        document, taken from the beginning of the "extra-info" line, through
+        the newline after the "router-signature" line.  An extra info
+        document is not valid unless the signature is performed with the
+        identity key whose digest matches FINGERPRINT.
+
+    The "extra-info" field is required and MUST appear first.  The
+    router-signature field is required and MUST appear last.  All others are
+    optional.  As for other documents, unrecognized fields must be ignored.
+
+  2. Existing formats
+
+     Implementations that use "read-history" and "write-history" SHOULD
+     accept router descriptors that contain them.  (Prior to 0.2.0.x, this
+     information was encoded in ordinary router descriptors.)
+
+     Add these field to router descriptors:
+       "extra-info-digest" DIGEST
+          DIGEST is a hex-encoded digest of the router's extra-info document,
+          as signed in the router's extra-info.  (If this field is absent,
+          no extra-info-digest exists.)
+
+       "caches-extra-info"
+          Present if this router is a directory cache that provides
+          extra-info documents.
+
+  3. New communications rules
+
+     Clients SHOULD generate and upload an extra-info document after each
+     descriptor they generate and upload; no more, no less.  Clients MUST
+     upload the new descriptor before they upload the new extra-info.
+
+     Authorities receiving an extra-info documents SHOULD verify all of the
+     following:
+       * They have a router descriptor for some server with a matching
+         nickname, IP, and identity fingerprint.
+       * That server's identity key has been used to sign the extra-info
+         document.
+       * The extra-info-digest field in the router descriptor matches
+         the digest of the extra-info document.
+
+     Authorities SHOULD try to fetch extra-info documents from one another if
+     they do not have one matching the digest declared in a router
+     descriptor.
+
+     Caches that are running locally with a tool that needs to use extra-info
+     documents MAY download and store extra-info documents.  They should do
+     so when they notice that the recommended descriptor has an
+     extra-info-digest not matching any extra-info document they currently
+     have.  (Caches not running on a host that needs to use extra-info
+     documents SHOULD NOT download or cache them.)
+
+  4. New URLs
+
+     http://<hostname>/tor/extra/d/...
+     http://<hostname>/tor/extra/fp/...
+     http://<hostname>/tor/extra/all.z
+        (As for /tor/server/ URLs: supports fetching extra-info documents
+        by their digest, by the fingerprint of their servers, or all at
+        once.  Only directory authorities are guaranteed to support these
+        URLs.)
+
+     http://<hostname>/tor/extra/authority.z
+        (The extra-info document for this router.)
+
+     Extra-info documents are uploaded to the same URLs as regular
+     router descriptors.
+
 
 Migration:
 
-  For long/short descriptor approach:
+  For extra info approach:
      * First:
-       * Authorities should accept both, now, and silently drop short
-         descriptors.
-       * Routers should upload both once authorities accept them.
-       * There should be a "long descriptor" url named
-         /tor/server/fp-detailed/ and the current "normal" URL.
-         Authorities should serve long descriptors from both URLs.
-         There's no such thing as asking for a long descriptor by
-         its digest.
-     * Once tools that want long descriptors support fetching them from the
-       "long descriptor" URL:
-       * Have authorities remember short descriptors, and serve them from the
-         'normal' URL.
-       These tools include:
-         lefkada's exit.py script.
-         tor26's noreply script and general directory cache.
-         https://nighteffect.us/tns/ for its graphs
-         and check with or-talk for the rest, once it's time.
-
-  For bandwidth info approach:
-     * First:
-       * Rename it; it won't be just bandwidth forever.
-       * Authorities should accept bandwidth info
+       * Authorities should accept extra info, and support downloading it.
        * Routers should upload bandwidth info once authorities accept it.
-       * There should be a way to download bandwidth info
+       * Caches should support an option to download and cache it, once
+         authorities serve it.
+       * Tools should be updated to use locally cached information.
+         These tools include:
+           lefkada's exit.py script.
+           tor26's noreply script and general directory cache.
+           https://nighteffect.us/tns/ for its graphs
+           and check with or-talk for the rest, once it's time.
+
      * Once tools that want bandwidth info support fetching it:
        * Have routers stop including bandwidth info in their router
          descriptors.