104-short-descriptors.txt 7.3 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182
  1. Filename: 104-short-descriptors.txt
  2. Title: Long and Short Router Descriptors
  3. Version: $Revision$
  4. Last-Modified: $Date$
  5. Author: Nick Mathewson
  6. Created:
  7. Status: Closed
  8. Overview:
  9. This document proposes moving unused-by-clients information from regular
  10. router descriptors into a new "extra info" router descriptor.
  11. Proposal:
  12. Some of the costliest fields in the current directory protocol are ones
  13. that no client actually uses. In particular, the "read-history" and
  14. "write-history" fields are used only by the authorities for monitoring the
  15. status of the network. If we took them out, the size of a compressed list
  16. of all the routers would fall by about 60%. (No other disposable field
  17. would save much more than 2%.)
  18. We propose to remove these fields from descriptors, and and have them
  19. uploaded as a part of a separate signed "extra info" to the authorities.
  20. This document will be signed. A hash of this document will be included in
  21. the regular descriptors.
  22. (We considered another design, where routers would generate and upload a
  23. short-form and a long-form descriptor. Only the short-form descriptor would
  24. ever be used by anybody for routing. The long-form descriptor would be
  25. used only for analytics and other tools. We decided against this because
  26. well-behaved tools would need to download short-form descriptors too (as
  27. these would be the only ones indexed), and hence get redundant info. Badly
  28. behaved tools would download only long-form descriptors, and expose
  29. themselves to partitioning attacks.)
  30. Other disposable fields:
  31. Clients don't need these fields, but removing them doesn't help bandwidth
  32. enough to be worthwhile.
  33. contact (save about 1%)
  34. fingerprint (save about 3%)
  35. We could represent these fields more succinctly, but removing them would
  36. only save 1%. (!)
  37. reject
  38. accept
  39. (Apparently, exit polices are highly compressible.)
  40. [Does size-on-disk matter to anybody? Some clients and servers don't
  41. have much disk, or have really slow disk (e.g. USB). And we don't
  42. store caches compressed right now. -RD]
  43. Specification:
  44. 1. Extra Info Format.
  45. An "extra info" descriptor contains the following fields:
  46. "extra-info" Nickname Fingerprint
  47. Identifies what router this is an extra info descriptor for.
  48. Fingerprint is encoded in hex (using upper-case letters), with
  49. no spaces.
  50. "published" As currently documented in dir-spec.txt. It MUST match the
  51. "published" field of the descriptor published at the same time.
  52. "read-history"
  53. "write-history"
  54. As currently documented in dir-spec.txt. Optional.
  55. "router-signature" NL Signature NL
  56. A signature of the PKCS1-padded hash of the entire extra info
  57. document, taken from the beginning of the "extra-info" line, through
  58. the newline after the "router-signature" line. An extra info
  59. document is not valid unless the signature is performed with the
  60. identity key whose digest matches FINGERPRINT.
  61. The "extra-info" field is required and MUST appear first. The
  62. router-signature field is required and MUST appear last. All others are
  63. optional. As for other documents, unrecognized fields must be ignored.
  64. 2. Existing formats
  65. Implementations that use "read-history" and "write-history" SHOULD
  66. continue accepting router descriptors that contain them. (Prior to
  67. 0.2.0.x, this information was encoded in ordinary router descriptors;
  68. in any case they have always been listed as opt, so they should be
  69. accepted anyway.)
  70. Add these fields to router descriptors:
  71. "extra-info-digest" Digest
  72. "Digest" is a hex-encoded digest (using upper-case characters)
  73. of the router's extra-info document, as signed in the router's
  74. extra-info. (If this field is absent, no extra-info-digest
  75. exists.)
  76. "caches-extra-info"
  77. Present if this router is a directory cache that provides
  78. extra-info documents, or an authority that handles extra-info
  79. documents.
  80. (Since implementations before 0.1.2.5-alpha required that the "opt"
  81. keyword precede any unrecognized entry, these keys MUST be preceded
  82. with "opt" until 0.1.2.5-alpha is obsolete.)
  83. 3. New communications rules
  84. Servers SHOULD generate and upload one extra-info document after each
  85. descriptor they generate and upload; no more, no less. Servers MUST
  86. upload the new descriptor before they upload the new extra-info.
  87. Authorities receiving an extra-info document SHOULD verify all of the
  88. following:
  89. * They have a router descriptor for some server with a matching
  90. nickname and identity fingerprint.
  91. * That server's identity key has been used to sign the extra-info
  92. document.
  93. * The extra-info-digest field in the router descriptor matches
  94. the digest of the extra-info document.
  95. * The published fields in the two documents match.
  96. Authorities SHOULD drop extra-info documents that do not meet these
  97. criteria.
  98. Extra-info documents MAY be uploaded as part of the same HTTP post as
  99. the router descriptor, or separately. Authorities MUST accept both
  100. methods.
  101. Authorities SHOULD try to fetch extra-info documents from one another if
  102. they do not have one matching the digest declared in a router
  103. descriptor.
  104. Caches that are running locally with a tool that needs to use extra-info
  105. documents MAY download and store extra-info documents. They should do
  106. so when they notice that the recommended descriptor has an
  107. extra-info-digest not matching any extra-info document they currently
  108. have. (Caches not running on a host that needs to use extra-info
  109. documents SHOULD NOT download or cache them.)
  110. 4. New URLs
  111. http://<hostname>/tor/extra/d/...
  112. http://<hostname>/tor/extra/fp/...
  113. http://<hostname>/tor/extra/all[.z]
  114. (As for /tor/server/ URLs: supports fetching extra-info documents
  115. by their digest, by the fingerprint of their servers, or all
  116. at once. When serving by fingerprint, we serve the extra-info
  117. that corresponds to the descriptor we would serve by that
  118. fingerprint. Only directory authorities are guaranteed to support
  119. these URLs.)
  120. http://<hostname>/tor/extra/authority[.z]
  121. (The extra-info document for this router.)
  122. Extra-info documents are uploaded to the same URLs as regular
  123. router descriptors.
  124. Migration:
  125. For extra info approach:
  126. * First:
  127. * Authorities should accept extra info, and support serving it.
  128. * Routers should upload extra info once authorities accept it.
  129. * Caches should support an option to download and cache it, once
  130. authorities serve it.
  131. * Tools should be updated to use locally cached information.
  132. These tools include:
  133. lefkada's exit.py script.
  134. tor26's noreply script and general directory cache.
  135. https://nighteffect.us/tns/ for its graphs
  136. and check with or-talk for the rest, once it's time.
  137. * Set a cutoff time for including bandwidth in router descriptors, so
  138. that tools that use bandwidth info know that they will need to fetch
  139. extra info documents.
  140. * Once tools that want bandwidth info support fetching extra info:
  141. * Have routers stop including bandwidth info in their router
  142. descriptors.