104-short-descriptors.txt 7.4 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183
  1. Filename: 104-short-descriptors.txt
  2. Title: Long and Short Router Descriptors
  3. Version: $Revision$
  4. Last-Modified: $Date$
  5. Author: Nick Mathewson
  6. Created:
  7. Status: Closed
  8. Implemented-In: 0.2.0.x
  9. Overview:
  10. This document proposes moving unused-by-clients information from regular
  11. router descriptors into a new "extra info" router descriptor.
  12. Proposal:
  13. Some of the costliest fields in the current directory protocol are ones
  14. that no client actually uses. In particular, the "read-history" and
  15. "write-history" fields are used only by the authorities for monitoring the
  16. status of the network. If we took them out, the size of a compressed list
  17. of all the routers would fall by about 60%. (No other disposable field
  18. would save much more than 2%.)
  19. We propose to remove these fields from descriptors, and and have them
  20. uploaded as a part of a separate signed "extra info" to the authorities.
  21. This document will be signed. A hash of this document will be included in
  22. the regular descriptors.
  23. (We considered another design, where routers would generate and upload a
  24. short-form and a long-form descriptor. Only the short-form descriptor would
  25. ever be used by anybody for routing. The long-form descriptor would be
  26. used only for analytics and other tools. We decided against this because
  27. well-behaved tools would need to download short-form descriptors too (as
  28. these would be the only ones indexed), and hence get redundant info. Badly
  29. behaved tools would download only long-form descriptors, and expose
  30. themselves to partitioning attacks.)
  31. Other disposable fields:
  32. Clients don't need these fields, but removing them doesn't help bandwidth
  33. enough to be worthwhile.
  34. contact (save about 1%)
  35. fingerprint (save about 3%)
  36. We could represent these fields more succinctly, but removing them would
  37. only save 1%. (!)
  38. reject
  39. accept
  40. (Apparently, exit polices are highly compressible.)
  41. [Does size-on-disk matter to anybody? Some clients and servers don't
  42. have much disk, or have really slow disk (e.g. USB). And we don't
  43. store caches compressed right now. -RD]
  44. Specification:
  45. 1. Extra Info Format.
  46. An "extra info" descriptor contains the following fields:
  47. "extra-info" Nickname Fingerprint
  48. Identifies what router this is an extra info descriptor for.
  49. Fingerprint is encoded in hex (using upper-case letters), with
  50. no spaces.
  51. "published" As currently documented in dir-spec.txt. It MUST match the
  52. "published" field of the descriptor published at the same time.
  53. "read-history"
  54. "write-history"
  55. As currently documented in dir-spec.txt. Optional.
  56. "router-signature" NL Signature NL
  57. A signature of the PKCS1-padded hash of the entire extra info
  58. document, taken from the beginning of the "extra-info" line, through
  59. the newline after the "router-signature" line. An extra info
  60. document is not valid unless the signature is performed with the
  61. identity key whose digest matches FINGERPRINT.
  62. The "extra-info" field is required and MUST appear first. The
  63. router-signature field is required and MUST appear last. All others are
  64. optional. As for other documents, unrecognized fields must be ignored.
  65. 2. Existing formats
  66. Implementations that use "read-history" and "write-history" SHOULD
  67. continue accepting router descriptors that contain them. (Prior to
  68. 0.2.0.x, this information was encoded in ordinary router descriptors;
  69. in any case they have always been listed as opt, so they should be
  70. accepted anyway.)
  71. Add these fields to router descriptors:
  72. "extra-info-digest" Digest
  73. "Digest" is a hex-encoded digest (using upper-case characters)
  74. of the router's extra-info document, as signed in the router's
  75. extra-info. (If this field is absent, no extra-info-digest
  76. exists.)
  77. "caches-extra-info"
  78. Present if this router is a directory cache that provides
  79. extra-info documents, or an authority that handles extra-info
  80. documents.
  81. (Since implementations before 0.1.2.5-alpha required that the "opt"
  82. keyword precede any unrecognized entry, these keys MUST be preceded
  83. with "opt" until 0.1.2.5-alpha is obsolete.)
  84. 3. New communications rules
  85. Servers SHOULD generate and upload one extra-info document after each
  86. descriptor they generate and upload; no more, no less. Servers MUST
  87. upload the new descriptor before they upload the new extra-info.
  88. Authorities receiving an extra-info document SHOULD verify all of the
  89. following:
  90. * They have a router descriptor for some server with a matching
  91. nickname and identity fingerprint.
  92. * That server's identity key has been used to sign the extra-info
  93. document.
  94. * The extra-info-digest field in the router descriptor matches
  95. the digest of the extra-info document.
  96. * The published fields in the two documents match.
  97. Authorities SHOULD drop extra-info documents that do not meet these
  98. criteria.
  99. Extra-info documents MAY be uploaded as part of the same HTTP post as
  100. the router descriptor, or separately. Authorities MUST accept both
  101. methods.
  102. Authorities SHOULD try to fetch extra-info documents from one another if
  103. they do not have one matching the digest declared in a router
  104. descriptor.
  105. Caches that are running locally with a tool that needs to use extra-info
  106. documents MAY download and store extra-info documents. They should do
  107. so when they notice that the recommended descriptor has an
  108. extra-info-digest not matching any extra-info document they currently
  109. have. (Caches not running on a host that needs to use extra-info
  110. documents SHOULD NOT download or cache them.)
  111. 4. New URLs
  112. http://<hostname>/tor/extra/d/...
  113. http://<hostname>/tor/extra/fp/...
  114. http://<hostname>/tor/extra/all[.z]
  115. (As for /tor/server/ URLs: supports fetching extra-info documents
  116. by their digest, by the fingerprint of their servers, or all
  117. at once. When serving by fingerprint, we serve the extra-info
  118. that corresponds to the descriptor we would serve by that
  119. fingerprint. Only directory authorities are guaranteed to support
  120. these URLs.)
  121. http://<hostname>/tor/extra/authority[.z]
  122. (The extra-info document for this router.)
  123. Extra-info documents are uploaded to the same URLs as regular
  124. router descriptors.
  125. Migration:
  126. For extra info approach:
  127. * First:
  128. * Authorities should accept extra info, and support serving it.
  129. * Routers should upload extra info once authorities accept it.
  130. * Caches should support an option to download and cache it, once
  131. authorities serve it.
  132. * Tools should be updated to use locally cached information.
  133. These tools include:
  134. lefkada's exit.py script.
  135. tor26's noreply script and general directory cache.
  136. https://nighteffect.us/tns/ for its graphs
  137. and check with or-talk for the rest, once it's time.
  138. * Set a cutoff time for including bandwidth in router descriptors, so
  139. that tools that use bandwidth info know that they will need to fetch
  140. extra info documents.
  141. * Once tools that want bandwidth info support fetching extra info:
  142. * Have routers stop including bandwidth info in their router
  143. descriptors.