dir-spec.txt 23 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522
  1. $Id$
  2. Tor directory protocol for 0.1.1.x series
  3. 0. Scope and preliminaries
  4. This document should eventually be merged to replace and supplement the
  5. existing notes on directories in tor-spec.txt.
  6. This is not a finalized version; what we actually wind up implementing
  7. may be different from the system described here.
  8. 0.1. Goals
  9. There are several problems with the way Tor handles directory information
  10. in version 0.1.0.x and earlier. Here are the problems we try to fix with
  11. this new design, already partially implemented in 0.1.1.x:
  12. 1. Directories are very large and use up a lot of bandwidth: clients
  13. download descriptors for all router several times an hour.
  14. 2. Every directory authority is a trust bottleneck: if a single
  15. directory authority lies, it can make clients believe for a time an
  16. arbitrarily distorted view of the Tor network.
  17. 3. Our current "verified server" system is kind of nonsensical.
  18. 4. Getting more directory authorities adds more points of failure and
  19. worsens possible partitioning attacks.
  20. There are two problems that remain unaddressed by this design.
  21. 5. Requiring every client to know about every router won't scale.
  22. 6. Requiring every directory cache to know every router won't scale.
  23. 1. Outline
  24. There is a small set (say, around 10) of semi-trusted directory
  25. authorities. A default list of authorities is shipped with the Tor
  26. software. Users can change this list, but are encouraged not to do so, in
  27. order to avoid partitioning attacks.
  28. Routers periodically upload signed "descriptors" to the directory
  29. authorities describing their keys, capabilities, and other information.
  30. Routers may act as directory mirrors (also called "caches"), to reduce
  31. load on the directory authorities. They announce this in their
  32. descriptors.
  33. Each directory authority periodically generates and signs a compact
  34. "network status" document that lists that authority's view of the current
  35. descriptors and status for known routers, but which does not include the
  36. descriptors themselves.
  37. Directory mirrors download, cache, and re-serve network-status documents
  38. to clients.
  39. Clients, directory mirrors, and directory authorities all use
  40. network-status documents to find out when their list of routers is
  41. out-of-date. If it is, they download any missing router descriptors.
  42. Clients download missing descriptors from mirrors; mirrors and authorities
  43. download from authorities. Descriptors are downloaded by the hash of the
  44. descriptor, not by the server's identity key: this prevents servers from
  45. attacking clients by giving them descriptors nobody else uses.
  46. All directory information is uploaded and downloaded with HTTP.
  47. Coordination among directory authorities is done client-side: clients
  48. compute a vote-like algorithm among the network-status documents they
  49. have, and base their decisions on the result.
  50. 1.1. What's different from 0.1.0.x?
  51. Clients used to download a signed concatenated set of router descriptors
  52. (called a "directory") from directory mirrors, regardless of which
  53. descriptors had changed.
  54. Between downloading directories, clients would download "network-status"
  55. documents that would list which servers were supposed to running.
  56. Clients would always believe the most recently published network-status
  57. document they were served.
  58. Routers used to upload fresh descriptors all the time, whether their keys
  59. and other information had changed or not.
  60. 2. Router operation
  61. The router descriptor format is unchanged from tor-spec.txt.
  62. ORs SHOULD generate a new router descriptor whenever any of the
  63. following events have occurred:
  64. - A period of time (18 hrs by default) has passed since the last
  65. time a descriptor was generated.
  66. - A descriptor field other than bandwidth or uptime has changed.
  67. - Bandwidth has changed by more than +/- 50% from the last time a
  68. descriptor was generated, and at least a given interval of time
  69. (20 mins by default) has passed since then.
  70. - Its uptime has been reset (by restarting).
  71. After generating a descriptor, ORs upload it to every directory
  72. authority they know, by posting it to the URL
  73. http://<hostname>/tor/
  74. 3. Network status format
  75. Directory authorities generate, sign, and compress network-status
  76. documents. Directory servers SHOULD generate a fresh network-status
  77. document when the contents of such a document would be different from the
  78. last one generated, and some time (at least one second, possibly longer)
  79. has passed since the last one was generated.
  80. The network status document contains a preamble, a set of router status
  81. entries, and a signature, in that order.
  82. We use the same meta-format as used for directories and router descriptors
  83. in "tor-spec.txt". Implementations MAY insert blank lines
  84. for clarity between sections; these blank lines are ignored.
  85. Implementations MUST NOT depend on blank lines in any particular location.
  86. As used here, "whitespace" is a sequence of 1 or more tab or space
  87. characters.
  88. The preamble contains:
  89. "network-status-version" -- A document format version. For this
  90. specification, the version is "2".
  91. "dir-source" -- The authority's hostname, current IP address, and
  92. directory port, all separated by whitespace.
  93. "fingerprint" -- A base16-encoded hash of the signing key's
  94. fingerprint, with no additional spaces added.
  95. "contact" -- An arbitrary string describing how to contact the
  96. directory server's administrator. Administrators should include at
  97. least an email address and a PGP fingerprint.
  98. "dir-signing-key" -- The directory server's public signing key.
  99. "client-versions" -- A comma-separated list of recommended client
  100. versions.
  101. "server-versions" -- A comma-separated list of recommended server
  102. versions.
  103. "published" -- The publication time for this network-status object.
  104. "dir-options" -- A set of flags, in any order, separated by whitespace:
  105. "Names" if this directory authority performs name bindings.
  106. "Versions" if this directory authority recommends software versions.
  107. The dir-options entry is optional. The "-versions" entries are required if
  108. the "Versions" flag is present. The other entries are required and must
  109. appear exactly once. The "network-status-version" entry must appear first;
  110. the others may appear in any order. Implementations MUST ignore
  111. additional arguments to the items above, and MUST ignore unrecognized
  112. flags.
  113. For each router, the router entry contains: (This format is designed for
  114. conciseness.)
  115. "r" -- followed by the following elements, in order, separated by
  116. whitespace:
  117. - The OR's nickname,
  118. - A hash of its identity key, encoded in base64, with trailing =
  119. signs removed.
  120. - A hash of its most recent descriptor, encoded in base64, with
  121. trailing = signs removed. (The hash is calculated as for
  122. computing the signature of a descriptor.)
  123. - The publication time of its most recent descriptor, in the form
  124. YYYY-MM-DD HH:MM:SS, in GMT.
  125. - An IP address
  126. - An OR port
  127. - A directory port (or "0" for none")
  128. "s" -- A series of whitespace-separated status flags, in any order:
  129. "Authority" if the router is a directory authority.
  130. "Exit" if the router is useful for building general-purpose exit
  131. circuits.
  132. "Fast" if the router is suitable for high-bandwidth circuits.
  133. "Guard" if the router is suitable for use as an entry guard.
  134. (Currently, this means 'fast' and 'stable'.)
  135. "Named" if the router's identity-nickname mapping is canonical,
  136. and this authority binds names.
  137. "Stable" if the router is suitable for long-lived circuits.
  138. "Running" if the router is currently usable.
  139. "Valid" if the router has been 'validated'.
  140. "V2Dir" if the router implements this protocol.
  141. The "r" entry for each router must appear first and is required. The
  142. 's" entry is optional. Unrecognized flags and extra elements on the
  143. "r" line must be ignored.
  144. The signature section contains:
  145. "directory-signature". A signature of the rest of the document using
  146. the directory authority's signing key.
  147. We compress the network status list with zlib before transmitting it.
  148. 3.1. Establishing server status
  149. [[XXXXX Describe how authorities actually decide Fast, Named, Stable,
  150. Running, Valid
  151. For each OR, a directory server remembers whether the OR was running and
  152. functional the last time they tried to connect to it, and possibly other
  153. liveness information.
  154. Directory server administrators may label some servers or IPs as
  155. blacklisted, and elect not to include them in their network-status lists.
  156. Thus, the network-status list includes all non-blacklisted,
  157. non-expired, non-superseded descriptors for ORs that the directory has
  158. observed at least once to be running.
  159. Directory server administrators may decide to support name binding. If
  160. they do, then they must maintain a file of nickname-to-identity-key
  161. mappings, and try to keep this file consistent with other directory
  162. servers. If they don't, they act as clients, and report bindings made by
  163. other directory servers (name X is bound to identity Y if at least one
  164. binding directory lists it, and no directory binds X to some other Y'.)
  165. ]]
  166. 4. Directory server operation
  167. All directory authorities and directory mirrors ("directory servers")
  168. implement this section, except as noted.
  169. 4.1. Accepting uploads (authorities only)
  170. When a router posts a signed descriptor to a directory authority, the
  171. authority first checks whether it is well-formed and correctly
  172. self-signed. If it is, the authority next verifies that the nickname
  173. question is already assigned to a router with a different public key.
  174. Finally, the authority MAY check that the router is not blacklisted
  175. because of its key, IP, or another reason.
  176. If the descriptor passes these tests, and the authority does not already
  177. have a descriptor for a router with this public key, it accepts the
  178. descriptor and remembers it.
  179. If the authority _does_ have a descriptor with the same public key, the
  180. newly uploaded descriptor is remembered if its publication time is more
  181. recent than the most recent old descriptor for that router, and either:
  182. - There are non-cosmetic differences between the old descriptor and the
  183. new one.
  184. - Enough time has passed between the descriptors' publication times.
  185. (Currently, 12 hours.)
  186. Differences between router descriptors are "non-cosmetic" if they would be
  187. sufficient to force an upload as described in section 2 above.
  188. Note that the "cosmetic difference" test only applies to uploaded
  189. descriptors, not to descriptors that the authority downloads from other
  190. authorities.
  191. 4.2. Downloading network-status documents
  192. All directory servers (authorities and mirrors) try to keep a fresh
  193. set of network-status documents from every authority. To do so,
  194. every 5 minutes, each authority asks every other authority for its
  195. most recent network-status document. Every 15 minutes, each mirror
  196. picks a random authority and asks it for the most recent network-status
  197. documents for all the authorities the authority knows about (including
  198. the chosen authority itself).
  199. Directory servers and mirrors remember and serve the most recent
  200. network-status document they have from each authority. Other
  201. network-status documents don't need to be stored. If the most recent
  202. network-status document is over 10 days old, it is discarded anyway.
  203. Mirrors SHOULD store and serve network-status documents from authorities
  204. they don't recognize, but SHOULD NOT use such documents for any other
  205. purpose.
  206. 4.3. Downloading and storing router descriptors
  207. Periodically (currently, every 10 seconds), directory servers check
  208. whether there are any specific descriptors (as identified by descriptor
  209. hash in a network-status document) that they do not have and that they
  210. are not currently trying to download.
  211. If so, the directory server launches requests to the authorities for these
  212. descriptors, such that each authority is only asked for descriptors listed
  213. in its most recent network-status. When more than one authority lists the
  214. descriptor, we choose which to ask at random.
  215. If one of these downloads fails, we do not try to download that descriptor
  216. from the authority that failed to serve it again unless we receive a newer
  217. network-status from that authority that lists the same descriptor.
  218. Directory servers must potentially cache multiple descriptors for each
  219. router. Servers must not discard any descriptor listed by any current
  220. network-status document from any authority. If there is enough space to
  221. store additional descriptors [XXXXXX then how do we pick.]
  222. Authorities SHOULD NOT download descriptors for routers that they would
  223. immediately reject for reasons listed in 3.1.
  224. 4.4. HTTP URLs
  225. "Fingerprints" in these URLs are base-16-encoded SHA1 hashes.
  226. The authoritative network-status published by a host should be available at:
  227. http://<hostname>/tor/status/authority.z
  228. The network-status published by a host with fingerprint
  229. <F> should be available at:
  230. http://<hostname>/tor/status/fp/<F>.z
  231. The network-status documents published by hosts with fingerprints
  232. <F1>,<F2>,<F3> should be available at:
  233. http://<hostname>/tor/status/fp/<F1>+<F2>+<F3>.z
  234. The most recent network-status documents from all known authorities,
  235. concatenated, should be available at:
  236. http://<hostname>/tor/status/all.z
  237. The most recent descriptor for a server whose identity key has a
  238. fingerprint of <F> should be available at:
  239. http://<hostname>/tor/server/fp/<F>.z
  240. The most recent descriptors for servers with identity fingerprints
  241. <F1>,<F2>,<F3> should be available at:
  242. http://<hostname>/tor/server/fp/<F1>+<F2>+<F3>.z
  243. (NOTE: Implementations SHOULD NOT download descriptors by identity key
  244. fingerprint. This allows a corrupted server (in collusion with a cache) to
  245. provide a unique descriptor to a client, and thereby partition that client
  246. from the rest of the network.)
  247. The server descriptor with (descriptor) digest <D> (in hex) should be
  248. available at:
  249. http://<hostname>/tor/server/d/<D>.z
  250. The most recent descriptors with digests <D1>,<D2>,<D3> should be
  251. available at:
  252. http://<hostname>/tor/server/d/<D1>+<D2>+<D3>.z
  253. The most recent descriptor for this server should be at:
  254. http://<hostname>/tor/server/authority.z
  255. [Nothing in the Tor protocol uses this resource yet, but it is useful
  256. for debugging purposes. Also, the official Tor implementations
  257. (starting at 0.1.1.x) use this resource to test whether a server's
  258. own DirPort is reachable.]
  259. A concatenated set of the most recent descriptors for all known servers
  260. should be available at:
  261. http://<hostname>/tor/server/all.z
  262. For debugging, directories SHOULD expose non-compressed objects at URLs like
  263. the above, but without the final ".z".
  264. Clients MUST handle compressed concatenated information in two forms:
  265. - A concatenated list of zlib-compressed objects.
  266. - A zlib-compressed concatenated list of objects.
  267. Directory servers MAY generate either format: the former requires less
  268. CPU, but the latter requires less bandwidth.
  269. Clients SHOULD use upper case letters (A-F) when base16-encoding
  270. fingerprints. Servers MUST accept both upper and lower case fingerprints
  271. in requests.
  272. 5. Client operation: downloading information
  273. Every Tor that is not a directory server (that is, clients and ORs that do
  274. not have a DirPort set) implements this section.
  275. 5.1. Downloading network-status documents
  276. Each client maintains an ordered list of directory authorities.
  277. Insofar as possible, clients SHOULD all use the same ordered list.
  278. Clients check whether they have enough recently published network-status
  279. documents (currently, this means that they must have a network-status
  280. published within the last 48 hours for over half of the authorities).
  281. If they do not, they download enough network-status documents so that this
  282. is so.
  283. Also, if the most recently published network-status document is over 30
  284. minutes old, the client downloads a network-status document.
  285. When choosing which documents to download, clients treat their list of
  286. directory authorities as a circular ring, and begin with the authority
  287. appearing immediately after the authority for their most recently
  288. published network-status document.
  289. If enough mirrors (currently 4) claim not to have a given network status,
  290. we stop trying to download that authority's network-status, until we
  291. download a new network-status that makes us believe that the authority in
  292. question is running.
  293. Network-status documents published over 10 hours in the past are
  294. discarded.
  295. 5.2. Downloading router descriptors
  296. Clients try to have the best descriptor for each router. A descriptor is
  297. "best" if:
  298. * it the most recently published descriptor listed for that router by
  299. at least two network-status documents.
  300. * OR, no descriptor for that router is listed by two or more
  301. network-status documents, and it is the most recently published
  302. descriptor listed by any network-status document.
  303. Periodically (currently every 10 seconds) clients check whether there are
  304. any "downloadable" descriptors. A descriptor is downloadable if:
  305. - It is the "best" descriptor for some router.
  306. - The descriptor was published at least 5 minutes (???) in the past.
  307. [This prevents clients from trying to fetch descriptors that the
  308. mirrors have not yet retrieved and cached.]
  309. - The client does not currently have it.
  310. - The client is not currently trying to download it.
  311. If at least 1/16 of known routers have downloadable descriptors, or if
  312. enough time (currently 10 minutes) has passed since the last time the
  313. client tried to download descriptors, it launches requests for all
  314. downloadable descriptors, as described in 5.3 below.
  315. When a descriptor download fails, the client notes it, and does not
  316. consider the descriptor downloadable again until a certain amount of time
  317. has passed. (Currently 0 seconds for the first failure, 60 seconds for the
  318. second, 5 minutes for the third, 10 minutes for the fourth, and 1 day
  319. thereafter.) Periodically (currently once an hour) clients reset the
  320. failure count.
  321. No descriptors are downloaded until the client has downloaded more than
  322. half of the network-status documents.
  323. 5.3. Managing downloads
  324. When a client has no live network-status documents, it downloads
  325. network-status documents from a randomly chosen authority. In all other
  326. cases, the client downloads from mirrors randomly chosen from among those
  327. believed to be V2 directory servers. (This information comes from the
  328. network-status documents; see 6 below.)
  329. When downloading multiple router descriptors, the client chooses multiple
  330. mirrors so that:
  331. - At least 3 different mirrors are used, except when this would result
  332. in more than one request for under 4 descriptors.
  333. - No more than 128 descriptors are requested from a single mirror.
  334. - Otherwise, as few mirrors as possible are used.
  335. After choosing mirrors, the client divides the descriptors among them
  336. randomly.
  337. After receiving any response client MUST discard any network-status
  338. documents and descriptors that it did not request.
  339. 6. Using directory information
  340. Everyone besides directory authorities uses the approaches in this section
  341. to decide which servers to use and what their keys are likely to be.
  342. (Directory authorities just believe their own opinions, as in 3.1 above.)
  343. 6.1. Choosing routers for circuits.
  344. Tor implementations only pay attention to "live" network-status documents.
  345. A network status is "live" if it is the most recently downloaded network
  346. status document for a given directory server, and the server is a
  347. directory server trusted by the client, and the network-status document is
  348. no more than 2 days old.
  349. For time-sensitive information, Tor implementations focus on "recent"
  350. network-status documents. A network status is "recent" if it is live, and
  351. if it was published in the last 60 minutes. If there are fewer
  352. than 3 such documents, the most recently published 3 are "recent." If
  353. there are fewer than 3 in all, all are "recent.")
  354. Circuits SHOULD NOT be built until the client has enough directory
  355. information: at least two live network-status documents, and descriptors
  356. for at least 1/4 of the servers believed to be running.
  357. A server is "listed" if it is included by more than half of the live
  358. network status documents. Clients SHOULD NOT use unlisted servers.
  359. A server is "valid" if it is listed as valid by more than half of the live
  360. network-status documents. Clients SHOULD NOT use non-valid servers unless
  361. specifically configured to do so.
  362. A server is "running" if it is listed as running by more than half of the
  363. recent network-status documents. Clients SHOULD NOT try to use
  364. non-running servers.
  365. A server is believed to be a directory mirror if it is listed as a V2
  366. directory by more than half of the recent network-status documents.
  367. 6.1. Managing naming
  368. In order to provide human-memorable names for individual server
  369. identities, some directory servers bind names to IDs. Clients handle
  370. names in two ways:
  371. When a client encounters a name it has not mapped before:
  372. If all the live "Naming" network-status documents the client has
  373. claim that the name binds to some identity ID, and the client has at
  374. least three live network-status documents, the client maps the name to
  375. ID.
  376. If a client encounters a name it has mapped before:
  377. It uses the last-mapped identity value, unless all of the "Naming"
  378. network status documents that list the name bind it to some other
  379. identity.
  380. When a user tries to refer to a router with a name that does not have a
  381. mapping under the above rules, the implementation SHOULD warn the user.
  382. After giving the warning, the implementation MAY use a router that at
  383. least one Naming authority maps the name to, so long as no other naming
  384. authority maps that name to a different router.
  385. 6.2. Software versions
  386. An implementation of Tor SHOULD warn when it has live network-statuses from
  387. more than half of the authorities, and it is running a software version
  388. not listed on more than half of the live "Versioning" network-status
  389. documents.
  390. TODO:
  391. - Resolve XXXXs
  392. - Are the magic numbers above sane?
  393. - Client-knowledge partitioning is worrisome. Most versions of this
  394. don't seem to be worse than the Danezis-Murdoch tracing attack, since
  395. an attacker can't do more than deduce probable exits from entries (or
  396. vice versa). But what about when the client connects to A and B but in
  397. a different order? How bad can it be partitioned based on its
  398. knowledge?