patch from karsten to clean up documentation and to integrate

more fixes into rend-spec.txt.


svn:r12715

doc/spec/proposals/000-index.txt

@@ -58,7 +58,6 @@ Proposals by status:
  OPEN:
    110  Avoiding infinite length circuits
    113  Simplifying directory authority administration
-   114  Distributed Storage for Tor Hidden Service Descriptors
    115  Two Hop Paths
    116  Two hop paths from entry guards
    117  IPv6 exits
@@ -88,6 +87,7 @@ Proposals by status:
    107  Uptime Sanity Checking
    108  Base "Stable" Flag on Mean Time Between Failures
    109  No more than one server per IP address
+   114  Distributed Storage for Tor Hidden Service Descriptors
    119  New PROTOCOLINFO command for controllers
    122  Network status entries need a new Unnamed flag
  SUPERSEDED:

doc/spec/proposals/114-distributed-storage.txt

@@ -4,7 +4,7 @@ Version: $Revision$
 Last-Modified: $Date$
 Author: Karsten Loesing
 Created: 13-May-2007
-Status: Open
+Status: Closed
 
 Change history:
 
@@ -16,6 +16,7 @@ Change history:
   11-Aug-2007  Updated implementation statuses, included non-consecutive
                replication to descriptor format
   20-Aug-2007  Renamed config option HSDir as HidServDirectoryV2
+  02-Dec-2007  Closed proposal
 
 Overview:
 
@@ -27,6 +28,17 @@ Overview:
   this proposal suggests changes to the hidden service descriptor format to
   prevent new security threats coming from decentralization and to gain even
   better security properties.
+
+Status:
+
+  As of December 2007, the new hidden service descriptor format is implemented
+  and usable. However, servers and clients do not yet make use of descriptor
+  cookies, because there are open usability issues of this feature that might
+  be resolved in proposal 121. Further, hidden service directories do not
+  perform replication by themselves, because (unauthorized) replica fetch
+  requests would allow any attacker to fetch all hidden service descriptors in
+  the system. As neither issue is critical to the functioning of v2
+  descriptors and their distribution, this proposal is considered as Closed.
   
 Motivation:
 
@@ -35,7 +47,7 @@ Motivation:
 
   First, the three hidden service authoritative directories constitute a
   performance bottleneck in the system. The directory nodes are responsible for
-  storing and serving all hidden service descriptors. At the moment there are
+  storing and serving all hidden service descriptors. As of May 2007 there are
   about 1000 descriptors at a time, but this number is assumed to increase in
   the future. Further, there is no replication protocol for descriptors between
   the three directory nodes, so that hidden services must ensure the
@@ -123,22 +135,6 @@ Design:
     at least 24 hours. A participant only trusts its own routing list and never
     learns about routing information from other parties.
 
-    - rend-spec.txt, section 1.4: Added description of how to obtain a routing
-      list of hidden service directories.
-
-    - routerparse.c: Changed routerstatus_parse_entry_from_string to parse the
-      "HSDir" flag in vote and consensus status documents.
-    - routerlist.c: Changed router_get_routerlist() to initialize routing list.
-    - or.h: Added hs_dirs member to routerlist_t.
-
-    - Changed routerlist_free() to free storage held by routing list.
-    - Added UPDATE_HS_DIRS_INTERVAL.
-    - Added update_hs_dir_routing_table().
-    - Changed run_scheduled_events().
-    - Added is_hs_dir member to routerstatus_t.
-    
-      [Aug 11: Specified and running.]
-
   /2/ Determine responsible hidden service directory
 
     All participants can determine the hidden service directory that is
@@ -148,40 +144,12 @@ Design:
     its predecessor, exclusive, to its own ID, inclusive. Further, a hidden
     service directory holds replicas for its n predecessors, where n denotes
     the number of consecutive replicas. (requires /1/)
-    
-    - rend-spec.txt, section 1.4: Added description of how to determine the
-      responsible node(s) for a given descriptor ID.
-
-    - routerlist.c: Added get_responsible_hs_dirs() to determine the routers
-      that are responsible for a given descriptor ID.
-      
-    - Added is_hs_dir member to routerstatus_t.
-    - Added have_enough_hs_dirs().
-    - Added next_hs_dir().
-    
-      [July 9: Specified and running.]
-    
-  Hidden service clients and providers:
 
-  /3/ Send tunneled HTTP request to hidden service directory in BEGIN_DIR cell
-
-    - rend-spec.txt, section 1.4: Added the requirement that requests need to
-      be sent via Tor.
-    - rend-spec.txt, section 1.6: Added the requirement that requests need to
-      be sent via Tor.
-
-    [July 9: Pending]
+  [/3/ and /4/ were requirements to use BEGIN_DIR cells for directory
+   requests which have not been fulfilled in the course of the implementation
+   of this proposal, but elsewhere.]
 
   Hidden service directory nodes:
-
-  /4/ Process tunneled HTTP request in BEGIN_DIR cell
-
-    - rend-spec.txt, section 3.2: Added the requirement that requests need to
-      be contained within BEGIN_DIR cells.
-    - rend-spec.txt, section 3.3: Added the requirement that requests need to
-      be contained within BEGIN_DIR cells.
-  
-    [July 9: Pending]
     
   /5/ Advertise hidden service directory functionality
 
@@ -191,19 +159,6 @@ Design:
     option being set includes the flag "hidden-service-dir" in its router
     descriptors that it sends to directory authorities.
 
-    - tor.1.in: Added the config option HidServDirectoryV2.
-    - dir-spec.txt, section 2.1: Added the flag hidden-service-dir to the
-      router descriptor format.
-    - rend-spec.txt, section 3.1: Added process of configuring a hidden service
-      directory.
-
-    - router.c: Changed router_dump_router_to_string() to include the
-      hidden-service-dir flag in a router descriptor if configured.
-    - or.h: Added HidServDirectoryV2 to or_options_t.
-    - config.c: Added config option HidServDirectoryV2.
-
-      [July 9: Specified and running.]
-      
   /6/ Accept v2 publish requests, parse and store v2 descriptors
 
     Hidden service directory nodes accept publish requests for hidden service
@@ -217,44 +172,12 @@ Design:
     descriptor based on its own routing table. Every hidden service directory
     node is responsible for the descriptor IDs in the interval of its n-th
     predecessor in the ID circle up to its own ID (n denotes the number of
-    consecutive replicas). (requires /1/ and /4/)
-
-    - rend-spec.txt, section 1.2: Added the new v2 hidden service descriptor
-      format.
-    - rend-spec.txt, section 3.2: Added the acceptance of v2 publish requests.
-
-    - routerparse.c: Added rend_parse_v2_service_descriptor() to parse a v2
-      hidden service descriptor.
-    - routerparse.c: Added desc_token_table[] to parse v2 hidden service
-      descriptors.
-    - routerparse.c: Added 8 keywords to directory_keyword to parse v2 hidden
-      service descriptors.
-    - rendcommon.c: Added rend_cache_store_v2_dir() to allow a hidden service
-      directory to parse a v2 descriptor and store it in the local cache under
-      its descriptor ID instead of its service ID.
-    - or.h: Added constant REND_DESC_ID_V2_LEN to reflect that v2 descriptor
-      IDs are longer than v0/1 onion addresses.
-
-    - Changed directory_handle_command_post().
-    
-      [Aug 11: Specified and running.]
+    consecutive replicas). (requires /1/)
 
   /7/ Accept v2 fetch requests
 
     Same as /6/, but with fetch requests for hidden service descriptors.
-    (requires /2/ and /4/)
-
-    - rend-spec.txt, section 3.3: Added the processing of v2 fetch requests.
-
-    - rendcommon.c: Added rend_cache_lookup_v2_dir() to allow a hidden service
-      directory to look up a v2 descriptor in the local cache under its
-      descriptor ID instead of its service ID.
-    - or.h: Added constant REND_DESC_ID_V2_LEN to reflect that v2 descriptor
-      IDs are longer than v0/1 onion addresses.
-
-    - Changed directory_handle_command_get().
-    
-      [Aug 11: Specified and running.]
+    (requires /2/)
 
   /8/ Replicate descriptors with neighbors
 
@@ -269,20 +192,10 @@ Design:
     descriptors in the interval of its (n+1)-th and its n-th predecessor.
     (requires /1/)
 
-    - rend-spec.txt, section 3.3: Added the replication of v2 descriptors.
-
-    - Added HS_DIR_REPLICATION_INTERVAL.
-    - Added next_hs_dir and previous_hs_dir.
-    - Changed directory_handle_command_get().
-    - Changed run_scheduled_events.
-    - Added hs_dir_perform_replication().
-    - Added rend_cache_lookup_v2_replicas.
-    - Added DIR_PURPOSE_REPLICATE_RENDDESC_V2.
-    - Changed directory_initiate_command.
-    - directory_send_command.
-    - Changed connection_dir_client_reached_eof.
-
-      [Aug 11: To some extend specified, running.]
+    [Dec 02: This function has not been implemented, because arbitrary nodes
+     what have been able to download the entire set of v2 descriptors. An
+     authorized replication request would be necessary. For the moment, the
+     system runs without any directory-side replication. -KL]
 
   Authoritative directory nodes:
 
@@ -294,30 +207,6 @@ Design:
     changing its onion key to become responsible for an identifier it wants to
     target.
 
-    - dir-spec.txt, section 3.2: Added the status flag "HSDir" to the vote and
-      consensus status document format.
-    - dir-spec.txt, section 3.3: Added a rule for how an authority decides
-      whether a router is assigned the flag "HSDir".
-    - rend-spec.txt, section 3.1: Added the decision on whether an onion router
-      is confirmed to act as hidden service directory or not.
-
-    - routerparse.c: Changed router_parse_entry_from_string() to parse the
-      "hidden-service-dir" flag in router descriptors.
-    - routerparse.c: Added an entry to routerdesc_token_table[] to parse the
-      "hidden-service-directory" flag in router descriptors.
-    - routerparse.c: Added 1 keyword to directory_keyword to parse the
-      "hidden-service-dir" flag in router descriptors.
-    - or.h: Added is_hs_dir and wants_to_be_hs_dir members to routerinfo_t.
-    - dirserv.c: Changed routerstatus_format_entry() to include the "HSDir"
-      flag in vote and consensus status documents.
-    - dirserv.c: Changed set_routerstatus_from_routerinfo() to set the "HSDir"
-      flag.
-
-    - Added dirserv_thinks_router_is_hs_dir().
-    - Added MIN_UPTIME_HS_DIR and HS_DIR_REACHABLE_TIMEOUT.
-
-      [Aug 11: Specified and running.]
-
   Hidden service provider:
 
   /10/ Configure v2 hidden service
@@ -329,20 +218,6 @@ Design:
     already created a random secret_cookie and a hostname2 file; if not, it
     creates both of them. (requires /2/)
 
-    - tor.1.in: Added the config option PublishV2HidServDescriptors.
-    - tor.1.in: Added the files hostname2 and secret_cookie.
-    - rend-spec.txt, section 1.1: Added requirement to create secret_cookie and
-      hostname2 file.
-
-    - rendservice.c: Added rend_get_hostname2() to assemble a v2 onion address.
-    - rendservice.c: Changed rend_service_load_keys() to write a secret_cookie
-      and a hostname2 file.
-    - rendservice.c: Extended rend_service_t by a member secret_cookie.
-    - or.h: Added PublishV2HidServDescriptors to or_options_t.
-    - config.c: Added config option PublishV2HidServDescriptors.
-
-      [July 9: Specified and running.]
-
   /11/ Establish introduction points with fresh key
 
     If configured to publish only v2 descriptors and no v0/v1 descriptors any
@@ -357,14 +232,6 @@ Design:
     rely on the fact that all introduction points accept the same public key,
     so that this new feature cannot be used.)
 
-    - rend-spec.txt, section 1.3: Instead of Bob's public key, the hidden
-      service provider uses a freshly generated public key for every
-      introduction point.
-
-    - TODO: Change in rend_encode_v2_descriptors.
-
-      [July 9: Specified, but not yet implemented.]
-
   /12/ Encode v2 descriptors and send v2 publish requests
 
     If configured to publish v2 descriptors, a hidden service provider
@@ -376,32 +243,7 @@ Design:
     the next period. Publication is performed by sending the descriptor to all
     hidden service directories that are responsible for keeping replicas for
     the descriptor ID. This includes two non-consecutive replicas that are
-    stored at 3 consecutive nodes each. (requires /1/, /2/, and /3/)
-
-    - rend-spec.txt, section 1.2: Added the new v2 hidden service descriptor
-      format.
-    - rend-spec.txt, section 1.4: Bob's OP does not only upload v0/v1 service
-      descriptors to the authoritative directories, but also v2 service
-      descriptors to the hidden service directories.
-
-    - rendservice.c: Changed upload_service_descriptor() to upload v2 hidden
-      service descriptors, if configured.
-    - rendservice.c: Changed rend_consider_services_upload() to also initiate
-      the upload of v2 descriptors, if configured.
-    - rendservice.c: Extended rend_service_t by a member secret_cookie.
-    - rendcommon.c: Added rend_encode_v2_descriptor() to encode a v2
-      descriptor.
-    - or.h: Added constant DIR_PURPOSE_UPLOAD_RENDDESC_V2.
-    - directory.c: Added directory_post_to_hs_dir().
-    - directory.c: Changed directory_initiate_command() to also recognize v2
-      publish requests.
-    - directory.c: Changed directory_send_command() to also prepare v2 publish
-      requests.
-    - crypto.c: Added implementation for crypto_cipher_encrypt_cbc().
-
-    - Changed connection_dir_client_reached_eof().
-
-      [Aug 11: Specified and running.]
+    stored at 3 consecutive nodes each. (requires /1/ and /2/)
 
   Hidden service client:
 
@@ -417,74 +259,13 @@ Design:
     fact that the availability will be the highest on the node with next higher
     ID. A hidden service client relies on the hidden service provider to store
     two sets of descriptors to compensate clock skew between service and
-    client. (requires /1/, /2/, and /3/)
-
-    - tor.1.in: Added the config option FetchV2HidServDescriptors.
-    - rend-spec.txt, section 1.5: Added the new v2 onion address format.
-    - rend-spec.txt, section 1.6: Alice's OP downloads the service descriptors
-      similarly as Bob's OP uploaded them in 1.4.
-
-    - rendcommon.c: Changed rend_cache_lookup_entry to enable it to also lookup
-      v2 descriptors.
-    - rendcommon.c: Added rend_compute_v2_desc_id() to generate v2 descriptor IDs
-      from v2 onion addresses.
-    - rendcommon.c: Changed rend_valid_service_id() to also consider v2 onion
-      addresses as valid and return the version number of the request (0 or 2).
-    - rendclient.c: Added rend_client_refetch_v2_renddesc() to fetch v2 service
-      descriptors using the secret cookie.
-    - rendclient.c: Changed rend_client_remove_intro_point() to copy the secret
-      cookie if the local descriptor has expired or there are no introduction
-      points left.
-    - or.h: Added FetchV2HidServDescriptors to or_options_t.
-    - or.h: Added constant REND_DESC_ID_V2_LEN to reflect that v2 descriptor
-      IDs are longer than v0/1 onion addresses.
-    - or.h: Added constant DIR_PURPOSE_FETCH_RENDDESC_V2.
-    - directory.c: Added directory_get_from_hs_dir().
-    - directory.c: Changed directory_initiate_command() to also recognize v2
-      fetch requests.
-    - directory.c: Changed directory_send_command() to also prepare v2 fetch
-      requests.
-    - connection_edge.c: Changed connection_ap_handshake_rewrite_and_attach()
-      to fetch v2 service descriptors.
-    - connection_edge.c: Changed parse_extended_hostname() to accept both,
-      current and v2 onion addresses.
-    - config.c: Added config options FetchV2HidServDescriptors.
-
-      [Aug 11: Base version specified and running, but no memory of failed
-       hidden service directories, yet.]
+    client. (requires /1/ and /2/)
 
   /14/ Process v2 fetch reply and parse v2 descriptors
 
     A hidden service client that has sent a request for a v2 descriptor can
     parse it and store it to the local cache of rendezvous service descriptors.
 
-    - rend-spec.txt, section 1.2: Added the new v2 hidden service descriptor
-      format.
-    - rend-spec.txt, section 1.6: Alice's OP parses the reply received from the
-      hidden service directory.
-
-    - routerparse.c: Added rend_parse_v2_service_descriptor() to parse a v2
-      hidden service descriptor.
-    - routerparse.c: Added rend_decrypt_introduction_points() to decrypt and
-      parse the list of introduction points.
-    - routerparse.c: Added ipo_token_table[] to parse the decrypted
-      introduction points of v2 hidden service descriptors.
-    - routerparse.c: Added desc_token_table[] to parse v2 hidden service
-      descriptors.
-    - routerparse.c: Added 8 keywords to directory_keyword to parse v2 hidden
-      service descriptors, and 5 to parse the decrypted list of introduction
-      points.
-    - rendcommon.c: Added rend_cache_store_v2_client() to parse a v2 descriptor
-      and parse the encrypted list of introduction points.
-    - or.h: Added rend_version and secret_cookie to edge_connection_t, to
-      dir_connection_t, and to origin_circuit_t to be able to decrypt
-      introduction points when receiving a v2 descriptor.
-    - directory.c: Changed connection_dir_client_reached_eof() to also parse v2
-      fetch replies.
-    - crypto.c: Added implementation for crypto_cipher_decrypt_cbc().
-
-      [July 9: Specified and running.]
-
   /15/ Establish connection to v2 hidden service
 
     A hidden service client can establish a connection to a hidden service
@@ -497,26 +278,6 @@ Design:
     that are included in the descriptor; by this, connection establishment is
     to a certain extend decoupled from fetching the descriptor.
 
-    - rend-spec.txt, section 1.8: Alice uses the public key that is included in
-      the descriptor instead of Bob's permanent service key.
-
-    - rendclient.c: Changed rend_client_introduction_acked() to copy the secret
-      cookie in case the introduction point denied the request.
-    - rendclient.c: Changed rend_client_remove_intro_point() to copy the secret
-      cookie if the local descriptor has expired or there are no introduction
-      points left.
-    - or.h: Added secret_cookie to edge_connection_t, to dir_connection_t, and
-      to origin_circuit_t to be able to decrypt introduction points when
-      receiving a v2 descriptor.
-    - circuitlist.c: Changed _circuit_mark_for_close() to pass the secret
-      cookie to rend_client_remove_intro_point() when an intro circ has failed.
-    - circuituse.c: Changed circuit_get_open_circ_or_launch() to fetch a v2
-      descriptor with the secret cookie, if no descriptor is available, or copy
-      the secret cookie to the circuit, in case it dies later, so that it can
-      be used to fetch a new descriptor.
-
-      [July 9: Base version specified and running, but without fresh key.]
-      
   Hidden service descriptor:
 
   (Requirements concerning the descriptor format are contained in /6/ and /7/.)
@@ -677,288 +438,3 @@ Compatibility:
   After the second transition phase, the authoritative directories should stop
   serving hidden service descriptors.
 
-Specification:
-
-  The proposed changes affect multiple sections in several specification
-  documents that are only mentioned in the following. (As for now, all changes
-  to specification documents are limited to the SVN branch 114-dist-storage.)
-
-  tor.1.in
-
-    Added the config options HidServDirectoryV2 (/5/),
-    PublishV2HidServDescriptors (/10/), and FetchV2HidServDescriptors (/13/).
-
-    Added the files hostname2 and secret_cookie (/10/).
-
-  dir-spec.txt
-
-    2.1  Added the flag hidden-service-dir to the router descriptor format
-    (/5/).
-
-    3.2  Added the status flag HSDir to the vote and consensus status
-    document format (/9/).
-
-    3.3  Added a rule for how an authority decides whether a router is assigned
-    the flag HSDir (/9/).
-
-  rend-spec.txt
-
-    0.4  Added history
-
-    1.1  Added requirement to create secret_cookie and hostname2 file (/10/).
-
-    1.2  Added the new v2 hidden service descriptor format (/6/, /12/ and
-    /14/).
-
-    1.3  Instead of Bob's public key, the hidden service provider uses a
-    freshly generated public key for every introduction point (/11/).
-
-    1.4  Added description of how to obtain a routing list of hidden service
-    directories (/1/).
-
-    1.4  Added description of how to determine the responsible node(s) for a
-    given descriptor ID (/2/).
-
-    1.4  Bob's OP does not only upload v0/v1 service descriptors to the
-    authoritative directories, but also v2 service descriptors to the hidden
-    service directories (/12/).
-
-    1.4  Added the requirement that requests need to be sent via Tor (/3/).
-
-    1.5  Added the new v2 onion address format (/13/).
-
-    1.6  Added the requirement that requests need to be sent via Tor (/3/).
-
-    1.6  Alice's OP downloads the service descriptors similarly as Bob's OP
-    uploaded them in 1.4 (/13/).
- 
-    1.6  Alice's OP parses the reply received from the hidden service directory
-    (/14/).
-
-    1.8  Alice uses the public key that is included in the descriptor instead
-    of Bob's permanent service key (/15/).
-
-    3.1: Added process of configuring a hidden service directory (/5/).
-
-    3.1: Added the decision on whether an onion router is confirmed to act as
-    hidden service directory or not (/9/).
-
-    3.2: Added the requirement that requests need to be contained within
-    BEGIN_DIR cells (/4/).
-
-    3.2: Added the acceptance of v2 publish requests (/6/).
-
-    3.3: Added the requirement that requests need to be contained within
-    BEGIN_DIR cells (/4/).
-  
-    3.3: Added the processing of v2 fetch requests (/7/).
-
-    3.3: Added the replication of v2 descriptors (/8/).
-
-Implementation:
-
-  The proposed changes affect the following changes in the source code. (As for
-  now, all changes to code are limited to the SVN branch 114-dist-storage.)
-
-  container.h
-
-    Added prototype for smartlist_digest_next_circular() (/2/).
-
-  container.c
-
-    Added implementation for smartlist_digest_next_circular() (/2/).
-
-  crypto.h
-
-    Added 3 prototypes according to the changes in crypto.c (various
-    requirements).
-
-  crypto.c
-
-    Added implementation for crypto_cipher_encrypt_cbc() (/12/).
-
-    Added implementation for crypto_cipher_decrypt_cbc() (/14/).
-
-    Added implementation for base32_decode() (various requirements).
-
-  circuitlist.c
-
-    Changed _circuit_mark_for_close() to pass the secret cookie to
-    rend_client_remove_intro_point() when an intro circ has failed (/15/).
-
-  circuituse.c
-
-    Changed circuit_get_open_circ_or_launch() to fetch a v2 descriptor with the
-    secret cookie, if no descriptor is available, or copy the secret cookie to
-    the circuit, in case it dies later, so that it can be used to fetch a new
-    descriptor (/15/).
-
-  config.c
-
-    Added config options FetchV2HidServDescriptors (/13/),
-    HidServDirectoryV2 (/5/), and PublishV2HidServDescriptors (/10/).
-
-  connection_edge.c
-
-    Changed connection_ap_handshake_rewrite_and_attach() to fetch v2 service
-    descriptors (/13/).
-
-    Changed parse_extended_hostname() to accept both, current and v2 onion
-    addresses (/13/).
-
-  directory.c
-
-    Added directory_post_to_hs_dir() (/12/).
-
-    Added directory_get_from_hs_dir() (/13/).
-
-    Changed directory_initiate_command() to also recognize v2 publish (/12/)
-    and fetch (/13/) requests.
-
-    Changed directory_send_command() to also prepare v2 publish (/12/) and
-    fetch (/13/) requests.
-
-    Changed connection_dir_client_reached_eof() to also parse v2 fetch replies
-    (/14/).
-
-    Changed directory_handle_command_get() to handle v2 fetch requests (/13/).
-
-    Changed directory_handle_command_post() to handle v2 publish requests
-    (/12/).
-
-  dirserv.c
-
-    Changed routerstatus_format_entry() to include the "HSDir" flag in vote and
-    consensus status documents (/9/).
-
-    Changed set_routerstatus_from_routerinfo() to set the "HSDir" flag (/9/).
-
-  or.h
-
-    Added constants DIR_PURPOSE_UPLOAD_RENDDESC_V2 (/12/) and
-    DIR_PURPOSE_FETCH_RENDDESC_V2 (/13/).
-
-    Added constant REND_DESC_ID_V2_LEN to reflect that v2 descriptor IDs are
-    longer than v0/1 onion addresses (/6/, /7/, and /13/).
-
-    Added rend_version and secret_cookie to edge_connection_t, to
-    dir_connection_t, and to origin_circuit_t to be able to decrypt
-    introduction points when receiving a v2 descriptor (/14/ and /15/).
-
-    Added is_hs_dir member to routerinfo_t and to routerstatus_t (/9/).
-
-    Added hs_dirs member to routerlist_t (/1/).
-
-    Added FetchV2HidServDescriptors (/13/), HidServDirectoryV2 (/5/), and
-    PublishV2HidServDescriptors (/10/) to or_options_t.
-
-    Added 7 new members to rend_service_descriptor_t to store v2-specific
-    information (/12/, /14/, and /15/).
-
-    Added 11 prototypes and changed the signature of 1 according to the
-    changes in .c files (various requirements).
-
-  rendclient.c
-
-    Changed rend_client_introduction_acked() to copy the secret cookie in case
-    the introduction point denied the request (/15/).
-
-    Added rend_client_refetch_v2_renddesc() to fetch v2 service descriptors
-    using the secret cookie (/13/).
-
-    Changed rend_client_remove_intro_point() to copy the secret cookie if the
-    local descriptor has expired or there are no introduction points left (/13/
-    and /15/).
-
-  rendcommon.c
-
-    Added rend_compute_v2_descriptor_fields() to prepare the encoding of a v2
-    descriptor (/12/).
-
-    Added rend_compute_desc_id() to generate v2 descriptor IDs from v2 onion
-    addresses (/13/).
-
-    Added rend_encode_v2_descriptor() to encode a v2 descriptor (/12/).
-
-    Changed rend_valid_service_id() to also consider v2 onion addresses as
-    valid and return the version number of the request (1 or 2) (/13/).
-
-    Changed rend_cache_lookup_entry to enable it to also lookup v2 descriptors
-    (/13/).
-
-    Added rend_cache_lookup_v2_dir() to allow a hidden service directory to
-    look up a v2 descriptor in the local cache under its descriptor ID instead
-    of its service ID (/7/).
-
-    Moved the parsing part from rend_cache_store() to the new function
-    rend_cache_store_parse() to reuse it for v2 descriptors (/6/).
-
-    Added rend_cache_store_v2_client() to parse a v2 descriptor and parse the
-    encrypted list of introduction points (/14/).
-
-    Added rend_cache_store_v2_dir() to allow a hidden service directory to
-    store a v2 descriptor in the local cache under its descriptor ID instead of
-    its service ID (/6/).
-
-  rendservice.c
-
-    Extended rend_service_t by a member secret_cookie (/10/ and /12/).
-
-    Added rend_get_hostname2() to assemble a v2 onion address (/10/).
-
-    Changed rend_service_load_keys() to write a secret_cookie and a hostname2
-    file (/10/).
-
-    Changed upload_service_descriptor() to upload v2 hidden service
-    descriptors, if configured (/12/).
-
-    Changed rend_consider_services_upload() to also initiate the upload of v2
-    descriptors, if configured (/12/).
-
-  router.c
-
-    Changed router_dump_router_to_string() to include the "hidden-service-dir"
-    flag in a router descriptor if configured (/5/).
-
-  routerlist.c
-
-    Changed router_get_routerlist() to initialize routing list (/1/).
-
-    Added get_responsible_hs_dir() to determine the router that is responsible
-    for a given descriptor ID (/2/).
-
-  routerparse.c
-
-    Added 14 keywords to directory_keyword; 1 to parse the "hidden-service-dir"
-    flag in router descriptors (/9/), 8 to parse v2 hidden service descriptors
-    (/6/ and /14/), and 5 to parse the decrypted list of introduction points
-    (/14/).
-
-    Added an entry to routerdesc_token_table[] to parse the
-    "hidden-service-directory" flag in router descriptors (/9/).
-
-    Added desc_token_table[] to parse v2 hidden service descriptors (/6/ and
-    /14/).
-
-    Added ipo_token_table[] to parse the decrypted introduction points of v2
-    hidden service descriptors (/14/).
-
-    Changed router_parse_entry_from_string() to parse the "hidden-service-dir"
-    flag in router descriptors (/9/).
-
-    Changed routerstatus_parse_entry_from_string to parse the "HSDir" flag in
-    vote and consensus status documents (/1/).
-
-    Added rend_parse_v2_service_descriptor() to parse a v2 hidden service
-    descriptor (/6/ and /14/).
-
-    Added rend_decrypt_introduction_points() to decrypt and parse the list of
-    introduction points (/14/).
-
-Test: 
-
-  The changes were tested via test functions in test.c for separate,
-  short-running functionality and using an automatic validation based on
-  PuppeTor.
-
-  

doc/spec/rend-spec.txt

@@ -109,6 +109,20 @@ $Id$
       39 -- RELAY_RENDEZVOUS_ESTABLISHED
       40 -- RELAY_COMMAND_INTRODUCE_ACK
 
+0.4. Version overview
+
+   There are several parts in the hidden service protocol that have
+   changed over time, each of them having its own version number, whereas
+   other parts remained the same. The following list of potentially
+   versioned protocol parts should help reduce some confusion:
+
+   - Hidden service descriptor: see 1.2. 
+
+   - Hidden service descriptor propagation mechanism: currently related to
+     the hidden service descriptor version; see 1.4 and 1.6.
+
+   - Introduction protocol: see 1.8.
+
 1. The Protocol
 
 1.1. Bob configures his local OP.
@@ -146,8 +160,9 @@ $Id$
     to do that if he previously advertised some introduction points,
     and now he doesn't have any. -RD]
 
-   The format of a "V2" descriptor, that will probably be used at some time
-   in the future, is as follows:
+   Beginning with 0.2.0.10-alpha, Bob's OP encodes "V2" descriptors in
+   addition to "V0" descriptors. The format of a "V2" descriptor is as
+   follows:
 
      "rendezvous-service-descriptor" descriptor-id NL
 
@@ -156,14 +171,14 @@ $Id$
        Indicates the beginning of the descriptor. "descriptor-id" is a
        periodically changing identifier of 160 bits formatted as 32 base32
        chars that is calculated by the hidden service and its clients. If
-       the optional "secret-cookie" is used, this "descriptor-id" cannot be
-       computed by anyone else. (Everyone can verify that this
+       the optional "descriptor-cookie" is used, this "descriptor-id"
+       cannot be computed by anyone else. (Everyone can verify that this
        "descriptor-id" belongs to the rest of the descriptor, even without
-       knowing the optional "secret-cookie", as described below.) The
+       knowing the optional "descriptor-cookie", as described below.) The
        "descriptor-id" is calculated by performing the following operation:
 
          descriptor-id =
-             H(permanent-id | H(time-period | secret-cookie | replica))
+             H(permanent-id | H(time-period | descriptor-cookie | replica))
 
        "permanent-id" is the permanent identifier of the hidden service,
        consisting of 80 bits. It can be calculated by computing the hash value
@@ -171,15 +186,15 @@ $Id$
 
          permanent-id = H(public-key)[:10]
 
-       "H(time-period | secret-cookie | replica)" is the (possibly secret)
-       id part that is
+       "H(time-period | descriptor-cookie | replica)" is the (possibly
+       secret) id part that is
        necessary to verify that the hidden service is the true originator
        of this descriptor. It can only be created by the hidden service
        and its clients, but the "signature" below can only be created by
        the service.
 
-       "secret-cookie" is an optional secret password of 128 bits that is
-       shared between the hidden service provider and its clients.
+       "descriptor-cookie" is an optional secret password of 128 bits that
+       is shared between the hidden service provider and its clients.
 
        "replica" denotes the number of the non-consecutive replica.
 
@@ -248,10 +263,10 @@ $Id$
 
        [At most once]
 
-       A list of introduction points. If the optional "secret-cookie" is
+       A list of introduction points. If the optional "descriptor-cookie" is
        used, this list is encrypted with AES in CTR mode with a random
        initialization vector of 128 bits that is written to
-       the beginning of the encrypted string, and the "secret-cookie" as
+       the beginning of the encrypted string, and the "descriptor-cookie" as
        secret key of 128 bits length.
 
        The string containing the introduction point data (either encrypted
@@ -260,9 +275,13 @@ $Id$
 
        The unencrypted string may begin with:
 
-        ["authentication" auth-type NL auth-data ... reserved]
+        ["service-authentication" auth-type NL auth-data ... reserved]
 
            [At start, any number]
+           
+           The service-specific authentication data can be used to perform
+           client authentication. This data is independent of the selected
+           introduction point as opposed to "intro-authentication" below.
 
        Subsequently, an arbitrary number of introduction point entries may
        follow, each containing the following data:
@@ -301,15 +320,14 @@ $Id$
            The public key that can be used to encrypt messages to the hidden
            service.
 
-        ["authentication" auth-type NL auth-data ... reserved]
+        ["intro-authentication" auth-type NL auth-data ... reserved]
 
            [Any number]
-           [XXXX this is valid at the start *and* at the end? -NM]
-           [These are two separate "authentication" fields. The one above
-            is global and independent from the introduction points, and
-            this one is specific for one introduction point. Should we use
-            different names for them? -KL]
-           [Probably. -NM]
+
+           The introduction-point-specific authentication data can be used
+           to perform client authentication. This data depends on the
+           selected introduction point as opposed to "service-authentication"
+           above.
 
         (This ends the fields in the encrypted portion of the descriptor.)
 
@@ -403,6 +421,17 @@ $Id$
    circuit with Bob's public key, and dissociates any other circuits
    currently associated with PK.  On success, the OR sends Bob a
    RELAY_INTRO_ESTABLISHED cell with an empty payload.
+ 
+   If a hidden service is configured to publish only v2 hidden service
+   descriptors, Bob's OP does not include its own public key in the
+   RELAY_ESTABLISH_INTRO cell, but the public key of a freshly generated
+   key pair. The OP also includes these fresh public keys in the v2 hidden
+   service descriptor together with the other introduction point
+   information. The reason is that the introduction point does not need to
+   and therefore should not know for which hidden service it works, so as
+   to prevent it from tracking the hidden service's activity. If the hidden
+   service is configured to publish both, v0 and v2 descriptors, two
+   separate sets of introduction points are established.
 
 1.4. Bob's OP advertises his service descriptor(s).
 
@@ -426,6 +455,34 @@ $Id$
    after its timestamp.  At least every 18 hours, Bob's OP uploads a
    fresh descriptor.
 
+   If Bob's OP is configured to publish v2 descriptors instead of or in
+   addition to v0 descriptors, it does so to a changing subset of all v2
+   hidden service directories instead of the authoritative directory
+   servers. Therefore, Bob's OP opens a stream via Tor to all
+   responsible hidden service directories. (He may re-use old circuits
+   for this.) Over this stream, Bob's OP makes an HTTP 'POST' request to a
+   URL "/tor/rendezvous2/publish" relative to the hidden service
+   directory's root, containing as its body Bob's service descriptor.
+
+   At any time, there are 6 hidden service directories responsible for
+   keeping replicas of a descriptor; they consist of 2 sets of 3 hidden
+   service directories with consecutive onion IDs. Bob's OP learns about
+   the complete list of hidden service directories by filtering the
+   consensus status document received from the directory authorities. A
+   hidden service directory is deemed responsible for all descriptor IDs in
+   the interval from its direct predecessor, exclusive, to its own ID,
+   inclusive; it further holds replicas for its 2 predecessors. A
+   participant only trusts its own routing list and never learns about
+   routing information from other parties.
+
+   Bob's OP publishes a new v2 descriptor once an hour or whenever its
+   content changes. V2 descriptors can be found by clients within a given
+   time period of 24 hours, after which they change their ID as described
+   under 1.2. If a published descriptor would be valid for less than 60
+   minutes (= 2 x 30 minutes to allow the server to be 30 minutes behind
+   and the client 30 minutes ahead), Bob's OP publishes the descriptor
+   under the ID of both, the current and the next publication period.
+
 1.5. Alice receives a x.y.z.onion address.
 
    When Alice receives a pointer to a location-hidden service, it is as a
@@ -475,6 +532,22 @@ $Id$
    [Caching may make her partitionable, but she fetched it anonymously,
     and we can't very well *not* cache it. -RD]
 
+   Alice's OP fetches v2 descriptors in parallel to v0 descriptors. Analog
+   to the description in section 1.4, the OP fetches a v2 descriptor from a
+   randomly chosen hidden service directory out of the changing subset of
+   6 nodes. If the request is unsuccessful, Alice retries the other
+   remaining responsible hidden service directories one after the other.
+   Alice relies on Bob to care about a potential clock skew between the two
+   by possibly storing two sets of descriptors.
+
+   Alice's OP opens a stream via Tor to the chosen v2 hidden service
+   directory. (She may re-use old circuits for this.) Over this stream,
+   Alice's OP makes an HTTP 'GET' request for the document
+   "/tor/rendezvous2/<z>", where z is replaced with the encoding of the
+   descriptor ID. The directory replies with a 404 HTTP response if it does
+   not recognize <z>, and otherwise returns Bob's most recently uploaded
+   service descriptor.
+
 1.7. Alice's OP establishes a rendezvous point.
 
    When Alice requests a connection to a given location-hidden service,
@@ -540,6 +613,10 @@ $Id$
    v1, and v2 since 0.1.1.x. As of Tor 0.2.0.7-alpha, clients switched
    to using the v2 intro format.
 
+   If Alice has downloaded a v2 descriptor, she uses the contained public
+   key ("service-key") instead of Bob's public key to create the
+   RELAY_INTRODUCE1 cell as described above.
+
 1.8.1. Other introduction formats we don't use.
 
     We briefly speculated about using the following format for the
@@ -633,3 +710,50 @@ $Id$
 2. Authentication and authorization.
 
 Foo.
+
+3. Hidden service directory operation
+
+   This section has been introduced with the v2 hidden service descriptor
+   format. It contains all operations of a v2 hidden service directory that
+   are required for the protocol described in section 1 to succeed with v2
+   hidden service descriptors.
+
+3.1. Configuring as hidden service directory
+
+   Every onion router that has its directory port open can decide whether it
+   wants to store and serve hidden service descriptors. An onion router which
+   is configured as such includes the "hidden-service-dir" flag in its router
+   descriptors that it sends to directory authorities.
+
+   The directory authorities include a new flag "HSDir" for routers that
+   decided to provide storage for hidden service descriptors and that are
+   running for at least 24 hours.
+
+3.2. Accepting publish requests
+
+   Hidden service directory nodes accept publish requests for v2 hidden service
+   descriptors and store them to their local memory. (It is not necessary to
+   make descriptors persistent, because after disconnecting, the onion router
+   would not be accepted as storing node anyway, because it has not been
+   running for at least 24 hours.) All requests and replies are formatted as
+   HTTP messages. Requests are contained within BEGIN_DIR cells, directed to
+   the router's directory port, and formatted as HTTP POST requests to the URL
+   "/tor/rendezvous2/publish" relative to the hidden service directory's root,
+   containing as its body a v2 service descriptor.
+
+   A hidden service directory node parses every received descriptor and only
+   stores it when it thinks that it is responsible for storing that descriptor
+   based on its own routing table. See section 1.4 for more information on how
+   to determine responsibility for a certain descriptor ID.
+
+3.3. Processing fetch requests
+
+   Hidden service directory nodes process fetch requests for hidden service
+   descriptors by looking them up in their local memory. (They do not need to
+   determine if they are responsible for the passed ID, because it does no harm
+   if they deliver a descriptor for which they are not (any more) responsible.)
+   All requests and replies are formatted as HTTP messages. Requests are
+   contained within BEGIN_DIR cells, directed to the router's directory port,
+   and formatted as HTTP GET requests for the document "/tor/rendezvous2/<z>",
+   where z is replaced with the encoding of the descriptor ID.
+

doc/tor.1.in

@@ -1025,6 +1025,11 @@ accepts and serves router descriptors, but it caches and serves the main
 networkstatus documents rather than generating its own. (Default: 0)
 .LP
 .TP
+\fBMinUptimeHidServDirectoryV2 \fR\fIN\fR \fBseconds\fR|\fBminutes\fR|\fBhours\fR|\fBdays\fR|\fBweeks\fP
+Minimum uptime of a v2 hidden service directory to be accepted as such by
+authoritative directories. (Default: 24 hours)
+.LP
+.TP
 \fBDirPort \fR\fIPORT\fP
 Advertise the directory service on this port.
 .LP

src/or/buffers.c

@@ -1440,7 +1440,8 @@ fetch_from_buf_socks(buf_t *buf, socks_request_t *req,
 "</p>\n"
 "<p>\n"
 "See <a href=\"https://www.torproject.org/documentation.html\">"
-           "https://www.torproject.org/documentation.html</a> for more information.\n"
+           "https://www.torproject.org/documentation.html</a> for more "
+           "information.\n"
 "<!-- Plus this comment, to make the body response more than 512 bytes, so "
 "     IE will be willing to display it. Comment comment comment comment "
 "     comment comment comment comment comment comment comment comment.-->\n"

src/or/command.c

@@ -776,3 +776,4 @@ command_process_link_auth_cell(cell_t *cell, or_connection_t *conn)
   connection_mark_for_close(TO_CONN(conn));
 }
 #endif
+

src/or/config.c

@@ -291,7 +291,6 @@ static config_var_t _option_vars[] = {
   VAR("__AllDirActionsPrivate",  BOOL,  AllDirActionsPrivate,     "0"),
   VAR("__DisablePredictedCircuits",BOOL,DisablePredictedCircuits, "0"),
   VAR("__LeaveStreamsUnattached",BOOL,  LeaveStreamsUnattached,   "0"),
-  /* XXX020 Karsten: this still needs a man page entry -RD */
   V(MinUptimeHidServDirectoryV2, INTERVAL, "24 hours"),
   { NULL, CONFIG_TYPE_OBSOLETE, 0, NULL }
 };
@@ -1901,7 +1900,8 @@ print_usage(void)
   printf(
 "Copyright 2001-2007 Roger Dingledine, Nick Mathewson.\n\n"
 "tor -f <torrc> [args]\n"
-"See man page for options, or https://www.torproject.org/ for documentation.\n");
+"See man page for options, or https://www.torproject.org/ for "
+"documentation.\n");
 }
 
 /** Print all non-obsolete torrc options. */

src/or/directory.c

@@ -1743,6 +1743,10 @@ connection_dir_client_reached_eof(dir_connection_t *conn)
           /* alice's ap_stream will notice when connection_mark_for_close
            * cleans it up */
           /*XXXX020 maybe retry quickly; timeout takes a while. */
+          /* This would require some kind of book-keeping which directories
+           * have been requested and which not. As directory servers are
+           * rather reliable, this should not be necessary, in constrast to
+           * v2 hidden service directories. -KL */
         } else {
           /* success. notify pending connections about this. */
           conn->_base.purpose = DIR_PURPOSE_HAS_FETCHED_RENDDESC;
@@ -1753,6 +1757,10 @@ connection_dir_client_reached_eof(dir_connection_t *conn)
         /* not there. pending connections will be notified when
          * connection_mark_for_close cleans it up. */
         /*XXXX020 maybe retry quickly; timeout takes a while. */
+        /* This would require some kind of book-keeping which directories
+         * have been requested and which not. As directory servers are
+         * rather reliable, this should not be necessary, in constrast to
+         * v2 hidden service directories. -KL */
         break;
       case 400:
         log_warn(LD_REND,
@@ -2706,7 +2714,8 @@ directory_handle_command_post(dir_connection_t *conn, const char *headers,
     switch (r) {
       case -1:
         log_notice(LD_DIRSERV,
-                   "Rejected router descriptor or extra-info from %s (\"%s\").",
+                   "Rejected router descriptor or extra-info from %s "
+                   "(\"%s\").",
                    conn->_base.address, msg);
         /* fall through */
       case 1:

src/or/or.h

@@ -1208,7 +1208,8 @@ typedef struct signed_descriptor_t {
   size_t annotations_len;
   /** Length of the server descriptor. */
   size_t signed_descriptor_len;
-  /** Digest of the server descriptor, computed as specified in dir-spec.txt. */
+  /** Digest of the server descriptor, computed as specified in
+   * dir-spec.txt. */
   char signed_descriptor_digest[DIGEST_LEN];
   /** Identity digest of the router. */
   char identity_digest[DIGEST_LEN];