eviscerate TODO and stitch it together into a far more attractive, albeit slightly less organic, creature

svn:r4402

doc/TODO

@@ -12,26 +12,56 @@ PHOBOS  - phobos claims
         D Deferred
         X Abandoned
 
+Non-Coding, Soon:
+  - FAQ entry: why gnutls is bad/not good for tor
+P - flesh out the rest of the section 6 of the faq
+P - gather pointers to livecd distros that include tor
+  - put the logo on the website, in source form, so people can put it on
+    stickers directly, etc.
+  - more pictures from ren. he wants to describe the tor handshake, i want to
+    talk about hidden services.
+  * clean up the places where our docs are redundant (or worse, obsolete in
+    one file and correct elsewhere). agl has a start on a global
+    list-of-tor-docs.
+P - update window's docs to clarify which versions of windows, and why a
+    DOS window, how it's used, for the less technical users
+NR- write a spec appendix for 'being nice with tor'
+  - Hunt for open socks ports on tor servers, send mail
+  - tor-in-the-media page
+  - Ask schanzle@cas.homelinux.org about a patch for  rpm spec fixes against
+    tor-0.1.0.7.rc
+  - Remove need for HACKING file.
+
+
 For 0.1.0.x:
   o Why do solaris cpuworks go dormant?
     (Apparently, disabling threads fixes this.)
-  - Why does kevent barf with EINVAL on some freebsd boxes?
-    (Lucky is trying a patch that should tell us why.)
   o Fix the remaining flyspray bugs marked for 0.1.0.9
   X Free remaining unfreed memory (arma will run valgrind)
     (Not for a stable release)
   o Note libevent/method/platform combos that are unlikely to work.
+  X change torrc to point to abuse-faq (once abuse-faq is posted)
   . Memory use on Linux: what's happening?
     - Is it threading?  (Maybe, maybe not)
     - Is it the buf_shrink bug? (Quite possibly)
-  - instrument the code to figure out where our memory is going. (all platforms?)
-  X change torrc to point to abuse-faq (once abuse-faq is posted)
+    - Instrument the 0.1.1 code to figure out where our memory is going;
+      apply the results. (all platforms?)
+  - Why does kevent barf with EINVAL on some freebsd boxes?
+    - Submit libevent patch to Niels
+    - Warn on non-repeated EINVAL in Tor (don't die.)
+    - Investigate why freebsd kernel actually does this: it doesn't seem
+      simple to trigger.
 
 for 0.1.1.x:
   - switch accountingmax to count total in+out, not either in or
     out. it's easy to move in this direction (not risky), but hard to
     back, out if we decide we prefer it the way it already is. hm.
-  - Start using create-fast cells as clients
+  . Come up with a coherent strategy for bandwidth buckets and TLS. (The
+    logic for reading from TLS sockets is likely to overrun the bandwidth
+    buckets under heavy load.  (Really, the logic was never right in the
+    first place.)  Also, we should audit all users of get_pending_bytes().)
+    - Make it harder to circumvent bandwidth caps: look at number of bytes
+      sent across sockets, not number sent inside TLS stream.
   - Handle rendezvousing with unverified nodes.
     - Specify: Stick rendezvous point's key in INTRODUCE cell.
       Bob should _always_ use key from INTRODUCE cell.
@@ -41,26 +71,31 @@ for 0.1.1.x:
   - it looks like tor_assert writes to stderr. what happens if
     stderr was closed and is now something else? uh.
   - new controller protocol
-  - Let more config options (e.g. ORPort) change dynamically.
+    - Specify
+    - Implement
   - controller should have an event to learn about new addressmappings,
     e.g. when we learn a hostname to IP mapping ?
   - christian grothoff's attack of infinite-length circuit.
     the solution is to have a separate 'extend-data' cell type
     which is used for the first N data cells, and only
     extend-data cells can be extend requests.
+    - Specify, including thought about
+    - Implement
   - Destroy and truncated cells should have reasons.
   - Add private:* alias in exit policies to make it easier to ban all the
     fiddly little 192.168.foo addresses.
+    (AGL had a patch; consider applying it.)
   - recommended-versions for client / server ?
-  - whine if your socks port is an open proxy.
+  - warn if listening for SOCKS on public IP.
+  - Forward-compatibility: add "needclientversion" option or "opt critical"
+    prefix.
   - cpu fixes:
-    M rewrite how libevent does select() on win32 so it's not so very slow.
-    - create-light
     - see if we should make use of truncate to retry
     - hardware accelerator support
-    - use openssl aes when available
-    - do the kernel buffer style design
+r   - kill dns workers more slowly
   - continue decentralizing the directory
+    - Specify and design all of the below before implementing any.
+    - Figure out what to do about hidden service descriptors.
     M have two router descriptor formats
     - dirservers verify reachability claims
     - find 10 dirservers. (what are criteria to be a dirserver?)
@@ -71,6 +106,8 @@ for 0.1.1.x:
         before we approve them
       - other?
     - dirservers publish router-status with all these flags.
+    - Servers publish new descriptors when options change, when 12-24 hours
+      have passed, when uptime is reset, or when bandwidth changes a lot.
     - alices fetch many router-statuses and update descriptors as needed.
     - add if-newer-than fetch options
     - dirservers allow people to lookup by N descriptors, or to fetch all.
@@ -80,6 +117,7 @@ for 0.1.1.x:
     - Compress router desc sets before transmitting them
     M Analyze how bad the partitioning is or isn't.
   - Naming:
+    - Specify and design all of the below before implementing any.
     - some dirservers announce that they manage bindings (a flag in
       router-status).
     - other dirservers mention a binding if there is no conflict for
@@ -92,177 +130,116 @@ for 0.1.1.x:
       the one she meant.
     - if the binding changes keys, the entry in her datadir will silently
       get corrected.
-  - helper nodes (at least preliminary)
-  - enclaves (at least preliminary)
   - packaging and ui stuff:
     - multiple sample torrc files (tyranix?)
     - uninstallers
       . for os x
     - something, anything, for sys tray on Windows.
-    - figure out how to make nt service stuff the default?
+    - figure out how to make nt service stuff work?
       . Document it.
+N   - Vet all pending installer patches
+      - Win32 installer plus privoxy, sockscap/freecap, etc.
+      - Vet win32 systray helper code
+N  . Make logs go into platform default locations.
+     o OSX
+     - Windows. (?)
+
+Reach (deferrable) items for 0.1.1.x:
+  - Start using create-fast cells as clients
+  - Let more config options (e.g. ORPort) change dynamically.
+  - start handling server descriptors without a socksport?
 
-
-
-For sometime soon:
-   - Server instructions for OSX and Windows operators.
-   - Audit all changes to bandwidth buckets for integer over/underflow.
-
- Refactoring and infrastructure:
-
-N . Switch to libevent
-       - Hold-open-until-flushed now works by accident; it should work by
-         design.
-       . The logic for reading from TLS sockets is likely to overrun the
-         bandwidth buckets under heavy load.  (Really, the logic was
-         never right in the first place.)  Also, we should audit all users
-         of get_pending_bytes().
-
- Documentation
-r - Correct and clarify the wiki entry on port forwarding.
-  o Document where OSX logs and torrc go.
-  o Document where windows logs and torrc go.
-    - (Make sure they actually go there.)
-
- Installers
-N - Vet all pending installer patches
-    - Win32 installer plus privoxy, sockscap/freecap, etc.
-    - Vet win32 systray helper code
-N . Make logs go into platform default locations.
-    o OSX
-    - Windows. (?)
-
- Correctness
-   - how do ulimits work on win32, anyway?  (We should handle WSAENOBUFS as
-     needed, look at the MaxConnections registry entry, look at the
-     MaxUserPort entry, and look at the TcpTimedWaitDelay entry. We may also
-     want to provide a way to set them as needed. See bug 98.)
-
-  - Bug: Why do idle cpuworkers sometimes get thought of as busy?
-
-N - tor-resolve script should use socks5 to get better error messages.
-  o Script to try pulling bytes through slow-seeming servers so they can
-    notice that they might be fast.
-N . Reverse DNS
-    o specify
-    - implement
-r - make min uptime a function of the available choices (say, choose 60th
+For 0.1.1.x, if we can figure out how:
+  - rewrite how libevent does select() on win32 so it's not so very slow.
+  - helper nodes (at least preliminary)
+  - enclaves (at least preliminary)
+  - Write limiting; separate token bucket for write
+  - Audit everything to make sure rend and intro points are just as likely to
+    be us as not.
+  - Do something to prevent spurious EXTEND cells from making middleman
+    nodes connect all over.  Rate-limit failed connections, perhaps?
+
+Future version:
+  - Limit to 2 dir, 2 OR, N SOCKS connections per IP.
+  - Handle full buffers without totally borking
+  - Rate-limit OR and directory connections overall and per-IP and
+    maybe per subnet.
+  - Hold-open-until-flushed now works by accident; it should work by
+    design.
+  - DoS protection: TLS puzzles, public key ops, bandwidth exhaustion.
+   - Specify?
+  - tor-resolve script should use socks5 to get better error messages.
+  - make min uptime a function of the available choices (say, choose 60th
     percentile, not 1 day.)
-r - kill dns workers more slowly
-r - build testing circuits? going through non-verified nodes?
   - config option to publish what ports you listen on, beyond ORPort/DirPort
-N - It would be nice to have a FirewalledIPs thing that works like
-    FirewallPorts.
-  - If we have a trusted directory on port 80, stop falling back to
-    forbidden ports when fascistfirewall blocks all good dirservers.
-N - Code cleanup
-    - Make configure.in handle cross-compilation
-      - Have NULL_REP_IS_ZERO_BYTES default to 1.
-      - Make with-ssl-dir disable search for ssl.
-  - Efficiency/speed improvements.
-    - Write limiting; configurable token buckets.
-    - Make it harder to circumvent bandwidth caps: look at number of bytes
-      sent across sockets, not number sent inside TLS stream.
-
-
-  No
-Todo: when you connect and get a guy you didn't expect, tell him hey i wasn't
-expecting you i'm going to go now bye, instead of just hanging up. This lets
-him know that he's doing something funny.
-  - choose entry node to be one you're already connected to?
-  X Convert man pages to pod, or whatever's right.
-  X support hostnames as well as IPs for authdirservers.
-  - GPSLocation optional config string.
-  - Windows
-    - Make millisecond accuracy work on win32
-  - IPv6 support
-     - teach connection_ap_handshake_socks_reply() about ipv6 and friends
-       so connection_ap_handshake_socks_resolved() doesn't also need
-       to know about them.
   - hidserv offerers shouldn't need to define a SocksPort
     * figure out what breaks for this, and do it.
+  - auth mechanisms to let hidden service midpoint and responder filter
+    connection requests.
+  - Relax clique assumptions.
+
+Blue-sky:
+  - Patch privoxy and socks protocol to pass strings to the browser.
+  - Standby/hotswap/redundant hidden services.
+  - Robust decentralized storage for hidden service descriptors.
+  - The "China problem"
+  - Allow small cells and large cells on the same network?
+  - Cell buffering and resending. This will allow us to handle broken
+    circuits as long as the endpoints don't break, plus will allow
+    connection (tls session key) rotation.
+  - Implement Morphmix, so we can compare its behavior, complexity, etc.
+  - Other transport. HTTP, udp, rdp, airhook, etc. May have to do our own
+    link crypto, unless we can bully openssl into it.
 
-   - Packaging
-     X Figure out how to make the rpm not strip the binaries it makes.
-   - Integrate an http proxy into Tor (maybe as a third class of worker
-     process), so we can stop shipping with the beast that is Privoxy.
-   - Implement If-Modified-Since for directories.
-   - Windows
-     - Get a controller to launch tor and keep it on the system tray.
-
-Tier two:
-
-N  - IPv6 support (For exit addresses)
-     - Spec issue: if a resolve returns an IP4 and an IP6 address,
-       which to use?
-     - Add to exit policy code
-     - Make tor_gethostbyname into tor_getaddrinfo
-     - Make everything that uses uint32_t as an IP address change to use
-       a generalize address struct.
-     - Change relay cell types to accept new addresses.
-     - Add flag to serverdescs to tell whether IPv6 is supported.
-
-   - Code cleanup
-     o fix router_get_by_* functions so they can get ourselves too ...
-     - and audit everything to make sure rend and intro points are
-       just as likely to be us as not.
-
-   - tor should be able to have a pool of outgoing IP addresses
-     that it is able to rotate through. (maybe)
-
-   Packaging, docs, etc:
+Volunteer projects:  [Phobos moves these to contribute.html]
+  - use openssl aes when available
+  - do the kernel buffer style design
+  - Server instructions for OSX and Windows operators.
+  - Improve and clarify the wiki entry on port forwarding.
+  - how do ulimits work on win32, anyway?  (We should handle WSAENOBUFS as
+    needed, look at the MaxConnections registry entry, look at the
+    MaxUserPort entry, and look at the TcpTimedWaitDelay entry. We may also
+    want to provide a way to set them as needed. See bug 98.)
+  - Implement reverse DNS (already specified)
+  - It would be nice to have a FirewalledIPs thing that works like
+    FirewallPorts.
+  - Make configure.in handle cross-compilation
+    - Have NULL_REP_IS_ZERO_BYTES default to 1.
+    - Make with-ssl-dir disable search for ssl.
+  - Packaging, docs, etc:
    - Exit node caching: tie into squid or other caching web proxy.
-
-   Deferred until needed:
-   - Do something to prevent spurious EXTEND cells from making middleman
-     nodes connect all over.  Rate-limit failed connections, perhaps?
-   - Limit to 2 dir, 2 OR, N SOCKS connections per IP.
-   - Handle full buffers without totally borking
-     * do this eventually, no rush.
-   - Rate-limit OR and directory connections overall and per-IP and
-     maybe per subnet.
-   - DoS protection: TLS puzzles, public key ops, bandwidth exhaustion.
-   - Have clients and dirservers preserve reputation info over
-     reboots.
-   - authdirserver lists you as running iff:
-     - he can connect to you
-     - he has successfully extended to you
-     - you have sufficient mean-time-between-failures
-     * keep doing nothing for now.
-
-   Blue sky or deferred indefinitely:
-   - Support egd or other non-OS-integrated strong entropy sources
-   - password protection for on-disk identity key
-   - Possible to get autoconf to easily install things into ~/.tor?
-   - server descriptor declares min log level, clients avoid servers
+  - Have clients and dirservers preserve reputation info over
+    reboots.
+  - Support egd or other non-OS-integrated strong entropy sources
+  - password protection for on-disk identity key
+  - Possible to get autoconf to easily install things into ~/.tor?
+  - server descriptor declares min log level, clients avoid servers
      that are too loggy.
-   - put expiry date on onion-key, so people don't keep trying
-     old ones that they could know are expired?
-   - Add a notion of nickname->Pubkey binding that's not 'verification'
-   - Conn key rotation.
-   - Need a relay teardown cell, separate from one-way ends.
-
-Big tasks that would demonstrate progress:
-
-   - Facility to automatically choose long-term helper nodes; perhaps
-     on by default for hidden services.
-   - patch privoxy and socks protocol to pass strings to the browser.
-   - patch tsocks with our current patches + gethostbyname, getpeername, etc.
-   - make freecap (or whichever) do what we want.
-   - scrubbing proxies for protocols other than http.
-     - Find an smtp proxy?
-     . Get socks4a support into Mozilla
-   - figure out enclaves, e.g. so we know what to recommend that people
-     do, and so running a tor server on your website is helpful.
-     - Do enclaves for same IP only.
-     - Resolve first, then if IP is an OR, extend to him first.
-   - implement a trivial fun gui to demonstrate our control interface.
-
-************************ Roadmap for 2004-2005 **********************
+  - Separate node discovery from routing to allow neat extensions. [Goodell?]
+    - Add SetServerStatus control event to adjust verified/running status of
+      nodes.
+    - Add NoDownload config option to prevent regular directory downloads
+      from happening.
+  - Choosing exit node by meta-data, e.g. country.
+  - What info squeaks by Privoxy? Are other scrubbers better?
+  - web proxy gateways to let normal people browse hidden services.
+    (This has been done a few times, but nobody has sent us code.)
+  - Use cpuworker for more heavy lifting.
+    - Signing (and verifying) hidserv descriptors
+    - Signing (and verifying) intro/rend requests
+    - Signing (and verifying) router descriptors
+    - Signing (and verifying) directories
+    - Doing TLS handshake (this is very hard to separate out, though)
+  - Buffer size pool: allocate a maximum size for all buffers, not a maximum
+    size for each buffer. So we don't have to give up as quickly (and kill
+    the thickpipe!) when there's congestion.
+  - Congestion control. Is our current design sufficient once we have heavy
+    use? Need to measure and tweak, or maybe overhaul.
+  - Add alternative versions of crypto.c and tortls.c to use libnss or
+    libgcrypt+gnutls.
 
-Hard problems that need to be solved:
 
-  - Separating node discovery from routing.
+Research projects: [Phobos moves these to contribute.html]
   - Arranging membership management for independence.
     Sybil defenses without having a human bottleneck.
     How to gather random sample of nodes.
@@ -270,109 +247,48 @@ Hard problems that need to be solved:
     Consider incremental switches: a p2p tor with only 50 users has
       different anonymity properties than one with 10k users, and should
       be treated differently.
-  - Measuring performance of other nodes. Measuring whether they're up.
-  - Choosing exit node by meta-data, e.g. country.
   - Incentives to relay; incentives to exit.
   - Allowing dissidents to relay through Tor clients.
-  - How to intercept, or not need to intercept, dns queries locally.
-  - Improved anonymity:
-    - Experiment with mid-latency systems. How do they impact usability,
-      how do they impact safety?
-    - Understand how powerful fingerprinting attacks are, and experiment
-      with ways to foil them (long-range padding?).
-    - Come up with practical approximations to picking entry and exit in
-      different routing zones.
-    - Find ideal churn rate for helper nodes; how safe is it?
-    - What info squeaks by Privoxy? Are other scrubbers better?
-    - Attacking freenet-gnunet/timing-delay-randomness-arguments.
-    - Is abandoning the circuit the only option when an extend fails, or
-      can we do something without impacting anonymity too much?
-    - Is exiting from the middle of the circuit always a bad idea?
-
-Sample Publicity Landmarks:
-
-  - we have N servers / N users
-  - we have servers at epic and aclu and foo
-  - hidden services are robust and fast
-  - a more decentralized design
-  - tor win32 installer works
-  - win32 tray icon for end-users
-  - tor server works on win32
-  - win32 service for servers
-  - mac installer works
-
-***************************Future tasks:****************************
-
-Rendezvous and hidden services:
-  make it fast:
-    o preemptively build and start rendezvous circs.
-    o preemptively build n-1 hops of intro circs?
-    o cannibalize general circs?
-  make it reliable:
-    - standby/hotswap/redundant services.
-    - store stuff to disk? dirservers forget service descriptors when
-      they restart; nodes offering hidden services forget their chosen
-      intro points when they restart.
-  make it robust:
-    - auth mechanisms to let midpoint and bob selectively choose
-      connection requests.
-  make it scalable:
-    - robust decentralized storage for hidden service descriptors.
-  make it accessible:
-    - web proxy gateways to let normal people browse hidden services.
-
-Tor scalability:
-  Relax clique assumptions.
-  Redesign how directories are handled.
-    - Resolve directory agreement somehow.
-  Find and remove bottlenecks
-    - Address linear searches on e.g. circuit and connection lists.
-  Reputation/memory system, so dirservers can measure people,
-    and so other people can verify their measurements.
-    - Need to measure via relay, so it's not distinguishable.
-  Let dissidents get to Tor servers via Tor users. ("Backbone model")
+  - Experiment with mid-latency systems. How do they impact usability,
+    how do they impact safety?
+  - Understand how powerful fingerprinting attacks are, and experiment
+    with ways to foil them (long-range padding?).
+  - Come up with practical approximations to picking entry and exit in
+    different routing zones.
+  - Find ideal churn rate for helper nodes; how safe is it?
+  - Attacking freenet-gnunet/timing-delay-randomness-arguments.
+  - Is exiting from the middle of the circuit always a bad idea?
+  - IPv6 support (For exit addresses)
+    - Spec issue: if a resolve returns an IP4 and an IP6 address,
+      which to use?
+    - Add to exit policy code
+    - Make tor_gethostbyname into tor_getaddrinfo
+    - Make everything that uses uint32_t as an IP address change to use
+      a generalize address struct.
+    - Change relay cell types to accept new addresses.
+    - Add flag to serverdescs to tell whether IPv6 is supported.
+  - patch tsocks with our current patches + gethostbyname, getpeername, etc.
+  - make freecap (or whichever) do what we want.
+  - scrubbing proxies for protocols other than http.
+  - We need better default privoxy configs to ship.
+  - We need a good scrubbing HTTP proxy; privoxy is unmaintained and sucky.
+  - A DNS proxy would let unmodified socks4/socks5 apps to work well.
+  - Add SOCKS support to more applications
+  - store hidden service information to disk: dirservers forget service
+    descriptors when they restart; nodes offering hidden services forget
+    their chosen intro points when they restart.
+
+
+Did we do these ones already? XXXXX
+  - If we have a trusted directory on port 80, stop falling back to
+    forbidden ports when fascistfirewall blocks all good dirservers.
 
-Make it more correct:
-  Handle half-open connections: right now we don't support all TCP
+I don't have any record of what we decided about these: XXXXX
+  - Make millisecond accuracy work on win32
+  - tor should be able to have a pool of outgoing IP addresses
+    that it is able to rotate through. (maybe)
+  - Conn key rotation.
+  - Need a relay teardown cell, separate from one-way ends.
+  - Handle half-open connections: right now we don't support all TCP
     streams, at least according to the protocol. But we handle all that
     we've seen in the wild.
-  Support IPv6.
-
-Efficiency/speed/robustness:
-  Congestion control. Is our current design sufficient once we have heavy
-    use? Need to measure and tweak, or maybe overhaul.
-  Allow small cells and large cells on the same network?
-  Cell buffering and resending. This will allow us to handle broken
-    circuits as long as the endpoints don't break, plus will allow
-    connection (tls session key) rotation.
-  Implement Morphmix, so we can compare its behavior, complexity, etc.
-  Use cpuworker for more heavy lifting.
-    - Signing (and verifying) hidserv descriptors
-    - Signing (and verifying) intro/rend requests
-    - Signing (and verifying) router descriptors
-    - Signing (and verifying) directories
-    - Doing TLS handshake (this is very hard to separate out, though)
-  Buffer size pool: allocate a maximum size for all buffers, not
-    a maximum size for each buffer. So we don't have to give up as
-    quickly (and kill the thickpipe!) when there's congestion.
-  Other transport. HTTP, udp, rdp, airhook, etc. May have to do our own
-    link crypto, unless we can bully openssl into it.
-
-*********** uncategorized
-
-    - why gnutls is bad/not good for tor
-P   - flesh out the rest of the section 6 of the faq
-P   - gather pointers to livecd distros that include tor
-    - desired contribute.html patch: patches for dir-servers to verify server reachability.
-P   - i want to put the logo on the website, in source form, so people can put it on stickers directly, etc.
-    - i want more pictures from ren. he wants to describe the tor
-      handshake, i want to talk about hidden services.
-    - clean up the places where our docs are redundant (or worse, obsolete in one file and correct elsewhere). agl has a start on a global list-of-tor-docs.
-P   - update window's docs to clarify which versions of windows, and why a DOS window, how it's used, for the less technical users
-
-    - write a spec appendix for 'being nice with tor'
-    - Hunt for open socks ports on tor servers, send mail
-    - tor-in-the-media page
-    - schanzle@cas.homelinux.org: rpm spec fixes for tor-0.1.0.7.rc
-    - start handling server descriptors without a socksport?
-