rearrange TODO a lot; still needs more.

svn:r3065

doc/TODO

@@ -11,101 +11,84 @@ ARMA    - arma claims
         X Abandoned
 
 For 0.0.9:
+
 N&R. bring tor-spec up to date
-   o cache and serve running-routers on other nodes?
-     o cache running-routers
-     o download running-routers from servers running rc5-cvs or later
-   o pump up periods for fetching things; figure out how to do this
-     backward-compatibily, so that people who did set dirfetchpostperiod
-     get the right behavior.
-     o If dirport is set, we should have a maximum dirfetchperiod and
-       a maximum statusfetchperiod, or else we'll serve very stale stuff.
-   o Adapt version parsing code to handle new version scheme; document new
-     version scheme.
 N&R. make loglevels info,debug less noisy
-   D fix dfc/weasel's intro point bug
-   o add goodell's .exit tld
 N  - Get win32 servers working, or find out why it isn't happening now.
 
-Beyond 0.0.9:
+************************ For Post 0.0.9 *****************************
+
+Tier one:
+   - fix dfc/weasel's intro point bug
    - support hostnames as well as IPs for authdirservers.
-   - server descriptor declares min log level, clients avoid servers
-     that are too loggy.
-N  - Clean up NT service code
 N  - OS X package (and bundle?)
-N  - Reverse DNS: specify and implement.
 N  - Make millisecond accuracy work on win32
+   - Make more configuration variables into CSVs.
+   - Once we have a trusted directory on port 80, stop falling back to
+     forbidden ports when fascistfirewall blocks all good dirservers.
+   - Convert man pages to pod, or whatever's right.
+   - Move to our new version system.
+   - Get more nodes running on 80 and 443.
+   - Get epic, aclu, etc running nodes.
+   - Start distributing an rpm with the new version scheme.
+   - Bug tracker.
+
+Tier two:
+
+   - Handle pools of waiting circuits better.
+   - Let more config options (e.g. ORPort) change dynamically.
+   - Write limiting; configurable token buckets.
+   - Only the top of a directory needs to be signed.
+   - Make sure logged information is 'safe'.
+   - make advertised_server_mode() ORs fetch dirs more often.
+
+N  - Clean up NT service code
+   - Work as an NT service; on system tray; etc.
+   - Win32 installer plus privoxy, sockscap/freecap, etc.
    - controller should have 'getinfo' command to query about rephist,
      about rendezvous status, etc.
-   - allow transition from ORPort to !ORPort, and back
-R  . bandwidth buckets for write as well as read.
-   - Limit to 2 dir, 2 OR, N SOCKS connections per IP.
    - Implement If-Modified-Since for directories.
-   - Make more configuration variables into CSVs.
 N  - Handle rendezvousing with unverified nodes.
      - Specify: Stick rendezvous point's key in INTRODUCE cell.
        Bob should _always_ use key from INTRODUCE cell.
      - Implement.
-R  - figure out enclaves, e.g. so we know what to recommend that people
-     do, and so running a tor server on your website is helpful.
-     - Do enclaves for same IP only.
-     - Resolve first, then if IP is an OR, connect to next guy.
-N  . the user interface interface
-     - Implement a trivial fun gui.
 N  - add ipv6 support.
      - Spec issue: if a resolve returns an IP4 and an IP6 address,
        which to use?
-N&R  - Update Spec
-R  X learn from ben about his openssl-reinitialization-trick to
-     rotate tls keys without making new connections.
-   - Do something to prevent spurious EXTEND cells from making middleman
-     nodes connect all over.  Rate-limit failed connections, perhaps?
    - christian grothoff's attack of infinite-length circuit.
      the solution is to have a separate 'extend-data' cell type
      which is used for the first N data cells, and only
      extend-data cells can be extend requests.
-   - have a pool of circuits available, cannibalize them
-     for your purposes (e.g. rendezvous, etc).
-   - Once we have a trusted directory on port 80, stop falling back to
-     forbidden ports when fascistfirewall blocks all good dirservers.
+   . rename/rearrange functions for what file they're in
+   - tor should be able to have a pool of outgoing IP addresses
+     that it is able to rotate through. (maybe)
+   - hidserv offerers shouldn't need to define a SocksPort
+     * figure out what breaks for this, and do it.
+   - should retry exitpolicy end streams even if the end cell didn't
+     resolve the address for you
+   - Make it harder to circumvent bandwidth caps: look at number of bytes
+     sent across sockets, not number sent inside TLS stream.
+   - fix router_get_by_* functions so they can get ourselves too,
+     and audit everything to make sure rend and intro points are
+     just as likely to be us as not.
 
-   o fix sprintf's to snprintf's?
-   . Make intro points and rendezvous points accept $KEYID in addition
-     to nicknames.
-           o Specify
-           o Implement parsing
-           - Generate new formats (Not till 007 is dead)
-   - Facility to automatically choose long-term helper nodes; perhaps
-     on by default for hidden services.
-   o Make command-line strict about checking options; make only certain
-     option prefixes work.
+   Packaging, docs, etc:
+   - Exit node caching: tie into squid or other caching web proxy.
+   - FAQ.
+   - Website spiffying. Logo. Pictures.
+   - Configuration walk-through with screenshots of each step.
+
+   Deferred until needed:
+   - Do something to prevent spurious EXTEND cells from making middleman
+     nodes connect all over.  Rate-limit failed connections, perhaps?
+   - Limit to 2 dir, 2 OR, N SOCKS connections per IP.
+   - Handle full buffers without totally borking
+     * do this eventually, no rush.
    - Rate-limit OR and directory connections overall and per-IP and
      maybe per subnet.
-   D put expiry date on onion-key, so people don't keep trying
-     old ones that they could know are expired?
-     * Leave on todo list, see if pre3 onion fixes helped enough.
-   D should the running-routers list put unverified routers at the
-     end?
-     * Cosmetic, don't do it yet.
-   D make advertised_server_mode() ORs fetch dirs more often.
-     * not necessary yet.
-   D Add a notion of nickname->Pubkey binding that's not 'verification'
-     * eventually, only when needed
-   D ORs use uniquer default nicknames
-     * Don't worry about this for now
-   D Handle full buffers without totally borking
-     * do this eventually, no rush.
-   D if destination IP is running a tor node, extend a circuit there
-     before sending begin.
-     * don't do this for now. figure out how enclaves work. but do
-       enclaves soon.
-   - Support egd or other non-OS-integrated strong entropy sources
-
-   more features, complex:
-   - password protection for on-disk identity key
+   - DoS protection: TLS puzzles, public key ops, bandwidth exhaustion.
    - Have clients and dirservers preserve reputation info over
      reboots.
-     * continue not doing until we have something we need to preserve
    - round detected bandwidth up to nearest 10KB?
    - client software not upload descriptor until:
      - you've been running for an hour
@@ -122,89 +105,77 @@ R  X learn from ben about his openssl-reinitialization-trick to
      * keep doing nothing for now.
    - Include HTTP status messages in logging (see parse_http_response).
 
-   blue sky:
+   Blue sky or deferred indefinitely:
+   - Support egd or other non-OS-integrated strong entropy sources
+   - password protection for on-disk identity key
    - Possible to get autoconf to easily install things into ~/.tor?
+   - server descriptor declares min log level, clients avoid servers
+     that are too loggy.
+   - put expiry date on onion-key, so people don't keep trying
+     old ones that they could know are expired?
+   - Add a notion of nickname->Pubkey binding that's not 'verification'
+   - Conn key rotation.
+   - Need a relay teardown cell, separate from one-way ends.
 
-   ongoing:
-   . rename/rearrange functions for what file they're in
-   - generalize our transport: add transport.c in preparation for
-     http, airhook, etc transport.
-   o investigate sctp for alternate transport.
-
-For September:
-N    . Windows port
-     o works as client
-       - deal with pollhup / reached_eof on all platforms
-     . robust as a client
-     . works as server
-       - can be configured
-     - robust as a server
-     . Usable as NT service
-     - docs for building in win
-     o installer, including all needed libs.
-       - and including privoxy
-       - and including a sockscap equivalent
-
-   - Docs
-     . FAQ
-     - a howto tutorial with examples
-       * put a stub on the wiki
-     o tutorial: how to set up your own tor network
-       o (need to not hardcode dirservers file in config.c)
-       o Make tutorial reflect this.
-     . port forwarding howto for ipchains, etc
-     . correct, update, polish spec
-     - document the exposed function api?
-     - Document where we differ from tor-design
-
-   . packages
-     . find a long-term rpm maintainer
-
-   - code
-     - better warn/info messages
-     - write howto for setting up tsocks, socat.
-       - including on osx and win32
-     - freecap handling
-     - tsocks
-       o gather patches, submit to maintainer
-         * send him a reminder mail and see what's up.
-       - intercept gethostbyname and others
-         * add this to tsocks
-       o do resolve via tor
-     - redesign and thorough code revamp, with particular eye toward:
-       - support half-open tcp connections
-       - conn key rotation
-       - other transports -- http, airhook
-       - modular introduction mechanism
-       - allow non-clique topology
+Big tasks that would demonstrate progress:
 
-Other details and small and hard things:
-   - tor should be able to have a pool of outgoing IP addresses
-     that it is able to rotate through. (maybe)
-   - tie into squid
-   - hidserv offerers shouldn't need to define a SocksPort
-     * figure out what breaks for this, and do it.
-   - when the client fails to pick an intro point for a hidserv,
-     it should refetch the hidserv desc.
-   . should maybe make clients exit(1) when bad things happen?
-     e.g. clock skew.
-   - should retry exitpolicy end streams even if the end cell didn't
-     resolve the address for you
-   o Make logs handle it better when writing to them fails.
-   o Dirserver shouldn't put you in running-routers list if you haven't
-     uploaded a descriptor recently
-   . Refactor: add own routerinfo to routerlist.  Right now, only
-     router_get_by_nickname knows about 'this router', as a hack to
-     get circuit_launch_new to do the right thing.
-   . Scrubbing proxies
-           - Find an smtp proxy?
-           . Get socks4a support into Mozilla
-   - Need a relay teardown cell, separate from one-way ends.
-   - Make it harder to circumvent bandwidth caps: look at number of bytes
-     sent across sockets, not number sent inside TLS stream.
-   - fix router_get_by_* functions so they can get ourselves too,
-     and audit everything to make sure rend and intro points are
-     just as likely to be us as not.
+   - Facility to automatically choose long-term helper nodes; perhaps
+     on by default for hidden services.
+   - patch privoxy and socks protocol to pass strings to the browser.
+   - patch tsocks with our current patches + gethostbyname, getpeername, etc.
+   - make freecap (or whichever) do what we want.
+   - scrubbing proxies for protocols other than http.
+     - Find an smtp proxy?
+     . Get socks4a support into Mozilla
+N  - Reverse DNS: specify and implement.
+   - figure out enclaves, e.g. so we know what to recommend that people
+     do, and so running a tor server on your website is helpful.
+     - Do enclaves for same IP only.
+     - Resolve first, then if IP is an OR, extend to him first.
+   - implement a trivial fun gui to demonstrate our control interface.
+
+************************ Roadmap for 2004-2005 **********************
+
+Hard problems that need to be solved:
+
+  - Separating node discovery from routing.
+  - Arranging membership management for independence.
+    Sybil defenses without having a human bottleneck.
+    How to gather random sample of nodes.
+    How to handle nodelist recommendations.
+    Consider incremental switches: a p2p tor with only 50 users has
+      different anonymity properties than one with 10k users, and should
+      be treated differently.
+  - Measuring performance of other nodes. Measuring whether they're up.
+  - Choosing exit node by meta-data, e.g. country.
+  - Incentives to relay; incentives to exit.
+  - Allowing dissidents to relay through Tor clients.
+  - How to intercept, or not need to intercept, dns queries locally.
+  - Improved anonymity:
+    - Experiment with mid-latency systems. How do they impact usability,
+      how do they impact safety?
+    - Understand how powerful fingerprinting attacks are, and experiment
+      with ways to foil them (long-range padding?).
+    - Come up with practical approximations to picking entry and exit in
+      different routing zones.
+    - Find ideal churn rate for helper nodes; how safe is it?
+    - What info squeaks by Privoxy? Are other scrubbers better?
+    - Attacking freenet-gnunet/timing-delay-randomness-arguments.
+    - Is abandoning the circuit the only option when an extend fails, or
+      can we do something without impacting anonymity too much?
+    - Is exiting from the middle of the circuit always a bad idea?
+
+Sample Publicity Landmarks:
+
+  - we have N servers / N users
+  - we have servers at epic and aclu and foo
+  - hidden services are robust and fast
+  - a more decentralized design
+  - tor win32 installer works
+  - win32 tray icon for end-users
+  - tor server works on win32
+  - win32 service for servers
+  - mac installer works
 
 ***************************Future tasks:****************************
 
@@ -222,42 +193,21 @@ Rendezvous and hidden services:
     - auth mechanisms to let midpoint and bob selectively choose
       connection requests.
   make it scalable:
-    - right now the hidserv store/lookup system is run by the dirservers;
-      this won't scale.
+    - robust decentralized storage for hidden service descriptors.
+  make it accessible:
+    - web proxy gateways to let normal people browse hidden services.
 
 Tor scalability:
   Relax clique assumptions.
   Redesign how directories are handled.
-    o Separate running-routers lookup from descriptor list lookup.
     - Resolve directory agreement somehow.
-    o Cache directory on all servers.
   Find and remove bottlenecks
     - Address linear searches on e.g. circuit and connection lists.
   Reputation/memory system, so dirservers can measure people,
     and so other people can verify their measurements.
     - Need to measure via relay, so it's not distinguishable.
-  Bandwidth-aware path selection. So people with T3's are picked
-    more often than people with DSL.
-  Reliability-aware node selection. So people who are stable are
-    preferred for long-term circuits such as intro and rend circs,
-    and general circs for irc, aim, ssh, etc.
   Let dissidents get to Tor servers via Tor users. ("Backbone model")
 
-Anonymity improvements:
-  Is abandoning the circuit the only option when an extend fails, or
-    can we do something without impacting anonymity too much?
-  Is exiting from the middle of the circuit always a bad idea?
-  Helper nodes. Decide how to use them to improve safety.
-  DNS resolution: need to make tor support resolve requests. Need to write
-    a script and an interface (including an extension to the socks
-    protocol) so we can ask it to do resolve requests. Need to patch
-    tsocks to intercept gethostbyname, else we'll continue leaking it.
-  Improve path selection algorithms based on routing-zones paper. Be sure
-    to start and end circuits in different ASs. Ideally, consider AS of
-    source and destination -- maybe even enter and exit via nearby AS.
-  Intermediate model, with some delays and mixing.
-  Add defensive dropping regime?
-
 Make it more correct:
   Handle half-open connections: right now we don't support all TCP
     streams, at least according to the protocol. But we handle all that
@@ -281,18 +231,6 @@ Efficiency/speed/robustness:
   Buffer size pool: allocate a maximum size for all buffers, not
     a maximum size for each buffer. So we don't have to give up as
     quickly (and kill the thickpipe!) when there's congestion.
-  Exit node caching: tie into squid or other caching web proxy.
   Other transport. HTTP, udp, rdp, airhook, etc. May have to do our own
     link crypto, unless we can bully openssl into it.
 
-P2P Tor:
-  Do all the scalability stuff above, first.
-  Incentives to relay. Not so hard.
-  Incentives to allow exit. Possibly quite hard.
-  Sybil defenses without having a human bottleneck.
-  How to gather random sample of nodes.
-  How to handle nodelist recommendations.
-  Consider incremental switches: a p2p tor with only 50 users has
-    different anonymity properties than one with 10k users, and should
-    be treated differently.
-