This is a list of evaluation criteria that have been applied by the authors of censorship circumvention systems and papers.
Many of these criteria deal with a specific threat model; not all of them are appropriate for every circumvention system. No one evaluation applied more than a fraction of the criteria. We tried to be comprehensive in our survey, and do not necessarily think that every criterion we found is good or necessary.
Questions? Contact the authors of the survey:
Revised 2018-06-25.
Assesses the system's resilience to unexpected blocking events and new kinds of blocking.
The amount of time it takes some programmer to create a new adaptation of the protocol.
This criterion considers the availability of the infrastructure used by an approach for users and the feasibility of deployment.
Examines the diversity of proxies in terms of geographic location.
Can deploy the circumvention approach without needing the help of third-parties, such as friendly ISPs.
The number of proxies usable with the tool.
Some tools rely on finding servers with a specific property at large in the Internet. For example, servers that accept user-posted content, or servers with specific network protocol quirks. This criterion means that the study measured how common such servers really are.
Measures or estimates the rate at which new proxies appear and old proxies go away.
Examines how long a decoy host is available to carry on a conversation.
Assesses an approach's client program efficiency in terms of CPU and memory usage.
Assesses the percentage of CPU cycles consumed by the client or server part of the system.
Assesses the memory requirements to run the system.
Measures how quickly client software starts up.
Evaluates the degree to which trust is centralized; i.e., all trust is placed in a single server/company, or spread out among many parties.
Whether the circumvention tool users can plausibly deny using it even when their computers are inspected.
Using the tool does not require installing special software.
Assesses whether the developers of the tool are known security experts.
Assesses the diversity of types of users of the system.
Assesses whether an approach protects user's privacy from intermediate and end nodes.
Does not log user information.
The tool hides information about the user from the end host (destination) to which the user connects to provide some degree of anonymity.
Assesses the cost of infrastructure required to deploy an approach in real world.
Estimates the cost of external services, e.g., cloud services, CDNs, that are required to deploy an approach.
Assesses the system's performance in terms of goodput, latency and overhead.
How many extra bytes are introduced by the tool.
Assesses the number of clients (source hosts) in the Internet that would be able to join a system.
The amount of useful throughput the tool enables.
Assesses the round-trip time for a request.
Assesses the number of requests that a requester must make to retrieve hidden messages.
Some systems need to apply a special distinguisher or mark to traffic destined for circumvention. For example, end-to-middle proxying systems need to tag flows at the client side and recognize them at the station. This criterion considers the performance of the registration method.
Assesses the time required to download a webpage. This is really a combination of goodput and latency, but it is specifically applied so often that we made it its own criterion.
The amount of throughput/bandwidth the tool enables.
How much extra time it takes to use the tool.
Assesses whether the source code available (client and server) and whether the design public or relies on security through obscurity.
The tool's source code is open.
Active probing attacks involve the censor initiating connections to hosts to determine whether the host runs a given circumvention protocol, typically then blocking the host's IP address upon finding that it does. A system is resistant to active probing if an adversary cannot discover the use of the system using this technique.
When probed, respond similar to how some allowed server would respond so that a censor deciding to block such responses will incur false positives.
Whether a client needs authentication to connect to the server.
A system resists blocking if it is hard to block the protocol or IP address of the infrastructure that the approach uses, even given a method of identifying it. For example, if blocking would cause substantial collateral damage.
Whether an approach sends traffic using a popular protocol, such as the Skype protocol, to force the censor to either block a popular protocol or identify the circumventing usage of the protocol from normal usage.
Whether an approach uses too many hosts to make it hard for a censor to block all of them.
Whether an approach uses infrastructure within a network, e.g., router, to avoid address blocking.
Whether an approach uses popular hosts, such as Skype nodes and CDNs, to resist address blocking.
Considers whether the system continues to work even if the censor joins the circumvention network and attempts to disrupt it.
This criterion considers different measures that a paper uses to avoid security attacks such as man-in-the-middle, denial of service, malicious proxy, key reuse and replay attack.
Whether an approach ignores invalid connections to avoid denial of service attacks.
Avoid DoS by limiting the amount of service the approach will provide to each user.
Whether an approach uses TLS to provide confidentiality.
Whether an approach uses authenticated key exchange.
Whether an approach uses block cipher to resist key reuse attack.
Whether an approach uses certificate pinning to avoid MiTM.
Require clients to solve a puzzle to prevent DoS.
Whether an approach uses encryption for confidentiality (and/or integrity).
Whether an approach uses shared secret to resist man-in-the-middle attacks.
A censor would have to overcome not just the circumvention deployment, but some third-party that hosts the deployment.
Whether an approach uses timestamp to resist replay attack.
By using a trustworthy proxy as the forwarder, the approach avoids the risks of a malicious proxy (as long as the proxy remains trustworthy).
An approach is resistant to traffic analysis if an adversary cannot statically use properties of the traffic generated by the approach to detect it. (Some of the metrics used for this goal can also be used for active probing, but they are not inherently active.)
Prevents detection by TLS characteristics, like TLS nonce, clienthello or serverhello messages.
They measured the number of simultaneous connections to a server.
Measures the duration of flows (e.g., TCP connections) and evaluates whether the duration can be a distinguisher. (If a connection is suspiciously long, for example.)
Measures the distribution of packet timing (interpacket times or packet rates) to assess whether it is unlike that of a blocked protocol, or like that of an allowed protocol.
This criterion is specific to TapDance and the way it sends half-finished HTTP requests. If a TapDance station exists along a path, the request is intercepted and a response comes back immediately. If there is no TapDance station, then no response ever comes (or perhaps comes only after a long timeout).
Considers the distribution of consecutive strings of symbols, for example bytes. This includes 1-grams (e.g., distribution of single byte values).
Assesses the fingerprintability of the network stack, for example comparing the TCP options of two different hosts. Relevant when one host spoofs packets on behalf of another.
Measures the total number of HTTP request-response pairs per TCP connection.
Measures the distribution of packet lengths to assess wehether it is unlike that of a blocked protocol, or like that of an allowed protocol.
Assesses the misclassification rate of the protocol classifiers to see how well the tool can evade the classifiers.
Counted the number of connections made in a row to a server.
The total number of TCP connections per session does not stick out.
The total payload length produced by the tool does not stick out.
Whether an approach uses encryption to resist traffic analysis.
Use a random port number for communications.
Evaluates the system's resistance to modification of packets, or injecting or dropping packets. This criterion is concerned only with manipulation of client-initiated flows.
Avoid DoS by limiting the amount of service the approach will provide to each user.
Whether an approach uses TLS to provide integrity.
Whether an approach uses UDP with reliability.
Whether a client needs authentication to connect to the server.
Whether an approach uses error correcting codes.
Evaluates whether an approach or tool promotes itself in a way that is likely to attract harmful attention (from the media or from the censor, for example).
Keeping the server used as a forwarder by the circumvention network hidden from the censor.
The circumventor's computer connects indirectly to the circumvention network's forwarders through an innocuous server.
Whether the system has funds and other resources to continue operating for the long term.
Assesses the additional effort that the circumvention tool client user must expand to use the system.
Assesses the system's usefulness for a wide variety of applications (e.g. web browsing, chat, email).
Assesses the quality of user and developer documentation.
Assesses whether the client software leaves traces after being uninstalled.
Cost in USD to use the system.
Assesses whether the client software has a graphical user interface.
Assesses whether the software and documentation are localized to relevant languages.
Using the tool does not require installing special software.
Assesses whether an approach artificially limits who can use it and for which service.
This criterion is specific to link-rewriting web proxies like CGIProxy that actually have to interpret HTML and JavaScript and change links so they point back into the proxy and not to their original location.
Assesses the system's portability to different operating systems and devices.
The size of the tool's client program file is small.
Assesses the availability of software updates.
Assesses real world usage of an approach.
Discusses the number unique of IP addresses that connect to the system on a daily basis.
The number of users the tool has.
An approach proposed in an academic paper that is deployed in the real world and used by users.
Evaluates whether the claims of about an approach by the its provider match reality.