To enable (public) many-to-many communication, we propagate the information that we believe that someone isn’t a spammer and add a blacklist to get rid of people who suddenly start to spam.

The big change with this scheme is that there is two-step authentication: Something expensive (solving a riddle) gets you seen by a few people, and if you then contribute constructively in a social context, they mark you as non-spammer and you get seen by more people.

The clever part about that scheme is that socializing is actually no cost to honest users (that’s why we use things like Sone or FMS), while it is a cost to attackers.

Let’s take Alice and Bob again, but add Carol. First Bob introduces himself to Alice, then Carol introduces herself to Alice. Thanks to propagating the riddle-information, Carol can directly write to Bob, without first solving a riddle. Scaling up that means that you only need to prove a single time that you are no spammer (or rather: not disruptive) if you want to enter a community.

To make it easier to follow, we will implement this with a bit of abstraction: People have a private key, can introduce themselves and publish lists of messages. Also they keep a public list of known people and a list of people they see as spammers who want to disrupt communication.

I got a bit carried away while implementing this, but please bear with me: It’ll work hard to make it this fun.

The finished program is available as alice_bob_carol.py. Just download and run it with python alice_bob_carol.py .

Let’s start with the minimal structure for any pyFreenet using program:

import fcp n = fcp.node.FCPNode() # for debugging add verbosity=5 <<body>> n.shutdown()

The body contains the definitions of a person with different actors, an update step (as simplification I use global stepwise updates) as well as the setup of the communication. Finally we need an event loop to run the system.

<<preparation>> <<person>> <<update>> <<setup>> <<event_loop>>

We start with some imports – and a bit of fun :)

import uuid import random try : import chatterbot # let's get a real conversation :) # https://github.com/guntherc/ChatterBot/wiki/Quick-Start # get with `pip install --user chatterbot` irc_loguri = "USK@Dtz9FjDPmOxiT54Wjt7JwMJKWaqSOS-UGw4miINEvtg,cuIx2THw7G7cVyh9PuvNiHa1e9BvNmmfTcbQ7llXh2Q,AQACAAE/irclogs/1337/" print "Getting the latest IRC log as base for the chatterbot" IRC_LOGLINES = n.get(uri=irc_loguri, realtime= True , priority=1, followRedirect= True )[1].splitlines() import re # what follows is an evil hack, but what the heck :) p = re. compile (r '<.*?>' ) q = re. compile (r '&.*?;' ) IRC_LOGLINES = [q.sub( '' , p.sub( '' , str ( unicode (i.strip(), errors= "ignore" )))) for i in IRC_LOGLINES] IRC_LOGLINES = [i[:-5] for i in IRC_LOGLINES # skip the time (last 5 letters) if (i[:-5] and # skip empty not "spam" in i # do not trigger spam-marking )][7:] # skip header except ImportError : chatterbot = None

The real code begins with some helper functions – essentially data definition.

def get_usk_namespace (key, name, version=0): """Get a USK key with the given namespace (foldername).""" return "U" + key[1:] + name + "/" + str (version) + "/" def extract_raw_from_usk (key): """Get an SSK key as used to identify a person from an arbitrary USK.""" return "S" + (key[1:]+ "/" ).split( "/" )[0] + "/" def deserialize_keylist (keys_data): """Parse a known file to get a list of keys. Reverse: serialize_keylist.""" return [i for i in keys_data.split( "

" ) if i] def serialize_keylist (keys_list): """Serialize the known keys into a text file. Reverse: parse_known.""" return "

" .join(keys_list)

Now we can define a person. The person is the primary actor. To keep everything contained, I use a class with some helper functions.

class Person ( object ): def __init__ ( self , myname, mymessage): self .name = myname self .message = mymessage self .introduced = False self .public_key, self .private_key = n.genkey() print self .name, "uses key" , self .public_key # we need a list of versions for the different keys self .versions = {} for name in [ "messages" , "riddles" , "known" , "spammers" ]: self .versions[name] = -1 # does not exist yet # and sets of answers, watched riddle-answer keys, known people and spammers. # We use sets for these, because we only need membership-tests and iteration. # The answers contain KSKs, the others the raw SSK of the person. # watched contains all persons whose messages we read. self .lists = {} for name in [ "answers" , "watched" , "known" , "spammers" , "knowntocheck" ]: self .lists[name] = set () # running requests per name, used for making all persons update asynchronously self .jobs = {} # and just for fun: get real conversations. Needs chatterbot and IRC_LOGLINES. # this is a bit slow to start, but fun. try : self .chatbot = chatterbot.ChatBot( self .name) self .chatbot.train(IRC_LOGLINES) except : self .chatbot = None def public_usk ( self , name, version=0): """Get the public usk of type name.""" return get_usk_namespace( self .public_key, name, version) def private_usk ( self , name, version=0): """Get the private usk of type name.""" return get_usk_namespace( self .private_key, name, version) def put ( self , key, data): """Insert the data asynchronously to the key. This is just a helper to avoid typing the realtime arguments over and over again. :returns: a job object. To get the public key, use job.wait(60).""" return n.put(uri=key, data=data, async= True , Global= True , persistence= "forever" , realtime= True , priority=1, IgnoreUSKDatehints= "true" ) def get ( self , key): """Retrieve the data asynchronously to the key. This is just a helper to avoid typing the realtime arguments over and over again. :returns: a job object. To get the public key, use job.wait(60).""" return n.get(uri=key, async= True , realtime= True , priority=1, IgnoreUSKDatehints= "true" , followRedirect= True ) def introduce_to_start ( self , other_public_key): """Introduce self to the other by solving a riddle and uploading the messages USK.""" riddlekey = get_usk_namespace(other_public_key, "riddles" , "-1" ) # -1 means the latest version try : self .jobs[ "getriddle" ].append( self .get(riddlekey)) except KeyError : self .jobs[ "getriddle" ] = [ self .get(riddlekey)] def introduce_start ( self ): """Select a person and start a job to get a riddle.""" known = list ( self .lists[ "known" ]) if known: # introduce to a random person to minimize # the chance of collisions k = random.choice(known) self .introduce_to_start(k) def introduce_process ( self ): """Get and process the riddle data.""" for job in self .jobs.get( "getriddle" , [])[:]: if job.isComplete(): try : riddle = job.wait()[1] except Exception as e: # try again next time print self .name, "getting the riddle from" , job.uri, "failed with" , e return self .jobs[ "getriddle" ].remove(job) answerkey = self .solve_riddle(riddle) messagekey = self .public_usk( "messages" ) try : self .jobs[ "answerriddle" ].append( self .put(answerkey, messagekey)) except KeyError : self .jobs[ "answerriddle" ] = [ self .put(answerkey, messagekey)] def introduce_finalize ( self ): """Check whether the riddle answer was inserted successfully.""" for job in self .jobs.get( "answerriddle" , [])[:]: if job.isComplete(): try : job.wait() self .jobs[ "answerriddle" ].remove(job) self .introduced = True except Exception as e: # try again next time print self .name, "inserting the riddle-answer failed with" , e return def new_riddle ( self ): """Create and upload a new riddle.""" answerkey = "KSK@" + str (uuid.uuid1()) + "-answered" self .lists[ "answers" ].add(answerkey) self .versions[ "riddles" ] += 1 next_riddle_key = self .private_usk( "riddles" , self .versions[ "riddles" ]) self .put(next_riddle_key, answerkey) def solve_riddle ( self , riddle): """Get the key for the given riddle. In this example we make it easy: The riddle is the key. For a real system, this needs user interaction. """ return riddle def update_info ( self ): for name in [ "known" , "spammers" ]: data = serialize_keylist( self .lists[name]) self .versions[name] += 1 key = self .private_usk(name, version= self .versions[name]) self .put(key, data) def publish ( self , data): self .versions[ "messages" ] += 1 messagekey = self .private_usk( "messages" , version= self .versions[ "messages" ]) print self .name, "published a message:" , data self .put(messagekey, data) def check_network_start ( self ): """start all network checks.""" # first cancel all running jobs which will be replaced here. for name in [ "answers" , "watched" , "known" , "knowntocheck" , "spammers" ]: for job in self .jobs.get(name, []): job.cancel() # start jobs for checking answers, for checking all known people and for checking all messagelists for new messages. for name in [ "answers" ]: self .jobs[name] = [ self .get(i) for i in self .lists[name]] for name in [ "watched" ]: self .jobs[ "messages" ] = [ self .get(get_usk_namespace(i, "messages" )) for i in self .lists[name]] self .jobs[ "spammers" ] = [] for name in [ "known" , "knowntocheck" ]: # find new nodes self .jobs[name] = [ self .get(get_usk_namespace(i, "known" )) for i in self .lists[name]] # register new nodes marked as spammers self .jobs[ "spammers" ].extend([ self .get(get_usk_namespace(i, "spammers" )) for i in self .lists[name]]) def process_network_results ( self ): """wait for completion of all network checks and process the results.""" for kind, jobs in self .jobs.items(): for job in jobs: if not kind in [ "getriddle" , "answerriddle" ]: try : res = job.wait(60)[1] self .handle(res, kind, job) except : continue def handle ( self , result, kind, job): """Handle a successful job of type kind.""" # travel the known nodes to find new ones if kind in [ "known" , "knowntocheck" ]: for k in deserialize_keylist(result): if ( not k in self .lists[ "spammers" ] and not k in self .lists[ "known" ] and not k == self .public_key): self .lists[ "knowntocheck" ].add(k) self .lists[ "watched" ].add(k) print self .name, "found and started to watch" , k # read introductions elif kind in [ "answers" ]: self .lists[kind].remove(job.uri) # no longer need to watch this riddle k = extract_raw_from_usk(result) if not k in self .lists[ "spammers" ]: self .lists[ "watched" ].add(k) print self .name, "discovered" , k, "through a solved riddle" # remove found spammers elif kind in [ "spammers" ]: for k in deserialize_keylist(result): if not result in self .lists[ "known" ]: self .lists[ "watched" ].remove(result) # check all messages for spam elif kind in [ "messages" ]: k = extract_raw_from_usk(job.uri) if not "spam" in result: if not k == self .public_key: print self .name, "read a message:" , result self .chat(result) # just for fun :) if not k in self .lists[ "known" ]: self .lists[ "known" ].add(k) self .update_info() print self .name, "marked" , k, "as known person" else : self .lists[ "watched" ].remove(k) if not k in self .lists[ "spammers" ]: self .lists[ "spammers" ].add(k) self .update_info() print self .name, "marked" , k, "as spammer" def chat ( self , message): if self .chatbot and not "spam" in self .message: msg = message[message.index( ":" )+1:-10].strip() # remove name and step self .message = self .name + ": " + self .chatbot.get_response(msg) # some helper functions; the closest equivalent to structure definition <<helper_functions>>

Note that nothing in here depends on running these from the same program. All communication between persons is done purely over Freenet. The only requirement is that there is a bootstrap key: One person known to all new users. This person could be anonymous, and even with this simple code there could be multiple bootstrap keys. In freenet we call these people “seeds”. They are the seeds from which the community grows. As soon as someone besides the seed adds a person as known, the seed is no longer needed to keep the communication going.

The spam detection implementation is pretty naive: It trusts people to mark others as spammers. In a real system, there will be disputes about what constitutes spam and the system needs to show who marks whom as spammer, so users can decide to stop trusting the spam notices from someone when they disagree. As example for a real-life system, the Web of Trust plugin uses trust ratings between -100 and 100 and calculates a score from the ratings of all trusted people to decide how much to trust people who are not rated explicitly by the user.

With this in place, we need the update system to be able to step through the simulation. We have a list of people who check keys of known other people.

We first start all checks for all people quasi-simultaneously and then check the results in serial to avoid long wait times from high latency. Freenet can check many keys simultaneously, but serial checking is slow.

people = [] def update (step): for p in people: if not p.introduced: p.introduce_start() for p in people: p.check_network_start() for p in people: if p.message: p.publish(p.name + ": " + p.message + " (step=%s)" % step) p.new_riddle() for p in people: if not p.introduced: p.introduce_process() for p in people: p.process_network_results() for p in people: if not p.introduced: p.introduce_finalize()

So that’s the update tasks - not really rocket science thanks to the fleshed out Persons. Only two things remain: Setting up the scene and actually running it.

For setup: We have Alice, Bob and Carol. Lets also add Chuck who wants to prevent the others from communicating by flooding them with spam.

def gen_person (name): try : return Person(myname=name, mymessage=random.choice(IRC_LOGLINES)) except : return Person(myname=name, mymessage= "Hi, it's me!" ) # start with alice alice = gen_person( "Alice" ) people.append(alice) # happy, friendly people for name in [ "Bob" , "Carol" ]: p = gen_person(name) people.append(p) # and Chuck p = Person(myname= "Chuck" , mymessage= "spam" ) people.append(p) # All people know Alice (except for Alice). for p in people: if p == alice: continue p.lists[ "known" ].add(alice.public_key) p.lists[ "watched" ].add(alice.public_key) # upload the first version of the spammer and known lists for p in people: p.update_info()

That’s it. The stage is set, let the trouble begin :)

We don’t need a while loop here, since we just want to know whether the system works. So the event loop is pretty simple: Just call the update function a few times.

for i in range (6): update(step=i)

That’s it. We have spam-resistant message-channels and community discussions. Now we could go on and implement more algorithms on this scheme, like the turn-based games specification (ever wanted to play against truly anonymous competitors?), Fritter (can you guess from its name what it is? :)), a truly privacy respecting dropbox or an anonymizing, censoriship resistant, self-hosting backend for a digital market like OpenBazaar (there’s a 4 Bitcoin bounty on that – info, wallet – you might want to give it a shot).

But that would go far beyond the goal of this article – which is to give you, my readers, the tools to create the next big thing by harnessing the capabilities of Freenet.

These capabilities have been there for years, but hidden beneath non-existing and outdated documentation, misleading claims of being in alpha-stage even though Freenet has been used in what amounts to production for over a decade and, not to forget, the ever-recurring, ever-damning suggestion to SGTFS (second-guess the friendly source). As written in Forgotten Cypherpunk Paradise, Freenet already solved many problems which researchers only begin to tackle now, but there are reasons why it was almost forgotten. With this series I intend fix some of them and start moving Freenet documentation towards the utopian vision laid out in Teach, Don’t Tell. It’s up to you to decide whether I succeeded. If I did, it will show up as a tiny contribution to the utilities and works of art and vision you create.

Note that this is not fast (i.e. enough for blogging but not enough for chat). We can make it faster by going back to SSKs instead of USKs with their additional logic for finding the newest version in O(1), but for USK there are very cheap methods to get notified of new versions for large numbers of keys (subscribing) which are used by more advanced tools like the Web of Trust and the Sone plugin, so this would be an optimization we would have to revert later. With these methods, Sone reaches round trip times of 5-15 minutes despite using large uploads.

Also since this uses Freenet as backend, it scales up: If Alice, Bob, Carol und Chuck used different computers instead of running on my single node, their communication would actually be faster, and if they called in all their alphabet and unicode friends, the system would still run fast. We’re harvesting part of the payoff from using a fully distributed backend :)

And with that, this installment ends. You can now implement really cool stuff using Freenet. In the next article I’ll describe how to avoid doing this stuff myself by interfacing with existing plugins. Naturally I could have done that from the start, but then how could I have explained the Freenet communication primitives these plugins use? :)

If you don’t want to wait, have a look at how Infocalypse uses wot to implement github-like access with user/repo , interfaces with Freemail to realize truly anonymous pull-requests from the command line and builds on FMS to provide automated updates of a DVCS wiki over Freenet.

Happy Hacking!

PS: You might ask “What is missing?”. You might have a nagging feeling that something we do every day isn’t in there. And you’re right. It’s scalable search. Or rather: scalable, spam- and censorship-resistant search. Scalable search would be Gnutella. Spam-resistance would be Credence on the social graph (the people you communicate with). Censorship-resistant is unsolved – even Google fails there. But seeing that Facebook just overtook Google as the main source of traffic, we might not actually need fully global search. Together with the cheap and easy update notifications in Freenet (via USKs), a social recommendation and bookmark-sharing system should make scalable search over Freenet possible. And until then there’s always the decentralized YaCy search engine which has been shown to be capable of crawling Freenet. Also there are the Library and Spider plugins, but they need some love to work well. Also there are the Library and Spider plugins, but they need some love to work well.

PPS: You can download the final example as alice_bob_carol.py