Freenet is an anonymous, secure, distributed datastore. I've written about using Freenet before, including using it as the backend for static websites. In this post I'll demonstrate how to use the Freenet API to push data into the Freenet network and retrieve data from it.

Unfortunately the freenet protocol documentation is in a state of flux as it moves from a previous wiki to a github based wiki. This means some of the protocol information may be incomplete. The old wiki data is stored in Freenet under the following keys:

CHK@San0hXZSyCEvvb7enNUIWrkiv8MDChn8peLJllnWt4s,MCNn4eUbl5NW9quOm4JTU~0rsWu6QlIdek9VtpFpXe4,AAMC--8/freenetproject-oldwiki.tar

CHK@4sff2MgvexbsfgSuqNOwqVDkP~GaZPsZ1rJVKUg87g8,unU3TZ93pYGYPCoH7LC53dlc5~Rmar8SKF9fsZnQX-8,AAMC--8/freenetproject-wiki.tar

The API for external programs to communicate with a running Freenet daemon is FCPv2. It's a text based protocol accessed using a TCP socket connection on port 9481. The FCP protocol can be enabled or disabled from the Freenet configuration settings but it is enabled by default so the examples here should work in a default installation.

The basic protocol of FCP uses a unit called a 'message'. Messages are sent over the socket starting with a line for the start of the message, followed by a series of 'Key=Value' lines, and ending the message with 'EndMessage'. Some messages containing binary data and these end differently and I'll discuss this after some basic examples.

For the examples that follow I'll be using bash. I debated picking from my usual toolkit of obscure languages but decided to use something that doesn't require installation on Linux and Mac OS X and may also run on Windows 10. The examples should be readable enough for non-bash users to pick up and translate to their favourite language, especially given the simplicity of the protocol. I've found the ability to throw together a quick bash script to do inserts and retrievals to be useful.

Hello

The FCPv2 documentation lists the messages that can be sent from the client to the Freenet node, and what can be expected to be received from the node to the client. On first connecting to the node the client must send a ClientHello message. This looks like:

ClientHello Name=My Client Name ExpectedVersion=2.0 EndMessage

The Name field uniquely identifies the client to the node. Disconnecting and reconnecting with the same Name retains access to a persistent queue of data being inserted and retrieved. An error is returned if an attempt to connect is made when a client with the same Name is already connected.

The node returns with a NodeHello Message. This looks like:

NodeHello Build=1477 ConnectionIdentifier=... ... EndMessage

The various fields are described in the NodeHello documentation. This interaction can be tested using netcat or telnet :

$ nc localhost 9481 ClientHello Name=My Client Name ExpectedVersion=2.0 EndMessage NodeHello CompressionCodecs=4 - GZIP(0), BZIP2(1), LZMA(2), LZMA_NEW(3) Revision=build01477 Testnet=false ... ExtRevision=v29 EndMessage

You can connect to a socket from bash using 'exec' and file redirection to a pseudo-path describing the tcp socket. See HACKTUX notes from the trenches for details. The above netcat interaction looks like this from bash:

#! /bin/bash function wait_for { local line local str = $1 while read -r line do >&2 echo "$line" if [ "$line" == "$str" ] ; then break fi done } exec 3<>/dev/tcp/127.0.0.1/9481 cat >&3 <<HERE ClientHello Name=My Client Name ExpectedVersion=2.0 EndMessage HERE wait_for "NodeHello" <&3 wait_for "EndMessage" <&3 exec 3<&- exec 3>&-

The exec line opens a socket on port 9481, the FCP port, and assigns it to file descriptor '3'. Then we use cat to write the ClientHello message to that file descriptor. wait_for reads lines from the socket, displaying them on standard error (file descriptor '2'), until it reaches a specifc line passed as an argument. Here we wait for the NodeHello line and then the EndMesage line to cover the NodeHello response from the server. The remaining two exec lines close the socket.

The full bash script is available in hello.sh.

Retrieving data inline

The FCP message ClientGet is used to retrieve data stored at a specific key. The data can be returned inline within a message or written to a file accessable by the node. An example message for retrieving a known key is:

ClientGet URI=CHK@otFYYKhLKFzkAKhEHWPzVAbzK9F3BRxLwuoLwkzefqA,AKn6KQE7c~8G5dLa4TuyfG16XIUwycWuFurNJYjbXu0,AAMC--8/example.txt Identifier=1234 Verbosity=0 ReturnType=direct EndMessage

This retrieves the contents of a particular CHK key where I stored example.txt . The Verbosity is set to not return any progress messages, just send messages when the entire contents are retrieved. A ReturnType of direct means return the data within the AllData message which is received when the retrieval is complete. The result messages are:

DataFound Identifier=1234 CompletionTime=1490614072644 StartupTime=1490614072634 DataLength=21 Global=false Metadata.ContentType=text/plain EndMessage AllData Identifier=1234 CompletionTime=1490614072644 StartupTime=1490614072634 DataLength=21 Global=false Metadata.ContentType=text/plain Data Hello Freenet World!

The first message received is DataFound giving information about the completed request. The following message, AllData, returns the actual data. Note that it does not include an EndMessage . Instead it has a Data terminator followed by the data as a sequence of bytes of length DataLength .

To process AllData from bash I use a function to extract the DataLength when it finds it:

function get_data_length { local line while read -r line do if [[ "$line" = ~ ^DataLength = .* ]] ; then echo "${line##DataLength=}" break fi done }

This is called from the script after the ClientHello and NodeHello exchange:

cat >&3 <<HERE ClientGet URI=CHK@otFYYKhLKFzkAKhEHWPzVAbzK9F3BRxLwuoLwkzefqA,AKn6KQE7c~8G5dLa4TuyfG16XIUwycWuFurNJYjbXu0,AAMC--8/example.txt Identifier=1234 Verbosity=0 ReturnType=direct EndMessage HERE wait_for "AllData" <&3 len = $( get_data_length <&3 ) wait_for "Data" <&3 dd status = none bs = "$len" count = 1 <&3 >&2

The dd command reads the specified number of bytes from the socket and outputs it to standard output. This is the contents of the key we requested:

$ ./getinline Hello Freenet World!

The full bash script is available in getinline.sh.

The main downside of using inline data requests is that large files can exhaust the memory of the node.

Request Direct Disk Access

A variant of ClientGet requests the node to write the result to a file on disk instead of sending it as part of the AllData message. This is useful for large files that don't fit in memory. The data is written to the filesystem that the node has access to so it's most useful when the FCP client and the freenet node are on the same system.

Being able to tell the server to write directly to the filesystem is a security issue so Freenet requires a negotiation to happen first to confirm that the client has access to the directory that you are requesting the server to write to. This negotiation requirement, known as TestDDA can be disabled in the configuration settings of the node but it's not recommended.

First the client must send a TestDDARequest message listing the directory it wants access to and whether read or write access is being requested.

TestDDARequest Directory=/tmp/ WantWriteDirectory=true WantReadDirectory=true EndMessage

The server replies with a TestDDAReply:

TestDDAReply Directory=/tmp/ ReadFilename=/tmp/testr.tmp WriteFilename=/tmp/testw.tmp ContentToWrite=RANDOM EndMessage

The script should now write the data contained in the ContentToWrite key into the file referenced by the WriteFilename key. It should read the data from the file referenced in the ReadFilename key and send that data in a TestDDAResponse:

TestDDAResponse Directory=/tmp/ ReadContent=...content from TestDDAReply... EndMessage

The server then responds with a TestDDAComplete:

TestDDAComplete Directory=/tmp/ ReadDirectoryAllowed=true WriteDirectoryAllowed=true EndMessage

Once that dance is complete then put and get requests can be done to that specific directory. The bash code for doing this is:

cat >&3 <<HERE TestDDARequest Directory=/tmp/ WantWriteDirectory=true WantReadDirectory=true EndMessage HERE wait_for "TestDDAReply" <&3 content = $( process_dda_reply <&3 ) cat >&3 <<HERE TestDDAResponse Directory=/tmp/ ReadContent=$content EndMessage HERE wait_for "TestDDAComplete" <&3 process_dda_complete <&3

It uses a helper function process_dda_reply to handle the TestDDAReply message from the server:

function process_dda_reply { local readfile = "" local writefile = "" local content = "" while read -r line do if [[ "$line" = ~ ^ReadFilename = .* ]] ; then readfile = "${line##ReadFilename=}" fi if [[ "$line" = ~ ^WriteFilename = .* ]] ; then writefile = "${line##WriteFilename=}" fi if [[ "$line" = ~ ^ContentToWrite = .* ]] ; then content = "${line##ContentToWrite=}" fi if [[ "$line" == "EndMessage" ]] ; then echo -n "$content" > "$writefile" cat "$readfile" break fi done }

This function reads the fields of the TestDDAReply and writes the required content to the write file and returns the data contained in the read file. This returned data is sent in the TestDDAResponse . The process_dda_complete function checks the TestDDAComplete fields to ensure that access was granted. The full bash script is available in testdda.sh.

Retrieving data to disk

The ReturnType field of the ClientGet message can be set to disk to write directly to a disk file once the TestDDA process is complete. The message looks like this:

cat >&3 <<HERE ClientGet URI=CHK@HH-OJMEBuwYC048-Ljph0fh11oOprLFbtB7QDi~4MWw,B~~NJn~XrJIYEOMPLw69Lc5Bv6BcGWoqJbEXrfX~VCo,AAMC--8/pitcairn_justice.jpg Identifier=1234 Verbosity=1 ReturnType=disk Filename=/tmp/pitcairn_justice.png EndMessage HERE

In this case we're retreving a file I've inserted previously. The Verbosity key is set to 1 to enable SimpleProgress messages to be received. These messages contain fields showing the total number of blocks that can be fetched for that file, the required number of blocks that we need to get, how many we've successfully retrieved so far, and a few other fields. The following bash function processes this and prints the result:

function handle_progress { local total = 0 local succeeded = 0 local required = 0 local final = "" while read -r line do if [[ "$line" = ~ ^Total = .* ]] ; then total = "${line##Total=}" fi if [[ "$line" = ~ ^Required = .* ]] ; then required = "${line##Required=}" fi if [[ "$line" == "FinalizedTotal=true" ]] ; then final = "final" fi if [[ "$line" = ~ ^Succeeded = .* ]] ; then succeeded = "${line##Succeeded=}" fi if [[ "$line" == "EndMessage" ]] ; then echo "Progress: retrieved $succeeded out of $required required and $total total ($final)" break fi done }

The FinalizedTotal field indicates if the Total field is accurate and will not change. Otherwise that field can increase as more knowledge about the file is gained. The Required field is the number of blocks that need to be received to reconstruct the file. It is less than Total due to redundancy in the way freenet stores data to account for nodes going away and data being lost.

The handle_progress function is called from within wait_with_progress , which waits for a particular message (usually the one indicating the end of the transfer), displays progress, and ignores everything else.

function wait_with_progress { while read -r line do if [ "$line" == "SimpleProgress" ] ; then handle_progress fi if [ "$line" == "$1" ] ; then break fi done }

These are called as follows:

cat >&3 <<HERE ClientGet URI=CHK@HH-OJMEBuwYC048-Ljph0fh11oOprLFbtB7QDi~4MWw,B~~NJn~XrJIYEOMPLw69Lc5Bv6BcGWoqJbEXrfX~VCo,AAMC--8/pitcairn_justice.jpg Identifier=1234 Verbosity=1 ReturnType=disk Filename=/tmp/pitcairn_justice.png EndMessage HERE wait_with_progress "DataFound" <&3 wait_for "EndMessage" <&3

The DataFound message is sent by the server when the file has been successfully retrieved. It can be found at the location specified in the Filename field of the ClientGet .

The full bash script is available in getdisk.sh.

$ bash getdisk.sh Progress: retrieved 0 out of 1 required and 1 total () Progress: retrieved 1 out of 1 required and 1 total () Progress: retrieved 1 out of 5 required and 10 total () Progress: retrieved 1 out of 5 required and 10 total (final) Progress: retrieved 5 out of 5 required and 10 total (final)

Inserting Data Inline

When storing data using FCP you can provide the data directly in the message or reference a file on disk that the node will read and store. They are both done using the ClientPut message. Sending this message looks like:

file = "$1" size = $( stat -c%s "$file" ) mime = $( file --mime-type "$file" |awk '{print $2}' ) cat >&3 <<HERE ClientPut URI=CHK@ Metadata.ContentType=$mime Identifier=1234 Verbosity=1 GetCHKOnly=false TargetFilename=$(basename "$file") DataLength=$size UploadFrom=direct Data HERE dd status = none if = "$file" bs = "$size" count = 1 |pv -L 500k >&3 wait_with_progress "PutSuccessful" <&3 uri = $( get_uri <&3 ) wait_for "EndMessage" <&3

ClientPut requires the mime type of the file and this is obtained using file . The size of the file is retrieved with stat . These are placed in the ClientPut message directly. The binary data for the file needs to be sent after a Data terminator similar to how we retrieved the data when doing an inline get. dd is again used for this but it's piped to pv to limit the data transfer rate otherwise the network gets swamped due to buffer bloat.

The URI for inserting is generated as a CHK key. This is a key based on a hash of the file content. Inserting the same file will result in the same key. get_uri reads the PutSuccessful message to find the full key of the insert. This is displayed to the user later in the script.

The full bash script is available in putinline.sh.

$ bash putinline.sh /tmp/example.txt Progress: inserted 0 out of 1 () Progress: inserted 0 out of 2 () Progress: inserted 0 out of 2 (final)

Inserting Data from Disk

Inserting data directly from a disk file works very similar to requesting from a disk file. It requires the TestDDA process followed by a ClientPut using a UploadFrom set to disk :

cat >&3 <<HERE ClientPut URI=CHK@ Metadata.ContentType=$mime Identifier=1234 Verbosity=1 GetCHKOnly=false TargetFilename=$(basename "$file") Filename=$file UploadFrom=disk EndMessage HERE wait_with_progress "PutSuccessful" <&3 uri = $( get_uri <&3 ) wait_for "EndMessage" <&3

The full bash script is available in putdisk.sh.

Other Messages

There are other interesting messages that are useful. Using ClientPut if you set the field GetCHKOnly to true then the file isn't inserted but the CHK key is generated. Since CHK is based on the file contents it will be the same key if the file is inserted using the same mime type, filename and other parameters. This allows generating a key, sending it to a third party and they can start a file retrieval before the file completes inserting. There are security issues with this in that if an attacker knows the key while it is being inserted they may be able to narrow down the location of the inserting node.

Another useful message is GenerateSSK:

GenerateSSK Identifier=1234 EndMessage

This results in a SSKKeypair reply containing a randomly generated SSK insert and request key:

SSKKeypair InsertURI=SSK@AK.../ RequestURI=SSK@Bn.../ Identifier=1234 EndMessage

These can be used to insert data with ClientPut by setting the URI to the InsertURI , and retrieved by a third party using the RequestURI as the URI in ClientGet .

ClientPutDiskDir inserts an entire directory. This is the basis of inserting 'Freesites' - Freenet websites. I wrote a mirror.sh utility that mirrors an online website or page and inserts it into Freenet. This is useful for linking to news articles in Freenet without having to leave the Freenet environment. It uses a putdir.sh script that inserts a directory.

Conclusion

The Freenet API has a lot of functionality beyond what I've shown here. I used bash for the examples because it has few dependancies but more robust scripts would be easier in a full featured programming language. I'm not a bash expert and welcome corrections and additions. I've put the code in an fcp-examples github repository.

There are some client libraries with pyFreenet being an example. I recommend the following articles for a deeper dive into Freenet programming: