logo


“...just as much about technological solutions as it is about the social tissue that we can nurture-culture, [as a way] to distribute ownership and establish a shared immune system.”


CoBox is an encrypted p2p file system and distributed back-up tool. With CoBox, there are no centralised servers, no data centers, and no surviellance architecture. Its just you and your peers, providing mutual support.

  • Want to create a real-time private box for you and your mates to drop in files, share projects and collaborate?
  • Want to back-up an extensive archive of videos or photos across several remove devices?
  • Scared your rampaging toddler is going to destroy your external hard drive?

Don't worry, CoBox has got you.

Our software creates encrypted private spaces on your laptop, which replicate and synchronise across a distributed network. A space is a private area accessible across multiple devices - a device can be another person, or it can be another computer you own. Each space contains its own file system, and, under the hood, comes log which is used to send space-specific messages. For example:

  • Declaring information about the current device to other members (e.g. a name)
  • Annotating files with custom metadata
  • Sending messages to other members of a space, privately or publicly
  • Informing other members that you have invited a new peer into the space

Why?


The cloud we have become used to looks nothing like the many scattered fluffy light weight balls of water vapour floating in the Earth's troposphere. Corporate cloud architectures are much more akin to a giant storm cloud, ever expanding, sucking in more and more electrical energy from its surrounding environment, with a singular giant eye sitting at their center obvserving all that passes through.

the all-seeing eye

Our team has worked with privacy tools and encryption for many years. We've tried our best to avoid using corporate architecture, and we understand, its hard. Corporate tools can be high quality and accessible - they have the money to pay for beautiful and intuitive user interfaces and enough servers to maintain a high speed service. But by having no alternatives to corporate tools, corporations get to determine how we use them and shape our very behaviour. By continuously using their tools, we're letting them shape and change how we think. And while they apparently provide these tools for 'free', they're making a tidy profit monetising the way we act on their platforms.

Our team is fed up of using proprietary software that surveilles our behaviour and transmutes the way we act online, the friends we make and the interactions we have - our data - into corporate financial gain. We want to use software built and maintained by our peers, by people we trust, that centers both our personal sovereignty and a common recognition that assets and wealth should be shared, not concentrated.

"What if co-operatives, that manifest self-governance and collaboration in their working relations could leverage these practices to provide data sovereignty, availability and resilience?

What if we could replace Google Docs or Microsoft Office with a suite of tools created by coops, for coops - designed to facilitate collaboration and serve co-operative governance needs?

What if we could create a distributed, co-operative cloud?"

So after discussing different ideas, talking to our peers, doing some research and surveying the available technologies, we built CoBox.

the all-seeing eye

CoBox wants to facilitate a transition away from giant data centers, huge storm clouds, towards a vision of cloud infrastructure that is light, distributed, and importantly, is offline-first. CoBox is the beginning of a sovereign commons-based data infrastructure and a co-operative distributed cloud architecture. Our vision sees friends, freelancers, small businesses and co-operatives, providing mutual assistance for each other, guaranteeing remote backup of their data and ensuring that they need not rely on giant corporations or flimsy hard drives to secure their memories.

How does it work?


CoBox aims to encourage small- and medium-scale infrastructure as a service. This means non-technical people can plug in a small computer in their home and begin providing back-up services for their friends, neighbours and themselves. How you ask?

CoBox creates augmented folders, what we call spaces, on your computer. Behaving much like a regular file system, you can drag and drop files into the space's folder, change the file's content, create sub-folders and build your own directory tree. The data inside a space is fully encrypted so only authenticated members can access it.

spaces
Each space is its own private bubble.

These folders are augmented with additional metadata or content specific to the space itself. At the moment, space members can give themselves a name, they can name the space itself, and they can tell other members that they've invited someone to join the space. space members will tell each other live of any updated information. So whenever a space member makes a change to a file, changes their name, or joins the space, all members will know about it pretty sharpish.

How about a story?

Think of a space a bit like a house. Each space or house has an address, a place where folks meet and hang out.

Your friend Bella told you to go there so you can work together on an important private document. She also gave you a magic pair of headphones, and tells you to put them on when you get there.

house
Each space has an address, which looks like this 98aa437f943948505ca28085cc9908ae3ce37eec333d671e13b49d143c416dd7.

You wander to the address Bella told you about and enter the house thats standing there. There are several people in the room. Bella is here, along with a few other people, and they're all talking to each other. But all the words coming of of everyone's mouths is total garbage - you can't understand anything anyone is saying. Observing the situation a bit more, you realise that some of the people in the room appear to be wearing headphones, and you remember that you too have headphones in your pocket. You put them on and, like something out of science fiction, suddenly everyone is speaking in a language you understand.

You and Bella sit down and begin putting together this important shared document. Everyone in the room copies what you're doing, although it appears that the people without the headphones don't really understand what they're writing down - it looks like gobbledeygook. Nonetheless, on they go, scribbling frantically.

In this analogy, the magic headphones are the encryption_key. This is a shared password between all space members. Anyone who holds the encryption_key can read and write to the space. The other slightly puzzled (and puzzling) people in the space – the ones writing things down who don't appear to understand whats going on – are what we call replicators. They don't have the encryption_key, but they keep a record of all the latest changes, so if space members aren't in the house at the same time, the replicators will tell the new arrival of the latest updates they know about. The replicators make sure that everyone is up to date and in the know, while themselves being totally ignorant of the content they possess. Ultimately, replicators are doing space members a favour, they've agreed to keep one eye on your space and make sure you can access it again if you ever need it.

When you start using the software, you'll find that you also have a public_key, which you can access from your profile page. This is a global identifier for only you. It is safe to share with anyone, but bear in mind that it identifies you and no-one else. If you want to gain access to a specific space, you'll need to send this public_key to a member of that space, and in return, they will give you an invite code. An invite code is just the address and the encryption_key locked in a safe using the public_key you sent them, so only you can open it. Only you can use this invite code. No-one else.

Getting Started


There are two important pieces of software that CoBox depends on. What you'll need to install depends to how you want to use CoBox.

DISCLAIMER: Please note that CoBox is experimental software still in development. We are currently in a peer review and testing phase. Therefore, use this software at your own risk. If you do use it, and/or would like to participate in giving feedback, we'd love to hear from you!

Magma Collective, 09.03.2020

NOTE: We have not yet published a release. We are planning to bundle our two top-level into easily installable image files with cross-platform support. In the meantime, bear with us. In order to run CoBox, you'll need to install Git and NodeJS version 11.12.0 or greater.

If you want to setup a development environment, we recommend you install Node Version Manager.

Magma Collective, 09.03.2020

Install the Hub

Backing up someone's data? You'll want to set up your own replication device.

Install the Client

No idea what a replication device is? Just want to install the app and create your own private space?

Install


You install the client on your laptop or desktop. Using the client software you can:

  1. Create private spaces for you and your peers
  2. Generate and accept invite codes to and from spaces
  3. Interact with a shared file system

Lets install the API

$ git clone https://gitlab.com/coboxcoop/cobox-server && cd cobox-server
$ npm install
$ npm start

Now lets install the UI

$ git clone https://gitlab.com/coboxcoop/cobox-ui && cd cobox-ui
$ npm install
$ npm run serve

Now navigate to http://localhost:8080

Create a Space


Once you have the client app started, you'll be prompted to give yourself a name. This is the name by which you will be known in all your spaces. Once you've picked a name you will land on the dashboard.

You can view a list of all your spaces, and you can go into each of them.

Now lets create a new space. Hit the + icon and create a space, give it a name, and hey presto, you now have your very own private space!

So lets add some files to the space.

  1. Hit the mount button. This will create a folder in your home directory under /cobox, followed by the space name. This is your space's file system and will synchronise across anyone with the space's address.

  2. Navigate to that directory and paste in the files you want to back-up

  3. Return to the interface and now at the top you can see the total space in bytes has gone up, equivalent to the size of the files you put in. You can now hit unmount and the folder will disappear.

Now we've got a space with some files in it, lets get a friend to backup your space.

The address is what you share with remote replicator devices so that they can begin backing up your space.

Copy the address and send it to the person you want to back your space up!

Sadly, at the moment, our UI doesn't support peers without a CoBox device doing backups, but this is coming really soon! If you don't have a CoBox device, but you know someone else who does, lets get them to back up your space.

Otherwise, lets setup a device.

Hub Install


The Hub is the software run by our blind replicator devices. CoBox Devices back-up yours and other's data without holding the encryption keys. Using the Hub software you can configure a stable computer, such as a Raspberry Pi in your home, to begin backing up your data.

In order to interact with your Hub, you'll need to install the client. Once installed, the Hub enables you to:

  1. Authenticate as an original administrator over a LAN
  2. Authenticate as an additional administrator using an invite code
  3. Authenticate as an additional administrator over a LAN
  4. Send remote commands to begin or stop replicating an address
$ git clone https://gitlab.com/coboxcoop/cobox-hub && cd cobox-hub
$ npm install
$ npm start

Setup a Device


You've just received your CoBox device and plugged it in, or you've installed the hub on a device on your local area network. And now we need to configure the device, and set it up so we can securely communicate with it from anywhere, not just on the LAN.

You'll want to hit the + icon on the dashboard next to devices, and select Setup a device.

Give your device a name, and hit Ok.

Thats it, you device is now configured to only talk to you and you alone. You can navigate to the admin dashboard for you device from the devices page.

If you want other people or computers to gain administrator privileges, there are a couple ways of doing this. Check out the next step, becoming and adding admins.

Becoming or Adding Admins


There are a couple of ways of becoming an admin for a CoBox replicator device. I'll run through the two methods here.

1. Over the Local Area Network

This requires you to be in the same physical location as the CoBox device itself, and having some kind of contact with the device's existing administrator.

You'll need to ask the existing device administrator to switch on the device's local broadcast. For them to do so, they'll simply need to hit the Announce button.

Once they've done that, you can simply follow the same steps in the setup a device instructions.

Once you've finished setting up the device, we strongly recommend you disable the broadcast. You can do so simply by hitting the Hide button. This means the device's authentication details are no longer openly visible on the network.

2. Using an Invite Code

This requires a two-step invite stage. You'll need to copy your public_key and send it to the existing device administrator. You can access your public_key either from your profile page - accessible if you click your name - or from the join a device page.

Once you've received an invite code, navigate to the join a device page, and paste in the code. If all went well and the invite code has not been tampered with, you are now a device administrator. Congratulations!

Backup a Space


So you've been sent an address to back-up someone's space. Lets go through what you need to do.

From the dashboard, you'll want to click into the devices page. Here you'll see a list of devices you are an administrator for. In the example below, nova is running her own cobox, nova's local cobox.

Click into your space and below 'Replicate Space', there's an input field for the address, and a name for the space you want to backup.

Simply paste the address, give it a memorable and easily recognisable name for any other administrators for in this space, and hit Ok.

Now we can see at the bottom of the page that our CoBox device is replicating this space. If we want to stop backing it up, all we need to do is hit the Unreplicate button, and it will stop replicating.

Thats it! You successfully backed up a space using CoBox. Congratulations!

Secure your Keys


Finally, its incredibly important to back up your space encryption keys, and your own secret_key, the counter-part to your public_key. If you lose this, you will no longer appear to other peers to be the same person.

Navigate to your profile page, and hit the 'Backup identity' button. This will enable you to download a PDF of your keys, which we recommend you store in a safe location in your home. Perhaps at the bottom of a filing cabinet, or behind a picture frame.

With CoBox, its important to remember, if you lose your secret_key, its not the end of the world. You can always create another account and gain access to your spaces as long as you have the encryption_key stored in a safe location, or a friend also has a copy.

In the future, we will make it so you can recover your old secret_key using a social key recovery mechanism - you'll be able to get your friends to restore your original keys!

Core Technologies


CoBox is built on top of dat. dat is a modular peer-to-peer technology stack. You can find a good explanation of how it works in the guide 'how dat works'.

dat

hypercore

dat's core database is an append-only log called hypercore. This log can be easily replicated between devices or peers in real-time, and it's integrity is preserved - ensuring other peers haven't tampered with the log's data - using cryptographically signed entries and merkle trees. To do so, each hypercore holds its own cryptographic keypair - a public key and a signing secret_key. Only the holder of the secret_key, the 'writer', can make changes to the log. But all peers with a copy of the log, without the keys, are allowed to 'read'.

hyperdrive

Lets step up the stack one step. On top of hypercore, dat have built a tool called hyperdrive. hyperdrive is a file system abstraction that uses two hypercore's - content and metadata - to represent a fully functioning unix-style file system. It can be mounted as a folder on your computer. When combined with a system of replication and peer discovery, 'writers' of a hyperdrive archive can dynamically update files on the remote 'reader' computers. Changes are replicated with all connected peers in real time, without the need for an authenticated server to guarantee the integrity of the data.

There are limitations to this setup. CoBox has taken steps to solve one limitation, which is the ability for multiple devices to participate as a 'writer', not just a 'reader', in a given file system.

With the default dat setup, if diverging changes are made using the same secret_key from different devices, the integrity of the log is broken and can be regarded as 'forked'. To prevent data corruption, its imperative that the secret_key is only ever used on a single device. This is a significant usability problem - it makes hyperdrive unsuitable for collaborative applications involving multiple peers or devices with write access - the fabled 'multi-writer' setup.

An additional usability issue dat struggles with is having many many secret_keys scattered across your file system, one for each hypercore, which, in our case, is quite a lot! We've gotten around this by using libsodium's key derivation function. All your hypercore keys are derived deterministically from a single parent_key.

multifeed

To resolve this issue, CoBox has made use of innovations by the kappa-db community. multifeed is an aggregation tool for multiple hypercores - it binds together a set of hypercores under a single public key, or as we call it, address. This public key does not directly correspond to a single hypercore, rather it corresponds to a dynamic set. When peers meet at this address on with their chosen networking tool, they first exchange a list of hypercore public keys they hold, then proceed to update each log with the latest data, as well as importing any new logs that may have appeared.

In a simple configuration, such as implemented in the IRC chat app Cabal, each peer maps to a single hypercore instance. For CoBox, each peer maps to three hypercores. If we need to, we can add to this dynamically in the future. Two of these hypercores - content and metadata feeds - are used by hyperdrive. The third is simply named log and is similar to Cabal's use, it stores binary-encoded JSON messages which are used for application-layer data.

kappa-core

To serve peers with live updates from a set of continuously sync'ing hypercores, the kappa-db community built a dynamic indexing system called kappa-core. kappa-core can be used to build custom materialised views over the datasets contained within a multifeed's collection of hypercores. We've used kappa-core to collect together the metadata from each peer's personal hyperdrive, to bind them together to act as a single multi-writer hyperdrive, called kappa-drive. To assist with possible collisions in drive state, we've implemented a conflict resolution mechanism based on 'vector clocks'. We've also built a module called kappa-view-query which enables us define a dynamic st of indexes for message types in our hypercores, over which we can perform scoped queries, and implement map/filter/reduce functions to reduce large datasets quickly and easily and help serve application layer data.

hyperswarm

CoBox uses the hyperswarm distributed hash table to connect peers over our multifeed address.

Modules

Our stack is built in NodeJS and we use libsodium for our cryptography. At the base of the stack lies hypercore, multifeed and kappa-core. And to access that intuitively, we have an ExpressJS API and a VueJS UI with Electron support.

We're currently investigating cross-platform support, including setting up FUSE mounts on Windows and easily packaged installers for Windows, MacOS and Linux.


Client

  • cobox-ui - a JavaScript front-end built with VueJS. Talks to cobox-server's REST API.

  • cobox-server - a JSON HTTP/S REST API to run on the client. Exposes end-points to a front-end to manage your groups and blind replicators, setup cobox devices, and perform admin functionality.

  • cobox-cli - a command-line interface build with javascript library Yargs. Talks to cobox-server's REST API.

  • cobox-group - adds content en/decryption and kappa-drive to cobox-replicator. Provides an additonal feed and indexes for group specific messages, and additional indexes for kappa-drive state / history.

  • paper-key-backup - export cryptographic keys to an easily printable format.

  • kappa-drive - multiwriter peer-to-peer filesystem, built on kappa-core and hyperdrive.

  • kappa-drive-mount - mount a kappa-drive using FUSE.


Hub

  • cobox-os - a devuan image builder for use on ARM devices or in VMs. Easy installation and configuration of your cobox device, installing and running cobox-hub

  • cobox-hub - utilises cobox-admin-group with additional setup functionality including UDP packet broacast over a LAN. Enables cobox device administation over both a local or remote connection


Shared

  • cobox-constants - system wide constants

  • cobox-config - local configuration settings stored in YAML

  • cobox-group-store - the repository pattern for finding a record based on a dynamic set of parameters from an existing collection. initialize and cache multiple cobox-group instances in a single node process.

  • cobox-replicator - base model for p2p private groups, containing storage and application layer data

  • cobox-admin-group - RPC-style command execution over a multifeed instance. Content en/decryption included

  • cobox-crypto - cryptographic primitives. All cryptography is compatible with the existing Dat ecosystem for future extensibility and interoperability.

  • hypercore-crypto-encoder - a content encryption encoder for hypercore using an encryption key.

  • kappa-view-query - a materialised view for kappa-core, aggregating all feed data in a multifeed instance, serving a dynamic and intuitive map-filter-reduce querying engine.


Package Tree

Here's how they're all laid out. We hope this helps you familiarise yourself with the project's structure, there's a lot of moving parts!

cobox-server
    │
    ├── cobox-config
    │
    └── cobox-group-store
            │
            ├──cobox-admin-group
            │      │
            │      └──cobox-replicator
            │
            └──cobox-group
                   │
                   ├── kappa-core
                   │
                   ├── cobox-replicator
                   │
                   ├── cobox-log
                   │       │
                   │       ├── kappa-sparse-indexer
                   │       │
                   │       ├── kappa-view-query
                   │       │
                   │       └── leveldb / subleveldown
                   │
                   └── kappa-drive
                           │
                           ├── hyperdrive
                           │       │
                           │       └── hypercore
                           │
                           ├── corestore
                           │
                           ├── mountable hypertrie
                           │
                           └── leveldb / subleveldown
                                   │
                                   └── kappa-view-kv

cobox-hub
    │
    └── cobox-admin-group
        │
        ├──cobox-log
        │
        └──cobox-replicator


cobox-replicator
    │
    ├── cobox-crypto
    │   │
    │   └── hypercore-crypto-encoder
    │
    ├── kappa-core
    │
    ├── multifeed
    │   │
    │   └── hypercore
    │
    ├── leveldb
    │   │
    │   └── kappa-view-query
    │
    ├── kappa-drive-mount
    │
    ├── hyperswarm
    │
    └── hypercore

Contributing


Want to get involved in developing CoBox? CoBox is a new open source project and all kinds of contributions are welcome.

Talk to us! You can find us on IRC in #dat on freenode (or in the browser through gitter).

Here you can find a list of our modules, all of which are hosted on our Gitlab. We encourage you to fork it, try it out, pick it apart and make pull requests!

For easier development, all our core modules have been bundled into a mono-repository, cobox-core.

Want to contribute to development and get involved in coding? Please get in contact by emailing info@magmacollective.org, we'd love to hear from you.

If you want to find out more about our future road-map, you can read the future section of our MVP document released in January 2020. Some of the features we want to develop we've spec'ed out. Here's what we already have planned...

  • identity-based authentication over the hypercore protocol
  • multifeed RPC
  • FUSE mount support for Windows
  • refactor polymorphic 'space' types by allowing pluggable middleware to a single space
  • network visualisation - an accessible map of your peers and backups
  • an allow-list - one-step and no-step invitations
  • space-level file permissions and nested mounted spaces
  • replication scheduling on your CoBox device, network settings and bandwidth limitation
  • sparse replication - pick and choose the data you want
  • multiple identities - no true names
  • a match-making platform to connect humans to store back-ups for each other
  • an address and public_key registry to simplify invitations and replication

Acknowledgements

We stand on the shoulders of giants. We've drawn on the amazing work that others have started and continue today. We've been inspired by many other projects, and we've had a lot of help along the way.

Thanks to:

Shout outs: