“...just as much about technological solutions as it is about the social tissue that we can nurture-culture, [as a way] to distribute ownership and establish a shared immune system.”
CoBox is an encrypted p2p file system and distributed back-up tool. With CoBox, there are no centralised servers, no data centers, and no surviellance architecture. Its just you and your peers, providing mutual support.
- Want to create a real-time private box for you and your mates to drop in files, share projects and collaborate?
- Want to back-up an extensive archive of videos or photos across several remove devices?
- Scared your rampaging toddler is going to destroy your external hard drive?
Don't worry, CoBox has got you.
Our software creates encrypted private
spaces on your laptop, which replicate and synchronise across a distributed network. A
space is a private area accessible across multiple devices - a device can be another person, or it can be another computer you own. Each space contains its own file system, and, under the hood, comes log which is used to send
space-specific messages. For example:
- Declaring information about the current device to other members (e.g. a name)
- Annotating files with custom metadata
- Sending messages to other members of a space, privately or publicly
- Informing other members that you have invited a new peer into the
The cloud we have become used to looks nothing like the many scattered fluffy light weight balls of water vapour floating in the Earth's troposphere. Corporate cloud architectures are much more akin to a giant storm cloud, ever expanding, sucking in more and more electrical energy from its surrounding environment, with a singular giant eye sitting at their center obvserving all that passes through.
Our team has worked with privacy tools and encryption for many years. We've tried our best to avoid using corporate architecture, and we understand, its hard. Corporate tools can be high quality and accessible - they have the money to pay for beautiful and intuitive user interfaces and enough servers to maintain a high speed service. But by having no alternatives to corporate tools, corporations get to determine how we use them and shape our very behaviour. By continuously using their tools, we're letting them shape and change how we think. And while they apparently provide these tools for 'free', they're making a tidy profit monetising the way we act on their platforms.
Our team is fed up of using proprietary software that surveilles our behaviour and transmutes the way we act online, the friends we make and the interactions we have - our data - into corporate financial gain. We want to use software built and maintained by our peers, by people we trust, that centers both our personal sovereignty and a common recognition that assets and wealth should be shared, not concentrated.
"What if co-operatives, that manifest self-governance and collaboration in their working relations could leverage these practices to provide data sovereignty, availability and resilience?
What if we could replace Google Docs or Microsoft Office with a suite of tools created by coops, for coops - designed to facilitate collaboration and serve co-operative governance needs?
What if we could create a distributed, co-operative cloud?"
So after discussing different ideas, talking to our peers, doing some research and surveying the available technologies, we built CoBox.
CoBox wants to facilitate a transition away from giant data centers, huge storm clouds, towards a vision of cloud infrastructure that is light, distributed, and importantly, is offline-first. CoBox is the beginning of a sovereign commons-based data infrastructure and a co-operative distributed cloud architecture. Our vision sees friends, freelancers, small businesses and co-operatives, providing mutual assistance for each other, guaranteeing remote backup of their data and ensuring that they need not rely on giant corporations or flimsy hard drives to secure their memories.
CoBox aims to encourage small- and medium-scale infrastructure as a service. This means non-technical people can plug in a small computer in their home and begin providing back-up services for their friends, neighbours and themselves. How you ask?
CoBox creates augmented folders, what we call
spaces, on your computer. Behaving much like a regular file system, you can drag and drop files into the
space's folder, change the file's content, create sub-folders and build your own directory tree. The data inside a
space is fully encrypted so only authenticated members can access it.
These folders are augmented with additional metadata or content specific to the
space itself. At the moment,
space members can give themselves a name, they can name the
space itself, and they can tell other members that they've invited someone to join the
space members will tell each other live of any updated information. So whenever a space member makes a change to a file, changes their name, or joins the space, all members will know about it pretty sharpish.
Think of a space a bit like a house. Each space or house has an
address, a place where folks meet and hang out.
Your friend Bella told you to go there so you can work together on an important private document. She also gave you a magic pair of headphones, and tells you to put them on when you get there.
You wander to the
address Bella told you about and enter the house thats standing there. There are several people in the room. Bella is here, along with a few other people, and they're all talking to each other. But all the words coming of of everyone's mouths is total garbage - you can't understand anything anyone is saying. Observing the situation a bit more, you realise that some of the people in the room appear to be wearing headphones, and you remember that you too have headphones in your pocket. You put them on and, like something out of science fiction, suddenly everyone is speaking in a language you understand.
You and Bella sit down and begin putting together this important shared document. Everyone in the room copies what you're doing, although it appears that the people without the headphones don't really understand what they're writing down - it looks like gobbledeygook. Nonetheless, on they go, scribbling frantically.
In this analogy, the magic headphones are the
encryption_key. This is a shared password between all
space members. Anyone who holds the
encryption_key can read and write to the
space. The other slightly puzzled (and puzzling) people in the
space – the ones writing things down who don't appear to understand whats going on – are what we call
replicators. They don't have the
encryption_key, but they keep a record of all the latest changes, so if space members aren't in the house at the same time, the
replicators will tell the new arrival of the latest updates they know about. The replicators make sure that everyone is up to date and in the know, while themselves being totally ignorant of the content they possess. Ultimately, replicators are doing
space members a favour, they've agreed to keep one eye on your
space and make sure you can access it again if you ever need it.
When you start using the software, you'll find that you also have a
public_key, which you can access from your profile page. This is a global identifier for only you. It is safe to share with anyone, but bear in mind that it identifies you and no-one else. If you want to gain access to a specific space, you'll need to send this
public_key to a member of that space, and in return, they will give you an invite code. An invite code is just the
address and the
encryption_key locked in a safe using the
public_key you sent them, so only you can open it. Only you can use this invite code. No-one else.
There are two important pieces of software that CoBox depends on. What you'll need to install depends to how you want to use CoBox.
DISCLAIMER: Please note that CoBox is experimental software still in development. We are currently in a peer review and testing phase. Therefore, use this software at your own risk. If you do use it, and/or would like to participate in giving feedback, we'd love to hear from you!
Magma Collective, 09.03.2020
NOTE: We have not yet published a release. We are planning to bundle our two top-level into easily installable image files with cross-platform support. In the meantime, bear with us. In order to run CoBox, you'll need to install Git and NodeJS version 11.12.0 or greater.
If you want to setup a development environment, we recommend you install Node Version Manager.
Magma Collective, 09.03.2020
Backing up someone's data? You'll want to set up your own replication device.
No idea what a replication device is? Just want to install the app and create your own private space?
You install the client on your laptop or desktop. Using the client software you can:
- Create private spaces for you and your peers
- Generate and accept invite codes to and from spaces
- Interact with a shared file system
Lets install the API
$ git clone https://gitlab.com/coboxcoop/cobox-server && cd cobox-server
$ npm install
$ npm start
Now lets install the UI
$ git clone https://gitlab.com/coboxcoop/cobox-ui && cd cobox-ui
$ npm install
$ npm run serve
Now navigate to http://localhost:8080
Once you have the client app started, you'll be prompted to give yourself a name. This is the name by which you will be known in all your
spaces. Once you've picked a name you will land on the dashboard.
You can view a list of all your
spaces, and you can go into each of them.
Now lets create a new
space. Hit the
+ icon and create a
space, give it a name, and hey presto, you now have your very own private
So lets add some files to the space.
mountbutton. This will create a folder in your home directory under
/cobox, followed by the
spacename. This is your
space's file system and will synchronise across anyone with the
Navigate to that directory and paste in the files you want to back-up
Return to the interface and now at the top you can see the total
spacein bytes has gone up, equivalent to the size of the files you put in. You can now hit
unmountand the folder will disappear.
Now we've got a
space with some files in it, lets get a friend to backup your
address is what you share with remote replicator devices so that they can begin backing up your
address and send it to the person you want to back your
Sadly, at the moment, our UI doesn't support peers without a CoBox device doing backups, but this is coming really soon! If you don't have a CoBox device, but you know someone else who does, lets get them to back up your space.
Otherwise, lets setup a device.
The Hub is the software run by our blind replicator devices. CoBox Devices back-up yours and other's data without holding the encryption keys. Using the Hub software you can configure a stable computer, such as a Raspberry Pi in your home, to begin backing up your data.
In order to interact with your Hub, you'll need to install the client. Once installed, the Hub enables you to:
- Authenticate as an original administrator over a LAN
- Authenticate as an additional administrator using an invite code
- Authenticate as an additional administrator over a LAN
- Send remote commands to begin or stop replicating an address
$ git clone https://gitlab.com/coboxcoop/cobox-hub && cd cobox-hub $ npm install $ npm start
You've just received your CoBox device and plugged it in, or you've installed the hub on a device on your local area network. And now we need to configure the device, and set it up so we can securely communicate with it from anywhere, not just on the LAN.
You'll want to hit the
+ icon on the dashboard next to devices, and select
Setup a device.
Give your device a name, and hit
Thats it, you device is now configured to only talk to you and you alone. You can navigate to the admin dashboard for you device from the devices page.
If you want other people or computers to gain administrator privileges, there are a couple ways of doing this. Check out the next step, becoming and adding admins.
There are a couple of ways of becoming an admin for a CoBox replicator device. I'll run through the two methods here.
This requires you to be in the same physical location as the CoBox device itself, and having some kind of contact with the device's existing administrator.
You'll need to ask the existing device administrator to switch on the device's local broadcast. For them to do so, they'll simply need to hit the
Once they've done that, you can simply follow the same steps in the setup a device instructions.
Once you've finished setting up the device, we strongly recommend you disable the broadcast. You can do so simply by hitting the
Hide button. This means the device's authentication details are no longer openly visible on the network.
This requires a two-step invite stage. You'll need to copy your
public_key and send it to the existing device administrator. You can access your
public_key either from your profile page - accessible if you click your name - or from the join a device page.
Once you've received an invite code, navigate to the join a device page, and paste in the code. If all went well and the invite code has not been tampered with, you are now a device administrator. Congratulations!
So you've been sent an
address to back-up someone's
space. Lets go through what you need to do.
From the dashboard, you'll want to click into the devices page. Here you'll see a list of devices you are an administrator for. In the example below, nova is running her own cobox,
nova's local cobox.
Click into your space and below 'Replicate Space', there's an input field for the
address, and a name for the space you want to backup.
Simply paste the
address, give it a memorable and easily recognisable name for any other administrators for in this space, and hit
Now we can see at the bottom of the page that our CoBox device is replicating this space. If we want to stop backing it up, all we need to do is hit the
Unreplicate button, and it will stop replicating.
Thats it! You successfully backed up a space using CoBox. Congratulations!
Finally, its incredibly important to back up your space encryption keys, and your own
secret_key, the counter-part to your
public_key. If you lose this, you will no longer appear to other peers to be the same person.
Navigate to your profile page, and hit the 'Backup identity' button. This will enable you to download a PDF of your keys, which we recommend you store in a safe location in your home. Perhaps at the bottom of a filing cabinet, or behind a picture frame.
With CoBox, its important to remember, if you lose your
secret_key, its not the end of the world. You can always create another account and gain access to your
spaces as long as you have the
encryption_key stored in a safe location, or a friend also has a copy.
In the future, we will make it so you can recover your old
secret_key using a social key recovery mechanism - you'll be able to get your friends to restore your original keys!
CoBox is built on top of
dat is a modular peer-to-peer technology stack. You can find a good explanation of how it works in the guide 'how dat works'.
dat's core database is an append-only log called
hypercore. This log can be easily replicated between devices or peers in real-time, and it's integrity is preserved - ensuring other peers haven't tampered with the log's data - using cryptographically signed entries and merkle trees. To do so, each
hypercore holds its own cryptographic keypair - a public key and a signing
secret_key. Only the holder of the
secret_key, the 'writer', can make changes to the log. But all peers with a copy of the log, without the keys, are allowed to 'read'.
Lets step up the stack one step. On top of
dat have built a tool called
hyperdrive is a file system abstraction that uses two
metadata - to represent a fully functioning unix-style file system. It can be mounted as a folder on your computer. When combined with a system of replication and peer discovery, 'writers' of a
hyperdrive archive can dynamically update files on the remote 'reader' computers. Changes are replicated with all connected peers in real time, without the need for an authenticated server to guarantee the integrity of the data.
There are limitations to this setup. CoBox has taken steps to solve one limitation, which is the ability for multiple devices to participate as a 'writer', not just a 'reader', in a given file system.
With the default
datsetup, if diverging changes are made using the same
secret_keyfrom different devices, the integrity of the log is broken and can be regarded as 'forked'. To prevent data corruption, its imperative that the
secret_keyis only ever used on a single device. This is a significant usability problem - it makes
hyperdriveunsuitable for collaborative applications involving multiple peers or devices with write access - the fabled 'multi-writer' setup.
An additional usability issue
datstruggles with is having many many
secret_keys scattered across your file system, one for each
hypercore, which, in our case, is quite a lot! We've gotten around this by using
libsodium's key derivation function. All your hypercore keys are derived deterministically from a single
To resolve this issue, CoBox has made use of innovations by the
multifeed is an aggregation tool for multiple hypercores - it binds together a set of
hypercores under a single public key, or as we call it,
address. This public key does not directly correspond to a single hypercore, rather it corresponds to a dynamic set. When peers meet at this address on with their chosen networking tool, they first exchange a list of
hypercore public keys they hold, then proceed to update each log with the latest data, as well as importing any new logs that may have appeared.
In a simple configuration, such as implemented in the IRC chat app Cabal, each peer maps to a single
hypercore instance. For CoBox, each peer maps to three
hypercores. If we need to, we can add to this dynamically in the future. Two of these
metadata feeds - are used by
hyperdrive. The third is simply named
log and is similar to Cabal's use, it stores binary-encoded JSON messages which are used for application-layer data.
To serve peers with live updates from a set of continuously sync'ing
kappa-db community built a dynamic indexing system called
kappa-core can be used to build custom materialised views over the datasets contained within a
multifeed's collection of
hypercores. We've used
kappa-core to collect together the metadata from each peer's personal
hyperdrive, to bind them together to act as a single multi-writer hyperdrive, called
kappa-drive. To assist with possible collisions in drive state, we've implemented a conflict resolution mechanism based on 'vector clocks'. We've also built a module called
kappa-view-query which enables us define a dynamic st of indexes for message types in our
hypercores, over which we can perform scoped queries, and implement
map/filter/reduce functions to reduce large datasets quickly and easily and help serve application layer data.
CoBox uses the
hyperswarm distributed hash table to connect peers over our
Our stack is built in NodeJS and we use
libsodium for our cryptography. At the base of the stack lies
kappa-core. And to access that intuitively, we have an ExpressJS API and a VueJS UI with Electron support.
We're currently investigating cross-platform support, including setting up FUSE mounts on Windows and easily packaged installers for Windows, MacOS and Linux.
cobox-server - a JSON HTTP/S REST API to run on the client. Exposes end-points to a front-end to manage your groups and blind replicators, setup cobox devices, and perform admin functionality.
cobox-group - adds content en/decryption and kappa-drive to cobox-replicator. Provides an additonal feed and indexes for group specific messages, and additional indexes for kappa-drive state / history.
paper-key-backup - export cryptographic keys to an easily printable format.
kappa-drive - multiwriter peer-to-peer filesystem, built on kappa-core and hyperdrive.
kappa-drive-mount - mount a kappa-drive using FUSE.
cobox-os - a devuan image builder for use on ARM devices or in VMs. Easy installation and configuration of your cobox device, installing and running cobox-hub
cobox-hub - utilises cobox-admin-group with additional setup functionality including UDP packet broacast over a LAN. Enables cobox device administation over both a local or remote connection
cobox-constants - system wide constants
cobox-config - local configuration settings stored in YAML
cobox-group-store - the repository pattern for finding a record based on a dynamic set of parameters from an existing collection. initialize and cache multiple cobox-group instances in a single node process.
cobox-replicator - base model for p2p private groups, containing storage and application layer data
cobox-admin-group - RPC-style command execution over a multifeed instance. Content en/decryption included
cobox-crypto - cryptographic primitives. All cryptography is compatible with the existing Dat ecosystem for future extensibility and interoperability.
hypercore-crypto-encoder - a content encryption encoder for hypercore using an encryption key.
kappa-view-query - a materialised view for kappa-core, aggregating all feed data in a multifeed instance, serving a dynamic and intuitive map-filter-reduce querying engine.
Here's how they're all laid out. We hope this helps you familiarise yourself with the project's structure, there's a lot of moving parts!
cobox-server │ ├── cobox-config │ └── cobox-group-store │ ├──cobox-admin-group │ │ │ └──cobox-replicator │ └──cobox-group │ ├── kappa-core │ ├── cobox-replicator │ ├── cobox-log │ │ │ ├── kappa-sparse-indexer │ │ │ ├── kappa-view-query │ │ │ └── leveldb / subleveldown │ └── kappa-drive │ ├── hyperdrive │ │ │ └── hypercore │ ├── corestore │ ├── mountable hypertrie │ └── leveldb / subleveldown │ └── kappa-view-kv cobox-hub │ └── cobox-admin-group │ ├──cobox-log │ └──cobox-replicator cobox-replicator │ ├── cobox-crypto │ │ │ └── hypercore-crypto-encoder │ ├── kappa-core │ ├── multifeed │ │ │ └── hypercore │ ├── leveldb │ │ │ └── kappa-view-query │ ├── kappa-drive-mount │ ├── hyperswarm │ └── hypercore
Want to get involved in developing CoBox? CoBox is a new open source project and all kinds of contributions are welcome.
Talk to us! You can find us on IRC in #dat on freenode (or in the browser through gitter).
For easier development, all our core modules have been bundled into a mono-repository,
Want to contribute to development and get involved in coding? Please get in contact by emailing email@example.com, we'd love to hear from you.
If you want to find out more about our future road-map, you can read the future section of our MVP document released in January 2020. Some of the features we want to develop we've spec'ed out. Here's what we already have planned...
- identity-based authentication over the hypercore protocol
- multifeed RPC
- FUSE mount support for Windows
- refactor polymorphic 'space' types by allowing pluggable middleware to a single
- network visualisation - an accessible map of your peers and backups
- an allow-list - one-step and no-step invitations
space-level file permissions and nested mounted spaces
- replication scheduling on your CoBox device, network settings and bandwidth limitation
- sparse replication - pick and choose the data you want
- multiple identities - no true names
- a match-making platform to connect humans to store back-ups for each other
public_keyregistry to simplify invitations and replication
We stand on the shoulders of giants. We've drawn on the amazing work that others have started and continue today. We've been inspired by many other projects, and we've had a lot of help along the way.
- Franz Frando
- Karissa McKelvey
- Dominic Tarr
- Kira a.k.a. Noffle
- Alex Cobleigh
- Mathias Buus
- Andrew Osheroff