BitTorrent is a protocol for downloading and distributing files across the Internet. In contrast with the traditional client/server relationship, in which downloaders connect to a central server (for example: watching a movie on Netflix, or loading the web page you're reading now), participants in the BitTorrent network, called **peers**, download pieces of files from *each other*—this is what makes it a **peer-to-peer** protocol. In this article we will investigate how this works, and build our own client that can find peers and exchange data between them.
BitTorrent is a protocol for downloading and distributing files across the Internet. In contrast with the traditional client/server relationship, in which downloaders connect to a central server (for example: watching a movie on Netflix, or loading the web page you're reading now), participants in the BitTorrent network, called **peers**, download pieces of files from *each other*—this is what makes it a **peer-to-peer** protocol. In this article we will investigate how this works, and build our own client that can find peers and exchange data between them.
The protocol evolved organically over the past 20 years, and various people and organizations added extensions for features like encryption, private torrents, and new ways of finding peers. We'll be implementing the [original spec](https://www.bittorrent.org/beps/bep_0003.html) from 2001 to keep this a weekend-sized project.
The protocol evolved organically over the past 20 years, and various people and organizations added extensions for features like encryption, private torrents, and new ways of finding peers. We'll be implementing the [original spec](https://www.bittorrent.org/beps/bep_0003.html) from 2001 to keep this a weekend-sized project.
@ -9,7 +9,7 @@ I'll be using a [Debian ISO](https://cdimage.debian.org/debian-cd/current/amd64/
# Finding peers
# Finding peers
Here’s a problem: we want to download a file with BitTorrent, but it’s a peer-to-peer protocol and we have no idea where to find peers to download it from. This is a lot like moving to a new city and trying to make friends—maybe we’ll hit up a local pub or a meetup group! Centralized locations like these are the big idea behind trackers, which are central servers that introduce peers to each other. They’re just web servers running over HTTP, and you can find Debian’s at http://bttracker.debian.org:6969/
Here’s a problem: we want to download a file with BitTorrent, but it’s a peer-to-peer protocol and we have no idea where to find peers to download it from. This is a lot like moving to a new city and trying to make friends—maybe we’ll hit up a local pub or a meetup group! Centralized locations like these are the big idea behind trackers, which are central servers that introduce peers to each other. They’re just web servers running over HTTP, and you can find Debian’s at http://bttracker.debian.org:6969/
Of course, these central servers are liable to get raided by the feds if they facilitate peers exchanging illegal content. You may remember reading about trackers like TorrentSpy, Popcorn Time, and KickassTorrents getting seized and shut down. New methods cut out the middleman by making even **peer discovery** a distributed process. We won't be implementing them, but if you're interested, some terms you can research are **DHT**, **PEX**, and **magnet links**.
Of course, these central servers are liable to get raided by the feds if they facilitate peers exchanging illegal content. You may remember reading about trackers like TorrentSpy, Popcorn Time, and KickassTorrents getting seized and shut down. New methods cut out the middleman by making even **peer discovery** a distributed process. We won't be implementing them, but if you're interested, some terms you can research are **DHT**, **PEX**, and **magnet links**.
@ -51,7 +51,7 @@ e
In this file, we can spot the URL of the tracker, the creation date (as a Unix timestamp), the name and size of the file, and a big binary blob containing the SHA-1 hashes of each **piece**, which are equally-sized parts of the file we want to download. The exact size of a piece varies between torrents, but they are usually somewhere between 256KB and 1MB. This means that a large file might be made up of *thousands* of pieces. We'll download these pieces from our peers, check them against the hashes from our torrent file, assemble them together, and boom, we've got a file!
In this file, we can spot the URL of the tracker, the creation date (as a Unix timestamp), the name and size of the file, and a big binary blob containing the SHA-1 hashes of each **piece**, which are equally-sized parts of the file we want to download. The exact size of a piece varies between torrents, but they are usually somewhere between 256KB and 1MB. This means that a large file might be made up of *thousands* of pieces. We'll download these pieces from our peers, check them against the hashes from our torrent file, assemble them together, and boom, we've got a file!
This mechanism allows us to verify the integrity of each piece as we go. It makes BitTorrent resistant to accidental corruption or intentional **torrent poisoning**. Unless an attacker is capable of breaking SHA-1 with a preimage attack, we will get exactly the content we asked for.
This mechanism allows us to verify the integrity of each piece as we go. It makes BitTorrent resistant to accidental corruption or intentional **torrent poisoning**. Unless an attacker is capable of breaking SHA-1 with a preimage attack, we will get exactly the content we asked for.
@ -89,7 +89,7 @@ Because I like to keep my structures relatively flat, and I like to keep my appl
Notably, I split `pieces` (previously a string) into a slice of hashes (each `[20]byte`) so that I can easily access individual hashes later. I also computed the SHA-1 hash of the entire bencoded `info` dict (the one which contained the name, size, and piece hashes). We know this as the **infohash** and it uniquely identifies files when we talk to trackers and peers. More on this later.
Notably, I split `pieces` (previously a string) into a slice of hashes (each `[20]byte`) so that I can easily access individual hashes later. I also computed the SHA-1 hash of the entire bencoded `info` dict (the one which contained the name, size, and piece hashes). We know this as the **infohash** and it uniquely identifies files when we talk to trackers and peers. More on this later.
```go
```go
type TorrentFile struct {
type TorrentFile struct {
@ -134,7 +134,7 @@ The important ones:
* **info_hash**: Identifies the *file* we're trying to download. It's the infohash we calculated earlier from the bencoded `info` dict. The tracker will use this to figure out which peers to show us.
* **info_hash**: Identifies the *file* we're trying to download. It's the infohash we calculated earlier from the bencoded `info` dict. The tracker will use this to figure out which peers to show us.
* **peer_id**: A 20 byte name to identify *ourselves* to trackers and peers. We'll just generate 20 random bytes for this. Real BitTorrent clients have IDs like `-TR2940-k8hj0wgej6ch` which identify the client software and version—in this case, TR2940 stands for Transmission client 2.94.
* **peer_id**: A 20 byte name to identify *ourselves* to trackers and peers. We'll just generate 20 random bytes for this. Real BitTorrent clients have IDs like `-TR2940-k8hj0wgej6ch` which identify the client software and version—in this case, TR2940 stands for Transmission client 2.94.
## Parsing the tracker response
## Parsing the tracker response
We get back a bencoded response:
We get back a bencoded response:
@ -152,7 +152,7 @@ e
`Peers` is another long binary blob containing the IP addresses of each peer. It's made out of **groups of six bytes**. The first four bytes in each group represent the peer's IP address—each byte represents a number in the IP. The last two bytes represent the port, as a big-endian `uint16`. **Big-endian**, or **network order**, means that we can interpret a group of bytes as an integer by just squishing them together left to right. For example, the bytes `0x1A`, `0xE1` make `0x1AE1`, or 6881 in decimal.
`Peers` is another long binary blob containing the IP addresses of each peer. It's made out of **groups of six bytes**. The first four bytes in each group represent the peer's IP address—each byte represents a number in the IP. The last two bytes represent the port, as a big-endian `uint16`. **Big-endian**, or **network order**, means that we can interpret a group of bytes as an integer by just squishing them together left to right. For example, the bytes `0x1A`, `0xE1` make `0x1AE1`, or 6881 in decimal.
```go
```go
// Peer encodes connection information for a peer
// Peer encodes connection information for a peer
@ -203,7 +203,7 @@ We've just set up a connection with a peer, but we want do a handshake to valida
* is able to understand and respond to our messages
* is able to understand and respond to our messages
* has the file that we want, or at least knows what we're talking about
* has the file that we want, or at least knows what we're talking about
My father told me that the secret to a good handshake is a firm grip and eye contact. The secret to a good BitTorrent handshake is that it's made up of five parts:
My father told me that the secret to a good handshake is a firm grip and eye contact. The secret to a good BitTorrent handshake is that it's made up of five parts:
@ -255,12 +255,12 @@ Once we've completed the initial handshake, we can send and receive **messages**
Once we've been unchoked, we can then begin sending **requests** for pieces, and they can send us messages back containing pieces.
Once we've been unchoked, we can then begin sending **requests** for pieces, and they can send us messages back containing pieces.
### Interpreting messages
### Interpreting messages
A message has a length, an **ID** and a **payload**. On the wire, it looks like:
A message has a length, an **ID** and a **payload**. On the wire, it looks like:
A message starts with a length indicator which tells us how many bytes long the message will be. It's a 32-bit integer, meaning it's made out of four bytes smooshed together in big-endian order. The next byte, the **ID**, tells us which type of message we're receiving—for example, a `2` byte means "interested." Finally, the optional **payload** fills out the remaining length of the message.
A message starts with a length indicator which tells us how many bytes long the message will be. It's a 32-bit integer, meaning it's made out of four bytes smooshed together in big-endian order. The next byte, the **ID**, tells us which type of message we're receiving—for example, a `2` byte means "interested." Finally, the optional **payload** fills out the remaining length of the message.
One of the most interesting types of message is the **bitfield**, which is a data structure that peers use to efficiently encode which pieces they are able to send us. A bitfield looks like a byte array, and to check which pieces they have, we just need to look at the positions of the *bits* set to 1. You can think of it like the digital equivalent of a coffee shop loyalty card. We start with a blank card of all `0`, and flip bits to `1` to mark their positions as "stamped."
One of the most interesting types of message is the **bitfield**, which is a data structure that peers use to efficiently encode which pieces they are able to send us. A bitfield looks like a byte array, and to check which pieces they have, we just need to look at the positions of the *bits* set to 1. You can think of it like the digital equivalent of a coffee shop loyalty card. We start with a blank card of all `0`, and flip bits to `1` to mark their positions as "stamped."
By working with *bits* instead of *bytes*, this data structure is super compact. We can stuff information about eight pieces in the space of a single byte—the size of a `bool`. The tradeoff is that accessing values becomes a little more tricky. The smallest unit of memory that computers can address are bytes, so to get to our bits, we have to do some bitwise manipulation:
By working with *bits* instead of *bytes*, this data structure is super compact. We can stuff information about eight pieces in the space of a single byte—the size of a `bool`. The tradeoff is that accessing values becomes a little more tricky. The smallest unit of memory that computers can address are bytes, so to get to our bits, we have to do some bitwise manipulation:
@ -395,7 +395,7 @@ close(workQueue)
We'll spawn a worker goroutine for each peer we've received from the tracker. It'll connect and handshake with the peer, and then start retrieving work from the `workQueue`, attempting to download it, and sending downloaded pieces back through the `results` channel.
We'll spawn a worker goroutine for each peer we've received from the tracker. It'll connect and handshake with the peer, and then start retrieving work from the `workQueue`, attempting to download it, and sending downloaded pieces back through the `results` channel.
@ -477,7 +477,7 @@ A peer is supposed to sever the connection if they receive a request for a block
### Pipelining
### Pipelining
Network round-trips are expensive, and requesting each block one by one will absolutely tank the performance of our download. Therefore, it's important to **pipeline** our requests such that we keep up a constant pressure of some number of unfulfilled requests. This can increase the throughput of our connection by an order of magnitude.
Network round-trips are expensive, and requesting each block one by one will absolutely tank the performance of our download. Therefore, it's important to **pipeline** our requests such that we keep up a constant pressure of some number of unfulfilled requests. This can increase the throughput of our connection by an order of magnitude.
Classically, BitTorrent clients kept a queue of five pipelined requests, and that's the value I'll be using. I found that increasing it can up to double the speed of a download. Newer clients use an [adaptive](https://luminarys.com/posts/writing-a-bittorrent-client.html) queue size to better accommodate modern network speeds and conditions. This is definitely a parameter worth tweaking, and it's pretty low hanging fruit for future performance optimization.
Classically, BitTorrent clients kept a queue of five pipelined requests, and that's the value I'll be using. I found that increasing it can up to double the speed of a download. Newer clients use an [adaptive](https://luminarys.com/posts/writing-a-bittorrent-client.html) queue size to better accommodate modern network speeds and conditions. This is definitely a parameter worth tweaking, and it's pretty low hanging fruit for future performance optimization.
Kamran Ahmed
5 years ago
