Distributed File System in Go

This project is a distributed file system implemented in Go, which allows for the storage and retrieval of files across multiple nodes in a peer-to-peer network.

Features

• Peer-to-peer Architecture: Files are distributed across multiple server nodes.
• Content-Addressable Storage: Files are stored using content hashing for efficient retrieval.
• File Encryption: Files are encrypted before transmission and storage.
• Automatic File Distribution: Files stored on one node are automatically distributed to other nodes.
• Transparent File Retrieval: Files can be retrieved from any node in the network.
• TCP Transport Layer: Communication between nodes is handled via TCP.

Project Structure

• main.go: Entry point that sets up and starts the file servers.
• server.go: Implements the FileServer which handles file operations and peer communication.
• store.go: Provides file storage and retrieval functionality.
• crypto.go: Contains cryptographic functions for file encryption/decryption.
• p2p: Package containing peer-to-peer networking components:
  • p2p/transport.go: Defines interfaces for network transport.
  • p2p/tcp_transport.go: TCP implementation of the Transport interface.
  • p2p/message.go: Defines message types for peer communication.
  • p2p/handshake.go: Handles peer connection handshakes.
  • p2p/encoding.go: Handles encoding/decoding of messages.

Getting Started

Prerequisites

• Go 1.18 or higher

Building the Project

make build

This will compile the project into a binary in the bin/ directory.

Running the Project

make run

This will build and start the distributed file system.

Running Tests

make test

How It Works

• Multiple file server nodes form a peer-to-peer network, connecting via TCP.
• When a file is stored on one node using the Store method:
  • The file is encrypted and stored locally.
  • The file is broadcast to other nodes in the network.
• When a file is requested using the Get method:
  • If the file exists locally, it's served directly.
  • If not, a request is broadcast to other nodes to retrieve the file.
  • Once retrieved, the file is stored locally for future requests.

Key Components

FileServer

Manages the core file operations and peer communication. It handles storing, retrieving, and distributing files across the network.

Store

Provides the functionality to write, read, and delete files from disk. It uses a path transformation function to organize files in a content-addressable manner.

TCP Transport

Implements the peer-to-peer communication layer using TCP, allowing nodes to exchange messages and stream file data.

Encryption

Files are encrypted before transmission and storage using AES encryption, ensuring data security across the network.