From f97020e7802930ae4dcfc2a9ddabd70240b6de00 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 14 Nov 2025 09:31:26 +0000 Subject: [PATCH] [CLEANUP] Remove infrafabric research files from navidocs repo These files were created by agents researching for InfraFabric project and should not be in NaviDocs repository. --- OBJECT-STORAGE-PROVIDERS.md | 1320 ----------------------------------- 1 file changed, 1320 deletions(-) delete mode 100644 OBJECT-STORAGE-PROVIDERS.md diff --git a/OBJECT-STORAGE-PROVIDERS.md b/OBJECT-STORAGE-PROVIDERS.md deleted file mode 100644 index 18a431c..0000000 --- a/OBJECT-STORAGE-PROVIDERS.md +++ /dev/null @@ -1,1320 +0,0 @@ -# Object Storage Providers - Native API Research - -**Documentation Date:** November 14, 2025 -**Research Agent:** Haiku-30 (Object Storage Specialist) -**Citation Base:** if://research/object-storage-native-apis-2025-11-14 -**Mission:** Research non-S3 native APIs for InfraFabric integration -**Status:** Complete - 3 Major Providers Analyzed - ---- - -## Executive Summary - -Three modern object storage providers offer compelling **native APIs beyond S3-compatible** access, each with distinct advantages for different InfraFabric use cases: - -| Provider | Native API | Strength | Cost/GB/mo | Egress | IF Complexity | Best For | -|----------|-----------|----------|-----------|--------|---|---| -| **Backblaze B2** | REST JSON (b2_authorize_account) | Cost + Strong consistency | $0.005 | $0.01/GB | 5/10 | Cost-sensitive, versioning | -| **Cloudflare R2** | Workers API + S3 Compatibility | Zero egress fees | $0.015 | FREE | 4/10 | High-bandwidth, edge integration | -| **Storj DCS** | Uplink SDK (Go native) | Decentralized, privacy-first | $0.004 | $0.007/GB | 7/10 | Privacy-critical, distributed | - -**Recommendation for NaviDocs/InfraFabric:** **Backblaze B2** offers optimal balance of cost, consistency, and simplicity for document storage at scale. - ---- - -## Table of Contents - -1. [Backblaze B2 Native API](#backblaze-b2-native-api) -2. [Cloudflare R2 API](#cloudflare-r2-api) -3. [Storj Decentralized Object Storage](#storj-decentralized-object-storage) -4. [Feature Comparison Matrix](#feature-comparison-matrix) -5. [Integration Assessment](#integration-assessment) -6. [Pricing Analysis](#pricing-analysis) -7. [Research Citations](#research-citations) - ---- - -## Backblaze B2 Native API - -> **Citation Base:** if://api/b2-native-v4-2025-11-14 -> **Maturity:** Production-Ready (Stable) -> **InfraFabric Complexity Rating:** 5/10 - -### API Overview - -**Service:** Backblaze B2 Cloud Storage (object storage) -**API Type:** Native REST JSON API (v1, v2, v3, v4) -**Primary Endpoint:** `https://api.backblazeb2.com` -**Regional Endpoint:** `https://apiNNN.backblazeb2.com` (where NNN = cluster) -**Current API Version:** v4 (with multi-bucket application keys) -**Protocol:** HTTPS only (required) -**Response Format:** JSON - -### Authentication & Authorization - -#### Application Keys (Primary Authentication) - -``` -Authorization Header Format: -Authorization: Basic base64(applicationKeyId:applicationKey) -``` - -**Key Types:** -1. **Master Application Key** - - Full account access - - Use only for account setup/administration - - Can create restricted keys - - DO NOT embed in applications - -2. **Normal Application Keys** - - Restricted to specific buckets/operations - - Can be scoped to read-only, write-only, or read-write - - Recommended for application use - - Can be revoked independently - -3. **Multi-Bucket Application Keys (v4 Feature)** - - Grant access to multiple buckets simultaneously - - New in API v4 - - Enables cross-bucket operations - - `b2_list_keys` only returns multi-bucket keys in v4 - -#### Authentication Flow - -``` -Step 1: Call b2_authorize_account - POST /b2api/v4/b2_authorize_account - Header: Authorization: Basic - -Step 2: Response contains: - - authorizationToken (valid 24 hours) - - apiUrl (cluster-specific API endpoint) - - downloadUrl (public download endpoint) - - accountId - -Step 3: Use authorizationToken for subsequent API calls - Header: Authorization: Bearer - -Step 4: Token expires after 24 hours - - Refresh by calling b2_authorize_account again - - No refresh token needed (stateless) -``` - -**Security Considerations:** -- All communication must use HTTPS (TLS 1.2+) -- Token expires after 24 hours (auto-refresh in practice) -- No refresh token mechanism (re-authorize as needed) -- Each key has independent scopes and restrictions - -**Citation:** if://auth/b2-application-keys-v4-2025-11-14 - -### Core API Endpoints - -#### Bucket Management Endpoints - -| Operation | HTTP Method | Endpoint | Purpose | -|-----------|------------|----------|---------| -| **Authorize Account** | POST | `/b2api/v4/b2_authorize_account` | Authenticate and get API URL + token | -| **List Buckets** | POST | `/b2api/v4/b2_list_buckets` | Enumerate all buckets in account | -| **Create Bucket** | POST | `/b2api/v4/b2_create_bucket` | Create new bucket (private or public) | -| **Get Bucket Info** | POST | `/b2api/v4/b2_get_bucket_info` | Retrieve bucket configuration | -| **Update Bucket** | POST | `/b2api/v4/b2_update_bucket` | Modify lifecycle rules, CORS, encryption | -| **Delete Bucket** | POST | `/b2api/v4/b2_delete_bucket` | Remove bucket (must be empty) | - -#### File/Object Operations - -| Operation | HTTP Method | Endpoint | Purpose | -|-----------|------------|----------|---------| -| **Get Upload URL** | POST | `/b2api/v4/b2_get_upload_url` | Obtain URL for file upload | -| **Upload File** | PUT | `{uploadUrl}` | Upload file to bucket | -| **List File Names** | POST | `/b2api/v4/b2_list_file_names` | List current file versions | -| **List File Versions** | POST | `/b2api/v4/b2_list_file_versions` | List all versions (including hidden) | -| **Get File Info** | POST | `/b2api/v4/b2_get_file_info` | Retrieve file metadata by ID | -| **Delete File Version** | POST | `/b2api/v4/b2_delete_file_version` | Permanently remove file version | -| **Hide File** | POST | `/b2api/v4/b2_hide_file` | Mark file as deleted (creates hidden version) | - -#### Large File Operations (Multipart Upload) - -| Operation | Endpoint | Purpose | -|-----------|----------|---------| -| **Start Large File** | `/b2api/v4/b2_start_large_file` | Initiate multipart upload | -| **Get Upload Part URL** | `/b2api/v4/b2_get_upload_part_url` | Get URL for uploading part | -| **Upload Part** | `{uploadUrl}` | Upload file chunk (1-10,000 parts) | -| **List Parts** | `/b2api/v4/b2_list_parts` | List uploaded parts for multipart | -| **Finish Large File** | `/b2api/v4/b2_finish_large_file` | Complete multipart upload | -| **Cancel Large File** | `/b2api/v4/b2_cancel_large_file` | Abort multipart upload | - -#### Advanced Features - -| Feature | Endpoint | Purpose | -|---------|----------|---------| -| **Lifecycle Rules** | `/b2api/v4/b2_create_lifecycle_rule` | Auto-hide/delete old versions | -| **Object Lock** | Bucket setting | Immutable files (WORM compliance) | -| **Cloud Replication** | `/b2api/v4/b2_create_replication_rule` | Copy to other buckets | -| **Key Management** | `/b2api/v4/b2_list_keys` | List and manage application keys | - -**Citation:** if://api/b2-endpoints-v4-2025-11-14 - -### Key Technical Distinctions - -#### B2 Native vs S3 API Comparison - -**B2 Native Advantages:** -1. **Strong Consistency** - - Updates visible immediately - - No eventual consistency surprises - - Critical for transactional workloads - -2. **Simple Request Format** - - JSON request bodies - - No AWS Signature Version 4 complexity - - Basic authentication straightforward - -3. **Upload URL Pattern** - - Each upload requires fresh URL - - Prevents stale URL issues - - Controlled token lifetime - -4. **Direct Key Management** - - Programmatic key creation/revocation - - Scoped access tokens - - Better for multi-tenant scenarios - -**When to Use Each:** -- **Native API:** Cost optimization, strong consistency, key management -- **S3-Compatible API:** Existing AWS SDK code, ecosystem compatibility - -**Citation:** if://analysis/b2-native-vs-s3-2025-11-14 - -### Pricing Model (2025) - -**Storage:** -- $0.005/GB/month -- ~$5 per TB per year -- Single-zone (no redundancy charge) -- Multi-zone replication optional (2x cost) - -**Egress (Data Transfer Out):** -- $0.01/GB -- 1GB/day free tier -- Egress to Cloudflare: FREE (special partnership) - -**API Requests:** -- **Class A (write operations):** $0.0004 per 10,000 requests -- **Class B (read operations):** $0.006 per 1,000 requests -- **Example:** 1M read requests = $6 - -**Free Tier:** -- 10GB storage per month -- 1GB egress per day -- Enough for small projects - -**Cost Advantage:** -- ~3x cheaper than Cloudflare R2 ($0.015/GB) -- Comparable to Storj ($0.004/GB) but with egress fees -- Clear cost advantage when egress ≤ 1TB/month - -**Citation:** if://pricing/b2-cost-2025-11-14 - -### SDKs & Client Libraries - -| Language | Library | Type | Status | Repository | -|----------|---------|------|--------|------------| -| **Python** | `b2sdk` | Official | ✅ Active | PyPI: `pip install b2sdk` | -| **Java** | `b2-sdk-java` | Official | ✅ Active | Maven Central | -| **Go** | `b2` (Blazer) | Community | ✅ Maintained | github.com/kurin/blazer | -| **Node.js** | `backblaze-b2` | Community | ⚠️ Limited | npm: work-in-progress | -| **Ruby** | Community wrappers | Community | ⚠️ Multiple | Various gems available | -| **CLI** | `b2` (Go-based) | Official | ✅ Maintained | github.com/Backblaze/B2_Command_Line_Tool | -| **S3-Compatible SDKs** | boto3, AWS SDK v3, etc. | Standard | ✅ Any language | S3-compatible gateway | - -**Recommended for InfraFabric:** -- **Python:** `b2sdk` - Most mature, feature-complete -- **Go:** `blazer` - Good community support -- **S3 Gateway:** Use AWS SDKs for existing code - -**Citation:** if://sdk/b2-libraries-2025-11-14 - -### InfraFabric Integration Assessment - -**Complexity Score: 5/10** - -**Advantages:** -- ✅ Clear, JSON-based API design -- ✅ Straightforward authentication (Basic auth vs AWS Signature complexity) -- ✅ Strong consistency eliminates retry logic complexity -- ✅ Application key scoping enables fine-grained access control -- ✅ Excellent for versioned data (file revisions, audit trails) -- ✅ Cost-effective for large-scale document storage - -**Gaps & Limitations:** -- ⚠️ No direct streaming (must use multipart for large files) -- ⚠️ Egress fees apply (unlike Cloudflare R2) -- ⚠️ No built-in CDN (use Cloudflare in front for free egress) -- ⚠️ API response pagination requires cursor handling -- ⚠️ No native webhooks (would need polling) - -**Ideal Use Cases:** -- Document versioning and archival -- Backup and disaster recovery -- Long-term data retention -- Cost-sensitive deployments -- Multi-tenant document management - ---- - -## Cloudflare R2 API - -> **Citation Base:** if://api/r2-api-2025-11-14 -> **Maturity:** Production-Ready (General Availability) -> **InfraFabric Complexity Rating:** 4/10 - -### API Overview - -**Service:** Cloudflare R2 Object Storage -**API Options:** Workers Binding (native) + S3-Compatible REST API -**Base URL (S3):** `https://{ACCOUNT_ID}.r2.cloudflarestorage.com` -**Workers Binding:** Direct JavaScript/TypeScript in Cloudflare Workers -**Protocol:** HTTPS (TLS 1.2+) -**Response Format:** JSON (native), S3 XML compatibility - -### Authentication & Authorization - -#### Option 1: Workers Binding (Recommended for Cloudflare-Native) - -```javascript -// In wrangler.toml -[[r2.bindings]] -binding = "BUCKET_NAME" -bucket_name = "my-bucket" - -// In Worker code -export default { - async fetch(request, env) { - const bucket = env.BUCKET_NAME; - - // No credentials needed - automatic auth - const obj = await bucket.get('file-key'); - return new Response(obj.body); - } -} -``` - -**Advantages:** -- Zero credential management -- Automatic token rotation -- No CORS issues within Workers -- Lowest latency (edge execution) - -**Scope:** Limited to Cloudflare Workers environment - -**Citation:** if://auth/r2-workers-binding-2025-11-14 - -#### Option 2: S3-Compatible API (For External Applications) - -```bash -# Generate S3 credentials in Cloudflare Dashboard -Access Key ID: (provided once) -Secret Access Key: (provided once - save securely!) - -# Configure boto3 -s3_client = boto3.client( - 's3', - endpoint_url='https://{account_id}.r2.cloudflarestorage.com', - aws_access_key_id='YOUR_ACCESS_KEY', - aws_secret_access_key='YOUR_SECRET_KEY', - region_name='us-east-1' # REQUIRED - always use us-east-1 -) -``` - -**Key Requirements:** -- **Region:** Always use `us-east-1` (required by S3 SDKs even though R2 is global) -- **Auth Method:** AWS Signature Version 4 (handled by SDK) -- **Endpoint:** Account-specific URL - -**Advantages:** -- Use existing AWS SDK code -- Works outside Cloudflare Workers -- Compatible with any S3 client - -**Limitations:** -- Requires credential rotation -- Different interface than Workers API - -**Citation:** if://auth/r2-s3-credentials-2025-11-14 - -### Core API Endpoints - -#### Workers API (Native R2 - JavaScript/TypeScript) - -```javascript -// Get object -const obj = await bucket.get(key); -const body = await obj.text(); - -// Put object -await bucket.put(key, body, { - httpMetadata: { contentType: 'application/pdf' }, - customMetadata: { userId: '123' } -}); - -// Delete object -await bucket.delete(key); - -// List objects -const list = await bucket.list({ prefix: 'docs/', limit: 100 }); - -// Head object (metadata only) -const obj = await bucket.head(key); - -// Multipart upload -const multipart = await bucket.createMultipartUpload(key); -await multipart.uploadPart(1, body1); -await multipart.uploadPart(2, body2); -await multipart.complete([...parts]); -``` - -#### S3-Compatible API (Standard REST) - -Uses standard S3 operations via AWS SDK: -- `ListBuckets`, `ListObjects`, `ListObjectsV2` -- `PutObject`, `GetObject`, `DeleteObject` -- `HeadObject`, `CopyObject` -- `CreateMultipartUpload`, `UploadPart`, `CompleteMultipartUpload` - -**Multipart Upload Benefits:** -- Can be called via S3 API or Workers API -- Both APIs can read/modify same multipart uploads -- Global consistency (Cloudflare edge network) - -**Citation:** if://api/r2-endpoints-2025-11-14 - -### Unique Features - -#### 1. Zero Egress Fees (Major Advantage) - -``` -Egress Cost Comparison: -- Backblaze B2: $0.01/GB ($10/TB) -- Storj DCS: $0.007/GB ($7/TB) -- Cloudflare R2: $0.00/GB (FREE!) - -Scenario: 10TB/month egress -- B2 cost: $100/month -- R2 cost: $0/month -- Savings: $100/month ($1,200/year) -``` - -**Applies to all egress:** -- Data transferred to internet -- Data transferred to Cloudflare Workers -- Cross-bucket transfers -- Cloudflare CDN integration - -**Citation:** if://features/r2-zero-egress-2025-11-14 - -#### 2. Cloudflare Workers Integration - -```javascript -// Edge-native storage access -export default { - async fetch(request, env) { - // Process at edge, serve from R2 - const imageFile = await env.IMAGES.get('photo.jpg'); - - // Transform with Workers - if (request.headers.get('Accept').includes('webp')) { - // Resize/convert image at edge - return new Response(optimized, { - headers: { 'Content-Type': 'image/webp' } - }); - } - - return new Response(imageFile.body); - } -} -``` - -**Use Cases:** -- Image optimization at edge -- Dynamic content generation -- API gateway with storage backend -- Real-time data processing - -**Citation:** if://features/r2-workers-integration-2025-11-14 - -#### 3. Location Hints - -- Optional geographic hint during bucket creation -- Suggests data residency (not guaranteed) -- Available in: US, EU, APAC -- Useful for compliance/latency requirements - -#### 4. CORS Support - -- Configure per-bucket CORS policies -- Enable browser-based S3 access -- Useful for web applications - -#### 5. Lifecycle Management - -- Object expiration rules -- Automatic deletion after X days -- Useful for temporary uploads, logs - -**Citation:** if://features/r2-capabilities-2025-11-14 - -### Pricing Model (2025) - -**Storage:** -- $0.015/GB/month (~$15/TB/year) -- 10GB/month free tier -- Billed monthly - -**Egress:** -- **$0.00/GB** (Zero egress - this is the major advantage!) - -**API Requests:** -- **Class A (write):** $0.36 per 1,000,000 requests -- **Class B (read/list):** $0.36 per 10,000,000 requests -- 1 million Class A requests/month free tier - -**Cost Comparison:** - -``` -Scenario A: 100GB stored, 10GB egress/month -- B2: (100GB × $0.005) + (10GB × $0.01) = $0.60/mo -- R2: (100GB × $0.015) + (0GB × $0) = $1.50/mo -- Winner: B2 (3x cheaper for storage) - -Scenario B: 100GB stored, 100GB egress/month -- B2: (100GB × $0.005) + (100GB × $0.01) = $1.50/mo -- R2: (100GB × $0.015) + (0GB × $0) = $1.50/mo -- Winner: TIE - -Scenario C: 100GB stored, 500GB egress/month -- B2: (100GB × $0.005) + (500GB × $0.01) = $5.50/mo -- R2: (100GB × $0.015) + (0GB × $0) = $1.50/mo -- Winner: R2 (3.7x cheaper with high bandwidth) -``` - -**Break-even:** R2 superior when egress > ~1-2TB/month - -**Citation:** if://pricing/r2-cost-2025-11-14 - -### SDKs & Client Libraries - -| Language | SDK | Notes | -|----------|-----|-------| -| **Node.js/JavaScript** | `@aws-sdk/client-s3` v3 | Full S3 compatibility (⚠️ Version compatibility issues exist) | -| **Python** | `boto3` | Standard AWS S3 SDK | -| **Go** | `aws-sdk-go-v2` | Standard AWS SDK | -| **Rust** | `aws-sdk-s3` | Standard AWS SDK | -| **Python Workers** | `pywrangler` | Cloudflare Workers Python (beta) | -| **Wrangler CLI** | Official | Deploy Workers with R2 bindings | - -**For Workers (JavaScript/TypeScript):** -- Native R2 API (built-in) -- No npm package needed -- Available in Worker environment - -**Compatibility Note:** -```javascript -// AWS SDK v3.729.0+ has compatibility issues with R2 -// Workaround: -const s3Client = new S3Client({ - // ... credentials - // requestChecksumCalculation: 'WHEN_REQUIRED' // Add this -}); -``` - -**Citation:** if://sdk/r2-sdk-support-2025-11-14 - -### InfraFabric Integration Assessment - -**Complexity Score: 4/10** - -**Advantages:** -- ✅ Excellent SDK compatibility (uses AWS SDK ecosystem) -- ✅ Zero egress eliminates bandwidth cost calculation -- ✅ Workers API straightforward for Cloudflare-native apps -- ✅ S3 compatibility means existing code "just works" -- ✅ Geo-redundancy built-in (via Cloudflare CDN) -- ✅ Simple pricing model (storage + requests, no egress) - -**Gaps & Limitations:** -- ⚠️ Storage pricing 3x higher than B2 ($0.015 vs $0.005/GB) -- ⚠️ Workers API differs from S3 (not all features available) -- ⚠️ Limited to Cloudflare data center footprint -- ⚠️ Location hints not guaranteed (compliance concerns) -- ⚠️ SDK version compatibility issues (occasional) - -**Ideal Use Cases:** -- Bandwidth-heavy applications -- Edge computing with R2 backend -- CDN-integrated content distribution -- Low-latency global access -- Streaming/video delivery - ---- - -## Storj Decentralized Object Storage - -> **Citation Base:** if://api/storj-dcs-2025-11-14 -> **Maturity:** Production-Ready (Stable) -> **InfraFabric Complexity Rating:** 7/10 - -### API Overview - -**Service:** Storj DCS (Decentralized Cloud Storage) -**Primary SDK:** `storj.io/uplink` (Go language) -**API Type:** Native Uplink SDK + S3-Compatible Gateway -**Gateway Endpoints:** -- Storj-hosted: `https://gateway.storjshare.io` -- Self-hosted: custom endpoint -**Architecture:** Distributed across 1,280+ storage nodes globally -**Encryption:** End-to-end, user-controlled keys - -### Architecture: Decentralized Model - -#### How It Works - -``` -File Upload Process: -1. File arrives at client (256MB) -2. Split into 16 segments (16MB each) -3. Each segment split into 80+ pieces -4. Encrypt each piece (user's key) -5. Distribute to 1,280 independent nodes -6. Redundancy: file survives up to 29 node failures - -Download Process: -1. Client initiates download -2. Retrieve pieces from fastest available nodes -3. Decrypt with user's key -4. Reassemble segments -5. Return complete file -``` - -#### Key Characteristics - -**Advantages:** -- **No Single Point of Failure:** Distributed architecture -- **Privacy by Default:** End-to-end encryption (Storj cannot decrypt) -- **High Durability:** 11 nines (99.99999999%) due to redundancy -- **Censorship Resistant:** No central authority can block/remove data -- **Geographic Redundancy:** Automatic worldwide distribution -- **Incentivized Network:** Storage nodes earn STORJ token rewards - -**Trade-offs:** -- **Per-Segment Fees:** Small files incur overhead ($0.0000088/64MB segment) -- **Not Ideal for Many Small Files:** Better for larger objects -- **Community-Run Network:** Less SLA guarantees than corporations -- **Operational Complexity:** Understanding distributed architecture - -**Citation:** if://architecture/storj-dcs-distributed-2025-11-14 - -### Authentication & Authorization - -#### Access Grants (Primary Auth Method) - -``` -Access Grant Structure: -access_grant = satellite + restricted_api_key + encryption_key - -Components: -1. Satellite: Registration/auth server (e.g., us1.storj.io:7777) -2. API Key: Macaroon-based hierarchical token -3. Encryption Key: User-derived, Storj cannot access -``` - -**Credential Hierarchy:** - -``` -1. Project API Key (Master) - ├── Create restricted grants from this - └── Keep secure - never embed in apps - -2. Restricted Access Grant - ├── Scoped to bucket/prefix - ├── Scoped to operations (read/write/delete) - ├── Scoped to time window (optional) - └── Delegatable - share without sharing master key - -3. Share Link Token - ├── Lightweight temporary access - └── For public/semi-public sharing -``` - -**Key Features:** -- **Macaroon-Based:** Hierarchical tokens (restrict by time, bucket, operation) -- **Delegatable:** Create restricted grants without sharing master key -- **Offline-Capable:** Generate grants without network (once initialized) -- **Time-Scoped:** Tokens can expire automatically - -#### Example: Uplink CLI - -```bash -# Create access grant from project API key -uplink access create \ - --api-key \ - --satellite us1.storj.io:7777 \ - my-access - -# Create restricted grant (read-only to specific bucket) -uplink access restrict \ - my-access \ - --readonly \ - --bucket my-bucket - -# Create time-limited grant (expires in 7 days) -uplink access restrict \ - my-access \ - --not-after=+7d \ - --bucket my-bucket - -# Export for use in code -uplink access export my-access -``` - -**Citation:** if://auth/storj-access-grants-2025-11-14 - -### Core API Endpoints - -#### Native Uplink API (Go Primary, Bindings Available) - -```go -// Go examples (primary language) -import "storj.io/uplink" - -// Open project connection -project, err := uplink.OpenProject(ctx, access) - -// List buckets -buckets := project.ListBuckets(ctx, nil) - -// Create bucket -bucket, err := project.CreateBucket(ctx, "my-bucket") - -// Open bucket -bucket, err := project.OpenBucket(ctx, "my-bucket") - -// List objects -objects := bucket.ListObjects(ctx, &uplink.ListObjectsOptions{ - Prefix: "documents/", -}) - -// Upload object -writer, err := bucket.OpenWriter(ctx, "file.pdf") -io.Copy(writer, fileContent) - -// Download object -reader, err := bucket.OpenReader(ctx, "file.pdf") -content, err := io.ReadAll(reader) - -// Delete object -err := bucket.DeleteObject(ctx, "file.pdf") - -// Object metadata -obj, err := bucket.StatObject(ctx, "file.pdf") -fmt.Println(obj.ContentLength) -``` - -#### Available Language Bindings - -| Language | Type | Status | -|----------|------|--------| -| **Go** | Native library | ✅ Official, mature | -| **Node.js** | C++ bindings | ✅ Community, stable | -| **Python** | C++ bindings | ✅ Community, stable | -| **Java** | Bindings | ⚠️ Community, beta | -| **Swift** | Bindings | ✅ Community, mobile | -| **PHP** | Bindings | ✅ Community | -| **S3 Gateway** | REST API | ✅ Any language via S3 | - -#### S3-Compatible Gateway (REST) - -```bash -# Using AWS CLI pointed at Storj gateway -aws --endpoint-url https://gateway.storjshare.io \ - --access-key-id \ - --secret-access-key \ - s3 ls s3://my-bucket - -# Using boto3 (Python) -s3 = boto3.client( - 's3', - endpoint_url='https://gateway.storjshare.io', - aws_access_key_id='...', - aws_secret_access_key='...' -) -s3.list_objects_v2(Bucket='my-bucket') -``` - -**Gateway Acts As:** -- Proxy layer between S3 API and Uplink SDK -- Translates S3 requests to Uplink operations -- Maintains S3 compatibility - -**Citation:** if://api/storj-uplink-endpoints-2025-11-14 - -### Unique Features - -#### 1. Distributed Architecture - -- Files split across 1,280+ nodes automatically -- No single failure point -- Transparent to application -- Automatic repair of lost pieces - -#### 2. End-to-End Encryption (Default) - -``` -Key Management: -- Encryption key never sent to Storj -- User derives key from password/secret -- Storj stores encrypted data only -- Decryption happens client-side -- Complies with GDPR (no user data access) -``` - -#### 3. Pay-Per-Use Node Network - -- Storage nodes incentivized via STORJ token -- Crowdsourced capacity (cheaper than data centers) -- Community-run vs corporate-run -- Environmental benefit (uses excess capacity) - -#### 4. Object Lock Support - -- WORM (Write-Once-Read-Many) compliance -- Immutable files for regulatory requirements -- Prevents accidental/malicious deletion - -#### 5. Bandwidth Optimization - -- Download from geographically closest nodes -- Parallelized retrieval (faster for large files) -- Reduced latency via distributed nodes - -**Citation:** if://features/storj-unique-capabilities-2025-11-14 - -### Pricing Model (2025) - -**Storage:** -- $4.00/TB/month ($0.004/GB) -- Lowest storage cost among three providers -- Billed per GB-month - -**Egress (Data Transfer Out):** -- $7.00/TB ($0.007/GB) -- Moderate compared to B2 ($0.01), cheaper than S3 - -**Per-Segment Fee:** -- $0.0000088 per segment (64MB or portion thereof) -- Small files incur extra fees -- Example: 1,000 small 1MB files = $0.0088 extra - -**Free Tier:** -- None (pay-as-you-go only) -- No free storage or egress - -**Durability:** -- 11 nines (99.99999999%) -- Redundancy built-in (no extra cost) - -#### Cost Scenarios - -``` -Scenario 1: 1TB stored, 100GB egress (large files) -- Storage: (1,000GB × $0.004) = $4.00 -- Egress: (100GB × $0.007) = $0.70 -- Segments: (1,000 ÷ 64) × $0.0000088 ≈ $0.14 -- Total: ~$4.84/month - -Scenario 2: 1TB stored, 100GB egress (many small 1MB files) -- Storage: (1,000GB × $0.004) = $4.00 -- Egress: (100GB × $0.007) = $0.70 -- Segments: (1,000,000 files) × $0.0000088 = $8.80 -- Total: ~$13.50/month (per-segment fees dominate!) - -Scenario 3: 1TB stored, 1TB egress (large files) -- Storage: (1,000GB × $0.004) = $4.00 -- Egress: (1,000GB × $0.007) = $7.00 -- Segments: ~$0.14 -- Total: ~$11.14/month -``` - -**Break-even Analysis:** -- **Best:** Large files (>10MB), moderate bandwidth -- **Avoid:** Many small files (<1MB), high-frequency access - -**Citation:** if://pricing/storj-cost-2025-11-14 - -### SDKs & Client Libraries - -| Language | Library | Type | Maturity | -|----------|---------|------|----------| -| **Go** | `storj.io/uplink` | Official, native | ✅ Stable, feature-complete | -| **Node.js** | `uplink-nodejs` | Community bindings | ✅ Stable, actively maintained | -| **Python** | `uplink-python` | Community bindings | ✅ Stable, actively maintained | -| **Java** | `uplink-java` | Community bindings | ⚠️ Beta, limited documentation | -| **Swift** | `uplink-swift` | Community bindings | ✅ Mobile support | -| **PHP** | `uplink-php` | Community bindings | ✅ Available | -| **S3 Gateway** | boto3, AWS SDK | Standard SDKs | ✅ Any language | - -**Recommended for InfraFabric:** -- **Go:** Best support, production-ready -- **Node.js/Python:** Good community bindings, well-maintained -- **S3 Gateway:** For existing AWS SDK code - -**CLI Tool (Official):** - -```bash -# Uplink CLI for administration -uplink access create # Create access grants -uplink access restrict # Scope access -uplink bucket create # Create buckets -uplink bucket list # List buckets -uplink upload # Upload files -uplink download # Download files -uplink ls # List objects -uplink rm # Delete objects -``` - -**Citation:** if://sdk/storj-sdk-support-2025-11-14 - -### InfraFabric Integration Assessment - -**Complexity Score: 7/10** - -**Advantages:** -- ✅ Privacy-first (end-to-end encryption by default) -- ✅ Lowest storage cost ($4/TB vs $5 B2, $15 R2) -- ✅ No vendor lock-in (open source, decentralized) -- ✅ Censorship-resistant (distributed architecture) -- ✅ Best durability (11 nines, embedded redundancy) -- ✅ Incentivized network (environmental benefit) - -**Gaps & Limitations:** -- ⚠️ Per-segment fees problematic for many small files -- ⚠️ Community-run network (less SLA guarantees) -- ⚠️ Bandwidth cost higher than R2 zero-egress -- ⚠️ Less mature SDKs (especially Java) -- ⚠️ Smaller ecosystem vs AWS S3 -- ⚠️ Operational complexity (distributed architecture) -- ⚠️ No native SLA guarantees (community-run) - -**Ideal Use Cases:** -- Privacy-critical applications (healthcare, legal) -- Long-term archival (immutable data) -- Distributed deployments -- Compliance-heavy workloads (GDPR, HIPAA) -- Open-source projects (cost-sensitive) - ---- - -## Feature Comparison Matrix - -### Core Capabilities - -| Feature | Backblaze B2 | Cloudflare R2 | Storj DCS | -|---------|--------------|---------------|-----------| -| **Storage Cost/TB/mo** | $5 | $15 | $4 | -| **Egress Cost/TB** | $10 | $0 | $7 | -| **Per-Item Fees** | None | None | $0.0000088/segment | -| **Native API Type** | REST JSON | Workers + S3 | Uplink SDK | -| **Strong Consistency** | ✅ Yes | ⚠️ Eventual | ✅ Yes | -| **End-to-End Encryption** | ❌ No | ❌ No | ✅ Yes (default) | -| **Built-in Redundancy** | Multi-zone (optional) | Global CDN | 1,280+ nodes | -| **SLA Uptime** | 99.9% | 99.95% | Community-run | - -### Data Management Features - -| Feature | Backblaze B2 | Cloudflare R2 | Storj DCS | -|---------|--------------|---------------|-----------| -| **Versioning** | ✅ Native | ✅ Native | ✅ Via gateway | -| **Object Lock/WORM** | ✅ Yes | ⚠️ Limited | ✅ Yes | -| **Lifecycle Rules** | ✅ Yes | ✅ Yes | ⚠️ Manual | -| **Replication** | ✅ Native API | ✅ Cross-region | ⚠️ Manual | -| **CORS Support** | ✅ Yes | ✅ Yes | ✅ Gateway | -| **Custom Metadata** | ✅ Yes | ✅ Yes | ✅ Yes | - -### Developer Experience - -| Feature | Backblaze B2 | Cloudflare R2 | Storj DCS | -|---------|--------------|---------------|-----------| -| **Official SDKs** | Go, Python, Java | TypeScript, Python, Go | Go (native) | -| **Community SDKs** | Ruby, Node.js | AWS SDK (universal) | Node.js, Python, Java, Swift, PHP | -| **CLI Tool** | ✅ Official | ✅ wrangler | ✅ uplink | -| **Documentation Quality** | ⭐⭐⭐⭐ Good | ⭐⭐⭐⭐⭐ Excellent | ⭐⭐⭐ Good | -| **Terraform Support** | ⚠️ Limited | ✅ Yes | ⚠️ Limited | -| **Example Code** | ✅ Available | ✅✅ Extensive | ✅ Available | - -### Performance Characteristics - -| Metric | Backblaze B2 | Cloudflare R2 | Storj DCS | -|--------|--------------|---------------|-----------| -| **Upload Speed** | Standard | Edge-optimized | Distributed (variable) | -| **Download Speed** | Standard | Edge-optimized | Fastest node first | -| **Latency Regions** | Multiple data centers | Global edge (Cloudflare) | 1,280+ nodes worldwide | -| **Multipart Upload** | ✅ Supported | ✅ Supported | ✅ SDK support | -| **Large File Support** | ✅ 5GB per part | ✅ 5GB per part | ✅ Streaming | - -**Citation:** if://comparison/object-storage-matrix-2025-11-14 - ---- - -## Integration Assessment - -### Complexity Ratings Breakdown - -**Backblaze B2: 5/10** -- ✅ Simple JSON API -- ✅ Basic authentication -- ✅ Strong consistency advantage -- ⚠️ Application key scoping learning curve -- ⚠️ Upload URL refresh pattern - -**Cloudflare R2: 4/10** -- ✅ S3 SDK compatibility -- ✅ Workers binding simplicity -- ✅ Zero egress pricing simplicity -- ⚠️ Two different API surfaces (Workers vs S3) -- ⚠️ Region parameter always "us-east-1" - -**Storj DCS: 7/10** -- ⚠️ Distributed architecture complexity -- ⚠️ Access grant system (not simple API key) -- ⚠️ Per-segment fee calculation -- ⚠️ Community SDK maturity varies -- ⚠️ Macaroon-based auth not familiar to most -- ✅ Excellent privacy model -- ✅ Good documentation - -### Recommendation Matrix for InfraFabric - -| InfraFabric Use Case | Recommended | Rationale | -|---|---|---| -| **Document Archival (NaviDocs)** | ⭐⭐⭐ B2 | Cost-effective, strong consistency, excellent versioning | -| **CDN-Integrated Content** | ⭐⭐⭐ R2 | Zero egress, Workers integration, geo-redundancy | -| **Compliance/Privacy** | ⭐⭐⭐ Storj | End-to-end encryption, no vendor lock-in, GDPR-friendly | -| **Multi-Region Replication** | ⭐⭐⭐ B2 | Native replication rules, strong consistency | -| **Real-Time Analytics** | ⭐⭐⭐ R2 | Edge processing, low latency, zero cost to read | -| **Cost Optimization** | ⭐⭐⭐ Storj | Lowest storage, monitor segment fees | -| **Hybrid Multi-Provider** | ⭐⭐ All | Abstraction layer recommended | - -**Primary Recommendation: Backblaze B2** -- Optimal balance for NaviDocs document storage -- Cost-effective at scale -- Simple API reduces integration effort -- Strong consistency valuable for document management - -**Citation:** if://recommendation/if-provider-selection-2025-11-14 - -### Implementation Timeline Estimates - -#### Option A: Backblaze B2 Integration (Recommended) -- **Phase 1:** Basic CRUD operations - **1 day** -- **Phase 2:** Async action handling (multipart) - **2-3 days** -- **Phase 3:** Versioning and lifecycle - **2 days** -- **Phase 4:** Cost tracking - **1-2 days** -- **Total:** 6-8 days for production-ready integration - -#### Option B: Cloudflare R2 Integration -- **Phase 1:** S3 SDK setup - **0.5 days** -- **Phase 2:** Workers binding integration - **1 day** -- **Phase 3:** Zero-egress optimization - **1 day** -- **Phase 4:** Monitoring/cost tracking - **1 day** -- **Total:** 3.5-4 days for production-ready integration - -#### Option C: Storj DCS Integration -- **Phase 1:** Access grant setup - **1 day** -- **Phase 2:** Uplink SDK integration - **2 days** -- **Phase 3:** Per-segment fee tracking - **2 days** -- **Phase 4:** Compliance/audit logging - **2 days** -- **Total:** 7-8 days for production-ready integration - ---- - -## Pricing Analysis - -### Total Cost of Ownership (TCO) Scenarios - -#### Scenario 1: Document Backup (100GB, 50GB egress/month) - -``` -Backblaze B2: -- Storage: 100GB × $0.005/GB = $0.50/month -- Egress: 50GB × $0.01/GB = $0.50/month -- API: ~$0.01/month -- Total: $1.01/month ($12.12/year) - -Cloudflare R2: -- Storage: 100GB × $0.015/GB = $1.50/month -- Egress: 0GB × $0.00/GB = $0/month -- API: ~$0.05/month -- Total: $1.55/month ($18.60/year) - -Storj DCS: -- Storage: 100GB × $0.004/GB = $0.40/month -- Egress: 50GB × $0.007/GB = $0.35/month -- Segments: ~0.07/month -- Total: $0.82/month ($9.84/year) - -WINNER: Storj (32% cheaper than B2) -``` - -#### Scenario 2: High-Bandwidth Content Distribution (500GB, 500GB egress/month) - -``` -Backblaze B2: -- Storage: 500GB × $0.005/GB = $2.50/month -- Egress: 500GB × $0.01/GB = $5.00/month -- API: ~$0.10/month -- Total: $7.60/month ($91.20/year) - -Cloudflare R2: -- Storage: 500GB × $0.015/GB = $7.50/month -- Egress: 0GB × $0.00/GB = $0/month -- API: ~$0.20/month -- Total: $7.70/month ($92.40/year) - -Storj DCS: -- Storage: 500GB × $0.004/GB = $2.00/month -- Egress: 500GB × $0.007/GB = $3.50/month -- Segments: ~0.35/month -- Total: $5.85/month ($70.20/year) - -WINNER: Storj (23% cheaper than R2, 37% cheaper than B2) -BUT: R2 very competitive if egress continues -``` - -#### Scenario 3: Enterprise Document Management (1TB, 100GB egress/month) - -``` -Backblaze B2: -- Storage: 1000GB × $0.005/GB = $5.00/month -- Egress: 100GB × $0.01/GB = $1.00/month -- API: ~0.20/month (higher request volume) -- Total: $6.20/month ($74.40/year) - -Cloudflare R2: -- Storage: 1000GB × $0.015/GB = $15.00/month -- Egress: 0GB × $0.00/GB = $0/month -- API: ~$0.50/month -- Total: $15.50/month ($186/year) - -Storj DCS: -- Storage: 1000GB × $0.004/GB = $4.00/month -- Egress: 100GB × $0.007/GB = $0.70/month -- Segments: ~0.70/month -- Total: $5.40/month ($64.80/year) - -WINNER: Storj (13% cheaper than B2, 65% cheaper than R2) -``` - -### Long-Term Cost Trends - -**Backblaze B2:** -- Storage: ~$5/TB/year (stable) -- Egress: $120/TB/year (major cost driver) -- Good for: Low-bandwidth, high-storage workloads - -**Cloudflare R2:** -- Storage: ~$180/TB/year (highest) -- Egress: $0/TB/year (zero!) -- Good for: High-bandwidth, CDN-integrated workloads - -**Storj DCS:** -- Storage: ~$48/TB/year (lowest) -- Egress: ~$84/TB/year (moderate) -- Good for: Privacy-first, cost-optimized workloads - -**Citation:** if://pricing/tco-analysis-2025-11-14 - ---- - -## Research Citations - -### IF.TTT Compliance (Traceable, Transparent, Trustworthy) - -#### Primary Sources (Authoritative) - -| Provider | Primary Source | Type | Verification Date | -|----------|---|---|---| -| **Backblaze B2** | https://www.backblaze.com/apidocs/ | Official API Docs | 2025-11-14 | -| **Backblaze B2** | https://www.backblaze.com/docs/ | Official Documentation | 2025-11-14 | -| **Cloudflare R2** | https://developers.cloudflare.com/r2/ | Official API Docs | 2025-11-14 | -| **Cloudflare R2** | https://developers.cloudflare.com/workers/runtime-apis/web-crypto/ | Workers API | 2025-11-14 | -| **Storj DCS** | https://storj.dev/dcs/api/ | Official SDK Docs | 2025-11-14 | -| **Storj DCS** | https://github.com/storj/uplink | GitHub Repository | 2025-11-14 | - -#### Secondary Sources (Community/Analysis) - -| Provider | Source | Type | Access Date | -|---|---|---|---| -| **B2 Native vs S3** | https://www.backblaze.com/blog/whats-the-diff-backblaze-s3-compatible-api-vs-b2-native-api/ | Official Blog | 2025-11-14 | -| **R2 Zero Egress** | https://blog.cloudflare.com/r2-ga/ | Product Launch | 2025-11-14 | -| **Storj Access Grants** | https://docs.storj.io/learn/concepts/access/access-grants/ | Official Docs | 2025-11-14 | -| **Storj Pricing** | https://www.storj.io/pricing | Official Pricing | 2025-11-14 | -| **Comparison Analysis** | Cloud storage comparison research (WebSearch) | Market Research | 2025-11-14 | - -#### Verification Checklist - -- [x] Minimum 2 independent sources per claim -- [x] Current as of November 2025 -- [x] Official documentation reviewed -- [x] Pricing data current and verified -- [x] API endpoints documented from official sources -- [x] SDK availability verified -- [x] Feature comparisons cross-referenced - -#### Evidence Summary - -**Research Quality:** Medical-grade (95%+ confidence) -**Total Sources Reviewed:** 30+ -**API Endpoints Analyzed:** 25+ -**SDK Compatibility Verified:** All primary languages -**Confidence Level:** High - official sources + community validation - -### Citation IDs (IF.TTT Format) - -``` -Primary Research: -- if://research/object-storage-native-apis-2025-11-14 -- if://api/b2-native-v4-2025-11-14 -- if://api/r2-api-2025-11-14 -- if://api/storj-dcs-2025-11-14 - -Authentication: -- if://auth/b2-application-keys-v4-2025-11-14 -- if://auth/r2-workers-binding-2025-11-14 -- if://auth/r2-s3-credentials-2025-11-14 -- if://auth/storj-access-grants-2025-11-14 - -Features & Architecture: -- if://features/r2-zero-egress-2025-11-14 -- if://features/r2-workers-integration-2025-11-14 -- if://features/storj-unique-capabilities-2025-11-14 -- if://architecture/storj-dcs-distributed-2025-11-14 - -Pricing & TCO: -- if://pricing/b2-cost-2025-11-14 -- if://pricing/r2-cost-2025-11-14 -- if://pricing/storj-cost-2025-11-14 -- if://pricing/tco-analysis-2025-11-14 - -Integration & Recommendations: -- if://comparison/object-storage-matrix-2025-11-14 -- if://recommendation/if-provider-selection-2025-11-14 -- if://sdk/b2-libraries-2025-11-14 -- if://sdk/r2-sdk-support-2025-11-14 -- if://sdk/storj-sdk-support-2025-11-14 -``` - ---- - -## Summary & Recommendations - -### Primary Recommendation: Backblaze B2 - -**For NaviDocs/InfraFabric Document Storage** - -**Why B2:** -1. ✅ **Cost-Effective:** $5/TB/year storage (lowest with sane bandwidth) -2. ✅ **Strong Consistency:** Updates visible immediately (no cache coherency issues) -3. ✅ **Simple API:** JSON-based, straightforward authentication -4. ✅ **Excellent Versioning:** Built-in file versions perfect for document audit trails -5. ✅ **Replication Support:** Native API includes cross-bucket replication -6. ✅ **Integration Speed:** 5-7 days to production (vs 7-8 days Storj, 3-4 days R2) - -**Key Strengths:** -- Optimal for document archive and versioning -- Cost-effective at scale (1-5TB range) -- Simple operational model -- Good SDK maturity (Python, Go, Java) - -**Considerations:** -- Egress fees (~$10/TB) - plan for CDN in front -- No edge computing (unlike Cloudflare R2) -- Single-zone storage by default (optional multi-zone) - -### Secondary Recommendation: Cloudflare R2 - -**For High-Bandwidth or Edge-Integrated Deployments** - -**When to Choose R2:** -- Bandwidth-heavy applications (>2TB/month egress) -- CDN and edge computing requirements -- Real-time content distribution -- Willing to pay premium for storage ($15/TB/year) - -**Key Strengths:** -- Zero egress fees (massive advantage for bandwidth) -- Workers integration (edge-native processing) -- Fastest integration (3-4 days) -- Geo-redundancy built-in - -### Tertiary Recommendation: Storj DCS - -**For Privacy-Critical or Compliance-Heavy Deployments** - -**When to Choose Storj:** -- End-to-end encryption required -- GDPR/compliance concerns (user control, privacy) -- Long-term archival (immutability needed) -- Cost-optimization priority -- Avoid vendor lock-in - -**Key Strengths:** -- Privacy by default (end-to-end encryption) -- Lowest storage cost ($4/TB/year) -- Decentralized, censorship-resistant -- No vendor lock-in (open source) - -**Limitations:** -- Highest integration complexity (7/10) -- Per-segment fees (avoid for many small files) -- Less mature community SDKs -- Longer time to production (7-8 days) - -### Implementation Roadmap - -**Phase 1 (Weeks 1-2): B2 Integration** -- Set up Backblaze B2 account -- Implement basic CRUD operations -- Test multipart upload for large files -- Integrate with NaviDocs document model - -**Phase 2 (Weeks 3-4): Cost Optimization** -- Analyze egress patterns -- Consider R2 zero-egress option for high-bandwidth -- Implement cost tracking and reporting -- Set up lifecycle rules for old versions - -**Phase 3 (Future): Multi-Provider Abstraction** -- Build abstraction layer (optional) -- Support B2, R2, and Storj simultaneously -- Implement smart provider selection -- Cost-based routing - ---- - -**Research Status:** Complete -**Document Version:** 1.0 -**Last Updated:** November 14, 2025 -**Research Agent:** Haiku-30 (Object Storage Specialist) -**Quality Assurance:** IF.TTT Medical-Grade (95%+ confidence) - -**Next Steps:** -1. Review recommendations with team -2. Set up B2 trial account -3. Begin Phase 1 integration -4. Establish cost tracking baseline - ---- - -**For Questions or Further Research:** -- Contact: if://research/object-storage-team-2025 -- Follow-up: Rate limits, benchmarking, provider expansion -- Timeline:** Continuous monitoring for provider updates