Observability & Metrics

The storage package provides two complementary observability layers:

Adapter-level metrics — structured counts, durations, and gauges emitted from every BaseStorage operation, with built-in OpenTelemetry support.
Facade-level hooks — fire-and-forget onAction / onError / onRetry callbacks on the Files facade, carrying per-operation context (type, key, from/to, durationMs, attempt, error).

Pick the layer that matches what you need: metrics for aggregation and dashboards; hooks for structured logs and tracing at the application boundary.

Overview

Metrics are collected for all storage operations including:

Operation counts - Track how many operations are performed
Operation durations - Measure latency (p50, p95, p99)
Error rates - Monitor failures by operation type
File sizes - Track file size distributions
Batch operation metrics - Monitor batch operation performance

`Files` facade hooks

The Files facade exposes three fire-and-forget hooks. Hooks are called, not awaited; exceptions thrown by a hook are swallowed so a hook can never fail the operation it observes.

import { Files } from "@visulima/storage";

const files = new Files({
    adapter,
    hooks: {
        onAction: ({ type, key, durationMs }) => log.info(type, key, durationMs),
        onError: ({ type, key, error, durationMs }) => log.error(type, key, error.message, durationMs),
        onRetry: ({ type, key, attempt, error }) => log.warn(type, key, "retry", attempt, error.message),
    },
});

Each event carries:

Field	Present on	Description
`type`	every event	`upload` \| `download` \| `head` \| `exists` \| `delete` \| `copy` \| `move` \| `list` \| `listAll` \| `url` \| `signedUploadUrl` \| `transfer`.
`key`	single-key operations	Caller-facing key (constructor prefix stripped off).
`keys`	bulk-delete	All caller-facing keys included in the batch.
`from` / `to`	`copy` / `move`	Source / destination keys.
`durationMs`	`onAction` / `onError`	Wall-clock time from the start of the operation to completion or failure.
`attempt`	`onRetry`	1-based retry attempt number (1 = first retry).
`error`	`onError`, `onRetry`	Normalized `Error`.

The listAll terminal event fires once at the end of the async-iterable walk — including the case where the consumer breaks out of the for await early.

When to use hooks vs. adapter metrics:

Use hooks when you want per-call structured logs with the caller's key, or a tracing span keyed by type + key.
Use adapter metrics when you want aggregate counters and histograms for dashboards and alerts.

Both layers can be enabled at once — they observe different concerns.

Metrics Interface

The storage package uses a simple Metrics interface that can be implemented by any metrics backend:

interface Metrics {
    increment(name: string, value?: number, attributes?: Record<string, string | number>): void;
    timing(name: string, duration: number, attributes?: Record<string, string | number>): void;
    gauge(name: string, value: number, attributes?: Record<string, string | number>): void;
}

The package includes built-in support for OpenTelemetry, the industry-standard observability framework. This allows integration with any OpenTelemetry-compatible backend (Prometheus, Datadog, New Relic, Grafana Cloud, etc.).

Installation

First, install the OpenTelemetry API:

pnpm add @opentelemetry/api

Basic Usage

import { metrics } from "@opentelemetry/api";
import { OpenTelemetryMetrics } from "@visulima/storage";
import { S3Storage } from "@visulima/storage/provider/aws";

// Initialize OpenTelemetry (typically done once in your app)
const meter = metrics.getMeter("@visulima/storage", "1.0.0");

// Create metrics instance
const storageMetrics = new OpenTelemetryMetrics(meter);

// Use with storage
const storage = new S3Storage({
    bucket: "my-bucket",
    metrics: storageMetrics,
});

Custom Meter

You can also provide a custom meter instance:

import { metrics } from "@opentelemetry/api";
import { OpenTelemetryMetrics } from "@visulima/storage";

// Create a meter with custom configuration
const meter = metrics.getMeter("my-app", "1.0.0", {
    // Custom meter options
});

const storageMetrics = new OpenTelemetryMetrics(meter);

Collected Metrics

The storage package automatically collects the following metrics:

Operation Metrics

storage.operations.{operation}.count - Counter for operation invocations
storage.operations.{operation}.duration - Histogram of operation durations (ms)
storage.operations.{operation}.error.count - Counter for operation errors

Operations tracked:

create - File creation
write - File write operations
get - File retrieval
delete - File deletion
copy - File copy operations
move - File move operations
update - Metadata updates

Batch Operation Metrics

storage.operations.batch.{operation}.count - Counter for batch operations
storage.operations.batch.{operation}.duration - Histogram of batch operation durations (ms)
storage.operations.batch.{operation}.success_count - Gauge of successful operations
storage.operations.batch.{operation}.failed_count - Gauge of failed operations

Batch operations tracked:

delete - Batch deletions
copy - Batch copies
move - Batch moves

File Metrics

storage.files.size - Gauge of file sizes (bytes)

Metric Attributes

All metrics include the following attributes:

storage - Storage backend type (e.g., "s3", "azure", "disk")
operation - Operation name (for file size metrics)
error - Error message (for error metrics)
batch_size - Number of items in batch (for batch operations)

Custom Metrics Implementation

You can implement your own metrics backend by implementing the Metrics interface:

import type { Metrics } from "@visulima/storage";
import { DiskStorage } from "@visulima/storage";

class CustomMetrics implements Metrics {
    increment(name: string, value = 1, attributes?: Record<string, string | number>): void {
        // Send to your metrics backend
        console.log(`Counter: ${name} = ${value}`, attributes);
    }

    timing(name: string, duration: number, attributes?: Record<string, string | number>): void {
        // Send to your metrics backend
        console.log(`Timing: ${name} = ${duration}ms`, attributes);
    }

    gauge(name: string, value: number, attributes?: Record<string, string | number>): void {
        // Send to your metrics backend
        console.log(`Gauge: ${name} = ${value}`, attributes);
    }
}

const storage = new DiskStorage({
    directory: "./uploads",
    metrics: new CustomMetrics(),
});

Examples

Prometheus Integration

import { metrics } from "@opentelemetry/api";
import { MeterProvider, PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";
import { PrometheusExporter } from "@opentelemetry/exporter-prometheus";
import { OpenTelemetryMetrics } from "@visulima/storage";
import { S3Storage } from "@visulima/storage/provider/aws";

// Setup Prometheus exporter
const exporter = new PrometheusExporter({ port: 9464 });

const meterProvider = new MeterProvider({
    readers: [new PeriodicExportingMetricReader({ exporter })],
});

metrics.setGlobalMeterProvider(meterProvider);

const meter = metrics.getMeter("@visulima/storage");
const storageMetrics = new OpenTelemetryMetrics(meter);

const storage = new S3Storage({
    bucket: "my-bucket",
    metrics: storageMetrics,
});

Datadog Integration

import { metrics } from "@opentelemetry/api";
import { MeterProvider, PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";
import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-http";
import { OpenTelemetryMetrics } from "@visulima/storage";
import { S3Storage } from "@visulima/storage/provider/aws";

const exporter = new OTLPMetricExporter({
    url: "https://api.datadoghq.com/api/v2/otlp/v1/metrics",
    headers: {
        "DD-API-KEY": process.env.DATADOG_API_KEY!,
    },
});

const meterProvider = new MeterProvider({
    readers: [new PeriodicExportingMetricReader({ exporter })],
});

metrics.setGlobalMeterProvider(meterProvider);

const meter = metrics.getMeter("@visulima/storage");
const storageMetrics = new OpenTelemetryMetrics(meter);

const storage = new S3Storage({
    bucket: "my-bucket",
    metrics: storageMetrics,
});

Using with Instrumentation Helper

For custom storage implementations, you can use the instrumentOperation helper method:

import { AbstractBaseStorage } from "@visulima/storage";

class MyStorage extends AbstractBaseStorage {
    public async create(config: FileInit): Promise<File> {
        return this.instrumentOperation(
            "create",
            async () => {
                // Your implementation
                const file = await this.doCreate(config);
                return file;
            },
            {
                custom_attribute: "value",
            },
        );
    }
}

Best Practices

Use OpenTelemetry for Production - OpenTelemetry is the industry standard and integrates with all major observability platforms.
Monitor Key Metrics - Focus on:
- Operation latencies (p95, p99)
- Error rates
- Batch operation success rates
- File size distributions
Set Up Alerts - Configure alerts for:
- High error rates (> 1%)
- Slow operations (p95 > 1s)
- Batch operation failures
Use Attributes Wisely - Attributes are automatically added but you can add custom ones for filtering and grouping.
No Metrics Overhead - If no metrics instance is provided, a no-op implementation is used, ensuring zero overhead when metrics are disabled.

Metric Naming Convention

All metrics follow this naming pattern:

storage.operations.{operation}.{type} - Operation-level metrics
storage.operations.batch.{operation}.{type} - Batch operation metrics
storage.files.{property} - File-level metrics

Where:

{operation} is the operation name (create, write, delete, etc.)
{type} is the metric type (count, duration, error.count)
{property} is the file property (size)

This consistent naming makes it easy to query and aggregate metrics across all storage operations.

Observability