@neural-trader/market-data


Enterprise-grade market data providers for Neural Trader supporting real-time streaming and historical data from Alpaca, Polygon, Yahoo Finance, and more. Built with Rust for maximum throughput and minimal latency.

Table of Contents

- Features
- Installation
- Quick Start
- Supported Providers
- Core Concepts
- Examples
- Integration Patterns
- API Reference
- Configuration
- Performance Optimization
- Best Practices
- License

Features

- Multiple Providers: Alpaca, Polygon, Yahoo Finance, IEX Cloud with unified API
- Real-Time Streaming: WebSocket-based live quotes and trades with sub-millisecond latency
- Historical Data: OHLCV bars with flexible timeframes (1Min, 5Min, 1Hour, 1Day, etc.)
- Batch Operations: Fetch quotes for hundreds of symbols in parallel
- Data Encoding: Efficient binary encoding for large datasets (10x compression)
- Smart Caching: Automatic caching with TTL for frequently accessed data
- Rate Limiting: Built-in rate limit management per provider requirements
- Rust Performance: Process millions of data points per second
- Type Safety: Full TypeScript definitions for all data structures
- Crypto & Equities: Support for both traditional and digital asset data
- Error Recovery: Automatic reconnection and data gap filling

Installation

``bash


Install with required dependencies

npm install @neural-trader/market-data @neural-trader/core

Or with yarn

yarn add @neural-trader/market-data @neural-trader/core

Install type definitions

npm install --save-dev @types/node
`

$3

This package requires:
- @neural-trader/core - Core functionality and types

Quick Start

$3

`typescript
import { MarketDataProvider, listDataProviders } from '@neural-trader/market-data';

// List available providers
console.log('Available providers:', listDataProviders());
// Output: ['alpaca', 'polygon', 'yahoo', 'iex']

// Create provider for Alpaca
const provider = new MarketDataProvider({
provider: 'alpaca',
apiKey: process.env.ALPACA_API_KEY,
apiSecret: process.env.ALPACA_SECRET,
websocketEnabled: true
});

await provider.connect();
console.log('Connected to market data provider');

// Fetch historical bars (OHLCV data)
const bars = await provider.fetchBars(
'AAPL',
'2024-01-01',
'2024-12-31',
'1Day'
);

console.log(Fetched ${bars.length} daily bars for AAPL);
console.log(First bar:, bars[0]);
// Output: { symbol: 'AAPL', timestamp: '2024-01-01T00:00:00Z',
// open: '185.00', high: '187.50', low: '184.00',
// close: '186.50', volume: '45234000' }

// Get real-time quote
const quote = await provider.getQuote('AAPL');
console.log(AAPL Quote:);
console.log( Bid: $${quote.bid} x ${quote.bidSize});
console.log( Ask: $${quote.ask} x ${quote.askSize});
console.log( Last: $${quote.last});

// Subscribe to real-time quotes
const subscription = provider.subscribeQuotes(['AAPL', 'MSFT', 'GOOGL'], (quote) => {
console.log(${quote.symbol}: $${quote.last} (${quote.timestamp}));
});

// Cleanup after 60 seconds
setTimeout(async () => {
subscription.unsubscribe();
await provider.disconnect();
}, 60000);
`

Supported Providers

$3


- Data Coverage: US Equities, ETFs, Crypto
- Real-Time: Yes (WebSocket)
- Historical: Yes (1Min to 1Day bars)
- Free Tier: Yes (delayed quotes)
- API Base: https://data.alpaca.markets

$3

- Data Coverage: US Equities, Options, Crypto, Forex
- Real-Time: Yes (WebSocket)
- Historical: Yes (tick-level to daily)
- Free Tier: Limited (15 min delay)
- API Base: https://api.polygon.io

$3

- Data Coverage: Global Equities, ETFs, Indices
- Real-Time: Delayed (15-20 minutes)
- Historical: Yes (daily bars)
- Free Tier: Yes
- API Base: https://query1.finance.yahoo.com

$3

- Data Coverage: US Equities
- Real-Time: Yes (WebSocket)
- Historical: Yes (intraday and daily)
- Free Tier: Limited
- API Base: https://cloud.iexapis.com

Core Concepts

$3

Supported timeframes for historical data:

`typescript
// Intraday timeframes
'1Min' // 1-minute bars
'5Min' // 5-minute bars
'15Min' // 15-minute bars
'30Min' // 30-minute bars
'1Hour' // 1-hour bars

// Daily and longer
'1Day' // Daily bars
'1Week' // Weekly bars
'1Month' // Monthly bars
`

$3

`typescript
// Bar (OHLCV) data
interface Bar {
symbol: string;
timestamp: string; // ISO 8601 format
open: string; // Decimal string
high: string;
low: string;
close: string;
volume: string;
}

// Real-time quote
interface Quote {
symbol: string;
timestamp: string;
bid: number;
ask: number;
bidSize: number;
askSize: number;
last: number;
lastSize: number;
volume: number;
}

// Trade data
interface Trade {
symbol: string;
timestamp: string;
price: number;
size: number;
conditions: string[];
}
`

Examples

$3

`typescript
import { MarketDataProvider } from '@neural-trader/market-data';

const provider = new MarketDataProvider({
provider: 'alpaca',
apiKey: process.env.ALPACA_API_KEY,
apiSecret: process.env.ALPACA_SECRET
});

await provider.connect();

// Fetch daily bars for multiple years
const dailyBars = await provider.fetchBars(
'AAPL',
'2020-01-01',
'2024-12-31',
'1Day'
);

console.log(Total bars: ${dailyBars.length});
console.log(Date range: ${dailyBars[0].timestamp} to ${dailyBars[dailyBars.length - 1].timestamp});

// Fetch intraday 5-minute bars
const intradayBars = await provider.fetchBars(
'MSFT',
'2024-11-01',
'2024-11-13',
'5Min'
);

console.log(5-minute bars: ${intradayBars.length});

// Process bars for analysis
dailyBars.forEach(bar => {
const close = parseFloat(bar.close);
const open = parseFloat(bar.open);
const change = ((close - open) / open) * 100;
console.log(${bar.timestamp}: ${change.toFixed(2)}%);
});

await provider.disconnect();
`

$3

`typescript
import { MarketDataProvider } from '@neural-trader/market-data';

const provider = new MarketDataProvider({
provider: 'polygon',
apiKey: process.env.POLYGON_API_KEY,
websocketEnabled: true
});

await provider.connect();

// Create price tracker
const priceTracker = new Map();

// Subscribe to multiple symbols
const symbols = ['AAPL', 'MSFT', 'GOOGL', 'AMZN', 'TSLA'];
const subscription = provider.subscribeQuotes(symbols, (quote) => {
const prevPrice = priceTracker.get(quote.symbol);
priceTracker.set(quote.symbol, quote.last);

if (prevPrice) {
const change = quote.last - prevPrice;
const changePercent = (change / prevPrice) * 100;
const direction = change > 0 ? '↑' : change < 0 ? '↓' : '→';

console.log(
${quote.symbol}: $${quote.last.toFixed(2)} ${direction} +
${changePercent.toFixed(2)}% (Vol: ${quote.volume})
);
}
});

// Monitor for 5 minutes
await new Promise(resolve => setTimeout(resolve, 300000));

subscription.unsubscribe();
await provider.disconnect();
`

$3

`typescript
import { MarketDataProvider } from '@neural-trader/market-data';

const provider = new MarketDataProvider({
provider: 'alpaca',
apiKey: process.env.ALPACA_API_KEY,
apiSecret: process.env.ALPACA_SECRET
});

await provider.connect();

// Fetch quotes for entire portfolio
const portfolio = [
'AAPL', 'MSFT', 'GOOGL', 'AMZN', 'TSLA',
'NVDA', 'META', 'NFLX', 'AMD', 'INTC'
];

const quotes = await provider.getQuotesBatch(portfolio);

// Calculate portfolio metrics
let totalValue = 0;
const positions = {
'AAPL': 100,
'MSFT': 50,
'GOOGL': 25,
// ... more positions
};

quotes.forEach(quote => {
const shares = positions[quote.symbol] || 0;
const value = quote.last * shares;
totalValue += value;

console.log(${quote.symbol}: ${shares} shares @ $${quote.last} = $${value.toFixed(2)});
});

console.log(\nTotal Portfolio Value: $${totalValue.toFixed(2)});

await provider.disconnect();
`

$3

`typescript
import { MarketDataProvider } from '@neural-trader/market-data';

// Use multiple providers for redundancy and coverage
const providers = [
new MarketDataProvider({
provider: 'alpaca',
apiKey: process.env.ALPACA_API_KEY,
apiSecret: process.env.ALPACA_SECRET
}),
new MarketDataProvider({
provider: 'polygon',
apiKey: process.env.POLYGON_API_KEY
})
];

// Connect to all providers
await Promise.all(providers.map(p => p.connect()));

// Fetch data from multiple sources
async function getQuoteWithFallback(symbol: string): Promise {
for (const provider of providers) {
try {
return await provider.getQuote(symbol);
} catch (error) {
console.warn(Provider ${provider.config.provider} failed, trying next...);
}
}
throw new Error(All providers failed for symbol: ${symbol});
}

// Usage
try {
const quote = await getQuoteWithFallback('AAPL');
console.log(AAPL: $${quote.last});
} catch (error) {
console.error('Failed to fetch quote:', error);
}

// Cleanup
await Promise.all(providers.map(p => p.disconnect()));
`

$3

`typescript
import {
MarketDataProvider,
encodeBarsToBuffer,
decodeBarsFromBuffer
} from '@neural-trader/market-data';
import fs from 'fs/promises';

const provider = new MarketDataProvider({
provider: 'alpaca',
apiKey: process.env.ALPACA_API_KEY,
apiSecret: process.env.ALPACA_SECRET
});

await provider.connect();

// Fetch large dataset
const bars = await provider.fetchBars(
'SPY',
'2020-01-01',
'2024-12-31',
'1Min'
);

console.log(Fetched ${bars.length} bars);

// Encode to binary for efficient storage
const encoded = encodeBarsToBuffer(bars);
console.log(Original size: ${JSON.stringify(bars).length} bytes);
console.log(Encoded size: ${encoded.length} bytes);
console.log(Compression ratio: ${(JSON.stringify(bars).length / encoded.length).toFixed(2)}x);

// Save to file
await fs.writeFile('spy_1min.bin', encoded);
console.log('Saved to spy_1min.bin');

// Later: decode from file
const fileBuffer = await fs.readFile('spy_1min.bin');
const decoded = decodeBarsFromBuffer(fileBuffer);
console.log(Decoded ${decoded.length} bars);

await provider.disconnect();
`

Integration Patterns

$3

`typescript
import { MarketDataProvider } from '@neural-trader/market-data';
import { Strategy } from '@neural-trader/strategies';
import { calculateRsi, calculateSma } from '@neural-trader/features';

// Create strategy
const strategy = new Strategy({
name: 'rsi-oversold-strategy',
symbols: ['AAPL', 'MSFT', 'GOOGL'],
parameters: {
rsiPeriod: 14,
rsiOversold: 30,
smaPeriod: 50
}
});

// Create market data provider
const provider = new MarketDataProvider({
provider: 'alpaca',
apiKey: process.env.ALPACA_API_KEY,
apiSecret: process.env.ALPACA_SECRET,
websocketEnabled: true
});

await provider.connect();

// Feed real-time data to strategy
strategy.on('barClose', async (bar) => {
console.log(New bar for ${bar.symbol});

// Fetch historical data for indicators
const endDate = new Date(bar.timestamp);
const startDate = new Date(endDate);
startDate.setDate(startDate.getDate() - 100); // 100 days lookback

const historicalBars = await provider.fetchBars(
bar.symbol,
startDate.toISOString().split('T')[0],
endDate.toISOString().split('T')[0],
'1Day'
);

// Calculate indicators
const closes = historicalBars.map(b => parseFloat(b.close));
const rsi = calculateRsi(closes, 14);
const sma = calculateSma(closes, 50);

const currentRsi = rsi[rsi.length - 1];
const currentSma = sma[sma.length - 1];
const currentClose = parseFloat(bar.close);

console.log(${bar.symbol}: RSI=${currentRsi.toFixed(2)}, Price=$${currentClose}, SMA50=$${currentSma.toFixed(2)});

// Generate signals
if (currentRsi < 30 && currentClose > currentSma) {
strategy.emit('signal', {
symbol: bar.symbol,
action: 'buy',
reason: 'RSI oversold + above SMA50',
quantity: 100
});
}
});

// Subscribe to real-time bars
const subscription = provider.subscribeQuotes(strategy.symbols, (quote) => {
// Convert quote to bar (simplified - would track full bar in production)
strategy.emit('barClose', {
symbol: quote.symbol,
timestamp: quote.timestamp,
close: quote.last.toString(),
volume: quote.volume.toString()
});
});

// Run for 1 hour
await new Promise(resolve => setTimeout(resolve, 3600000));

subscription.unsubscribe();
await provider.disconnect();
`

$3

`typescript
import { MarketDataProvider } from '@neural-trader/market-data';
import { BacktestEngine } from '@neural-trader/backtesting';
import { calculateSma, calculateRsi } from '@neural-trader/features';

// Create backtest engine
const backtest = new BacktestEngine({
initialCapital: 100000,
commission: 0.001,
startDate: '2023-01-01',
endDate: '2023-12-31'
});

// Create market data provider
const provider = new MarketDataProvider({
provider: 'alpaca',
apiKey: process.env.ALPACA_API_KEY,
apiSecret: process.env.ALPACA_SECRET
});

await provider.connect();

// Fetch historical data for backtesting
const symbols = ['AAPL', 'MSFT', 'GOOGL'];
const historicalData = new Map();

for (const symbol of symbols) {
const bars = await provider.fetchBars(
symbol,
'2023-01-01',
'2023-12-31',
'1Day'
);
historicalData.set(symbol, bars);
console.log(Loaded ${bars.length} bars for ${symbol});
}

// Add strategy using historical data
backtest.addStrategy(async (context) => {
const { symbol, date, portfolio } = context;
const bars = historicalData.get(symbol);

if (!bars) return;

// Get bars up to current date
const currentBars = bars.filter(b => new Date(b.timestamp) <= date);
if (currentBars.length < 50) return; // Need minimum history

// Calculate indicators
const closes = currentBars.map(b => parseFloat(b.close));
const rsi = calculateRsi(closes, 14);
const sma50 = calculateSma(closes, 50);

const currentRsi = rsi[rsi.length - 1];
const currentSma = sma50[sma50.length - 1];
const currentPrice = parseFloat(currentBars[currentBars.length - 1].close);

// Trading logic
if (currentRsi < 30 && currentPrice > currentSma) {
// Buy signal
const position = portfolio.positions.get(symbol);
if (!position || position.quantity === 0) {
await context.placeOrder({
symbol,
side: 'buy',
orderType: 'market',
quantity: 100,
timeInForce: 'day'
});
}
} else if (currentRsi > 70) {
// Sell signal
const position = portfolio.positions.get(symbol);
if (position && position.quantity > 0) {
await context.placeOrder({
symbol,
side: 'sell',
orderType: 'market',
quantity: position.quantity,
timeInForce: 'day'
});
}
}
});

// Run backtest
const results = await backtest.run(symbols);

console.log('\nBacktest Results:');
console.log( Initial Capital: $${backtest.initialCapital.toFixed(2)});
console.log( Final Value: $${results.finalValue.toFixed(2)});
console.log( Total Return: ${(results.totalReturn * 100).toFixed(2)}%);
console.log( Sharpe Ratio: ${results.sharpeRatio.toFixed(2)});
console.log( Max Drawdown: ${(results.maxDrawdown * 100).toFixed(2)}%);
console.log( Total Trades: ${results.totalTrades});
console.log( Win Rate: ${(results.winRate * 100).toFixed(2)}%);

await provider.disconnect();
`

API Reference

$3

Main class for fetching market data.

`typescript
class MarketDataProvider {
constructor(config: MarketDataConfig);

// Connection Management
connect(): Promise;
disconnect(): Promise;
isConnected(): Promise;

// Historical Data
fetchBars(
symbol: string,
start: string,
end: string,
timeframe: string
): Promise;

// Real-Time Data
getQuote(symbol: string): Promise;
getQuotesBatch(symbols: string[]): Promise;

// WebSocket Streaming
subscribeQuotes(
symbols: string[],
callback: (quote: Quote) => void
): Subscription;

subscribeTrades(
symbols: string[],
callback: (trade: Trade) => void
): Subscription;

// Configuration
get config(): MarketDataConfig;
updateConfig(updates: Partial): void;
}
`

$3

`typescript
interface MarketDataConfig {
provider: 'alpaca' | 'polygon' | 'yahoo' | 'iex';
apiKey: string;
apiSecret?: string;
websocketEnabled?: boolean;
timeout?: number;
cacheTTL?: number;
}

interface Bar {
symbol: string;
timestamp: string; // ISO 8601
open: string; // Decimal string
high: string;
low: string;
close: string;
volume: string;
}

interface Quote {
symbol: string;
timestamp: string;
bid: number;
ask: number;
bidSize: number;
askSize: number;
last: number;
lastSize: number;
volume: number;
}

interface Trade {
symbol: string;
timestamp: string;
price: number;
size: number;
conditions: string[];
exchange?: string;
}

interface Subscription {
unsubscribe(): void;
isActive(): boolean;
}
`

$3

`typescript
// List all supported providers
function listDataProviders(): string[];

// Binary encoding for large datasets
function encodeBarsToBuffer(bars: Bar[]): Buffer;
function decodeBarsFromBuffer(buffer: Buffer): Bar[];

// Data validation
function validateBar(bar: Bar): boolean;
function validateQuote(quote: Quote): boolean;

// Timeframe utilities
function parseTimeframe(timeframe: string): { value: number; unit: string };
function isValidTimeframe(timeframe: string): boolean;
`

Configuration

$3

`bash


Alpaca

ALPACA_API_KEY=your_alpaca_key
ALPACA_SECRET=your_alpaca_secret

Polygon

POLYGON_API_KEY=your_polygon_key

IEX Cloud

IEX_API_KEY=your_iex_key
`

$3

`typescript
// config/market-data.ts
import { MarketDataConfig } from '@neural-trader/market-data';

export const dataProviderConfigs: Record = {
alpaca: {
provider: 'alpaca',
apiKey: process.env.ALPACA_API_KEY!,
apiSecret: process.env.ALPACA_SECRET!,
websocketEnabled: true,
timeout: 30000,
cacheTTL: 5000
},
polygon: {
provider: 'polygon',
apiKey: process.env.POLYGON_API_KEY!,
websocketEnabled: true,
timeout: 10000,
cacheTTL: 1000
},
yahoo: {
provider: 'yahoo',
apiKey: '', // No key required
websocketEnabled: false,
timeout: 15000,
cacheTTL: 60000
}
};
`

Performance Optimization

$3

`typescript
// Use batch operations instead of sequential
const symbols = ['AAPL', 'MSFT', 'GOOGL', 'AMZN', 'TSLA'];

// ❌ Slow: Sequential fetching
for (const symbol of symbols) {
const quote = await provider.getQuote(symbol);
console.log(quote);
}

// ✅ Fast: Batch fetching
const quotes = await provider.getQuotesBatch(symbols);
quotes.forEach(quote => console.log(quote));
`

$3

`typescript
import { encodeBarsToBuffer, decodeBarsFromBuffer } from '@neural-trader/market-data';

// Fetch large dataset
const bars = await provider.fetchBars('SPY', '2020-01-01', '2024-12-31', '1Min');

// Encode to binary (10x smaller than JSON)
const encoded = encodeBarsToBuffer(bars);

// Store or transmit encoded data
await fs.writeFile('data.bin', encoded);

// Decode when needed
const decoded = decodeBarsFromBuffer(encoded);
`

$3

`typescript
const provider = new MarketDataProvider({
provider: 'alpaca',
apiKey: process.env.ALPACA_API_KEY,
apiSecret: process.env.ALPACA_SECRET,
cacheTTL: 5000 // Cache quotes for 5 seconds
});

// First call: fetches from API
const quote1 = await provider.getQuote('AAPL');

// Second call within 5 seconds: returns cached data
const quote2 = await provider.getQuote('AAPL');
`

Best Practices

$3

`typescript
// For backtesting: Use daily bars
const dailyBars = await provider.fetchBars('AAPL', '2023-01-01', '2023-12-31', '1Day');

// For intraday trading: Use minute bars
const minuteBars = await provider.fetchBars('AAPL', '2024-11-13', '2024-11-13', '1Min');
`

$3

`typescript
import pLimit from 'p-limit';

const limit = pLimit(5); // Max 5 concurrent requests

const symbols = ['AAPL', 'MSFT', / ... 100 more symbols /];
const quotes = await Promise.all(
symbols.map(symbol =>
limit(() => provider.getQuote(symbol))
)
);
`

$3

`typescript
async function fetchBarsWithRetry(
provider: MarketDataProvider,
symbol: string,
start: string,
end: string,
timeframe: string,
maxRetries = 3
): Promise {
for (let i = 0; i < maxRetries; i++) {
try {
return await provider.fetchBars(symbol, start, end, timeframe);
} catch (error) {
console.error(Attempt ${i + 1} failed:, error.message);
if (i === maxRetries - 1) throw error;
await new Promise(resolve => setTimeout(resolve, 1000 * (i + 1)));
}
}
throw new Error('Should not reach here');
}
`

$3

`typescript
const provider = new MarketDataProvider({
provider: 'alpaca',
apiKey: process.env.ALPACA_API_KEY,
apiSecret: process.env.ALPACA_SECRET,
websocketEnabled: true
});

await provider.connect();

// Monitor connection status
setInterval(async () => {
const connected = await provider.isConnected();
if (!connected) {
console.warn('Connection lost, reconnecting...');
await provider.connect();
}
}, 10000);
`

$3

`typescript
// Always disconnect when done
try {
await provider.connect();
const bars = await provider.fetchBars('AAPL', '2024-01-01', '2024-12-31', '1Day');
// Process bars...
} finally {
await provider.disconnect();
}

// Or use with automatic cleanup
async function withProvider(
config: MarketDataConfig,
callback: (provider: MarketDataProvider) => Promise
): Promise {
const provider = new MarketDataProvider(config);
try {
await provider.connect();
return await callback(provider);
} finally {
await provider.disconnect();
}
}
``

License

Dual-licensed under MIT OR Apache-2.0.

See LICENSE-MIT and LICENSE-APACHE for details.