Console logging interceptor that sends logs to Google BigQuery
npm install bqtapConsole logging interceptor that automatically sends logs to Google BigQuery.
``bash`
npm install bqtap
1. A Google Cloud project with BigQuery API enabled
2. A BigQuery dataset created (the table will be created automatically by default)
3. Google Cloud credentials (one of the following):
- Service account credentials as JSON string (recommended for production)
- Application Default Credentials (ADC)
- Google Cloud SDK authentication
Note: The BigQuery table will be created automatically with the correct schema when you first log. You can disable this by setting autoCreateTable: false in the configuration.
`javascript
const { BQTap } = require('bqtap');
const bqtap = new BQTap({
projectId: 'your-gcp-project-id',
dataset: 'your-dataset-name',
table: 'your-table-name',
credentials: process.env.BQTAP_GOOGLE_API_CREDENTIALS, // JSON string (optional)
});
// Start intercepting console methods
bqtap.start();
// Now all console logs will be sent to BigQuery
console.log('Hello, BigQuery!');
console.error('An error occurred', { userId: 123 });
console.warn('Warning message');
// Stop intercepting when done
bqtap.stop();
`
`typescript
import { BQTap, BQTapConfig } from 'bqtap';
const config: BQTapConfig = {
projectId: 'your-gcp-project-id',
dataset: 'your-dataset-name',
table: 'your-table-name',
};
const bqtap = new BQTap(config);
bqtap.start();
`
`javascript
const bqtap = new BQTap({
projectId: 'your-gcp-project-id',
dataset: 'logs',
table: 'console_logs',
// Google Cloud credentials as JSON string
credentials: process.env.BQTAP_GOOGLE_API_CREDENTIALS,
// Enable/disable (defaults to BQTAP_ENABLED env var)
enabled: true,
// Auto-create table if it doesn't exist (default: true)
autoCreateTable: true,
// Only intercept specific methods
methods: ['error', 'warn'],
// Batch size before sending to BigQuery
batchSize: 20,
// Flush interval in milliseconds
flushInterval: 10000,
// Disable passthrough to original console
passthrough: false,
// Add custom metadata to all logs
metadata: {
service: 'api-server',
version: '1.0.0',
},
// Handle BigQuery insertion errors
onError: (error) => {
console.error('Failed to send logs to BigQuery:', error);
},
});
bqtap.start();
`
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| projectId | string | required | Google Cloud project ID |dataset
| | string | required | BigQuery dataset name |table
| | string | required | BigQuery table name |credentials
| | string | optional | Google Cloud credentials as JSON string |enabled
| | boolean | true | Enable/disable console override (also via BQTAP_ENABLED env var) |autoCreateTable
| | boolean | true | Automatically create BigQuery table if it doesn't exist |methods
| | string[] | ['log', 'warn', 'error', 'info', 'debug'] | Console methods to intercept |batchSize
| | number | 10 | Number of logs to batch before sending |flushInterval
| | number | 5000 | Flush interval in milliseconds |passthrough
| | boolean | true | Whether to also output to original console |metadata
| | object | optional | Custom metadata attached to all logs |onError
| | function | optional | Error handler for BigQuery failures |
Start intercepting console methods and sending logs to BigQuery.
Stop intercepting console methods and restore original behavior. Flushes any pending logs.
Manually flush all buffered logs to BigQuery. Returns a Promise.
`javascript`
await bqtap.flush();
1. Handle process termination: Ensure logs are flushed before the process exits
`javascript`
process.on('beforeExit', async () => {
await bqtap.flush();
bqtap.stop();
});
2. Use environment variables: Store configuration in environment variables
`javascript`
const bqtap = new BQTap({
projectId: process.env.BQTAP_PROJECT_ID,
dataset: process.env.BQTAP_DATASET,
table: process.env.BQTAP_TABLE,
credentials: process.env.BQTAP_GOOGLE_API_CREDENTIALS, // JSON string
});
3. Control activation with BQTAP_ENABLED: Use environment variable to enable/disable logging
`javascript
// Set BQTAP_ENABLED=false to disable, BQTAP_ENABLED=true to enable
const bqtap = new BQTap({
projectId: process.env.BQTAP_PROJECT_ID,
dataset: process.env.BQTAP_DATASET,
table: process.env.BQTAP_TABLE,
credentials: process.env.BQTAP_GOOGLE_API_CREDENTIALS,
// Will automatically check BQTAP_ENABLED env var
});
// Or explicitly control it via config
const bqtap = new BQTap({
// ... other config
enabled: true, // or false
});
`
4. Add contextual metadata: Include service information in metadata
`javascript`
const bqtap = new BQTap({
// ... other config
metadata: {
hostname: os.hostname(),
pid: process.pid,
service: 'my-service',
},
});
By default, BQTap will automatically create the BigQuery table with the correct schema if it doesn't exist. The schema used is:
`sqlyour-project.your-dataset.logs
CREATE TABLE (`
timestamp TIMESTAMP NOT NULL,
level STRING NOT NULL,
message STRING,
args STRING, -- JSON-stringified array of arguments
metadata STRING -- JSON-stringified metadata object
);
If you prefer to create the table manually or want to use advanced features like partitioning, you can set autoCreateTable: false and create the table yourself:
`sqlyour-project.your-dataset.logs
CREATE TABLE (`
timestamp TIMESTAMP NOT NULL,
level STRING NOT NULL,
message STRING,
args STRING, -- JSON-stringified array of arguments
metadata STRING -- JSON-stringified metadata object
)
PARTITION BY DATE(timestamp)
CLUSTER BY level;
The package includes tests using Node.js built-in test runner:
`bash`
npm test
This will build the TypeScript code and run all tests in the test/ directory.
`bash`
npm run build
This compiles the TypeScript code to JavaScript in the dist/` directory.
MIT