MapColonies utility library for http, geo, arrays and interfaces
npm install @map-colonies/mc-utils this class constructor requires the following parameters:
- ILogger instance.
- base url to use for all requests.
- optional name for target service to be used in logs.
- optional retry configuration.
the class have the following protected attributes
-``axiosOptions` the options used by axios when sending requests
the class have the protected methods for sending http requests:
- `protected async get`
- protected async post`
- protected async put`
- protected async delete`
- protected async head`
- protected async options`
- protected async patch`
function parameters list:
- : url path to send the request to (not including the base url).body
- : optional object to be used as request body (in relevant request types).queryParams
- : optional dictionary with query parameters and value.retryConfig
- : optional override to the class retry configuration.auth
- : optional basic authentication object (username, password).headers
- : optional headers to proceed to the request.
usage example:
`typescript
class myServiceClient extends HttpClient {
public constructor(logger: ILogger){
super(logger,'https://myService.com','myService',{
attempts: 3,
delay: 'exponential',
shouldResetTimeout: true
})
}
public async getName(): string {
const name = await this.get
return name;
}
}
`models
ILogger
logger interface
with the flowing log methods
- error(message:string) `
- warn(message:string)`
- info(message: string)` `
- debug(message: string)`
attempts` - the number of request to send until valid response (not 500+ status code). this value must be integer and greater then 0.
- `delay` - the amount of time in ms to wait between attempts (for constant delay) or `'exponential'` for exponential backoff.
- `shouldResetTimeout` boolean value to indicate if the request timeout should be for each request (true) or global for all attempts (false) Shapefile Processing with State Management
The
ShapefileChunkReader provides robust shapefile processing with built-in state management and progress tracking capabilities.$3
- Resumable Processing: Automatically saves processing state and can resume from the last processed chunk
- Progress Tracking: Real-time progress information including features processed, vertices counted, and time estimates
- Metrics Collection: Optional performance metrics collection for monitoring resource usage
- Chunk-based Processing: Processes large shapefiles in configurable chunks to manage memory usage
$3
`typescript
import { ShapefileChunkReader, ReaderOptions, ChunkProcessor } from '@map-colonies/mc-utils';// Define your chunk processor
const processor: ChunkProcessor = {
process: async (chunk) => {
console.log(
Processing chunk ${chunk.id} with ${chunk.features.length} features);
// Your processing logic here
}
};// Configure the reader
const options: ReaderOptions = {
maxVerticesPerChunk: 10000,
generateFeatureId: false, // Optional: for generating an UUID v4 for missing IDs
logger: myLogger,
stateManager: myStateManager, // Optional: for resumable processing
metricsCollector: myMetricsCollector, // Optional: for performance monitoring
};
// Create and use the reader
const reader = new ShapefileChunkReader(options);
await reader.readAndProcess('/path/to/shapefile.shp', processor);
`$3
Implement the
interface to enable resumable processing:`typescript
import { StateManager, ProcessingState } from '@map-colonies/mc-utils';class FileStateManager implements StateManager {
async saveState(state: ProcessingState): Promise {
// Save state to file, database, etc.
await fs.writeFile('processing-state.json', JSON.stringify(state));
}
async loadState(): Promise {
try {
const data = await fs.readFile('processing-state.json', 'utf8');
return JSON.parse(data);
} catch {
return null; // No previous state found
}
}
}
`$3
The reader automatically tracks processing progress and provides detailed information:
typescript
// Progress information is included in the saved state
const progressInfo = state.progress; // Contains:
// - processedFeatures, totalFeatures
// - processedChunks, totalChunks
// - processedVertices, totalVertices
// - percentage, elapsedTimeMs, estimatedRemainingTimeMs
// - Processing speeds (features/sec, vertices/sec, chunks/sec)
`$3
Implement the
interface to monitor performance:`typescript
import { MetricsCollector, ChunkMetrics, FileMetrics } from '@map-colonies/mc-utils';const metricsCollector: MetricsCollector = {
onChunkMetrics: (metrics: ChunkMetrics) => {
console.log(
Chunk ${metrics.chunkIndex}: ${metrics.featuresCount} features, ${metrics.totalTimeMs}ms);
},
onFileMetrics: (metrics: FileMetrics) => {
console.log(File complete: ${metrics.totalFeatures} features, ${metrics.totalTimeMs}ms total);
}
};
`$3
| Option | Type | Default | Description |
|--------|------|---------|-------------|
|
| number | Required | Maximum vertices per chunk to control memory usage |
| generateFeatureId | boolean | false | Optional determines whether a unique feature identifier should be automatically generated for each feature missing an indentifier |
| logger | Logger | undefined | Optional logger for debugging and monitoring |
| stateManager | StateManager | undefined | Optional state manager for resumable processing |
| metricsCollector` | MetricsCollector | undefined | Optional metrics collector for performance monitoring |