Updated LLM Management Service using ENAPSO MongoDB ORM
npm install llm-management-updatedbash
npm install
`
4. Create a .env file in the root directory with the following configuration:
`
PORT=3200
MONGO_URL=mongodb://localhost:27017/llm-management
LOG_LEVEL=debug
VASTAI_API_KEY=your_vastai_api_key_here
`
Adjust the values according to your environment. The VASTAI_API_KEY is optional - if not provided, the service will fall back to using mock machine data.
Usage
$3
`bash
npm start
`
For development with auto-reload:
`bash
npm run dev
`
$3
To test just the ORM functionality:
`bash
npm run orm
`
This will start the server with the ORM endpoint exposed on port 3200 and print usage examples.
To test the ORM client that interacts with the endpoint:
`bash
npm run orm-client
`
This will send a test request to the ORM endpoint and display the results.
ORM Endpoint
The primary way to interact with the service is through the ORM endpoint at http://localhost:3200/orm. This accepts POST requests to interact with the Machine model using the ENAPSO Mongo ORM.
$3
The Machine model supports standard CRUD operations:
1. Read (Find Machines):
`json
{
"class": "Machine",
"method": "read",
"filter": [
{ "property": "provider", "value": "vastai" },
{ "property": "gpu_count", "operator": "$gte", "value": 1 }
],
"sort": [ { "property": "price_per_hour", "direction": "ASC" } ],
"limit": 5
}
`
2. Get by ID:
`json
{
"class": "Machine",
"method": "getById",
"_id": "mongoose_object_id_string"
}
`
3. Create Machine (Manual):
`json
{
"class": "Machine",
"method": "create",
"args": {
"data": {
"id": "manual-machine-01",
"name": "My Manual Machine",
"provider": "manual",
"price_per_hour": 0.5,
"gpu_name": "Custom GPU",
"gpu_count": 2
// ... other fields
}
}
}
`
4. Update Machine:
`json
{
"class": "Machine",
"method": "update",
"args": {
"_id": "mongoose_object_id_string",
"data": {
"name": "Updated Machine Name",
"instanceStatus": "idle"
}
}
}
`
5. Delete Machine:
`json
{
"class": "Machine",
"method": "delete",
"args": {
"_id": "mongoose_object_id_string"
}
}
`
$3
The Machine model provides custom methods for managing Vast.ai instances:
1. Start Machine:
`json
{
"class": "Machine",
"method": "startMachine",
"args": {
"machineId": "12345",
"options": {
"model": "llama3",
"diskSpace": 215,
"timeout": 300000,
"checkInterval": 10000
}
}
}
`
You can use either machineId or ask_contract_id to identify the machine. Both now use the original Vast.ai ID without the "vastai-" prefix that was used in earlier versions.
The response will return immediately with a success status and instance ID. The service then continues monitoring the instance startup in the background and updates the machine's record in the database once running.
When the machine reaches 'running' status, the system will automatically create an accessUrl field that provides direct access to the WebUI (typically on port 9090). This URL is constructed using the machine's public IP address and exposed port, and can be used to access the Ollama WebUI interface directly from a browser.
2. Stop Machine:
`json
{
"class": "Machine",
"method": "stopMachine",
"args": {
"machineId": "12345"
}
}
`
You can use any of these parameters to identify the machine:
- machineId: The machine ID in the database (same as the original Vast.ai ID)
- ask_contract_id: The original Vast.ai offer ID (same as machineId)
- instanceId: The ID returned when starting a machine
3. Get Machine Status:
`json
{
"class": "Machine",
"method": "getMachineStatus",
"args": {
"machineId": "12345"
}
}
`
This will return the current status of the machine (idle, starting, running, stopping, stopped, error). If the machine has an instanceId, it will query Vast.ai for the latest status and update the database record if needed.
Machine Synchronization
The service automatically syncs machine data from Vast.ai to the MongoDB database on startup and every 2 hours. If you provide a valid VastAI API key in the .env file, the service will fetch real-time machine offers from Vast.ai.
The synchronization process:
1. Fetches available machine offers from Vast.ai
2. Maps them to the Machine schema format
3. Stores them in MongoDB through the ORM layer
To manually trigger synchronization, you can restart the service.
Development
$3
- lib/
- OrmService.js - Core service that defines the Machine model and its methods
- VastAIService.js - Handles Vast.ai API interactions (machine start/stop/status)
- MachineSyncService.js - Fetches and syncs machine offers from Vast.ai
- LLMService.js - Main service that orchestrates all other services
- config.js - Application configuration
- index.js - Application entry point
- .env - Environment variables (ignored by git)
- .env.example - Template for environment variables
- delete-machines.js - Utility script to delete all machines from the database
- package.json - Project metadata and dependencies
$3
The Machine schema includes fields for machine specifications, location, pricing, and instance state:
- Basic Info: id, name, provider
- Hardware: gpu_name, gpu_count, total_flops, cuda_cores, cpu_cores, cpu_ram, disk_space
- Pricing: price_per_hour
- Location: country, city, latitude, longitude
- Instance Status: instanceStatus, instanceId, instanceHost, instancePorts, accessUrl, instanceStartTime, instanceStopTime, instanceError
- Metadata: lastSyncTime, timestamps
$3
You can extend the Machine schema with more methods in OrmService.js by adding them to the methods object in the schema definition. For example:
`javascript
// Add this to the methods object in machineSchemaDefinition.methods
async findByGpuName(body) {
const { args } = body;
const gpuName = args?.gpuName;
if (!gpuName) {
return {
success: false,
status: 400,
message: 'gpuName is required in args'
};
}
try {
// 'this.constructor' refers to the Model when called via ORM
const results = await this.constructor.find({ gpu_name: gpuName });
return {
success: true,
status: 200,
message: 'Ok',
data: results,
count: results.length
};
} catch (error) {
return {
success: false,
status: 500,
message: error.message
};
}
}
`
You can then call this method via the ORM endpoint:
`json
{
"class": "Machine",
"method": "findByGpuName",
"args": {
"gpuName": "RTX 4090"
}
}
``