Skip to content

Hyperparameter Tuning Farm

This pattern demonstrates the true power of AerolVM: horizontal scaling. By spinning up multiple sandboxes, you can benchmark different hyperparameters (like learning rates or model architectures) across a fleet of isolated workers simultaneously.

  • Speed: Reduce total tuning time from hours to minutes by running jobs in parallel.
  • Isolation: A crash in one training job doesn't affect the others.
  • Resource Management: Each sandbox gets its own dedicated CPU and RAM, preventing resource contention.

Code URL: https://github.com/aerol-ai/aerolvm-examples/tree/main/ml-data-engineering/hyperparameter-tuning-farm

This script creates 3 concurrent sandboxes, each training a Scikit-Learn model with a different n_estimators value.

import { MicroVM } from "@aerol-ai/aerolvm-sdk";
const apiUrl = process.env.SB_API_URL ?? "http://127.0.0.1:21212";
const patToken = process.env.SB_PAT_TOKEN;
const trainingScript = `
import os
import time
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
n_estimators = int(os.environ.get("N_ESTIMATORS", 10))
print(f"Training RandomForest with n_estimators={n_estimators}...")
X, y = make_classification(n_samples=1000, n_features=20)
clf = RandomForestClassifier(n_estimators=n_estimators)
scores = cross_val_score(clf, X, y, cv=5)
print(f"RESULT: accuracy={scores.mean():.4f}")
`;
async function trainModel(client: MicroVM, nEstimators: number) {
console.log(`[Job n_estimators=${nEstimators}] Creating sandbox...`);
const sandbox = await client.create({
image: "python:3.11-bookworm",
cpu: 1,
memoryMB: 1024,
env: { N_ESTIMATORS: String(nEstimators) }
});
console.log(`[Job n_estimators=${nEstimators}] Sandbox created: ${sandbox.id}`);
console.log(`[Job n_estimators=${nEstimators}] Installing scikit-learn...`);
await sandbox.exec("pip install scikit-learn");
console.log(`[Job n_estimators=${nEstimators}] Uploading training script...`);
await sandbox.uploadFile("/train.py", trainingScript);
console.log(`[Job n_estimators=${nEstimators}] Running training script...`);
const result = await sandbox.exec("python3 /train.py");
console.log(`[Job n_estimators=${nEstimators}] Script finished (exit code: ${result.exitCode})`);
const output = result.stdout.split("RESULT: ")[1]?.trim();
console.log(`[Job n_estimators=${nEstimators}] Destroying sandbox...`);
await sandbox.destroy();
console.log(`[Job n_estimators=${nEstimators}] Sandbox destroyed.`);
return { nEstimators, output };
}
async function main() {
if (!patToken) throw new Error("Set SB_PAT_TOKEN.");
console.log("Initializing AerolVM client...");
const client = new MicroVM({ apiUrl, patToken });
const hyperparams = [10, 50, 100];
console.log(`Spinning up ${hyperparams.length} parallel training jobs...`);
const jobs = hyperparams.map(val => trainModel(client, val));
const results = await Promise.all(jobs);
console.log("\nAll training jobs completed successfully! Results:");
console.table(results);
}
main().catch(console.error);