Skip to content

Add Azure Benchmark Orchestration Tools - #2132

Merged
Vasileios Zois (vazois) merged 21 commits into
mainfrom
vazois/azure-bench
Sep 23, 2026
Merged

Add Azure Benchmark Orchestration Tools#2132
Vasileios Zois (vazois) merged 21 commits into
mainfrom
vazois/azure-bench

Conversation

@vazois

@vazois Vasileios Zois (vazois) commented Sep 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Adds benchmark/Azure.benchmark: an end-to-end toolkit for provisioning Azure infrastructure and running Garnet/Valkey performance benchmarks in the cloud.

The suite is organized into three stages:

  • 01-resources – Bicep templates and PowerShell to deploy shared Azure resources (network, key vault, storage) and manage SSH keys.
  • 02-platform – VMSS provisioning (manage-vmss.ps1, vmss.bicep), cloud-init configs, and bootstrap tooling (dotnet install, repo clone, build, NVMe/ramdisk/network setup, SSH key distribution). Includes Garnet and Valkey server configs for cache/disk/AOF/replication scenarios and cluster deployment helpers.
  • 03-workload – Workload drivers (memtier-bench.ps1, resp-bench.ps1, cluster.ps1) with read/write/mixed workload configs and an experimental-setup writeup.

Full TLS support

  • Generates a private benchmark CA plus separate server and client certificates, stores them in Azure Key Vault, and installs role-specific material on server and client VMSS instances.
  • Starts Garnet clusters in TLS mode with CA validation and the stable azurebench-server certificate identity. Both automatic and manual cluster creation are TLS-aware.
  • Builds and installs side-by-side TLS-enabled Valkey and Redis server/CLI binaries without replacing plaintext binaries.
  • Adds dedicated bench-tls.conf and memtier-tls.conf workload configurations.
  • Runs Resp.benchmark, including cluster-bench, over TLS using a protected password file and explicit CA/hostname validation. TLS-aware Redis CLI probes cover server information, replication, cluster topology, and database-size collection.
  • Runs both memtier load and timed benchmark phases over TLS using CA validation and SNI. An OpenSSL hostname preflight compensates for memtier's lack of certificate-name enforcement.
  • Rejects missing role metadata, certificate material, unsupported binaries, incorrect certificate identities, and attempts to override required TLS options.
  • Does not use insecure or skip-verification modes. The current Garnet configuration uses server-auth TLS and does not require client certificates.

Notes

  • AzureBench remains self-contained under benchmark/Azure.benchmark; the only existing benchmark-code changes add secure TLS password-file and issuer-certificate support to Resp.benchmark.
  • Scripts follow the PowerShell-first convention for the benchmark tooling.

Copilot AI balanced review requested due to automatic review settings September 14, 2026 16:32
Copilot AI reviewed Sep 14, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Bootstrap error suppression, orchestration races, credential exposure, and invalid benchmark configurations can produce insecure deployments or unreliable results.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds a self-contained Azure toolkit for provisioning VMSS environments and benchmarking Garnet or Valkey. The title and description match the implementation, though testing details and an issue link are absent.

Changes:

  • Deploys shared Azure networking, storage, Key Vault, and VMSS resources.
  • Bootstraps hosts and manages Garnet/Valkey processes and clusters.
  • Runs and aggregates RESP and memtier workloads.
File summaries
File Description
benchmark/Azure.benchmark/.gitattributes Configures file handling.
benchmark/Azure.benchmark/.gitignore Excludes generated artifacts.
benchmark/Azure.benchmark/README.md Documents the suite workflow.
01-resources/README.md Documents shared infrastructure.
01-resources/deploy-common-resources.ps1 Orchestrates resource deployment.
01-resources/network/network-parameters.json Supplies network parameters.
01-resources/network/network.bicep Defines networking resources.
01-resources/security/.gitignore Excludes local key material.
01-resources/security/keyvault.bicep Defines Key Vault resources.
01-resources/security/keyvault.json Contains generated vault deployment JSON.
01-resources/security/manifest.json Configures SSH key locations.
01-resources/security/ssh-key-utils.ps1 Resolves and manages SSH keys.
01-resources/storage/storage.bicep Defines shared storage.
02-platform/README.md Documents VMSS management.
02-platform/cloud-config-azurelinux.yml Bootstraps Azure Linux hosts.
02-platform/cloud-config.yml Bootstraps Ubuntu hosts.
02-platform/manage-vmss.ps1 Manages VMSS lifecycle and tools.
02-platform/vmss.bicep Defines VMSS infrastructure.
02-platform/tools/manifest.json Lists bootstrap actions.
02-platform/tools/update.ps1 Executes bootstrap actions.
02-platform/tools/bench/memtier-bench.ps1 Runs memtier on guest VMs.
02-platform/tools/bootstrap/build.ps1 Builds benchmark repositories.
02-platform/tools/bootstrap/clone-repos.ps1 Clones required repositories.
02-platform/tools/bootstrap/config.env Provides bootstrap defaults.
02-platform/tools/bootstrap/install-dotnet.ps1 Installs .NET.
02-platform/tools/bootstrap/list-ips.ps1 Lists VM network addresses.
02-platform/tools/bootstrap/setup-cluster-dir.ps1 Creates cluster directories.
02-platform/tools/bootstrap/setup-network.ps1 Tunes guest networking.
02-platform/tools/bootstrap/setup-nvme.ps1 Configures NVMe storage.
02-platform/tools/bootstrap/setup-ramdisk.ps1 Configures tmpfs storage.
02-platform/tools/bootstrap/setup-ssh-keys.ps1 Installs intra-cluster SSH keys.
02-platform/tools/cluster/cluster-deploy.ps1 Orchestrates remote clusters.
02-platform/tools/cluster/mcluster.ps1 Manages per-VM processes.
02-platform/tools/cluster/simple-setup.ps1 Provides simplified cluster setup.
02-platform/tools/config/garnet/defaults.conf Bundles Garnet defaults.
02-platform/tools/config/garnet/garnet-aof-mlog.conf Configures multi-log AOF.
02-platform/tools/config/garnet/garnet-aof-nvme.conf Configures NVMe AOF.
02-platform/tools/config/garnet/garnet-aof.conf Configures standard AOF.
02-platform/tools/config/garnet/garnet-aofx4-disk.conf Configures four-way disk AOF.
02-platform/tools/config/garnet/garnet-aofx4.conf Configures four-way AOF.
02-platform/tools/config/garnet/garnet-aofx8-disk.conf Configures eight-way disk AOF.
02-platform/tools/config/garnet/garnet-aofx8m.conf Configures manual eight-way AOF.
02-platform/tools/config/garnet/garnet-aofx16-disk.conf Configures 16-way disk AOF.
02-platform/tools/config/garnet/garnet-aofx16m.conf Configures manual 16-way AOF.
02-platform/tools/config/garnet/garnet-cache-replication.conf Configures cache replication.
02-platform/tools/config/garnet/garnet-cache.conf Configures in-memory Garnet.
02-platform/tools/config/garnet/garnet-disk.conf Configures disk-backed Garnet.
02-platform/tools/config/valkey/valkey-aof.conf Configures Valkey AOF.
02-platform/tools/config/valkey/valkey-cache.conf Configures Valkey caching.
02-platform/tools/config/valkey/valkey-disk.conf Configures disk-backed Valkey.
02-platform/tools/config/valkey/valkey-nvme.conf Configures NVMe-backed Valkey.
02-platform/tools/config/valkey/valkey-ramdisk.conf Configures ramdisk-backed Valkey.
02-platform/tools/config/valkey/valkey-repl.conf Configures Valkey replication.
03-workload/README.md Documents workload operation.
03-workload/cluster.ps1 Coordinates clusters from workstations.
03-workload/bench/EXPERIMENTAL_SETUP.md Describes benchmark methodology.
03-workload/bench/bench.conf Supplies RESP benchmark defaults.
03-workload/bench/memtier-bench.ps1 Drives distributed memtier workloads.
03-workload/bench/memtier.conf Supplies memtier defaults.
03-workload/bench/read.conf Defines read workloads.
03-workload/bench/resp-bench.ps1 Drives distributed RESP workloads.
03-workload/bench/utils.ps1 Provides shared workload helpers.
03-workload/bench/write.conf Defines write workloads.
Review details
  • Files reviewed: 63/63 changed files
  • Comments generated: 17
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread benchmark/Azure.benchmark/02-platform/tools/bootstrap/clone-repos.ps1 Outdated
Comment thread benchmark/Azure.benchmark/02-platform/tools/bootstrap/setup-ssh-keys.ps1 Outdated
Comment thread benchmark/Azure.benchmark/02-platform/tools/cluster/cluster-deploy.ps1
Comment thread benchmark/Azure.benchmark/02-platform/tools/cluster/simple-setup.ps1
Comment thread benchmark/Azure.benchmark/02-platform/tools/config/garnet/garnet-disk.conf
Comment thread benchmark/Azure.benchmark/02-platform/tools/config/garnet/garnet-aofx8m.conf Outdated
Comment thread benchmark/Azure.benchmark/02-platform/tools/config/garnet/garnet-aofx8m.conf Outdated
Comment thread benchmark/Azure.benchmark/03-workload/bench/memtier-bench.ps1
Comment thread benchmark/Azure.benchmark/03-workload/bench/memtier-bench.ps1 Outdated
Comment thread benchmark/Azure.benchmark/01-resources/README.md Outdated
@vazois
Add Azure benchmark suite
b995c9a 
End-to-end toolkit for provisioning Azure infrastructure and running
Garnet/Valkey benchmarks: 01-resources (Bicep for network/key vault/storage),
02-platform (VMSS provisioning, bootstrap tooling, server configs, cluster
helpers) and 03-workload (memtier/resp workload drivers).

All personal and account-specific identifiers are replaced with __OWNER__ and
__SSH_USER_KEY__ placeholders; run initialize-placeholders.ps1 to fill them.
Compiled Bicep ARM templates are git-ignored (the .bicep files are the source
of truth). See SECURITY-REVIEW.md for the open-source review notes.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f62e2e55-c61d-4fcb-8a0b-1b40b04a65f3
@vazois Vasileios Zois (vazois) changed the title Add Azure benchmark suite Add Azure Benchmark Orchestration Tools Sep 15, 2026
@kevin-montrose kevin-montrose self-assigned this Sep 18, 2026
@vazois
Vasileios Zois (vazois) merged commit e04ff2f into main Sep 23, 2026
169 checks passed
@vazois
Vasileios Zois (vazois) deleted the vazois/azure-bench branch September 23, 2026 23:15
x@01 (x-at-01) added a commit to webc-fork/garnet that referenced this pull request Oct 4, 2026
@x-at-01
merge: upstream 2026-09-23 — SetupFixtures/TestBase to Tsavorite + ma…
023f7ea 
…intenance loop cancellation (microsoft#2162), github-actions bump (microsoft#2169), Azure Benchmark Orchestration Tools (microsoft#2132)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@kevin-montrose kevin-montrose kevin-montrose approved these changes

Assignees

@kevin-montrose kevin-montrose

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@vazois @kevin-montrose