feat: add prometheus by miekg · Pull Request #304 · psviderski/uncloud

miekg · 2026-04-07T05:20:10Z

Add the prometheus server and control its lifetime just like other services.

miekg · 2026-04-07T09:13:53Z

Defines 3 metrics, and hooks up 2 metrics; dns and uncloudd version.

miekg · 2026-04-08T13:17:35Z

For grpc I can go as far as using a metrics interceptor to get all data out of the grpc calls; that generate a lot of metrics and might be too lowlevel. That also excludes non-grpc, such as dns requests

miekg · 2026-04-26T05:11:08Z

This does the bare minium for adding prometheus and example metrics. It listens on the cluster network, so metrics can be scraped from a prometheus running in the cluster.

miekg · 2026-04-30T19:35:00Z

ok the e2e test fails because m.IP() is not there yet. So this needs to be moved to where the dns server is started.

miekg · 2026-05-01T09:42:50Z

locally it works:

root@debug-xqa9-d6ik:/# curl -s http://10.210.0.1:51004/metrics | grep uncloud
go_build_info{checksum="",path="github.com/psviderski/uncloud",version="(devel)"} 1
# HELP uncloud_dns_query_total Counter of DNS queries.
# TYPE uncloud_dns_query_total counter
uncloud_dns_query_total{internal="false",status="ok"} 2
# HELP uncloud_uncloudd_build_info Build information.
# TYPE uncloud_uncloudd_build_info gauge
uncloud_uncloudd_build_info{version="999.0.0-dev"} 1

miekg · 2026-05-02T05:52:02Z

ready for review. PTAL

miekg · 2026-05-19T09:19:26Z

currently trying figure out that I prolly need uc in a container to get the current machine list to auto-update the list of machines so I can scrape uncloud cluster wide.

Add the prometheus server and control its lifetime just like other services Signed-off-by: Miek Gieben <[email protected]>

Signed-off-by: Miek Gieben <[email protected]>

… closed' Checked start and stop, now logs: ``` uncloudd[1233408]: INFO Starting prometheus server. uncloudd[1233408]: INFO Prometheus server stopped. ``` Metrics are available on on port 51004 ``` go_build_info{checksum="",path="github.com/psviderski/uncloud",version="v0.0.0-20260407052333-46be9c590e02+dirty"} 1 ... ``` Signed-off-by: Miek Gieben <[email protected]>

Signed-off-by: Miek Gieben <[email protected]>

For ContainerExec add exec and pull Signed-off-by: Miek Gieben <[email protected]>

Signed-off-by: Miek Gieben <[email protected]>

Rename label to resolver for dns metrics Signed-off-by: Miek Gieben <[email protected]>

Signed-off-by: Miek Gieben <[email protected]>

This makes prometheus listen on port 51004 in the cluster network. This allows scraping prometheus from an incluster prometheus Signed-off-by: Miek Gieben <[email protected]>

This fixes the test failure by pushing promtheus into the cluster controller. If we want to listen on the same address as the dns server we should do it like the dns server. This means moving it to the cluster controler. Also expand it to look more like the dns's server.go with Run() and stop() methods. Signed-off-by: Miek Gieben <[email protected]>

Signed-off-by: Miek Gieben <[email protected]>

psviderski

Overall looks good, just need to decide where to init and run the metrics server and a few minor comments

Use port 51090 and rename prometheus to metrics Signed-off-by: Miek Gieben <[email protected]>

Signed-off-by: Miek Gieben <[email protected]>

Dont use a new image, reuse the one we likely already have and move to wget Signed-off-by: Miek Gieben <[email protected]>

Signed-off-by: Miek Gieben <[email protected]>

miekg · 2026-05-27T06:59:41Z

I have trouble relating this test failure to my change:

== NAME  TestComposeDeployment
    cluster_test.go:52: 
        	Error Trace:	/home/runner/work/uncloud/uncloud/test/e2e/cluster_test.go:52
        	            				/home/runner/work/uncloud/uncloud/test/e2e/compose_deploy_test.go:22
        	Error:      	Received unexpected error:
        	            	wait for machine "machine-1" to be ready: inspect machine: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:36933: connect: connection refused"
        	Test:       	TestComposeDeployment
--- FAIL: TestComposeDeployment (93.98s)
=== CONT  TestComposeBuild
=== NAME  TestServiceLifecycle

starting uncloudd locally just starts up and shuts down

miekg · 2026-05-27T08:16:42Z

I have trouble relating this test failure to my change:

== NAME  TestComposeDeployment
    cluster_test.go:52: 
        	Error Trace:	/home/runner/work/uncloud/uncloud/test/e2e/cluster_test.go:52
        	            				/home/runner/work/uncloud/uncloud/test/e2e/compose_deploy_test.go:22
        	Error:      	Received unexpected error:
        	            	wait for machine "machine-1" to be ready: inspect machine: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:36933: connect: connection refused"
        	Test:       	TestComposeDeployment
--- FAIL: TestComposeDeployment (93.98s)
=== CONT  TestComposeBuild
=== NAME  TestServiceLifecycle

starting uncloudd locally just starts up and shuts down

seeing this locally, this hangs (likely on starting the metrics server):

% ./ucind cluster create metrics                                                                                                                          ~miekg/uncloud/cmd/ucind miek/26/apr07di/prom2+

Error: create cluster 'metrics': wait for machine "machine-1" to be ready: inspect machine: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:38653: connect: connection refused

Signed-off-by: Miek Gieben <[email protected]>

miekg · 2026-05-27T09:31:42Z

ok, lots slushing code back and forth, but the end result looks neater, which is all that counts.

miekg · 2026-05-29T07:17:50Z

Yep, exactly. This is what I have running atm. Makes all that stuff fall outside of the core of uncloud. Cheers

…

On Fri, May 29, 2026, 08:58 Pasha Sviderski ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In internal/machine/cluster.go <#304 (comment)>: > @@ -194,6 +199,14 @@ func (cc *clusterController) Run(ctx context.Context) error { return nil }) + errGroup.Go(func() error { + slog.Info("Starting prometheus server.") + if err := cc.prometheusServer.Run(ctx); err != nil { + return fmt.Errorf("prometheus server failed: %w", err) + } + return nil + }) + Got it, thanks for clarifying. And is the reason to run on a cluster IP to be able to scrape all endpoints from prometheus/collector running only on one machine? — Reply to this email directly, view it on GitHub <#304?email_source=notifications&email_token=AACWIWZYMKTSGVFMCN674TD45EYKLA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTIMZYG4ZTIMZSGU42M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#discussion_r3322662263>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACWIWYCCLRSNCF233NRPEL45EYKLAVCNFSM6AAAAACXO6ONS2VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHM2DGOBXGM2DGMRVHE> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/AACWIW6U5NG7G6JO2TXKTXT45EYKLA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTIMZYG4ZTIMZSGU42M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KUZTPN52GK4S7NFXXG> and Android <https://github.com/notifications/mobile/android/AACWIWY5JCVMFWKFAQC2J2345EYKLA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTIMZYG4ZTIMZSGU42M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>. Download it today! You are receiving this because you authored the thread.Message ID: ***@***.***>

psviderski · 2026-05-29T08:20:15Z

Where are all these other metrics coming from?

# HELP go_build_info Build information about the main Go module.
# TYPE go_build_info gauge
go_build_info{checksum="",path="github.com/psviderski/uncloud",version="(devel)"} 1
# HELP go_gc_duration_seconds A summary of the wall-time pause (stop-the-world) duration in garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 4.2291e-05
go_gc_duration_seconds{quantile="0.25"} 5.9292e-05
go_gc_duration_seconds{quantile="0.5"} 0.000156834
go_gc_duration_seconds{quantile="0.75"} 0.000672292
go_gc_duration_seconds{quantile="1"} 0.000714917
go_gc_duration_seconds_sum 0.002137252
go_gc_duration_seconds_count 7
# HELP go_gc_gogc_percent Heap size target percentage configured by the user, otherwise 100. This value is set by the GOGC environment variable, and the runtime/debug.SetGCPercent function. Sourced from /gc/gogc:percent.
# TYPE go_gc_gogc_percent gauge
go_gc_gogc_percent 100
# HELP go_gc_gomemlimit_bytes Go runtime memory limit configured by the user, otherwise math.MaxInt64. This value is set by the GOMEMLIMIT environment variable, and the runtime/debug.SetMemoryLimit function. Sourced from /gc/gomemlimit:bytes.
# TYPE go_gc_gomemlimit_bytes gauge
go_gc_gomemlimit_bytes 9.223372036854776e+18
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 78
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.26.1"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated in heap and currently in use. Equals to /memory/classes/heap/objects:bytes.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 6.118408e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated in heap until now, even if released already. Equals to /gc/heap/allocs:bytes.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 1.830528e+07
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table. Equals to /memory/classes/profiling/buckets:bytes.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.464347e+06
# HELP go_memstats_frees_total Total number of heap objects frees. Equals to /gc/heap/frees:objects + /gc/heap/tiny/allocs:objects.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 121853
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata. Equals to /memory/classes/metadata/other:bytes.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 4.014832e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and currently in use, same as go_memstats_alloc_bytes. Equals to /memory/classes/heap/objects:bytes.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 6.118408e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used. Equals to /memory/classes/heap/released:bytes + /memory/classes/heap/free:bytes.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 5.431296e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use. Equals to /memory/classes/heap/objects:bytes + /memory/classes/heap/unused:bytes
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 9.904128e+06
# HELP go_memstats_heap_objects Number of currently allocated objects. Equals to /gc/heap/objects:objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 34432
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS. Equals to /memory/classes/heap/released:bytes.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 3.588096e+06
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system. Equals to /memory/classes/heap/objects:bytes + /memory/classes/heap/unused:bytes + /memory/classes/heap/released:bytes + /memory/classes/heap/free:bytes.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 1.5335424e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.7800424269864492e+09
# HELP go_memstats_mallocs_total Total number of heap objects allocated, both live and gc-ed. Semantically a counter version for go_memstats_heap_objects gauge. Equals to /gc/heap/allocs:objects + /gc/heap/tiny/allocs:objects.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 156285
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures. Equals to /memory/classes/metadata/mcache/inuse:bytes.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 18368
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system. Equals to /memory/classes/metadata/mcache/inuse:bytes + /memory/classes/metadata/mcache/free:bytes.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 32144
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures. Equals to /memory/classes/metadata/mspan/inuse:bytes.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 231360
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system. Equals to /memory/classes/metadata/mspan/inuse:bytes + /memory/classes/metadata/mspan/free:bytes.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 244800
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place. Equals to /gc/heap/goal:bytes.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 1.2570746e+07
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations. Equals to /memory/classes/other:bytes.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.737773e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes obtained from system for stack allocator in non-CGO environments. Equals to /memory/classes/heap/stacks:bytes.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 1.441792e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator. Equals to /memory/classes/heap/stacks:bytes + /memory/classes/os-stacks:bytes.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 1.441792e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system. Equals to /memory/classes/total:byte.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 2.4271112e+07
# HELP go_sched_gomaxprocs_threads The current runtime.GOMAXPROCS setting, or the number of operating system threads that can execute user-level Go code simultaneously. Sourced from /sched/gomaxprocs:threads.
# TYPE go_sched_gomaxprocs_threads gauge
go_sched_gomaxprocs_threads 8
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 13
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.27
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_network_receive_bytes_total Number of bytes received by the process over the network.
# TYPE process_network_receive_bytes_total counter
process_network_receive_bytes_total 1.170792e+06
# HELP process_network_transmit_bytes_total Number of bytes sent by the process over the network.
# TYPE process_network_transmit_bytes_total counter
process_network_transmit_bytes_total 308464
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 28
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 5.4980608e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.78004239106e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.355751424e+09
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 0
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0
# HELP registry_proxy_hits_total The number of total proxy request hits
# TYPE registry_proxy_hits_total counter
registry_proxy_hits_total{type="blob"} 0
registry_proxy_hits_total{type="manifest"} 0
# HELP registry_proxy_misses_total The number of total proxy request misses
# TYPE registry_proxy_misses_total counter
registry_proxy_misses_total{type="blob"} 0
registry_proxy_misses_total{type="manifest"} 0
# HELP registry_proxy_pulled_bytes_total The size of total bytes pulled from the upstream
# TYPE registry_proxy_pulled_bytes_total counter
registry_proxy_pulled_bytes_total{type="blob"} 0
registry_proxy_pulled_bytes_total{type="manifest"} 0
# HELP registry_proxy_pushed_bytes_total The size of total bytes pushed to the client
# TYPE registry_proxy_pushed_bytes_total counter
registry_proxy_pushed_bytes_total{type="blob"} 0
registry_proxy_pushed_bytes_total{type="manifest"} 0
# HELP registry_proxy_requests_total The number of total incoming proxy request received
# TYPE registry_proxy_requests_total counter
registry_proxy_requests_total{type="blob"} 0
registry_proxy_requests_total{type="manifest"} 0
# HELP registry_storage_cache_errors_total The number of cache request errors
# TYPE registry_storage_cache_errors_total counter
registry_storage_cache_errors_total 0
# HELP registry_storage_cache_hits_total The number of cache request received
# TYPE registry_storage_cache_hits_total counter
registry_storage_cache_hits_total 0
# HELP registry_storage_cache_requests_total The number of cache request received
# TYPE registry_storage_cache_requests_total counter
registry_storage_cache_requests_total 0
# HELP uncloud_uncloudd_build_info Build information.
# TYPE uncloud_uncloudd_build_info gauge
uncloud_uncloudd_build_info{version="999.0.0-dev"} 1

registry_proxy_ is probably what unregistry registers. I thought that you would need to explicitly declare something to expose internal process/go/gc metrics. And they are not prefixed with uncloud_. Is this a problem?

miekg · 2026-05-29T09:08:22Z

Where are all these other metrics coming from?

# HELP go_build_info Build information about the main Go module.
# TYPE go_build_info gauge
go_build_info{checksum="",path="github.com/psviderski/uncloud",version="(devel)"} 1
# HELP go_gc_duration_seconds A summary of the wall-time pause (stop-the-world) duration in garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 4.2291e-05
go_gc_duration_seconds{quantile="0.25"} 5.9292e-05
go_gc_duration_seconds{quantile="0.5"} 0.000156834
go_gc_duration_seconds{quantile="0.75"} 0.000672292
go_gc_duration_seconds{quantile="1"} 0.000714917
go_gc_duration_seconds_sum 0.002137252
go_gc_duration_seconds_count 7
# HELP go_gc_gogc_percent Heap size target percentage configured by the user, otherwise 100. This value is set by the GOGC environment variable, and the runtime/debug.SetGCPercent function. Sourced from /gc/gogc:percent.
# TYPE go_gc_gogc_percent gauge
go_gc_gogc_percent 100
# HELP go_gc_gomemlimit_bytes Go runtime memory limit configured by the user, otherwise math.MaxInt64. This value is set by the GOMEMLIMIT environment variable, and the runtime/debug.SetMemoryLimit function. Sourced from /gc/gomemlimit:bytes.
# TYPE go_gc_gomemlimit_bytes gauge
go_gc_gomemlimit_bytes 9.223372036854776e+18
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 78
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.26.1"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated in heap and currently in use. Equals to /memory/classes/heap/objects:bytes.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 6.118408e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated in heap until now, even if released already. Equals to /gc/heap/allocs:bytes.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 1.830528e+07
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table. Equals to /memory/classes/profiling/buckets:bytes.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.464347e+06
# HELP go_memstats_frees_total Total number of heap objects frees. Equals to /gc/heap/frees:objects + /gc/heap/tiny/allocs:objects.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 121853
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata. Equals to /memory/classes/metadata/other:bytes.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 4.014832e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and currently in use, same as go_memstats_alloc_bytes. Equals to /memory/classes/heap/objects:bytes.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 6.118408e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used. Equals to /memory/classes/heap/released:bytes + /memory/classes/heap/free:bytes.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 5.431296e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use. Equals to /memory/classes/heap/objects:bytes + /memory/classes/heap/unused:bytes
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 9.904128e+06
# HELP go_memstats_heap_objects Number of currently allocated objects. Equals to /gc/heap/objects:objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 34432
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS. Equals to /memory/classes/heap/released:bytes.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 3.588096e+06
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system. Equals to /memory/classes/heap/objects:bytes + /memory/classes/heap/unused:bytes + /memory/classes/heap/released:bytes + /memory/classes/heap/free:bytes.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 1.5335424e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.7800424269864492e+09
# HELP go_memstats_mallocs_total Total number of heap objects allocated, both live and gc-ed. Semantically a counter version for go_memstats_heap_objects gauge. Equals to /gc/heap/allocs:objects + /gc/heap/tiny/allocs:objects.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 156285
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures. Equals to /memory/classes/metadata/mcache/inuse:bytes.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 18368
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system. Equals to /memory/classes/metadata/mcache/inuse:bytes + /memory/classes/metadata/mcache/free:bytes.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 32144
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures. Equals to /memory/classes/metadata/mspan/inuse:bytes.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 231360
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system. Equals to /memory/classes/metadata/mspan/inuse:bytes + /memory/classes/metadata/mspan/free:bytes.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 244800
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place. Equals to /gc/heap/goal:bytes.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 1.2570746e+07
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations. Equals to /memory/classes/other:bytes.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.737773e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes obtained from system for stack allocator in non-CGO environments. Equals to /memory/classes/heap/stacks:bytes.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 1.441792e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator. Equals to /memory/classes/heap/stacks:bytes + /memory/classes/os-stacks:bytes.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 1.441792e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system. Equals to /memory/classes/total:byte.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 2.4271112e+07
# HELP go_sched_gomaxprocs_threads The current runtime.GOMAXPROCS setting, or the number of operating system threads that can execute user-level Go code simultaneously. Sourced from /sched/gomaxprocs:threads.
# TYPE go_sched_gomaxprocs_threads gauge
go_sched_gomaxprocs_threads 8
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 13
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.27
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_network_receive_bytes_total Number of bytes received by the process over the network.
# TYPE process_network_receive_bytes_total counter
process_network_receive_bytes_total 1.170792e+06
# HELP process_network_transmit_bytes_total Number of bytes sent by the process over the network.
# TYPE process_network_transmit_bytes_total counter
process_network_transmit_bytes_total 308464
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 28
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 5.4980608e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.78004239106e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.355751424e+09
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 0
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0
# HELP registry_proxy_hits_total The number of total proxy request hits
# TYPE registry_proxy_hits_total counter
registry_proxy_hits_total{type="blob"} 0
registry_proxy_hits_total{type="manifest"} 0
# HELP registry_proxy_misses_total The number of total proxy request misses
# TYPE registry_proxy_misses_total counter
registry_proxy_misses_total{type="blob"} 0
registry_proxy_misses_total{type="manifest"} 0
# HELP registry_proxy_pulled_bytes_total The size of total bytes pulled from the upstream
# TYPE registry_proxy_pulled_bytes_total counter
registry_proxy_pulled_bytes_total{type="blob"} 0
registry_proxy_pulled_bytes_total{type="manifest"} 0
# HELP registry_proxy_pushed_bytes_total The size of total bytes pushed to the client
# TYPE registry_proxy_pushed_bytes_total counter
registry_proxy_pushed_bytes_total{type="blob"} 0
registry_proxy_pushed_bytes_total{type="manifest"} 0
# HELP registry_proxy_requests_total The number of total incoming proxy request received
# TYPE registry_proxy_requests_total counter
registry_proxy_requests_total{type="blob"} 0
registry_proxy_requests_total{type="manifest"} 0
# HELP registry_storage_cache_errors_total The number of cache request errors
# TYPE registry_storage_cache_errors_total counter
registry_storage_cache_errors_total 0
# HELP registry_storage_cache_hits_total The number of cache request received
# TYPE registry_storage_cache_hits_total counter
registry_storage_cache_hits_total 0
# HELP registry_storage_cache_requests_total The number of cache request received
# TYPE registry_storage_cache_requests_total counter
registry_storage_cache_requests_total 0
# HELP uncloud_uncloudd_build_info Build information.
# TYPE uncloud_uncloudd_build_info gauge
uncloud_uncloudd_build_info{version="999.0.0-dev"} 1

registry_proxy_ is probably what unregistry registers. I thought that you would need to explicitly declare something to expose internal process/go/gc metrics. And they are not prefixed with uncloud_. Is this a problem?

these are the std go metrics that the prometheus go_client exposes. Not a problem and very good to have, each metric when scraped will have a label job="uncloud" which means you still know what they are.

All the uncloud_ should be the metrics we add.

miekg changed the title ~~WIP: feat: add prometheus~~ feat: add prometheus Apr 7, 2026

miekg mentioned this pull request Apr 8, 2026

add prometheus metrics #294

Closed

psviderski reviewed Apr 14, 2026

View reviewed changes

Comment thread internal/metrics/metrics.go

Comment thread pkg/client/container.go Outdated

Comment thread internal/metrics/metrics.go Outdated

miekg mentioned this pull request Apr 20, 2026

Faculty of Science wishlist #320

Closed

miekg force-pushed the miek/26/apr07di/prom2 branch from d89aae4 to 5a8fe59 Compare April 26, 2026 05:09

miekg mentioned this pull request Apr 26, 2026

feat: uc list, structured output #335

Closed

miekg mentioned this pull request May 2, 2026

[draft] feat: Add labels on machines (with cli for manual add/rm/ls) #164

Draft

miekg added 16 commits May 26, 2026 06:35

feat: add prometheus

f2619dd

Add the prometheus server and control its lifetime just like other services Signed-off-by: Miek Gieben <[email protected]>

go mod tidy

0b9b931

Signed-off-by: Miek Gieben <[email protected]>

Define initial set of metrics

f7e3a42

Signed-off-by: Miek Gieben <[email protected]>

Set version metric value

9057777

Signed-off-by: Miek Gieben <[email protected]>

Hookup dns metrics

a59b313

Signed-off-by: Miek Gieben <[email protected]>

Hook up all current metrics

e3e0120

For ContainerExec add exec and pull Signed-off-by: Miek Gieben <[email protected]>

Actually increment the metrics

641a01a

Signed-off-by: Miek Gieben <[email protected]>

Inc() all metrics

3892bfc

Signed-off-by: Miek Gieben <[email protected]>

Remove ContainerExec metrics for now

5bcc366

Rename label to resolver for dns metrics Signed-off-by: Miek Gieben <[email protected]>

go mod tidy

b16362e

Signed-off-by: Miek Gieben <[email protected]>

Remove config list on cluster address

8475263

This makes prometheus listen on port 51004 in the cluster network. This allows scraping prometheus from an incluster prometheus Signed-off-by: Miek Gieben <[email protected]>

dont double log

747ff99

Signed-off-by: Miek Gieben <[email protected]>

make port an int, just like for the dns server

b5dcb6d

Signed-off-by: Miek Gieben <[email protected]>

dns ->prometheus

96287f9

Signed-off-by: Miek Gieben <[email protected]>

miekg added 5 commits May 26, 2026 06:40

hardcode address

df1d2c3

Signed-off-by: Miek Gieben <[email protected]>

Get the ip address programmatically

f221a76

Signed-off-by: Miek Gieben <[email protected]>

silent curl

60d2a0c

Signed-off-by: Miek Gieben <[email protected]>

remove comment

0b3428e

Signed-off-by: Miek Gieben <[email protected]>

silent curl

3a98df2

Signed-off-by: Miek Gieben <[email protected]>

miekg force-pushed the miek/26/apr07di/prom2 branch from 46094c8 to 3a98df2 Compare May 26, 2026 04:44

miekg added 2 commits May 26, 2026 08:01

compile again

c9205cc

Signed-off-by: Miek Gieben <[email protected]>

no hardcoded ips

dae20db

Signed-off-by: Miek Gieben <[email protected]>

psviderski reviewed May 27, 2026

View reviewed changes

miekg added 9 commits May 27, 2026 05:56

Port and rename

e8b3098

Use port 51090 and rename prometheus to metrics Signed-off-by: Miek Gieben <[email protected]>

Use correct logging

f3d399d

Signed-off-by: Miek Gieben <[email protected]>

Use busybox with wget

17f4970

Dont use a new image, reuse the one we likely already have and move to wget Signed-off-by: Miek Gieben <[email protected]>

Move metrics server start/stop to machine.Run

8c473ed

Signed-off-by: Miek Gieben <[email protected]>

Move e2e test

382240d

Signed-off-by: Miek Gieben <[email protected]>

rerun tst

eefb62c

Signed-off-by: Miek Gieben <[email protected]>

re-run tests

f63465e

Signed-off-by: Miek Gieben <[email protected]>

noop

15860d2

Signed-off-by: Miek Gieben <[email protected]>

noop

1e6c0c8

Signed-off-by: Miek Gieben <[email protected]>

miekg added 3 commits May 27, 2026 11:06

Move metrics back to controller

5ebb06e

Signed-off-by: Miek Gieben <[email protected]>

remove space

17151bf

Signed-off-by: Miek Gieben <[email protected]>

ignore cleanup errors

3ea5fcb

Signed-off-by: Miek Gieben <[email protected]>

fix: shutdown metrics server only once on context cancellation

2e297c7

psviderski merged commit 24b81af into psviderski:main May 29, 2026
4 checks passed

miekg deleted the miek/26/apr07di/prom2 branch May 29, 2026 09:08

Conversation

miekg commented Apr 7, 2026

Uh oh!

miekg commented Apr 7, 2026

Uh oh!

miekg commented Apr 8, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

miekg commented Apr 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

miekg commented Apr 30, 2026

Uh oh!

miekg commented May 1, 2026

Uh oh!

miekg commented May 2, 2026

Uh oh!

miekg commented May 19, 2026

Uh oh!

psviderski left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

miekg commented May 27, 2026

Uh oh!

miekg commented May 27, 2026

Uh oh!

miekg commented May 27, 2026

Uh oh!

miekg commented May 29, 2026 via email

Uh oh!

psviderski commented May 29, 2026

Uh oh!

Uh oh!

miekg commented May 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

miekg commented Apr 26, 2026 •

edited

Loading