Prometheus Unix Socket Scraping (3.15): 6 Easy Tests, Errors

Prometheus 3.15.0 can scrape a target over a Unix domain socket. Set the special label __unix_socket__ to the socket path, either in static_configs labels or with relabel_configs, and Prometheus connects to that file instead of a host and port. A Prometheus Unix socket scrape lets you monitor an exporter that has no TCP port at all. Older versions accept the same config file without error but ignore the label and try to resolve the target name with DNS.

Every Prometheus Unix socket example below was tested on 2026-10-19 with the official images prom/prometheus:v3.15.0 and v3.13.4, node_exporter v1.12.1 and socat 1.8.0.3 in rootless podman containers. All output is copied from the lab.

Contents

• What changed in Prometheus 3.15.0
• Prometheus Unix socket lab: an exporter without a network
• Prometheus Unix socket scrape with static_configs
• Same file on Prometheus 3.13.4
• Many sockets with file_sd and relabel_configs
• Expose your own app metrics on a Unix socket
• Prometheus Unix socket errors and fixes
• Docker and Podman notes
• Other changes in 3.15.0: --log.level and GOMEMLIMIT

What changed in Prometheus 3.15.0

The Prometheus 3.15.0 release (2026-09-25) lists "[FEATURE] Scraping: Add support for scraping targets via Unix Domain Sockets" (pull request 18091, issue 12024). The configuration docs describe it in one paragraph:

• __unix_socket__, when set, makes the scrape client connect via that Unix domain socket path instead of the target's __address__.
• __scheme__ (or scheme: in the job) still chooses http or https over the socket; the default is http.

There is no new unix_sd_config and no unix:// address format: the Prometheus Unix socket support is just this one label. The target address stays as the instance name and as the Host part of the URL.

Prometheus Unix socket lab: an exporter without a network

To prove that the scrape really goes through the socket, node_exporter runs in a container with --network none, listening on 127.0.0.1:9100 inside its own network namespace. socat in the same namespace turns that into a socket file in a shared volume. The volume is mounted into Prometheus as /sock:

docker run -d --name ne --network none quay.io/prometheus/node-exporter:v1.12.1 --web.listen-address=127.0.0.1:9100
docker run -d --name ne-socat --network container:ne -v promsock:/sock docker.io/alpine/socat:1.8.0.3 \
  UNIX-LISTEN:/sock/node.sock,fork,mode=660,user=65534,group=65534 TCP:127.0.0.1:9100

$ docker inspect -f '{{.HostConfig.NetworkMode}}' ne
none

$ docker run --rm -v promsock:/sock docker.io/library/alpine:3.22 ls -l /sock
total 0
srw-rw----    1 nobody   nobody           0 Oct 19 07:35 node.sock

The socket answers normal HTTP, which you can test with curl --unix-socket:

$ curl -s --unix-socket /sock/node.sock http://localhost/metrics | grep -E "^node_exporter_build_info"
node_exporter_build_info{branch="HEAD",goarch="amd64",goos="linux",goversion="go1.26.5",revision="6044da783597cc3b57aef7580ddcdcff58a4ee99",tags="unknown",version="1.12.1"} 1

The socket is owned by uid 65534 with mode 660 on purpose: the official Prometheus image runs as nobody:

$ docker run --rm --entrypoint id docker.io/prom/prometheus:v3.15.0
uid=65534(nobody) gid=65534(nobody) groups=65534(nobody)

Prometheus Unix socket scrape with static_configs

The smallest working Prometheus Unix socket config. The target node-sock is only a name here; it does not need to resolve:

global:
  scrape_interval: 5s
scrape_configs:
  - job_name: node-socket
    static_configs:
      - targets: ["node-sock"]
        labels:
          __unix_socket__: /sock/node.sock

Both versions pass promtool check config, so promtool cannot tell you whether your Prometheus understands the label:

$ docker run --rm -v /tmp/cfg:/cfg:z --entrypoint promtool docker.io/prom/prometheus:v3.15.0 check config /cfg/prometheus.yml
Checking /cfg/prometheus.yml
 SUCCESS: /cfg/prometheus.yml is valid prometheus config file syntax

$ docker run --rm -v /tmp/cfg:/cfg:z --entrypoint promtool docker.io/prom/prometheus:v3.13.4 check config /cfg/prometheus.yml
Checking /cfg/prometheus.yml
 SUCCESS: /cfg/prometheus.yml is valid prometheus config file syntax

The checks below use the HTTP API. targets and query are two small shell functions:

targets() { curl -s "http://127.0.0.1:$1/api/v1/targets" | jq -r '.data.activeTargets[] | "job=\(.labels.job) instance=\(.labels.instance) health=\(.health)\n  scrapeUrl=\(.scrapeUrl)\n  lastError=\(.lastError)"'; }
query() { curl -s "http://127.0.0.1:$1/api/v1/query" --data-urlencode "query=$2" | jq -c '[.data.result[] | {metric: .metric, value: .value[1]}]'; }

Prometheus 3.15.0 scrapes the exporter through the Prometheus Unix socket path:

$ targets 19015
job=node-socket instance=node-sock health=up
  scrapeUrl=http://node-sock/metrics
  lastError=

$ query 19015 'up'
[{"metric":{"__name__":"up","instance":"node-sock","job":"node-socket"},"value":"1"}]

$ query 19015 'scrape_duration_seconds'
[{"metric":{"__name__":"scrape_duration_seconds","instance":"node-sock","job":"node-socket"},"value":"0.146234367"}]

Same config file: Prometheus 3.15.0 scrapes the Unix socket, 3.13.4 ignores the label and fails on DNS.

Note that scrapeUrl shows http://node-sock/metrics; the Targets page does not show the socket path, so add it as a normal label (see the next section) if you want to see it.

Same file on Prometheus 3.13.4

The old version starts fine and silently drops the label, so there is no Prometheus Unix socket support before 3.15. It treats node-sock as a host name:

$ targets 19013
job=node-socket instance=node-sock health=down
  scrapeUrl=http://node-sock/metrics
  lastError=Get "http://node-sock/metrics": dial tcp: lookup node-sock on 169.254.1.1:53: no such host

$ query 19013 'up'
[{"metric":{"__name__":"up","instance":"node-sock","job":"node-socket"},"value":"0"}]

If you see dial tcp: lookup ... no such host for a target that should use a socket, check the version with prometheus --version:

$ docker run --rm docker.io/prom/prometheus:v3.15.0 --version 2>&1 | head -1
prometheus, version 3.15.0 (branch: HEAD, revision: 5241a27fe3c6983549fccc32f6e65917408c63cd)

Many sockets with file_sd and relabel_configs

For a Prometheus Unix socket setup with several sockets, keep the path as a normal label in a file and copy it to __unix_socket__ with relabeling. The normal label socket also stays visible on every series:

$ cat sockets.json
[
  {"targets": ["node-sock"], "labels": {"socket": "/sock/node.sock", "role": "node"}},
  {"targets": ["app-sock"],  "labels": {"socket": "/sock/app.sock",  "role": "app"}}
]

scrape_configs:
  - job_name: sockets
    file_sd_configs:
      - files: [/cfg/sockets.json]
    relabel_configs:
      - source_labels: [socket]
        target_label: __unix_socket__

After a reload (SIGHUP), the second socket did not exist yet:

job=sockets instance=node-sock health=up
  scrapeUrl=http://node-sock/metrics
  lastError=
job=sockets instance=app-sock health=down
  scrapeUrl=http://app-sock/metrics
  lastError=Get "http://app-sock/metrics": dial unix /sock/app.sock: connect: no such file or directory

Expose your own app metrics on a Unix socket

Any HTTP server that can listen on a socket works as a Prometheus Unix socket target. This Python standard-library server (running as uid 65534) answers /metrics on /sock/app.sock:

import socketserver, http.server, os
class H(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        body = b"# TYPE app_requests_total counter\napp_requests_total{path=\"/\"} 42\n"
        self.send_response(200); self.send_header("Content-Type","text/plain; version=0.0.4"); self.end_headers(); self.wfile.write(body)
    def address_string(self): return "unix"
class S(socketserver.ThreadingMixIn, socketserver.UnixStreamServer): pass
p="/sock/app.sock"
os.path.exists(p) and os.unlink(p)
s=S(p,H); os.chmod(p,0o660); s.serve_forever()

Once it was running, Prometheus picked it up on the next scrape, and the socket and role labels came along:

$ docker logs app 2>&1 | tail -2
unix - - [19/Oct/2026 07:36:14] "GET /metrics HTTP/1.1" 200 -
unix - - [19/Oct/2026 07:36:19] "GET /metrics HTTP/1.1" 200 -

$ query 19015 'app_requests_total'
[{"metric":{"__name__":"app_requests_total","instance":"app-sock","job":"sockets","path":"/","role":"app","socket":"/sock/app.sock"},"value":"42"}]

Prometheus Unix socket errors and fixes

Every Prometheus Unix socket problem shows up in lastError on the Targets page or in /api/v1/targets. Three file-level problems from the lab, a wrong path, a socket file with no process behind it and a socket Prometheus may not open:

job=sockets instance=typo health=down
  scrapeUrl=http://typo/metrics
  lastError=Get "http://typo/metrics": dial unix /sock/nodes.sock: connect: no such file or directory
job=sockets instance=stale health=down
  scrapeUrl=http://stale/metrics
  lastError=Get "http://stale/metrics": dial unix /sock/stale.sock: connect: connection refused

job=sockets instance=node-sock health=down
  scrapeUrl=http://node-sock/metrics
  lastError=Get "http://node-sock/metrics": dial unix /sock/node.sock: connect: permission denied

lastError	Cause	Fix
dial tcp: lookup ... no such host	Prometheus older than 3.15 ignores __unix_socket__	Upgrade to 3.15.0+
dial unix ...: no such file or directory	Typo in the path, exporter not started, or volume not mounted in the Prometheus container	Check the path inside the Prometheus container
dial unix ...: connection refused	Socket file left behind, nothing listening	Restart the exporter; remove stale files on start
dial unix ...: permission denied	Prometheus user (nobody in the image) has no write permission on the socket	Owner/group 65534 or a shared group, mode 660
server gave HTTP response to HTTPS client	scheme: https but the socket speaks plain HTTP	Remove scheme: https
server returned HTTP status 404 Not Found	Wrong metrics_path	Use the path the exporter serves

The Prometheus Unix socket permission error needs a closer look. When the socket was changed to root:root mode 600, Prometheus kept scraping successfully, because it reused the connection that was already open:

$ ls -l /sock/node.sock
srw-------    1 root     root             0 Oct 19 07:35 /sock/node.sock

job=sockets instance=node-sock health=up
  scrapeUrl=http://node-sock/metrics
  lastError=

Only after a Prometheus restart, which needs a new connect(), did the target go down with permission denied (shown above). Restoring owner 65534 and mode 660 brought it back to health=up. So a wrong permission on a new socket may only appear after the next restart. Test with a fresh connection, for example curl --unix-socket run as the Prometheus user.

The https and metrics_path cases came from this job definition:

  - job_name: tls-over-socket
    scheme: https
    static_configs:
      - targets: ["node-sock"]
        labels:
          __unix_socket__: /sock/node.sock
  - job_name: custom-path
    metrics_path: /probe
    ...

job=custom-path instance=node-sock health=down
  scrapeUrl=http://node-sock/probe
  lastError=server returned HTTP status 404 Not Found
job=tls-over-socket instance=node-sock health=down
  scrapeUrl=https://node-sock/metrics
  lastError=Get "https://node-sock/metrics": http: server gave HTTP response to HTTPS client

Docker and Podman notes

• The Prometheus Unix socket must be inside a volume or bind mount that both containers share. A socket in the exporter container's own /tmp is invisible to Prometheus. See Docker volumes for the volume basics.
• A fresh named volume is owned by root. The exporter must be able to create the socket file there (the lab ran chmod 1777 /sock once), and Prometheus needs write permission on the socket itself.
• On SELinux hosts (Fedora, RHEL), add :z to bind mounts of config directories, otherwise Prometheus fails with open /cfg/prometheus.yml: permission denied, which is what this lab got before the flag was added.
• With a Prometheus Unix socket you do not need to publish the exporter port or find the container IP address.

Other changes in 3.15.0: --log.level and GOMEMLIMIT

Two more items from the same release are visible in --help. --log.level is deprecated in favour of runtime.log_level in the configuration file (the docs say this setting can be changed by reloading the configuration), and --auto-gomemlimit.refresh-interval is new:

$ docker run --rm docker.io/prom/prometheus:v3.15.0 --help 2>&1 | grep -A2 -E 'log.level|auto-gomemlimit.refresh'
      --auto-gomemlimit.refresh-interval=0s
                                 Interval at which to re-detect the container
                                 or system memory limit and update GOMEMLIMIT
--
      --log.level=info           Only log messages with the given severity
                                 or above. One of: [debug, info, warn,
                                 error] Deprecated: set runtime.log_level in the
                                 configuration file instead.

Starting 3.15.0 with --log.level=debug did not print a deprecation warning in the log in this lab (docker logs | grep -i deprecat returned nothing), so check your unit files and compose files yourself. The new form:

runtime:
  log_level: debug

For a full Prometheus and Grafana setup, see Prometheus Grafana: a comprehensive guide. To summarise the Prometheus Unix socket feature: upgrade to 3.15.0, set __unix_socket__, give the Prometheus user write access to the socket, and read lastError when a target is down.