skywalking

Download

https://skywalking.apache.org/downloads/

Notice

**Don't download source codes from this page.

Please follow build document, if you want to build source codes by yourself.**

Metrics and Topology from OTLP Traces

Applications instrumented with the OpenTelemetry SDKs no longer need a SkyWalking agent for service dashboards, topology and trace search. OAP stores OTLP traces as they arrive, and the new Trace Analysis Language (TAL) turns every span into service, instance and endpoint metrics, a topology, and dashboards for the databases, caches and message queues the services call. Read OpenTelemetry Traces in SkyWalking for a one-command quick start with the OpenTelemetry Astronomy Shop.

OTLP Service dashboard of the cart service

Topology of the OTLP services

Project

• Update the e2e mock sender's OTLP proto to v1.11.1. A current OpenTelemetry Collector adds fields the old copy lacked, such as a metric's metadata, and the sender's strict JSON parser refused a recording that carries them. The Windows e2e now also replays such a recording, of windows_exporter 0.31.8 through OpenTelemetry Collector 0.158.0 taken on a GitHub-hosted Windows runner, and asserts its memory metrics.
• Replace the GenAI e2e cases' Spring AI application with an in-repo e2e-spring-ai-service module pinned to the released Spring AI 2.0.1, and move the mock LLM endpoints out of e2e-service-provider into a dedicated e2e-mock-llm-server module that all three GenAI cases share. The Spring AI application was previously built at test time by cloning spring-projects/spring-ai-examples and running Maven inside the image build, which resolved spring-ai:2.0.0-SNAPSHOT: a line that has since been abandoned, whose surviving builds all predate the 2.0.0 GA by a day and whose older builds have been pruned, so the fixture was pinned to nothing released and could no longer be rebuilt from the same bytes. Both modules are built by the existing e2e reactor and published to ghcr.io/apache/skywalking alongside the other e2e service images; e2e-spring-ai-service needs Java 17 and Spring Boot 4, so it is gated behind a new jdk-17 profile in the e2e reactor and the jobs that build it now pin JDK 17 explicitly; make -C test docker refuses up front on an older JDK rather than packaging a missing or stale jar, since clean never reaches a module the profile excluded and compose would turn the absent jar into a directory.
• Add the Envoy project's NOTICE to NOTICE and to the binary distribution's NOTICE, for the Envoy proto files SkyWalking bundles in receiver-proto.
• The BanyanDB CA rotation e2e rotates the way an operator does: once the topology shows the consumer's calls to the provider, the OAP trusts the old and the new CA, BanyanDB reloads a certificate the new CA signed, and the OAP then trusts the new CA only, before the storage assertions run. Each step writes its file once, and the OAP's trust file is renamed into place, so the file monitor never reads it half-written. The rotation to an unrelated CA, which proves a rebuilt channel uses the trust file, moves to the end. It used to run first, and when the writes it fails included a network address alias's first write, the alias stayed out of storage until its next refresh five minutes later, so the topology assertions timed out.
• The BanyanDB credentials rotation e2e waits for the topology to show the consumer's calls to the provider before it rotates, as the CA rotation e2e does. The incomplete credentials it applies fail the writes in that window, and when they included a network address alias's first write, the alias stayed out of storage until its next refresh five minutes later, so the topology assertions timed out.
• The BanyanDB credentials and CA rotation e2e cases now also prove that writes go through after the rotation. Their topology assertions use the default range, the last 30 minutes, which rows written before the rotation satisfy too, so they only showed that queries work on the rotated credentials or channel. Each case now keeps the time its last rotation took effect and asserts the provider topology drawn only from the minutes after it.
• Clear the Dependabot CVE alerts in the e2e test fixtures, which no OAP artifact ships. The Go fixtures (cases/go/service and cases/profiling/ebpf/network) move to google.golang.org/grpc 1.79.3 → 1.83.2, golang.org/x/crypto 0.51.0 → 0.55.0 and github.com/klauspost/compress 1.15.9 → 1.18.7, and the Java test services to guava 32.0.0-jre → 33.7.2-jre and log4j 2.25.4 → 2.25.5.

OAP Server

• Add customizable LLM-as-judge support for AI evaluation, with OpenAI-compatible endpoint / model / API key configuration, and persist the evaluation result as queryable GenAIEvaluationRecord rows for later inspection.
• Support hot-reloading the BanyanDB credentials from a new secretsManagementFile setting in bydb.yml (SW_STORAGE_BANYANDB_SECRETS_MANAGEMENT_FILE), matching the ElasticSearch storage plugin. The properties file carrying user/password is watched, so a rotation performed by a 3rd party tool such as Vault is applied without restarting the OAP. The credentials are read per RPC by the gRPC auth interceptor, so they are swapped without re-establishing the channel and without interrupting in-flight queries or writes. Whatever the file contains is applied, including an incomplete pair, so that a mistake in it fails visibly rather than being masked by credentials that silently keep working; a username and a password are only ever sent together, so an incomplete file means requests carry no credentials and are answered with UNAUTHENTICATED rather than authenticating as the named user.
• Support hot-reloading the BanyanDB TLS trust CA. The file at sslTrustCAPath is now watched, and a change rebuilds the gRPC channel so the new CA takes effect without restarting the OAP. Previously the CA was only re-read after the certificate in use had already caused requests to fail, which meant a rotation was paid for with an outage. The replacement channel is created before the old one is released, so a failed rebuild leaves the current channel serving; requests already in flight finish on the old channel. The replacement is not health-checked before the swap — a gRPC channel connects lazily — so rotating to a CA that does not validate the server interrupts traffic until valid material is written back; the storage doc describes the old+new overlap procedure that avoids this. Note the replacement re-picks an address from targets, so the OAP may connect to a different node after a rotation.
• Fix MultipleFilesChangeMonitor being able to silently disable every file watch in the OAP. Its registry of monitors was a plain ArrayList that scanChanges() iterated from the scheduler thread without holding the lock that start() / stop() take, so starting a monitor while a scan was in flight could raise a ConcurrentModificationException from the iterator. That exception escapes past the per-monitor catch, and an uncaught exception cancels a scheduleAtFixedRate task permanently — after which no secrets file, keystore, or TLS certificate is ever reloaded again, with nothing in the log to say so. The registry is now copy-on-write. The failure log in the same scan loop also now names the monitor that failed instead of printing an empty gourp = .
• Fix MultipleFilesChangeMonitor never honouring its watching period. lastCheckTimestamp was declared and compared against, but never assigned, so the guard always measured against 0 and passed — every registered monitor re-stat'd its watched files on each 200ms tick of the shared scheduler thread, and the watchingPeriodInSec constructor argument had no effect at all. This affects every file watch in the OAP: the ElasticSearch storage secrets / truststore / keystore watch, the BanyanDB credentials and trust CA watches, and the TLS certificate watches behind each OAP HTTP and gRPC server, all of which ask for 10 seconds. Change detection is now paced as configured, which also means it is no longer near-instant: a rotated file is picked up within the requested period rather than within ~200ms.
• Add BanyanDB trace tail sampling metrics to the BanyanDB self-observability layer, in a new otel-rules/banyandb/banyandb-trace-sampling.yaml rule file. It covers the whole banyandb_trace_pipeline_* / banyandb_trace_tst_pipeline_* catalog a sampler plugin chain emits — pipeline reconciliation, per-plugin Decide execution rate and latency, chain batching, the trace-level evaluated / retained / dropped / immature outcomes, every fail-open guard and bounded-retention counter, drop-set capacity and finalization state, the plugin telemetry-host safety bounds, and the first-party sw-trace-sampler / zipkin-trace-sampler decision and row metrics. The plugin chain is optional, and the metrics follow it: on a cluster with no sampler configured the wire families are never registered, so every metric here stays absent rather than reading zero. Modeled at Service scope with group kept as a metric label rather than at Endpoint scope, so one cluster-wide page can render per-group series and cluster totals alike — OAP does no cross-scope rollup, so an Endpoint-scope metric could not have been aggregated back up to the cluster.
• Fix a second CounterWindow key collision in the v2 MAL engine, this time ACROSS rules. rate() / increase() / irate() resolve their lower bound from a process-wide window keyed on the counter's own name plus its post-.sum(...) label set, with nothing identifying the rule doing the evaluation. Two rules that read one wire family, tell their streams apart with tagEqual(...), and then .sum(...) away the label they filtered on therefore collapse onto one window slot and difference against each other's values. The queue is ordered by (timestamp, value), so the smaller counter wins the lower-bound lookup and still reads correctly while its partner is inflated by the gap between them — which is why this went unnoticed. A collision needs the discriminating label to be DROPPED by the .sum(...): where it survives, the rules' label values differ and the window keeps them apart. Auditing the shipped rules on that basis gives 10 colliding keys over ~25 rules — meter_activemq_cluster_gc_parallel_young_collection_count reported ~9000/min of young-gen collections from a completely idle broker (differencing against the old-gen counter); MySQL commands_* / tps rate against each other; so do the GenAI gateway input/output token rates, four Envoy cluster_* counters, APISIX matched/unmatched instance bandwidth, and BanyanDB's own network_recv / network_sent, which drop the kind label that separates bytes-received from bytes-sent on one interface. Measured against two live scrapes of the demo cluster's FODC proxy, that last pair was wrong on every interface: network_sent read a flat 0 B/s and network_recv read large negative values (down to -778 MB/s) from differencing against the sent counter, where both now match the byte delta exactly. No rule changes were needed for any of these -- each rule already reduces to the labels it should; only the window key was wrong. The window is now keyed by (owning rule, counter name, labels). This is the complement of the within-rule collision fixed earlier by keying on the counter's own name: neither name alone is sufficient, because the two collisions are independent. RunningContext.metricName — written on every rule evaluation and read by nobody since that earlier fix — is what supplies the rule identity, so no code generation or MAL syntax changes. Note the whole-rule-set comparison suite could not have caught this: it resets the shared window before every rule, the one condition under which the collision cannot appear.
• Fix meter_rabbitmq_node_outgoing_messages_total double-counting one of its terms. The rule summed six delivery-rate terms but rabbitmq_global_messages_delivered_get_auto_ack_total appeared twice, so auto-ack basic.get deliveries were counted once more than the other four delivery paths and the reported outgoing rate ran high whenever polling consumers were in use. The duplicate term is removed, leaving the five distinct families (redelivered, consume auto/manual ack, get auto/manual ack).
• Add AI agent conversations landed by the AI Sessionizer: the AI_AGENT layer, the bundled lal/ai-agent.yaml rule with the ConversationFile output builder that verifies and stores Session Data and Session Flow files, the ai_agent_session_data and ai_agent_session_flow models in a new BanyanDB group recordsAIAgent, the ai-agent-conversation module that folds a conversation into one asz.view document, the listConversations GraphQL query (with optional conversation and title conditions), and two streamed HTTP routes beside /graphql: GET /ai-agent/conversations/{conversation}/v1/view, which serves the document, and GET /ai-agent/conversations/{conversation}/v1/files, which serves up to 32 Session Data files of a session, each chosen by its landed seq, as application/vnd.skywalking.asz.files+ndjson, a line naming each file followed by the file's bytes, so a page loads the files a step points at only when a reader opens it; there is no read of every file. Both routes require the service and the sender's instance, as a list row names them, so every read is a full series lookup. The view and the files route leave BanyanDB's stage selection unspecified by default and select only the cold stage when the caller explicitly sets coldStage to true. A round from before the list attributes existed lands and lists with zero counts, and the view shows as much as landed: the chain resumes after a missing, unreadable or refused round, and the absent rounds and files are named once as ranges. A file over maxFileBytes, 15 MiB by default, is rejected at ingest and counted under the reason size, because one file over BanyanDB's 16 MiB gRPC message limit fails the bulk write it travels in and every record behind it; on MySQL the body column is LONGTEXT, since a body that size outgrows MEDIUMTEXT as Base64. A conversation's rounds and files are read readWindow at a time, 16 by default, and each of those reads is capped at maxResponseBytes, 100 MiB by default, as a per-call option on the BanyanDB client in place of its 50 MB default, so the module's reads are bounded by its own settings and nothing else's read changes. The document also carries the session's workspace changes: the Sessionizer's Claude Code plugin lands a Session Data file of kind changes beside a stream's transcript, streams//changes--.sd, with one change record per observed tool call, stored like any other file, and the runtime's own patch for its editing tools travels as the second data part of the call's result record; workspace_changes lists every record joined to its step by tool-use id with the record's fields as changes/1 lists them, summary.changes counts them, and a tool step names its records under changes. The document also names the session's provider bodies: the request and response bodies a runtime exchanged with its model provider land as Session Data files of kind provider_body, /provider_body/provider_body--.sd, stored like any other file; an llm.call step lists where its request and response landed under provider_bodies, as the round carries the join the Sessionizer made when it parsed the round, and summary.provider_bodies and summary.captured_prompts count them. A round's header counts, changes, lines_added, lines_removed, llm_calls, subagents and bash_runs, land on ai_agent_session_flow and reach the conversation list as ConversationRow fields, absent rather than zero when the round did not carry them. The layer also takes the agent runtime's metrics, in the otel-rules/ai-agent/ rule set enabled by default: agent.token.usage, which the Sessionizer derives from the landed transcripts, per service and per sender, by type, model and query source with session.id summed away, plus the cost, active time, sessions, lines of code, commits, pull requests and edit decisions that Claude Code's own exporter sends when it is pointed at the OAP directly; its own token metric is not read, so a call is never counted twice.
• Support querying Zipkin traces from the BanyanDB cold stage through the Zipkin HTTP query API. /api/v2/traces takes an optional coldStage parameter, and /api/v2/trace/{traceId} and /api/v2/traceMany take optional coldStage, endTs and lookback parameters; the two by-id lookups had no time range at all before. All of them are SkyWalking additions to the Zipkin API and default to the previous behavior, so existing Zipkin clients such as the Lens UI keep working unchanged, and storages other than BanyanDB ignore coldStage. The admin debugging endpoints /debugging/query/zipkin/api/v2/traces and /debugging/query/zipkin/api/v2/trace accept the same parameters. The BanyanDB stages e2e now seeds Zipkin traces into the generated cold data and verifies the three endpoints against it.
• BanyanDB: a query without a time range now covers everything the group's hot/warm stages retain instead of only the last 24 hours. The hard-coded day was a fallback added with the trace model, which requires a time range; it made queryTrace(traceId) without a duration, the Zipkin /api/v2/trace/{traceId} and /api/v2/traceMany lookups, and the TraceQL by-id lookup return "not found" for any trace older than a day even though it was still retained. The unbounded range is now bound instead, and BanyanDB's own retention limits the scan. queryTraces now rejects a condition with neither queryDuration nor traceId instead of silently searching the last day; a traceId lookup without queryDuration searches everything the hot/warm stages retain and reports a 0-to-now retrievedTimeRange.
• Fix the JDBC storages, MySQL, PostgreSQL and H2, reading only the first day's table of a time range shorter than a day that crosses midnight. TableHelper walked the range in 24-hour steps from its first instant, so a range that began late on one day and ended early on the next never reached the second day; every time-ranged read, metrics, traces, logs, alarms, records and the AI agent conversation list, was missing the newer day's rows for the length of the range after each midnight, the last thirty minutes read at 00:10 among them. The range is now walked by calendar day from the start of its first day.
• Fix a minute of metrics on BanyanDB losing the points of an earlier persistence round. PersistenceTimer counted the delay to the next round from the moment a round's writes were queued, not answered: the per-worker future completed when batchDAO.flush returned, the flush's own future was used only to close a latency histogram, and endOfFlush ran after that. A metric enters the persistence session cache only when the storage acknowledges its write, and the BanyanDB bulk processor holds a queued write for up to flushInterval (15s) before sending it, so under the 25s persistentPeriod an acknowledgement that lagged ten seconds, as it can on a BanyanDB still busy with the boot's schema installs, let the next round find the metric absent from the cache, take the minute for a new one and write it again with only its own points; a measure write replaces the series at that time, so the earlier points we