Setup OpenTelemetry
Backstage uses OpenTelemetery to instrument its components by reporting traces and metrics.
This tutorial shows how to setup exporters in your Backstage backend package. For demonstration purposes we will use a Prometheus exporter, but you can adjust your solution to use another one that suits your needs; see for example the article on OTLP exporters.
Install dependencies
We will use the OpenTelemetry Node SDK and the auto-instrumentations-node
packages.
Backstage packages, such as the catalog, use the OpenTelemetry API to send custom traces and metrics.
The auto-instrumentations-node
will automatically create spans for code called in libraries like Express.
yarn --cwd packages/backend add \
@opentelemetry/sdk-node \
@opentelemetry/auto-instrumentations-node \
@opentelemetry/exporter-prometheus
Configure
In your packages/backend/src
folder, create an instrumentation.js
file.
const { NodeSDK } = require('@opentelemetry/sdk-node');
const {
getNodeAutoInstrumentations,
} = require('@opentelemetry/auto-instrumentations-node');
const { PrometheusExporter } = require('@opentelemetry/exporter-prometheus');
// By default exports the metrics on localhost:9464/metrics
const prometheus = new PrometheusExporter();
const sdk = new NodeSDK({
// You can add a traceExporter field here too
metricReader: prometheus,
instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();
You probably won't need all of the instrumentation inside getNodeAutoInstrumentations()
so make sure to
check the documentation and tweak it properly.
Local Development Setup
It's important to setup the NodeSDK and the automatic instrumentation before
importing any library. This is why we will use the nodejs
--require
flag when we
start up the application.
For local development, you can add the required flag in your packages/backend/package.json
.
"scripts": {
"start": "backstage-cli package start --require ./src/instrumentation.js",
...
You can now start your Backstage instance as usual, using yarn dev
.
Production Setup
In your .dockerignore
, add this line:
!packages/backend/src/instrumentation.js
This ensures that Docker build will not ignore the instrumentation file if you are following the recommended .dockerignore
setup.
In your Dockerfile
, copy instrumentation.js
file into the root of the working directory.
COPY packages/backend/src/instrumentation.js ./
And then add the --require
flag that points to the file to the CMD array.
CMD ["node", "packages/backend", "--config", "app-config.yaml"]
CMD ["node", "--require", "./instrumentation.js", "packages/backend", "--config", "app-config.yaml"]
If you need to disable/configure some OpenTelemetry feature there are lots of environment variables which you can tweak.
Available Metrics
The following metrics are available:
catalog_entities_count
: Total amount of entities in the catalogcatalog_registered_locations_count
: Total amount of registered locations in the catalogcatalog_relations_count
: Total amount of relations between entitiescatalog.processed.entities.count
: Amount of entities processedcatalog.processing.duration
: Time spent executing the full processing flowcatalog.processors.duration
: Time spent executing catalog processorscatalog.processing.queue.delay
: The amount of delay between being scheduled for processing, and the start of actually being processedcatalog.stitched.entities.count
: Amount of entities stitchedcatalog.stitching.duration
: Time spent executing the full stitching flowcatalog.stitching.queue.length
: Number of entities currently in the stitching queuecatalog.stitching.queue.delay
: The amount of delay between being scheduled for stitching, and the start of actually being stitchedscaffolder.task.count
: Count of task runsscaffolder.task.duration
: Duration of a task runscaffolder.step.count
: Count of step runsscaffolder.step.duration
: Duration of a step runsbackend_tasks.task.runs.count
: Total number of times a task has been runbackend_tasks.task.runs.duration
: Histogram of task run durations