Writing Custom Collators
You can create your own collator to index any data source and make it searchable in Backstage. The recommended approach is to use the built-in template to scaffold a collator module, then implement your data fetching logic.
Scaffolding a collator module
Run the following command from your Backstage root directory:
yarn new --select search-collator-module
You are prompted for a module ID, which is used to name the package and the
generated collator class. For this example, enter blog-posts.
The template creates a new package at
plugins/search-backend-module-blog-posts/ with the following structure and
automatically adds the module to your backend:
backend.add(import('@internal/plugin-search-backend-module-blog-posts'));
The generated package contains these files:
plugins/search-backend-module-blog-posts/
├── config.d.ts
├── package.json
└── src/
├── collator/
│ ├── BlogPostsCollatorFactory.test.ts
│ └── BlogPostsCollatorFactory.ts
├── index.ts
└── module.ts
Understanding the generated code
The template generates two files worth noting: the backend module that wires the collator into the search system, and the collator factory that fetches and yields documents.
The backend module
The src/module.ts file creates a backend module that registers the collator
with the search index. It reads an optional schedule from configuration and
falls back to a default schedule of every 10 minutes:
import {
coreServices,
createBackendModule,
readSchedulerServiceTaskScheduleDefinitionFromConfig,
} from '@backstage/backend-plugin-api';
import { searchIndexRegistryExtensionPoint } from '@backstage/plugin-search-backend-node/alpha';
import { BlogPostsCollatorFactory } from './collator/BlogPostsCollatorFactory';
const DEFAULT_SCHEDULE = {
frequency: { minutes: 10 },
timeout: { minutes: 15 },
initialDelay: { seconds: 3 },
};
export const searchModuleBlogPosts = createBackendModule({
pluginId: 'search',
moduleId: 'blog-posts-collator',
register({ registerInit }) {
registerInit({
deps: {
config: coreServices.rootConfig,
logger: coreServices.logger,
scheduler: coreServices.scheduler,
indexRegistry: searchIndexRegistryExtensionPoint,
},
async init({ config, logger, scheduler, indexRegistry }) {
const scheduleConfig = config
.getOptionalConfig('search.collators.blogPosts')
?.getOptionalConfig('schedule');
const schedule = scheduleConfig
? readSchedulerServiceTaskScheduleDefinitionFromConfig(scheduleConfig)
: DEFAULT_SCHEDULE;
indexRegistry.addCollator({
schedule: scheduler.createScheduledTaskRunner(schedule),
factory: BlogPostsCollatorFactory.fromConfig(config, { logger }),
});
},
});
},
});
The collator factory
The src/collator/BlogPostsCollatorFactory.ts file implements the
DocumentCollatorFactory interface. The execute() method is an async
generator that yields IndexableDocument objects. Each document must include
title, text, and location fields:
import { LoggerService } from '@backstage/backend-plugin-api';
import { Config } from '@backstage/config';
import {
DocumentCollatorFactory,
IndexableDocument,
} from '@backstage/plugin-search-common';
import { Readable } from 'node:stream';
export type BlogPostsCollatorFactoryOptions = {
logger: LoggerService;
};
export class BlogPostsCollatorFactory implements DocumentCollatorFactory {
public readonly type = 'blog-posts';
private readonly logger: LoggerService;
static fromConfig(
_config: Config,
options: BlogPostsCollatorFactoryOptions,
): BlogPostsCollatorFactory {
return new BlogPostsCollatorFactory(options);
}
private constructor(options: BlogPostsCollatorFactoryOptions) {
this.logger = options.logger;
}
async getCollator(): Promise<Readable> {
return Readable.from(this.execute());
}
private async *execute(): AsyncGenerator<IndexableDocument> {
this.logger.info('Collating documents for blog-posts');
// TODO: Replace with your data fetching logic
yield* [];
}
}
Implementing the collator
To make the collator useful, replace the placeholder execute() method with
your data fetching logic. Each yielded object must include at minimum a
title, text, and location.
The following example fetches blog posts from an internal API:
import { LoggerService } from '@backstage/backend-plugin-api';
import { Config } from '@backstage/config';
import {
DocumentCollatorFactory,
IndexableDocument,
} from '@backstage/plugin-search-common';
import { Readable } from 'node:stream';
type BlogPost = {
id: string;
title: string;
body: string;
author: string;
};
export type BlogPostsCollatorFactoryOptions = {
logger: LoggerService;
};
export class BlogPostsCollatorFactory implements DocumentCollatorFactory {
public readonly type = 'blog-posts';
private readonly baseUrl: string;
private readonly logger: LoggerService;
static fromConfig(
config: Config,
options: BlogPostsCollatorFactoryOptions,
): BlogPostsCollatorFactory {
const baseUrl = config.getString('blogPosts.baseUrl');
return new BlogPostsCollatorFactory(baseUrl, options);
}
private constructor(
baseUrl: string,
options: BlogPostsCollatorFactoryOptions,
) {
this.baseUrl = baseUrl;
this.logger = options.logger;
}
async getCollator(): Promise<Readable> {
return Readable.from(this.execute());
}
private async *execute(): AsyncGenerator<IndexableDocument> {
this.logger.info('Collating documents for blog-posts');
const response = await fetch(`${this.baseUrl}/blog-posts`);
const posts: BlogPost[] = await response.json();
for (const post of posts) {
yield {
title: post.title,
text: post.body,
location: `/blog-posts/${post.id}`,
};
}
}
}
For large data sets, use cursor-based pagination in the execute() method to
avoid loading all records into memory at once:
private async *execute(): AsyncGenerator<IndexableDocument> {
let cursor: string | undefined = undefined;
do {
const url = cursor
? `${this.baseUrl}/blog-posts?cursor=${cursor}`
: `${this.baseUrl}/blog-posts`;
const response = await fetch(url);
const { items, nextCursor } = await response.json();
for (const item of items) {
yield {
title: item.title,
text: item.body,
location: `/blog-posts/${item.id}`,
};
}
cursor = nextCursor;
} while (cursor);
}
Testing the collator
The generated test file at
src/collator/BlogPostsCollatorFactory.test.ts uses TestPipeline from
@backstage/plugin-search-backend-node to run the collator and verify its
output. Update the tests to match your implementation:
import { BlogPostsCollatorFactory } from './BlogPostsCollatorFactory';
import { mockServices } from '@backstage/backend-test-utils';
import { TestPipeline } from '@backstage/plugin-search-backend-node';
const mockPosts = [
{
id: '1',
title: 'Getting Started',
body: 'Welcome to our engineering blog',
author: 'Alice',
},
{
id: '2',
title: 'Best Practices',
body: 'Tips for writing great code',
author: 'Bob',
},
];
describe('BlogPostsCollatorFactory', () => {
beforeEach(() => {
global.fetch = jest.fn().mockResolvedValue({
json: async () => mockPosts,
});
});
it('returns a collator with the correct type', async () => {
const factory = BlogPostsCollatorFactory.fromConfig(
mockServices.rootConfig({
data: { blogPosts: { baseUrl: 'http://localhost' } },
}),
{ logger: mockServices.logger.mock() },
);
expect(factory.type).toBe('blog-posts');
});
it('runs the collator and returns documents', async () => {
const factory = BlogPostsCollatorFactory.fromConfig(
mockServices.rootConfig({
data: { blogPosts: { baseUrl: 'http://localhost' } },
}),
{ logger: mockServices.logger.mock() },
);
const collator = await factory.getCollator();
const { error, documents } = await TestPipeline.fromCollator(
collator,
).execute();
expect(error).toBeUndefined();
expect(documents).toHaveLength(2);
expect(documents[0]).toMatchObject({
title: 'Getting Started',
text: 'Welcome to our engineering blog',
location: '/blog-posts/1',
});
});
});
Configuring the schedule
The generated module reads an optional schedule from app-config.yaml.
Without configuration, the collator runs every 10 minutes. To customize the
schedule:
search:
collators:
blogPosts:
schedule: # same options as in SchedulerServiceTaskScheduleDefinition
# supports cron, ISO duration, "human duration" as used in code
initialDelay: { seconds: 90 }
# supports cron, ISO duration, "human duration" as used in code
frequency: { hours: 6 }
# supports ISO duration, "human duration" as used in code
timeout: { minutes: 3 }
Customizing search result presentation
Search results from a custom collator appear automatically using the default result list item. To customize how results are presented, create a frontend module that extends the search plugin with a custom result list item.
Scaffolding a frontend module
Run the following command from your Backstage root directory:
yarn new --select frontend-plugin-module
When prompted, enter search for the plugin ID and blog-posts for the
module ID. The template creates a new package at
plugins/search-module-blog-posts/ and adds it as a dependency in
packages/app/package.json. The new frontend system auto-discovers the
module from there.
Creating the result list item component
Add a component that renders a single search result. Each result includes
title, text, and location fields from the collator:
import { Link } from '@backstage/core-components';
import ListItemIcon from '@material-ui/core/ListItemIcon';
import ListItemText from '@material-ui/core/ListItemText';
import { SearchDocument } from '@backstage/plugin-search-common';
import { ReactNode } from 'react';
export interface BlogPostSearchResultListItemProps {
icon?: ReactNode;
result?: SearchDocument;
rank?: number;
}
export function BlogPostSearchResultListItem(
props: BlogPostSearchResultListItemProps,
) {
const { icon, result } = props;
if (!result) return null;
return (
<>
{icon && <ListItemIcon>{icon}</ListItemIcon>}
<ListItemText
primaryTypographyProps={{ variant: 'h6' }}
primary={
<Link noTrack to={result.location}>
{result.title}
</Link>
}
secondary={result.text}
/>
</>
);
}
Registering the result list item
Update the generated src/module.tsx to register a
SearchResultListItemBlueprint with a predicate that matches results from
your collator. The predicate checks the result type field, which must
match the type property set in your collator factory — in this example,
blog-posts:
import { createFrontendModule } from '@backstage/frontend-plugin-api';
import { SearchResultListItemBlueprint } from '@backstage/plugin-search-react/alpha';
const blogPostSearchResultListItem = SearchResultListItemBlueprint.make({
name: 'blog-posts',
params: {
predicate: result => result.type === 'blog-posts',
component: () =>
import('./components/BlogPostSearchResultListItem').then(
m => m.BlogPostSearchResultListItem,
),
},
});
export const searchModuleBlogPosts = createFrontendModule({
pluginId: 'search',
extensions: [blogPostSearchResultListItem],
});