Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 36 additions & 18 deletions src/content/docs/en/guides/content-collections.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -166,8 +166,8 @@ The [`glob()` loader](/en/reference/content-loader-reference/#glob-loader) fetch
This loader accepts a `pattern` of entry files to match using glob patterns supported by [micromatch](https://github.com/micromatch/micromatch#matching-features), and a base file path of where your files are located. Each entry's `id` will be automatically generated from its file name.

```ts title="src/content.config.ts" {5}
import { defineCollection, z } from 'astro:content';
import { glob, file } from 'astro/loaders';
import { defineCollection } from 'astro:content';
import { glob } from 'astro/loaders';

const blog = defineCollection({
loader: glob({ pattern: "**/*.md", base: "./src/data/blog" }),
Expand All @@ -183,7 +183,7 @@ The [`file()` loader](/en/reference/content-loader-reference/#file-loader) fetch
It accepts a `base` file path to your file and optionally a [`parser` function](#parser-function) for data files it cannot parse automatically. Use this loader when your data file can be parsed as an array of objects.

```ts title="src/content.config.ts" {5}
import { defineCollection, z } from 'astro:content';
import { defineCollection } from 'astro:content';
import { file } from 'astro/loaders';

const dogs = defineCollection({
Expand All @@ -201,13 +201,15 @@ The `file()` loader will automatically detect and parse (based on their file ext

The following example shows importing a CSV parser, then loading a `cats` collection into your project by passing both a file path and `parser` function to the `file()` loader:

```typescript title="src/content.config.ts" {3} "{ parser: (text) => parseCsv(text, { columns: true, skipEmptyLines: true })}"
```typescript title="src/content.config.ts" {3} "parser: (text) => parseCsv(text, { columns: true, skipEmptyLines: true })"
import { defineCollection } from "astro:content";
import { file } from "astro/loaders";
import { parse as parseCsv } from "csv-parse/sync";

const cats = defineCollection({
loader: file("src/data/cats.csv", { parser: (text) => parseCsv(text, { columns: true, skipEmptyLines: true })})
loader: file("src/data/cats.csv", {
parser: (text) => parseCsv(text, { columns: true, skipEmptyLines: true }),
}),
});
```

Expand All @@ -222,6 +224,9 @@ The `parser` argument also allows you to load a single collection from a nested
You can separate these collections by passing a custom `parser` to the `file()` loader for each collection:

```typescript title="src/content.config.ts"
import { file } from "astro/loaders";
import { defineCollection } from "astro:content";

const dogs = defineCollection({
loader: file("src/data/pets.json", { parser: (text) => JSON.parse(text).dogs })
});
Expand All @@ -247,6 +252,8 @@ You can define a loader inline, inside your collection, as an async function tha
This is useful for loaders that don't need to manually control how the data is loaded and stored. Whenever the loader is called, it will clear the store and reload all the entries.

```ts title="src/content.config.ts"
import { defineCollection } from "astro:content";

const countries = defineCollection({
loader: async () => {
const response = await fetch("https://restcountries.com/v3.1/all");
Expand Down Expand Up @@ -740,8 +747,9 @@ Astro will generate some errors itself, depending on the response from the live
- If a schema is defined for the collection and the data does not match the schema, Astro will return a `LiveCollectionValidationError`.
- If the loader returns an invalid cache hint, Astro will return a `LiveCollectionCacheHintError`. The `cacheHint` field is optional, so if you do not have valid data to return, you can simply omit it.

```ts title="my-loader.ts" {9-11}
```ts title="my-loader.ts" {10-12}
import type { LiveLoader } from 'astro/loaders';
import type { MyData } from "./types";
import { MyLoaderError } from './errors.js';

export function myLoader(config): LiveLoader<MyData, undefined, undefined, MyLoaderError> {
Expand Down Expand Up @@ -861,6 +869,9 @@ export function productLoader(config: {
You can create custom error types for [errors returned by your loader](#error-handling-in-live-loaders) and pass them as a generic to get proper typing:

```ts title="my-loader.ts"
import type { LiveLoader } from "astro/loaders";
import type { MyData } from "./types"

class MyLoaderError extends Error {
constructor(
message: string,
Expand Down Expand Up @@ -911,13 +922,18 @@ if (error) {
Live loaders can provide cache hints to help with response caching. You can use this data to send HTTP cache headers or otherwise inform your caching strategy.

```ts title="my-loader.ts"
import type { LiveLoader } from "astro/loaders";
import { byLastModified } from "./sort";
import { loadStoreProduct, loadStoreProducts } from "./store";
import type { MyData } from "./types";

export function myLoader(config): LiveLoader<MyData> {
return {
name: 'cached-loader',
loadCollection: async ({ filter }) => {
// ... fetch data
const products = await loadStoreProducts(filter);
return {
entries: data.map((item) => ({
entries: products.map((item) => ({
id: item.id,
data: item,
// You can optionally provide cache hints for each entry
Expand All @@ -930,14 +946,14 @@ export function myLoader(config): LiveLoader<MyData> {
// tags are merged from all entries
// maxAge is the shortest maxAge of all entries and the collection
// lastModified is the most recent lastModified of all entries and the collection
lastModified: new Date(item.lastModified),
lastModified: new Date(products.sort(byLastModified)[0].lastModified),
tags: ['products'],
maxAge: 300, // 5 minutes
},
};
},
loadEntry: async ({ filter }) => {
// ... fetch single item
const item = await loadStoreProduct(filter);
return {
id: item.id,
data: item,
Expand All @@ -954,7 +970,7 @@ export function myLoader(config): LiveLoader<MyData> {

You can then use these hints in your pages:

```astro
```astro title="src/pages/store/[id].astro"
---
export const prerender = false; // Not needed in 'server' mode
Expand Down Expand Up @@ -1026,11 +1042,12 @@ export const collections = { products };

When using Zod schemas with live collections, validation errors are automatically caught and returned as `AstroError` objects:

```astro
```astro title="src/pages/store/index.astro"
---
export const prerender = false; // Not needed in 'server' mode
import { getLiveEntry, LiveCollectionValidationError } from 'astro:content';
import { LiveCollectionValidationError } from 'astro/content/runtime';
import { getLiveEntry } from 'astro:content';
const { entry, error } = await getLiveEntry('products', '123');
Expand Down Expand Up @@ -1059,7 +1076,7 @@ These return entries with a unique `id`, and `data` object with all defined prop

You can use these functions to access your live data, passing the name of the collection and optionally filtering conditions.

```astro
```astro title="src/pages/store/[slug].astro"
---
export const prerender = false; // Not needed in 'server' mode
Expand All @@ -1082,7 +1099,7 @@ If your live loader [returns a `rendered` property](#providing-rendered-content)

You also have access to any [error returned by the live loader](#error-handling-in-live-loaders), for example, to rewrite to a 404 page when content cannot be displayed:

```astro "render(entry)" "<Content />"
```astro title="src/pages/store/[id].astro" "render(entry)" "<Content />"
---
export const prerender = false; // Not needed in 'server' mode
Expand Down Expand Up @@ -1113,11 +1130,12 @@ When you call `getLiveCollection()` or `getLiveEntry()`, the error will be one o

These errors have a static `is()` method that you can use to check the type of error at runtime:

```astro "LiveEntryNotFoundError.is(error)"
```astro title="src/pages/store/[id].astro" "LiveEntryNotFoundError.is(error)"
---
export const prerender = false; // Not needed in 'server' mode
import { getLiveEntry, LiveEntryNotFoundError } from 'astro:content';
import { LiveEntryNotFoundError } from 'astro/content/runtime';
import { getLiveEntry } from 'astro:content';
const { entry, error } = await getLiveEntry('products', Astro.params.id);
Expand Down Expand Up @@ -1267,4 +1285,4 @@ In the following example, all YAML files in the `src/data/authors/` directory wi
}
```

See [“Associating schemas”](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml#associating-schemas) in the Red Hat YAML extension documentation for more details.
See [“Associating schemas”](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml#associating-schemas) in the Red Hat YAML extension documentation for more details.
53 changes: 28 additions & 25 deletions src/content/docs/en/reference/content-loader-reference.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,16 @@ Astro's Content Loader API allows you to load your data from any source, local

## What is a loader?

Astro loaders allow you to load data into [content collections](/en/guides/content-collections/), which can then be used in pages and components. The [built-in `glob()` and `file()` loaders](/en/guides/content-collections/#build-time-collection-loaders) are used to load content from the file system, and you can create your own loaders to load content from other sources.
Astro loaders allow you to load data into [content collections](/en/guides/content-collections/), which can then be used in pages and components. The [built-in `glob()` and `file()` loaders](#built-in-loaders) are used at built time to load content from the file system. You can also create your own [built-time loaders](/en/guides/content-collections/#custom-build-time-loaders) or [live loaders](/en/guides/content-collections/#creating-a-live-loader) to load content from other sources.

Each collection needs a loader defined in its schema. You can define a loader inline in your project's `src/content.config.ts` file, share one loader between multiple collections, or even [publish your loader to NPM as a package](/en/reference/publish-to-npm/) to share with others and be included in our integrations library.
Each collection needs to define a loader and an optional schema. For build-time collections, you can define a loader inline in your project's `src/content.config.ts` file or share an object loader between multiple collections. For live collections, you'll need to create your own live loader and import it in your `src/live.config.ts` file. In both cases, you can even [publish your loader to NPM as a package](/en/reference/publish-to-npm/) to share with others and be included in our integrations library.

## Built-in loaders

Astro provides two built-in loaders to help you fetch your collections. Both offer options to suit a wide range of use cases.

<ReadMore>Learn more about [using built-time loaders](/en/guides/content-collections/#build-time-content-collections) in the “Content Collections” guide.</ReadMore>

### `glob()` loader

<p>
Expand All @@ -31,19 +33,17 @@ The `glob()` loader creates entries from directories of files from anywhere on t

This loader accepts an object with the following properties: `pattern`, `base` (optional), and `generateId` (optional).

```ts title="src/content.config.ts" {2,6,11,17-21}
```ts title="src/content.config.ts" {2,6,10,15-19}
import { defineCollection } from 'astro:content';
import { glob } from 'astro/loaders';

const pages = defineCollection({
/* Retrieve all Markdown files in your pages directory. */
loader: glob({ pattern: "**/*.md", base: "./src/data/pages" }),
schema: /* ... */
});
const blog = defineCollection({
/* Retrieve all Markdown and MDX files in your blog directory. */
loader: glob({ pattern: "**/*.(md|mdx)", base: "./src/data/blog" }),
schema: /* ... */
});
const authors = defineCollection({
/* Retrieve all JSON files in your authors directory while retaining
Expand All @@ -53,7 +53,6 @@ const authors = defineCollection({
base: "./src/data/authors",
generateId: ({ entry }) => entry.replace(/\.json$/, ''),
}),
schema: /* ... */
});
```

Expand Down Expand Up @@ -104,21 +103,19 @@ The `file()` loader creates entries from a single file that contains an array of

This loader accepts a `fileName` property and an optional object as second argument:

```ts title="src/content.config.ts" {2,6,11-13}
```ts title="src/content.config.ts" {2,6,10-12}
import { defineCollection } from 'astro:content';
import { file } from 'astro/loaders';

const authors = defineCollection({
/* Retrieve all entries from a JSON file. */
loader: file("src/data/authors.json"),
schema: /* ... */
});
const products = defineCollection({
/* Retrieve all entries from a CSV file using a custom parser. */
loader: file("src/data/products.csv", {
parser: (fileContent) => { /* your parser logic */ },
}),
schema: /* ... */
});
```

Expand Down Expand Up @@ -160,6 +157,8 @@ An inline loader is an async function that returns an array or object containing
The function can be async and must return either an array of entries that each contain a unique `id` field, or an object where each key is a unique ID and each value is the entry. Whenever the loader is invoked, it will clear the store and reload all the entries.

```ts title="src/content.config.ts"
import { defineCollection } from "astro:content";

const countries = defineCollection({
loader: async () => {
const response = await fetch("https://restcountries.com/v3.1/all");
Expand All @@ -171,7 +170,6 @@ const countries = defineCollection({
...country,
}));
},
schema: /* ... */
});
```

Expand All @@ -182,7 +180,7 @@ A loader is an object with a [`load()` method](#load) that is called at build ti
The recommended pattern is to define a function that accepts configuration options and returns the loader object, in the same way that you would normally define an Astro integration or Vite plugin.


```ts title=loader.ts
```ts title="src/loader.ts"
import type { Loader, LoaderContext } from 'astro/loaders';
import { z } from 'astro:content';
import { loadFeedData } from "./feed.js";
Expand Down Expand Up @@ -211,16 +209,15 @@ export function myLoader(options: { url: string, apiKey: string }): Loader {
These configuration options can then be set when defining a collection:

```ts title="src/content.config.ts" {2,5-8}
import { defineCollection, z } from 'astro:content';
import myLoader from '../../loader.ts';
import { defineCollection } from 'astro:content';
import { myLoader } from './loader.ts';

const blog = defineCollection({
const blog = defineCollection({
loader: myLoader({
url: "https://api.example.com/posts",
apiKey: "my-secret",
}),
schema: /* ... */
});
}),
});
```

### Live loaders
Expand Down Expand Up @@ -290,7 +287,7 @@ export function articleLoader(config: { apiKey: string }): LiveLoader<Article> {
}
```

<ReadMore>[See the `Content Collection` guide](/en/guides/content-collections/#creating-a-live-loader) for more information about creating a live loader and example usage.</ReadMore>
<ReadMore>See [how to create a live loader](/en/guides/content-collections/#creating-a-live-loader) with example usage.</ReadMore>

## Object loader API

Expand Down Expand Up @@ -393,7 +390,7 @@ The full, resolved Astro configuration object with all defaults applied. See [th

Validates and parses the data according to the collection schema. Pass data to this function to validate and parse it before storing it in the data store.

```ts title=loader.ts {14-17}
```ts title=loader.ts {15-18}
import type { Loader } from "astro/loaders";
import { loadFeed } from "./feed.js";

Expand All @@ -407,8 +404,9 @@ export function feedLoader({ url }): Loader {
store.clear();

for (const item of feed.items) {
const id = item.guid;
const data = await parseData({
id: item.guid,
id,
data: item,
});
store.set({
Expand Down Expand Up @@ -469,7 +467,7 @@ export function myLoader(settings): Loader {

Generates a non-cryptographic content digest of an object or string. This can be used to track if the data has changed by setting [the `digest` field](#digest) of an entry.

```ts title=loader.ts {19,24}
```ts title=loader.ts {20,25}
import type { Loader } from "astro/loaders";
import { loadFeed } from "./feed.js";

Expand All @@ -483,8 +481,9 @@ export function feedLoader({ url }): Loader {
store.clear();

for (const item of feed.items) {
const id = item.guid;
const data = await parseData({
id: item.guid,
id,
data: item,
});

Expand Down Expand Up @@ -537,7 +536,10 @@ return {

If the loader has been triggered by an integration, this may optionally contain extra data set by that integration. It is only set when the loader is triggered by an integration. See the [`astro:server:setup`](/en/reference/integrations-reference/#refreshcontent-option) hook reference for more information.

```ts title=loader.ts {5-8}
```ts title=loader.ts {8-11}
import type { Loader } from "astro/loaders";
import { processWebhook } from "./lib/webhooks";

export function myLoader(options: { url: string }): Loader {
return {
name: "my-loader",
Expand Down Expand Up @@ -580,10 +582,11 @@ The returned object is a [`DataEntry`](#dataentry) object.

Used after data has been [validated and parsed](#parsedata) to add an entry to the store, returning `true` if the entry was set. This returns `false` when the [`digest`](#digest) property determines that an entry has not changed and should not be updated.

```ts title=loader.ts {7-14}
```ts title=loader.ts {8-15}
for (const item of feed.items) {
const id = item.guid;
const data = await parseData({
id: item.guid,
id,
data: item,
});
const digest = generateDigest(data);
Expand Down
Loading