Consider utilizing 'fetch' or alternative techniques in place of 'fs' when reading markdown files specifically within NextJS

Question

Consider utilizing 'fetch' or alternative techniques in place of 'fs' when reading markdown files specifically within NextJS

I'm currently working on a blog that includes math formulas in markdown pages. The markdown files are stored locally in a folder. Here is the code snippet from my Blogpage.tsx file-


import React from 'react'
import fs from "fs"
import Markdown from "react-markdown"
import rehypeKatex from "rehype-katex"
import remarkMath from "remark-math"
import rehypeRaw from "rehype-raw"
import matter from "gray-matter"
import 'katex/dist/katex.min.css'

// export const runtime = 'edge'
const getPostContent = (postID: string) => {
  const folder = "src/blogPosts";
  const file = `${folder}/${postID}.md`;
  const content = fs.readFileSync(file, "utf8");
  const matterResult = matter(content);
  return matterResult;
};

export const generateStaticParams = async () => {
  return [{ postID: "blog-post1" }]
};

const BlogPage = (props: any) => {

  const postID = props.params.postID;
  const post = getPostContent(postID)

  return (
    <>
      <h1>{post.data.title}</h1>
      <Markdown remarkPlugins={[remarkMath]} rehypePlugins={[rehypeKatex, rehypeRaw]}>
        {post.content}
      </Markdown>
    </>
  )
}

export default BlogPage

Localhost runs smoothly, but encountering an error when trying to deploy on Cloudflare pages.


19:18:58.294    ⚡️ Completed `npx vercel build`.
19:18:58.342    ⚡️ Invalid prerender config for /allblogs/[postID]
...
(remaining log messages)
...

Introducing export const runtime = 'edge' on the page leads to an error related to the 'fs' module:

Error: Module not found: Can't resolve 'fs'

Looks like there's a conflict between the edge runtime and the nodejs module. Trying an alternate approach using fetch:


const getPostContent = async (postID: string) => {
  const resource = await import(`${folder}/${postID}.md`);
  const res = await fetch(resource.default);
  if (!res.ok) {
      throw new Error(`${resource.default}: ${res.status} ${res.statusText}`);
  }
  return res.text();
};

This method throws an error -

⨯ unhandledRejection: Error: Cannot find module

Seeking assistance with reading markdown files without utilizing 'fs'. Also looking for alternative ways to import markdown files without conflicting with the edge runtime.

Update

Solved issues with uploading to Cloudflare pages by utilizing NextJS static export feature.

Note: For NextJS version 13 or newer, use generateStaticParams() instead of getStaticProps() and getStaticPaths(). Details can be found here.

Still in search of a good alternative for fetching markdown files from local directories without relying on 'fs'. Open to suggestions!

typescript next.js fetch-api node.js-fs cloudflare-pages

Answer 1

Answer №1

If you're utilizing the @cloudflare/next-on-pages package along with dynamic routes using the app router (e.g., src\app\posts\[slug]\page.tsx), then making sure to set the dynamicParams flag to false might be necessary.

I encountered a similar issue to what you described, where my blog content consisted of mdx files on disk and I aimed to statically generate each blog page. However, during the build process, I started receiving errors prompting me to enable the edge runtime. Subsequently, when I did that, my library for loading the mdx content stopped working due to the unavailability of fs.

In my case, setting

export const dynamicParams = false;

resolved the issue by directing nextjs not to generate a cloudflare worker for handling requests for pages that weren't statically rendered. This action eliminated concerns about which runtime was being utilized.

Please refer to the examples provided below.

You can also check .

src\app\posts\[slug]\page.tsx

import { getArticleBySlug, getAllArticles } from "@/lib/articles"

/*
 * https://developers.cloudflare.com/pages/framework-guides/nextjs/deploy-a-nextjs-site/#generatestaticparams
 * 
 * When doing static site generation (SSG) in the /app directory and using the generateStaticParams function,
 * Next.js by default tries to handle requests for non statically generated routes on-demand. It does so by
 * creating a Node.js serverless function (which, as such, is incompatible with @cloudflare/next-on-pages).
 * 
 * In such cases you need to instruct Next.js not to do so by specifying dynamicParams = false.
 */
export const dynamicParams = false;

export async function generateStaticParams() {
    const posts = await getAllArticles();

    return posts.map((post) => ({
        slug: post.slug,
    }));
}

export default async function Page({ params }: { params: { slug: string } }) {
    const article = await getArticleBySlug(params.slug);
    return (
        <article.component />
    );
}

src\lib\articles.ts

import glob from 'fast-glob';

interface Article {
    title: string
    description: string
    authors: string[]
    date: string,
    featuredImage?: {
        url: string
    }
}

export interface ArticleWithSlug extends Article {
    slug: string
}

export interface ArticleWithComponent extends ArticleWithSlug {
    component: React.ComponentType
}

async function importArticle(
    articleFilename: string,
): Promise<ArticleWithComponent> {
    let post = (
        await import(`../content/posts/${articleFilename}`)
    ) as {
        default: React.ComponentType
        article: Article,
        metadata: Record<string, string>
    }

    const slug = articleFilename.replace(/(\/page)?\.mdx$/, '');

    return {
        component: post.default,
        slug,
        ...post.article
    }
}

export async function getArticleBySlug(slug: string) {
    return importArticle(`${slug}/page.mdx`);
}

export async function getAllArticles(): Promise<ArticleWithComponent[]> {
    const articleFilenames = await glob('*/page.mdx', {
        cwd: './src/content/posts',
    })

    const articles = (await Promise.all(
        articleFilenames.map(importArticle)
    ));

    return articles.sort((a, z) => +new Date(z.date) - +new Date(a.date))
}

Answer 2

If you're utilizing the @cloudflare/next-on-pages package along with dynamic routes using the app router (e.g., src\app\posts\[slug]\page.tsx), then making sure to set the dynamicParams flag to false might be necessary.

I encountered a similar issue to what you described, where my blog content consisted of mdx files on disk and I aimed to statically generate each blog page. However, during the build process, I started receiving errors prompting me to enable the edge runtime. Subsequently, when I did that, my library for loading the mdx content stopped working due to the unavailability of fs.

In my case, setting

export const dynamicParams = false;

resolved the issue by directing nextjs not to generate a cloudflare worker for handling requests for pages that weren't statically rendered. This action eliminated concerns about which runtime was being utilized.

Please refer to the examples provided below.

You can also check .

src\app\posts\[slug]\page.tsx

import { getArticleBySlug, getAllArticles } from "@/lib/articles"

/*
 * https://developers.cloudflare.com/pages/framework-guides/nextjs/deploy-a-nextjs-site/#generatestaticparams
 * 
 * When doing static site generation (SSG) in the /app directory and using the generateStaticParams function,
 * Next.js by default tries to handle requests for non statically generated routes on-demand. It does so by
 * creating a Node.js serverless function (which, as such, is incompatible with @cloudflare/next-on-pages).
 * 
 * In such cases you need to instruct Next.js not to do so by specifying dynamicParams = false.
 */
export const dynamicParams = false;

export async function generateStaticParams() {
    const posts = await getAllArticles();

    return posts.map((post) => ({
        slug: post.slug,
    }));
}

export default async function Page({ params }: { params: { slug: string } }) {
    const article = await getArticleBySlug(params.slug);
    return (
        <article.component />
    );
}

src\lib\articles.ts

import glob from 'fast-glob';

interface Article {
    title: string
    description: string
    authors: string[]
    date: string,
    featuredImage?: {
        url: string
    }
}

export interface ArticleWithSlug extends Article {
    slug: string
}

export interface ArticleWithComponent extends ArticleWithSlug {
    component: React.ComponentType
}

async function importArticle(
    articleFilename: string,
): Promise<ArticleWithComponent> {
    let post = (
        await import(`../content/posts/${articleFilename}`)
    ) as {
        default: React.ComponentType
        article: Article,
        metadata: Record<string, string>
    }

    const slug = articleFilename.replace(/(\/page)?\.mdx$/, '');

    return {
        component: post.default,
        slug,
        ...post.article
    }
}

export async function getArticleBySlug(slug: string) {
    return importArticle(`${slug}/page.mdx`);
}

export async function getAllArticles(): Promise<ArticleWithComponent[]> {
    const articleFilenames = await glob('*/page.mdx', {
        cwd: './src/content/posts',
    })

    const articles = (await Promise.all(
        articleFilenames.map(importArticle)
    ));

    return articles.sort((a, z) => +new Date(z.date) - +new Date(a.date))
}

Answer 3

Answer №2

If you're encountering a familiar challenge when deploying Next.js applications that utilize Node.js-specific modules such as fs to static hosting services like Cloudflare Pages, you're not alone.

Unfortunately, the fs module isn't accessible in browsers or serverless functions used by Cloudflare Pages for dynamic routes.

To resolve this issue, consider utilizing Next.js's getStaticProps and getStaticPaths functions to retrieve markdown files during the build phase rather than runtime execution.

This method allows you to pre-render each blog post as a static page, enabling direct serving by Cloudflare Pages without file system access requirements.

import React from 'react';
import fs from 'fs';
import path from 'path';
import Markdown from 'react-markdown';
import rehypeKatex from 'rehype-katex';
import remarkMath from 'remark-math';
import rehypeRaw from 'rehype-raw';
import matter from 'gray-matter';
import 'katex/dist/katex.min.css';

const postsDirectory = path.join(process.cwd(), 'src/blogPosts');

export async function getStaticPaths() {
  // Read all markdown files in the postsDirectory
  const filenames = fs.readdirSync(postsDirectory);

  // Create a route for each markdown file
  const paths = filenames.map((filename) => {
    return {
      params: {
        postID: filename.replace(/\.md$/, ''),
      },
    };
  });

  return {
    paths,
    fallback: false,
  };
}

export async function getStaticProps({ params }) {
  // Read the markdown file for the current postID
  const fullPath = path.join(postsDirectory, `${params.postID}.md`);
  const fileContents = fs.readFileSync(fullPath, 'utf8');

  // Parse the post metadata section
  const matterResult = matter(fileContents);

  // Pass the post data to the page via props
  return {
    props: {
      post: matterResult,
    },
  };
}

const BlogPage = ({ post }) => {
  return (
    <>
      <h1>{post.data.title}</h1>
      <Markdown
        remarkPlugins={[remarkMath]}
        rehypePlugins={[rehypeKatex, rehypeRaw]}
      >
        {post.content}
      </Markdown>
    </>
  );
};

export default BlogPage;

Let me break down the code above:

getStaticPaths scans through your src/blogPosts folder to identify markdown files and create paths for them. This informs Next.js about the pages to generate during the build process.
getStaticProps utilizes the provided postID from getStaticPaths, reads the associated markdown file, and delivers the content to your page as props.
The BlogPage component accepts the post content as props and displays it on the page.

Hopefully, this explanation proves helpful to you!!!

Answer 4

If you're encountering a familiar challenge when deploying Next.js applications that utilize Node.js-specific modules such as fs to static hosting services like Cloudflare Pages, you're not alone.

Unfortunately, the fs module isn't accessible in browsers or serverless functions used by Cloudflare Pages for dynamic routes.

To resolve this issue, consider utilizing Next.js's getStaticProps and getStaticPaths functions to retrieve markdown files during the build phase rather than runtime execution.

This method allows you to pre-render each blog post as a static page, enabling direct serving by Cloudflare Pages without file system access requirements.

import React from 'react';
import fs from 'fs';
import path from 'path';
import Markdown from 'react-markdown';
import rehypeKatex from 'rehype-katex';
import remarkMath from 'remark-math';
import rehypeRaw from 'rehype-raw';
import matter from 'gray-matter';
import 'katex/dist/katex.min.css';

const postsDirectory = path.join(process.cwd(), 'src/blogPosts');

export async function getStaticPaths() {
  // Read all markdown files in the postsDirectory
  const filenames = fs.readdirSync(postsDirectory);

  // Create a route for each markdown file
  const paths = filenames.map((filename) => {
    return {
      params: {
        postID: filename.replace(/\.md$/, ''),
      },
    };
  });

  return {
    paths,
    fallback: false,
  };
}

export async function getStaticProps({ params }) {
  // Read the markdown file for the current postID
  const fullPath = path.join(postsDirectory, `${params.postID}.md`);
  const fileContents = fs.readFileSync(fullPath, 'utf8');

  // Parse the post metadata section
  const matterResult = matter(fileContents);

  // Pass the post data to the page via props
  return {
    props: {
      post: matterResult,
    },
  };
}

const BlogPage = ({ post }) => {
  return (
    <>
      <h1>{post.data.title}</h1>
      <Markdown
        remarkPlugins={[remarkMath]}
        rehypePlugins={[rehypeKatex, rehypeRaw]}
      >
        {post.content}
      </Markdown>
    </>
  );
};

export default BlogPage;

Let me break down the code above:

getStaticPaths scans through your src/blogPosts folder to identify markdown files and create paths for them. This informs Next.js about the pages to generate during the build process.
getStaticProps utilizes the provided postID from getStaticPaths, reads the associated markdown file, and delivers the content to your page as props.
The BlogPage component accepts the post content as props and displays it on the page.

Hopefully, this explanation proves helpful to you!!!

Consider utilizing 'fetch' or alternative techniques in place of 'fs' when reading markdown files specifically within NextJS

Answer №1

Answer №2

Similar questions

Using dual index variables in Angular 4's ngFor loop for iterating through arrays

What is the process for obtaining the directory path in a Next.js project that has been deployed to Vercel?

Initiate and terminate server using supertest

I am looking to create a counter in NextJS that will store its value in a database for persistent storage. How can

How can I assign two different colors based on the type in Typescript?

Register with your Facebook account - Angular 4 Typescript

Issue with Google Adsense - adsbygoogle.push() error: Pages can only support one AdSense head tag. The additional tag will be disregarded

NextJs/Express Server Encounters Domain HTTPS Request Error

Is there a way to access window.location.hash through a Next.js application?

`AngularJS Voice Recognition Solutions`

Exploring TypeScript Heartbeat functionality with Nodejs ws module

Attempting to incorporate alert feedback into an Angular application

Setting dynamic values within the constructor of a TypeScript class to the object instance

Is there a way to apply a consistent style to all the fields of an object at once?

How to link observables in HTTP requests using Angular 5?

"Converting array into a string in TypeScript/Javascript, but unable to perform operations

The struggle of implementing useReducer and Context in TypeScript: A type error saga

Unending loop detected in the React useEffect hook during testing

Ways to showcase information based on the chosen option?

A TypeScript method for accessing deeply nested properties within an object