How to set up fastify multipart to upload documents to google cloud - Stack Overflow

admin管理员组

文章数量:1023221

I am trying to set up a backend using a Fastify server with the fastify-multipart plugin to handle file uploads. I have configured the Fastify server and registered the plugin, but when I try to upload a file using both curl and Postman, I get an error. Below are the details of my setup and the error I am encountering:

Fastify Setup:

I am using Fastify to create a simple server that will accept file uploads. I have installed the fastify-multipart plugin to handle multipart form data, which includes file uploads. The file upload functionality is mapped to a specific route, and I am using the multer storage engine to store the uploaded files on the server.

Error Details:

When using Postman, I am sending the file as multipart/form-data, but the server responds with an error message which I have higlighted below.

When using curl, I get a similar error

The Fastify server logs indicate that there is an issue with processing the file upload, but the error is not very clear, and I am unsure how to resolve it. Steps I’ve Tried:

I’ve made sure that I am sending the correct headers in both curl and Postman for file uploads (Content-Type: multipart/form-data). I’ve checked that the uploads folder exists and is writable, and that there is enough disk space. I’ve verified that the Fastify server is properly handling file uploads via other methods, such as testing with smaller, simpler data, but the issue seems to be isolated to file uploads. Questions:

What could be the reason the server is not handling file uploads correctly? Are there any specific configuration options for fastify-multipart or Multer that I might be missing? How can I troubleshoot this further to find the exact cause of the error?

my index.ts:

import fastify from 'fastify'
import dotenv from 'dotenv'
import multipart from '@fastify/multipart'
import formbody from '@fastify/formbody'
import { registerIndexRoute } from './routes/indexRoute'
import { registerGraphQLRoute } from './routes/graphqlRoute'
import { registerGetRedisDataRoute } from './routes/getRedisDataRoute'
import { registerUploadFilesRoute } from './routes/uploadFilesRoute'

dotenv.config()

export function createServer() {
  const server = fastify({
    logger: true,
  })

  // Register a global hook to log requests
  server.addHook('onRequest', (req, reply, done) => {
    console.log('Request received:', req.headers);
    done();
  });

  // Register the multipart plugin globally (before routes)
  server.register(multipart, {
    limits: { fileSize: 50 * 1024 * 1024 }, // 50MB file size limit
  });

  server.register(formbody);

  // Registered routes
  registerIndexRoute(server)
  registerGraphQLRoute(server)
  registerGetRedisDataRoute(server)
  registerUploadFilesRoute(server)

  return server
}

if (require.main === module) {
  ; (async () => {
    try {
      const server = createServer()
      await server.listen({ port: 8080, host: '0.0.0.0' })
      console.log(`Server is running on http://localhost:8080/`)
    } catch (error) {
      console.error(error)
      process.exit(1)
    }
  })()
}

My uploadFilesRoute.ts:

import path from 'path';
import fs from 'fs';
import util from 'util';
import { pipeline } from 'stream';
import { Storage } from '@google-cloud/storage';
import multipart from '@fastify/multipart';
import { bucketName, locallDir } from '../config/config';

const pump = util.promisify(pipeline);

// Register multipart plugin globally (only once)
export function registerUploadFilesRoute(server: FastifyInstance) {

  // Ensure local directory exists in development mode
  if (process.env.NODE_ENV === 'development') {
    const uploadDir = path.resolve(locallDir || './uploads');
    fs.promises.mkdir(uploadDir, { recursive: true }).catch((err) => {
      server.log.error(`Failed to create upload directory: ${err.message}`);
    });
  }


  // File upload route
  server.post('/api/upload/files', async (req, reply) => {

    req.log.info('Request received with headers: ', req.headers);
    req.log.info('Content-Type:', req.headers['content-type']);

    if (!req.isMultipart()) {
      return reply.status(415).send({
        error: 'Unsupported Media Type',
        message: 'Request must be multipart/form-data',
      });
    }

    try {
      const parts = req.files();
      const uploadedFiles: Array<{ filename: string; path?: string; url?: string }> = [];
      let fileCount = 0;

      // Iterate through the parts (files)
      for await (const part of parts) {
        if (!part.file) continue; // Skip if no file content

        fileCount++;
        const filename = part.filename || `unnamed_file_${Date.now()}`;
        const mimetype = part.mimetype || 'application/octet-stream';

        // Check for allowed file types (optional validation step)
        const allowedMimeTypes = ['application/pdf'];
        if (!allowedMimeTypes.includes(mimetype)) {
          return reply.status(400).send({
            error: 'Invalid file type',
            message: `File type ${mimetype} not supported`,
          });
        }

        if (process.env.NODE_ENV === 'development') {
          // Save files locally in development mode
          const localDir = locallDir || './uploads';
          const localFilePath = path.resolve(localDir, filename);
          await pump(part.file, fs.createWriteStream(localFilePath));
          uploadedFiles.push({ filename, path: localFilePath });
          server.log.info(`File uploaded locally: ${filename}`);
        } else {
          // Upload file to Google Cloud Storage in production mode
          const storage = new Storage();
          const bucket = storage.bucket(bucketName);
          const blob = bucket.file(filename);
          const blobStream = blob.createWriteStream({
            resumable: true,
            gzip: true,
            metadata: { contentType: mimetype },
          });

          await new Promise<void>((resolve, reject) => {
            part.file
              .pipe(blobStream)
              .on('finish', () => {
                const publicUrl = `/${bucketName}/${filename}`;
                uploadedFiles.push({ filename, url: publicUrl });
                server.log.info(`File uploaded to GCS: ${filename}`);
                resolve();
              })
              .on('error', (err) => {
                server.log.error(`Error uploading to GCS: ${err.message}`);
                reject(err);
              });
          });
        }
      }

      if (fileCount === 0) {
        return reply.status(400).send({
          error: 'No files provided for upload',
        });
      }

      reply.send({
        message: 'Files uploaded successfully',
        uploadedFiles,
      });
    } catch (error) {
      // Handle known error types
      if (error instanceof Error) {
        server.log.error(`File upload failed: ${error.message}`);
        reply.status(500).send({ error: 'File upload failed', details: error.message });
      } else {
        // Catch all unknown errors
        server.log.error('An unknown error occurred during file upload');
        reply.status(500).send({ error: 'File upload failed', details: 'Unknown error' });
      }
    }
  });
}

I am trying to set up a backend using a Fastify server with the fastify-multipart plugin to handle file uploads. I have configured the Fastify server and registered the plugin, but when I try to upload a file using both curl and Postman, I get an error. Below are the details of my setup and the error I am encountering:

Fastify Setup:

I am using Fastify to create a simple server that will accept file uploads. I have installed the fastify-multipart plugin to handle multipart form data, which includes file uploads. The file upload functionality is mapped to a specific route, and I am using the multer storage engine to store the uploaded files on the server.

Error Details:

When using Postman, I am sending the file as multipart/form-data, but the server responds with an error message which I have higlighted below.

When using curl, I get a similar error

The Fastify server logs indicate that there is an issue with processing the file upload, but the error is not very clear, and I am unsure how to resolve it. Steps I’ve Tried:

I’ve made sure that I am sending the correct headers in both curl and Postman for file uploads (Content-Type: multipart/form-data). I’ve checked that the uploads folder exists and is writable, and that there is enough disk space. I’ve verified that the Fastify server is properly handling file uploads via other methods, such as testing with smaller, simpler data, but the issue seems to be isolated to file uploads. Questions:

What could be the reason the server is not handling file uploads correctly? Are there any specific configuration options for fastify-multipart or Multer that I might be missing? How can I troubleshoot this further to find the exact cause of the error?

my index.ts:

import fastify from 'fastify'
import dotenv from 'dotenv'
import multipart from '@fastify/multipart'
import formbody from '@fastify/formbody'
import { registerIndexRoute } from './routes/indexRoute'
import { registerGraphQLRoute } from './routes/graphqlRoute'
import { registerGetRedisDataRoute } from './routes/getRedisDataRoute'
import { registerUploadFilesRoute } from './routes/uploadFilesRoute'

dotenv.config()

export function createServer() {
  const server = fastify({
    logger: true,
  })

  // Register a global hook to log requests
  server.addHook('onRequest', (req, reply, done) => {
    console.log('Request received:', req.headers);
    done();
  });

  // Register the multipart plugin globally (before routes)
  server.register(multipart, {
    limits: { fileSize: 50 * 1024 * 1024 }, // 50MB file size limit
  });

  server.register(formbody);

  // Registered routes
  registerIndexRoute(server)
  registerGraphQLRoute(server)
  registerGetRedisDataRoute(server)
  registerUploadFilesRoute(server)

  return server
}

if (require.main === module) {
  ; (async () => {
    try {
      const server = createServer()
      await server.listen({ port: 8080, host: '0.0.0.0' })
      console.log(`Server is running on http://localhost:8080/`)
    } catch (error) {
      console.error(error)
      process.exit(1)
    }
  })()
}

My uploadFilesRoute.ts:

import path from 'path';
import fs from 'fs';
import util from 'util';
import { pipeline } from 'stream';
import { Storage } from '@google-cloud/storage';
import multipart from '@fastify/multipart';
import { bucketName, locallDir } from '../config/config';

const pump = util.promisify(pipeline);

// Register multipart plugin globally (only once)
export function registerUploadFilesRoute(server: FastifyInstance) {

  // Ensure local directory exists in development mode
  if (process.env.NODE_ENV === 'development') {
    const uploadDir = path.resolve(locallDir || './uploads');
    fs.promises.mkdir(uploadDir, { recursive: true }).catch((err) => {
      server.log.error(`Failed to create upload directory: ${err.message}`);
    });
  }


  // File upload route
  server.post('/api/upload/files', async (req, reply) => {

    req.log.info('Request received with headers: ', req.headers);
    req.log.info('Content-Type:', req.headers['content-type']);

    if (!req.isMultipart()) {
      return reply.status(415).send({
        error: 'Unsupported Media Type',
        message: 'Request must be multipart/form-data',
      });
    }

    try {
      const parts = req.files();
      const uploadedFiles: Array<{ filename: string; path?: string; url?: string }> = [];
      let fileCount = 0;

      // Iterate through the parts (files)
      for await (const part of parts) {
        if (!part.file) continue; // Skip if no file content

        fileCount++;
        const filename = part.filename || `unnamed_file_${Date.now()}`;
        const mimetype = part.mimetype || 'application/octet-stream';

        // Check for allowed file types (optional validation step)
        const allowedMimeTypes = ['application/pdf'];
        if (!allowedMimeTypes.includes(mimetype)) {
          return reply.status(400).send({
            error: 'Invalid file type',
            message: `File type ${mimetype} not supported`,
          });
        }

        if (process.env.NODE_ENV === 'development') {
          // Save files locally in development mode
          const localDir = locallDir || './uploads';
          const localFilePath = path.resolve(localDir, filename);
          await pump(part.file, fs.createWriteStream(localFilePath));
          uploadedFiles.push({ filename, path: localFilePath });
          server.log.info(`File uploaded locally: ${filename}`);
        } else {
          // Upload file to Google Cloud Storage in production mode
          const storage = new Storage();
          const bucket = storage.bucket(bucketName);
          const blob = bucket.file(filename);
          const blobStream = blob.createWriteStream({
            resumable: true,
            gzip: true,
            metadata: { contentType: mimetype },
          });

          await new Promise<void>((resolve, reject) => {
            part.file
              .pipe(blobStream)
              .on('finish', () => {
                const publicUrl = `/${bucketName}/${filename}`;
                uploadedFiles.push({ filename, url: publicUrl });
                server.log.info(`File uploaded to GCS: ${filename}`);
                resolve();
              })
              .on('error', (err) => {
                server.log.error(`Error uploading to GCS: ${err.message}`);
                reject(err);
              });
          });
        }
      }

      if (fileCount === 0) {
        return reply.status(400).send({
          error: 'No files provided for upload',
        });
      }

      reply.send({
        message: 'Files uploaded successfully',
        uploadedFiles,
      });
    } catch (error) {
      // Handle known error types
      if (error instanceof Error) {
        server.log.error(`File upload failed: ${error.message}`);
        reply.status(500).send({ error: 'File upload failed', details: error.message });
      } else {
        // Catch all unknown errors
        server.log.error('An unknown error occurred during file upload');
        reply.status(500).send({ error: 'File upload failed', details: 'Unknown error' });
      }
    }
  });
}

本文标签： How to set up fastify multipart to upload documents to google cloudStack Overflow