Unable to send data to response after pipeline failed

Issue

I’m trying to query some data from mysql, transform it to csv and then finally download it as csv. Goal is to achieve this using streams.

    res.set('Content-Type', 'text/csv')
    res.set('Content-Disposition', `attachment; filename="${req.query?.filename || 'export'}.csv"`)
    const sqlStream = connection.query('SELECT * FROM whatever_table').stream({highWaterMark: 5}) // --> using mysql2 adapter
    const formatCsvStream = formatCsv({delimiter: ';'}) // --> using @fast-csv/format
    try {
        // using {pipeline} from require('stream/promises')
        await pipeline(
            sqlStream,
            formatCsvStream,
            res // --> express response
        )
    } catch (err) {
        console.error(err)
        return res.status(500).json({message: 'Please check the application logs for more details.'}).end()
    }

Everything is working as expected if there are no errors before it pipes to response. But if errors do happen (for example we select data from table that is not existing in db) browser will respond with ERR_EMPTY_RESPONSE instead of sending json error message defined in catch block.

If I remove res from pipeline, browser will show correct response (obviously download will not work in that case).

I tried removing Content-Type and Content-Disposition headers before returning .json. I also tried regular .pipe() syntax without any luck.

How to continue using express response object after fail in pipeline? Any suggestions would be great!

Solution

By the time the catch block is reached, the socket of the res object has already been destroyed by the pipeline. A pipeline "forwards errors" and destroys the destination if an error occurs in the source or along the intermediate transformations.

Try the following:

sqlStream.on("error", function(err) {
  console.error(err);
  if (!res.headersSent)
    res.status(500).json({message: 'Please check the application logs for more details.'}).end();
});
try {
  await pipeline(sqlStream, formatCsvStream, res);
} catch (err) {} // to avoid unhandled promise rejections. No further output here.

Answered By – Heiko Theißen

This Answer collected from stackoverflow, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0