Feature: Where should we put this documentation on reconnection and serverSelectionTimeoutMs?

[titanism]

Prerequisites

• I have written a descriptive issue title
• I have searched existing issues to ensure the feature has not already been requested

🚀 Feature Proposal

We've done some complex connection management, and wanted to share some patterns we used.

For example, if you have an Express or Koa app, such as app.js, you can set it up such as:

For single connection:

const express = require('express');
const mongoose = require('mongoose');
const Graceful = require('graceful');
const pRetry = require('p-retry'); // use v4.x for CJS and v5.x+ for ESM
const getPort = require('get-port'); // use v5.x for CJS and v6.x+ for ESM

try {
  const app = express();
  const port = await getPort();
  const server = await new Promise((resolve, reject) => {
    app.listen(port, function (err) {
      if (err) return reject(err);
      resolve(this);
    });
  });
  const graceful = new Graceful({ servers: [server] });
  graceful.listen();

  console.log('express app started on %d', port);

  //
  // this will attempt to reconnect with exponential backoff up to 10x
  // however if there is not a MongooseServerSelectionError, it will throw
  // (this is useful to keep http server running and retry db connection)
  // (apps such as pm2 are recommended to cause app to reboot if this eventually throws)
  //
  await pRetry(
    () => mongoose.connection.asPromise(),
    {
      // <https://github.com/sindresorhus/p-retry/issues/58>
      // <https://github.com/tim-kos/node-retry/issues/84>
      // forever: true,
      // retries: Infinity,
      onFailedAttempt(err) {
        console.error(err);
        if (!(err instanceof mongoose.Error.MongooseServerSelectionError))
          throw err;
      }
    }
  );
} catch (err) {
  console.error(err);
  process.exit(1);
}

For multiple connections:

const express = require('express');
const mongoose = require('mongoose');
const Graceful = require('graceful');
const pRetry = require('p-retry'); // use v4.x for CJS and v5.x+ for ESM
const getPort = require('get-port'); // use v5.x for CJS and v6.x+ for ESM

try {
  const app = express();
  const port = await getPort();
  const server = await new Promise((resolve, reject) => {
    app.listen(port, function (err) {
      if (err) return reject(err);
      resolve(this);
    });
  });
  const graceful = new Graceful({ servers: [server] });
  graceful.listen();

  console.log('express app started on %d', port);

  //
  // this will attempt to reconnect with exponential backoff up to 10x
  // however if there is not a MongooseServerSelectionError, it will throw
  // (this is useful to keep http server running and retry db connection)
  // (apps such as pm2 are recommended to cause app to reboot if this eventually throws)
  //
  await pRetry(
    () => Promise.all(mongoose.connections.map((connection) => connection.asPromise())),
    {
      // <https://github.com/sindresorhus/p-retry/issues/58>
      // <https://github.com/tim-kos/node-retry/issues/84>
      // forever: true,
      // retries: Infinity,
      onFailedAttempt(err) {
        console.error(err);
        if (!(err instanceof mongoose.Error.MongooseServerSelectionError))
          throw err;
      }
    }
  );
} catch (err) {
  console.error(err);
  process.exit(1);
}

Did you want to put this in documentation and as an example somewhere for users? If so let us know where - or feel free to add it wherever you deem best fit!

Motivation

Lack of real documentation and examples surrounding connection on application boot with serverSelectionTimeoutMS and how to properly handle it in real-world Express, Koa, Fastify, etc.

[titanism]

See above @vkarpov15 - this is also going to be pushed up to the codebase at https://github.com/forwardemail/forwardemail.net later today most likely.

[titanism]

Another thing we do is alert users if we have an issue with database connections (website outage):

    app.use((ctx, next) => {
      // if either mongoose or redis are not connected
      // then render website outage message to users
      const isMongooseDown = mongoose.connections.some(
        (conn) => conn.readyState !== mongoose.ConnectionStates.connected
      );
      const isRedisDown = !ctx.client || ctx.client.status !== 'ready';

      if (isMongooseDown || isRedisDown)
        ctx.logger.fatal(new Error('Website outage'), {
          mongoose: mongoose.connections.map((conn) => ({
            id: conn.id,
            readyState: conn.readyState,
            name: conn.name,
            host: conn.host,
            port: conn.port
          })),
          redis: {
            status: ctx.client.status,
            description: ctx.client._getDescription()
          }
        });

      if (
        ctx.method === 'GET' &&
        ctx.accepts('html') &&
        (isMongooseDown || isRedisDown)
      )
        ctx.flash('warning', ctx.translate('WEBSITE_OUTAGE'));

      return next();
    });

[titanism]

Okay! 🎉

We actually finally figured out how to make this work and implemented it... 🤦

It's super hacky right now because of the following discoveries:

• Bug: Anti-pattern with usage of class/prototype inheritance #12971
• Feature: createConnection should allow option to _not_ call conn.openUri, e.g. options.openUri: false #12970
• Bug: DisconnectedError should be removed as it is deprecated (?) #12968

This required us to create a custom wrapper for createConnection at https://github.com/ladjs/mongoose/blob/31ed19e14e41241a9b8940df4ae95820ac877774/index.js#L81-L111.

Here's the working code:

// eslint-disable-next-line import/no-unassigned-import
require('#config/env');
// eslint-disable-next-line import/no-unassigned-import
require('#config/mongoose');

const process = require('process');

const Graceful = require('@ladjs/graceful');
const Redis = require('@ladjs/redis');
const Web = require('@ladjs/web');
const ip = require('ip');
const mongoose = require('mongoose');
const pRetry = require('p-retry');
const sharedConfig = require('@ladjs/shared-config');

const Users = require('#models/users');
const config = require('#config');
const logger = require('#helpers/logger');
const webConfig = require('#config/web');

const webSharedConfig = sharedConfig('WEB');
const redis = new Redis(
  webSharedConfig.redis,
  logger,
  webSharedConfig.redisMonitor
);

const web = new Web(webConfig(redis), Users);
const graceful = new Graceful({
  mongooses: [mongoose],
  servers: [web.server],
  redisClients: [redis],
  logger
});
graceful.listen();

(async () => {
  try {
    await web.listen(web.config.port);
    if (process.send) process.send('ready');
    const { port } = web.server.address();
    logger.info(
      `Lad web server listening on ${port} (LAN: ${ip.address()}:${port})`,
      { hide_meta: true }
    );
    if (config.env === 'development')
      logger.info(
        `Please visit ${config.urls.web} in your browser for testing`,
        { hide_meta: true }
      );

    //
    // this will attempt to reconnect with exponential backoff up to 10x
    // however if there is not a MongooseServerSelectionError, it will throw
    // (this is useful to keep http server running and retry db connection)
    // (apps such as pm2 are recommended to cause app to reboot if this eventually throws)
    // <https://github.com/Automattic/mongoose/issues/12967>
    // <https://github.com/Automattic/mongoose/issues/12965>
    // <https://github.com/Automattic/mongoose/issues/12966>
    // <https://github.com/Automattic/mongoose/issues/12968>
    // <https://github.com/Automattic/mongoose/issues/12970>
    // <https://github.com/Automattic/mongoose/issues/12971>
    //
    await pRetry(
      () =>
        Promise.all(
          mongoose.connections
            .filter(
              (c) => c.readyState === mongoose.ConnectionStates.disconnected
            )
            //
            // NOTE: our version of `asPromise` contains magic per <https://github.com/Automattic/mongoose/issues/12970>
            //       see @ladjs/mongoose package source code for more insight into how this works
            //
            .map((c) => c.asPromise())
        ),
      {
        // <https://github.com/sindresorhus/p-retry/issues/58>
        // <https://github.com/tim-kos/node-retry/issues/84>
        // forever: true,
        // retries: Infinity,
        onFailedAttempt(err) {
          logger.fatal(err);
          if (!(err instanceof mongoose.Error.MongooseServerSelectionError))
            throw err;
        }
      }
    );
  } catch (err) {
    logger.fatal(err);

    process.exit(1);
  }
})();

[vkarpov15]

You could always just increase serverSelectionTimeoutMS, although 30 seconds is typically more than enough time for any intermittent connectivity issues to resolve. But feel free to put a PR in for using p-retry to retry initial connection failures, it could be useful for cases where you want to keep serverSelectionTimeoutMS low during runtime, but allow retrying initial connection.

[hasezoey]

Did you want to put this in documentation and as an example somewhere for users? If so let us know where - or feel free to add it wherever you deem best fit!

Lack of real documentation and examples surrounding connection on application boot with serverSelectionTimeoutMS and how to properly handle it in real-world Express, Koa, Fastify, etc.

if you have documentation about something like options, like usage of serverSelectionTimeoutMS, then feel free to add a guide in docs/ and just look at other existing guides for the style to use, if something is wrong or needs improvement it will be noted in the PR

mongoose does not currently list integration examples (aside from typescript) on the documentation website itself, but feel free to add your guide in docs/further_reading.md