Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Model.insertMany with lean: true is very slow compared to native driver #14372

Closed
1 task done
berkay-dincer opened this issue Feb 23, 2024 · 0 comments
Closed
1 task done
Labels
help wanted help This issue can likely be resolved in GitHub issues. No bug fixes, features, or docs necessary
Milestone

Comments

@berkay-dincer
Copy link

Prerequisites

  • I have written a descriptive issue title

Mongoose version

6.6.1

Node.js version

16.x

MongoDB version

6.0.1

Operating system

macOS

Operating system version (i.e. 20.04, 11.3, 10)

No response

Issue

Hey 👋

I am currently working on a piece of functionality that requires bulk inserting of documents using Mongoose. In the process, I've noticed a considerable performance difference between Model.collection.insertMany(docs) method and the Model.insertMany(docs, {lean: true}) method with the {lean: true} flag. Based on the documentation, I understand that the {lean: true} option helps to skip the process of hydrating and validating the documents before insertion. I had thought that this would effectively make these two methods roughly equivalent in terms of performance.

However, upon performing benchmark tests, I found that Model.collection.insertMany is significantly faster than Model.insertMany({lean: true}). I am hoping to better understand why this is the case. I am also attaching a script to replicate the issue.

const mongoose = require('mongoose');

const Schema = mongoose.Schema;

const MyModelSchema = new Schema({
    field1: String,
    field2: Number,
    field3: Boolean,
    field4: Date,
    field5: Array
});

const MyModel = mongoose.model('MyModel', MyModelSchema);

async function clearCollection() {
    await MyModel.deleteMany({});
}
async function mongooseInsert(docs) {
    console.time("Mongoose Insert");
    await MyModel.insertMany(docs, { ordered: false, lean: true });
    console.timeEnd("Mongoose Insert");
}

async function nativeInsert(docs) {
    console.time("Native Insert");
    await MyModel.collection.insertMany(docs, { ordered: false });
    console.timeEnd("Native Insert");
}

async function chunkedNativeInsert(docs) {
    console.time("Chunked Native Insert");
    for (let i = 0; i < docs.length; i += 1000) {
        await MyModel.collection.insertMany(docs.slice(i, i + 1000), { ordered: false });
    }
    console.timeEnd("Chunked Native Insert");

}
async function main() {
    await mongoose.connect("mongodb://localhost:27017/test");
    let documents = [];

    for (let i = 0; i < 50000; i++) {
        documents.push({
            field1: `String${i}`,
            field2: i,
            field3: i % 2 === 0,
            field4: new Date(),
            field5: [i, i+1, i+2]
        });
    }
    await mongooseInsert(documents);
    await clearCollection();
    await chunkedNativeInsert(documents);
    await clearCollection();
    await nativeInsert(documents);
    await clearCollection();
}


main();

Thanks!

@berkay-dincer berkay-dincer added help This issue can likely be resolved in GitHub issues. No bug fixes, features, or docs necessary help wanted labels Feb 23, 2024
@vkarpov15 vkarpov15 added this to the 6.12.7 milestone Feb 25, 2024
vkarpov15 added a commit that referenced this issue Feb 28, 2024
perf(model): make `insertMany()` `lean` option skip hydrating Mongoose docs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted help This issue can likely be resolved in GitHub issues. No bug fixes, features, or docs necessary
Projects
None yet
Development

No branches or pull requests

2 participants