mongodb: insert if not exists

Cover Image for mongodb: insert if not exists
Matheus Mello
Matheus Mello
published a few days ago. updated a few hours ago

MongoDB: Insert if Not Exists - A Fast and Efficient Solution

šŸ“¢ Hey there tech enthusiasts! Are you struggling with the tedious process of inserting new documents in MongoDB while ensuring no duplicates and maintaining updated timestamps? Well, worry no more! In this blog post, we'll dive into the problem our fellow developer is facing and provide you with a super-fast and efficient solution using the Python driver (pymongo). Let's get started! šŸš€

The Problem

Our friend receives a daily stock of documents and needs to insert each item that does not already exist. Pretty straightforward, right? But here are the specific requirements:

  1. He wants to track the first time a document is inserted and its last update time within the update.

  2. No duplicate documents should be stored.

  3. Existing documents, not present in the update, should remain untouched.

  4. A significant portion of the records remains unmodified day-to-day.

The Current Approach

Here's the pseudo-code our friend is using:

for each document in update:
    existing_document = collection.find_one(document)
    if not existing_document:
        document['insertion_date'] = now
        document = existing_document
    document['last_update_date'] = now

Now, this approach gets the job done, but it's painstakingly slow. For less than 100,000 records, it takes around 40 minutes! With millions of records in the update, it becomes an unbearable process. šŸ˜«

A Faster Solution

Fortunately, MongoDB provides a built-in method to handle such scenarios - update_many() with the upsert option, which inserts a document if it doesn't exist and updates it otherwise. Let's revamp our friend's code using this efficient approach:

for each document in update:
    filter = document.copy()
    filter.pop('_id', None)  # Exclude '_id' field from check
    update = {
        '$set': {
            'last_update_date': now
        '$setOnInsert': {
            'insertion_date': now
    collection.update_many(filter, update, upsert=True)

šŸŽ‰ That's it! With just a few changes, our friend's insert process will go from minutes to mere seconds.

In the updated code:

  • We create a filter from the document by making a copy and excluding the _id field. This ensures that the _id field, if present, won't affect the upsert behavior.

  • We define the update object with two modifiers:

    • $set sets the last_update_date to the current timestamp.

    • $setOnInsert sets the insertion_date only during the insert operation, not during updates.

  • Finally, with the update_many() method and upsert=True, MongoDB handles the insert/update operation for each document efficiently.

Time to Fly āœˆļø

And there you have it folks - a lightning-fast solution to the problem of inserting documents in MongoDB without duplicates, keeping track of timestamps, and handling updates with ease! Say goodbye to lengthy waiting times and start leveraging the power of update_many() with upsert.

If you found this guide helpful, we'd love to hear your thoughts! Have you faced similar challenges with MongoDB? Do you have any other cool tips and tricks for optimizing database operations? Don't hesitate to share your experiences and suggestions in the comments below. Let's dive into the conversation! šŸ’¬

Keep coding! šŸ‘Øā€šŸ’»šŸ‘©ā€šŸ’»

More Stories

Cover Image for How can I echo a newline in a batch file?

How can I echo a newline in a batch file?

updated a few hours ago

šŸ”„ šŸ’» šŸ†’ Title: "Getting a Fresh Start: How to Echo a Newline in a Batch File" Introduction: Hey there, tech enthusiasts! Have you ever found yourself in a sticky situation with your batch file output? We've got your back! In this exciting blog post, we

Matheus Mello
Matheus Mello
Cover Image for How do I run Redis on Windows?

How do I run Redis on Windows?

updated a few hours ago

# Running Redis on Windows: Easy Solutions for Redis Enthusiasts! šŸš€ Redis is a powerful and popular in-memory data structure store that offers blazing-fast performance and versatility. However, if you're a Windows user, you might have stumbled upon the c

Matheus Mello
Matheus Mello
Cover Image for Best way to strip punctuation from a string

Best way to strip punctuation from a string

updated a few hours ago

# The Art of Stripping Punctuation: Simplifying Your Strings šŸ’„āœ‚ļø Are you tired of dealing with pesky punctuation marks that cause chaos in your strings? Have no fear, for we have a solution that will strip those buggers away and leave your texts clean an

Matheus Mello
Matheus Mello
Cover Image for Purge or recreate a Ruby on Rails database

Purge or recreate a Ruby on Rails database

updated a few hours ago

# Purge or Recreate a Ruby on Rails Database: A Simple Guide šŸš€ So, you have a Ruby on Rails database that's full of data, and you're now considering deleting everything and starting from scratch. Should you purge the database or recreate it? šŸ¤” Well, my

Matheus Mello
Matheus Mello