Golang mongo driver performance

Question

I have written some code for testing mongodb. But its speed is so bad. What is wrong with is?

func mgoSpeedTest(N int) int64 {
    session, err := mgo.Dial(MongoHost)
    if err != nil {
        panic(err)
    }
    defer session.Close()
    session.SetMode(mgo.Monotonic, true)

    start := time.Now()
    for i := 0; i < N; i++ {
        sessionCopy := session.Copy()
        c := session.DB(MongoDB).C(MongoCol)
        _, err = c.Find(bson.M{"id": 1}).Count()
        if err != nil {
            log.Fatal(err)
        }
        sessionCopy.Close()
    }
    t := time.Now()
    elapsed := t.Sub(start)
    return elapsed.Milliseconds()
}

func main() {
    // speed test
    N := 1000
    ms = mgoSpeedTest(N)
    println(fmt.Sprintf("mgo: %d", ms))
}

Answer ~~ 3500ms I tried to use mongo-client but speed is the same ~3700-3800

You execute 1 thousand queries, averaging 3.5 ms (this includes network transfer back and forth, database execution, result handling, session copying and closing). What is considered good? Given what's happening under the hood, I don't think it's bad. Use the same session if you want better performance. — icza
– icza, Commented Oct 16, 2019 at 9:39
I also tried insert 1000 documents to localhost database. 1000 inserts = 36 seconds. But in official speed performance speed is the difference — Egor Kozelskij
– Egor Kozelskij, Commented Oct 16, 2019 at 9:44
I would also execute an mgo.Dial() call before starting the timer, to make sure your session is ready. — icza
– icza, Commented Oct 16, 2019 at 9:45
And you preclude the possibility that it is your code behaving badly? I would not. Disclose the code and we can find out where the problem is. Hint: If that was the general experience, nobody would use the mongo driver. Not even MongoDB Inc. And all their tools (mongodump etc) are written in Go. — Markus W Mahlberg
– Markus W Mahlberg, Commented Oct 16, 2019 at 10:25
Furthermore, you a looking up id (not _id, mind you). That basically means a collection scan is happening. Gist: MongoDB needs to open each and every document, read it, compare the value of the field id with 1. Btw, your code does not even compile - I have the feeling that you withhold information either accidentally or intentionally. If code is missing, that precludes us from helping you properly. — Markus W Mahlberg
– Markus W Mahlberg, Commented Oct 17, 2019 at 22:43

Community · Accepted Answer · 2020-06-20 09:12:55Z

Important

Much to my dismay, I cannot suggest to use mgo any more, since it seems to be no longer maintained as of the time of this writing.

I was curious and compiled a little benchmark of my own. It uses a single instance MongoDB via docker:

$ docker run -d --name mongobench -p 27017:27017 mongo
731e5f57d677718244c2304a992abd44a5a4bbad6f1fd8e5a23e53b3c4f9ada4

Note: The hash you get will be different.

As per the benchmark, it is twofold: simple insert and bulk inserts. Bulk inserts are the preferred way of dealing with mass insertions.

It tests both the mgo driver and the mongo-go-driver:

package mongobench

// Use
//  docker run -d --name mongobench -p 27017:27017 mongo
// to create a suitable test instance and access it via
//  docker exec -it mongobench mongo

import (
    "context"
    "log"
    "testing"

    "github.com/globalsign/mgo"
    "github.com/globalsign/mgo/bson"
    "go.mongodb.org/mongo-driver/mongo"
    "go.mongodb.org/mongo-driver/mongo/options"
)

type Data struct {
    ID      primitive.ObjectID `bson:"_id"`
    Counter int
}

const DefaultHost = "localhost:27017"
const DefaultDB = "test"
const DefaultCollection = "bench"

var sess *mgo.Session
var client *mongo.Client

func init() {
    var err error

    sess, err = mgo.Dial(DefaultHost)
    if err != nil {
        log.Fatalf("setting up session: %s", err)
    }

    client, err = mongo.NewClient(options.Client().ApplyURI("mongodb://" + DefaultHost))
    if err != nil {
        log.Fatalf("setting up client: %s", err)
    }

    if err = client.Connect(context.Background()); err != nil {
        log.Fatalf("connecting with client: %s", err)
    }
}
func BenchmarkMgoInsert(b *testing.B) {

    c := sess.DB(DefaultDB).C("simple")
    if _, err := c.RemoveAll(bson.M{}); err != nil {
        b.Logf("cleaning collection 'simple': %s", err)
        b.FailNow()
    }

    b.ResetTimer()

    for i := 0; i < b.N; i++ {
        if err := c.Insert(&Data{ID: bson.NewObjectId(), Counter: i}); err != nil {
            b.Logf("error inserting: %s", err)
            b.FailNow()
        }
    }
}

func BenchmarkMgoBulk(b *testing.B) {
    c := sess.DB(DefaultDB).C("bulk")
    if _, err := c.RemoveAll(bson.M{}); err != nil {
        b.Logf("cleaning collection 'simple': %s", err)
        b.FailNow()
    }

    b.ResetTimer()

    bulk := c.Bulk()

    for i := 0; i < b.N; i++ {
        bulk.Insert(&Data{ID: bson.NewObjectId(), Counter: i})
    }

    if _, err := bulk.Run(); err != nil {
        b.Logf("executing bulk: %s", err)
        b.FailNow()
    }
}

func BenchmarkMongoInsert(b *testing.B) {
    c := client.Database(DefaultDB).Collection("mongosimple")
    if _, err := c.DeleteMany(context.Background(), bson.M{}); err != nil {
        b.Logf("cleaning collection 'mongosimple': %s", err)
        b.FailNow()
    }

    b.ResetTimer()

    for i := 0; i < b.N; i++ {
        c.InsertOne(context.Background(), &Data{ID: bson.NewObjectId(), Counter: i})
    }
}

func BenchmarkMongoBulk(b *testing.B) {

    c := client.Database(DefaultDB).Collection("mongobulk")
    if _, err := c.DeleteMany(context.Background(), bson.M{}); err != nil {
        b.Logf("cleaning collection 'mongosimple': %s", err)
        b.FailNow()
    }

    d := make([]mongo.WriteModel, b.N)

    b.ResetTimer()

    for i := 0; i < b.N; i++ {
        d[i] = mongo.NewInsertOneModel().SetDocument(Data{ID: bson.NewObjectId(), Counter: i})
    }

    if _, err := c.BulkWrite(context.Background(), d); err != nil {
        b.Logf("inserting bulk: %s", err)
        b.FailNow()
    }

}

goos: darwin
goarch: amd64
pkg: github.com/mwmahlberg/so-mongobench
BenchmarkMgoInsert-4            1164       1100919 ns/op        1501 B/op         44 allocs/op
BenchmarkMgoBulk-4            201560          6512 ns/op         258 B/op          4 allocs/op
BenchmarkMongoInsert-4          1171       1019140 ns/op        3642 B/op         66 allocs/op
BenchmarkMongoBulk-4          181040          7251 ns/op        1151 B/op         15 allocs/op

What we can see is that both drivers are orders of magnitude faster than what you describe, hence it is safe to assume that it is not the driver causing the delays.

@ScottStensland To be fair, I did not, since I thought it would be too obvious. Will add it.

Collectives™ on Stack Overflow

Golang mongo driver performance

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related