Go SQlite Concurrency Issue

Question

I've been playing around with this SQLite library for Go, and in doing some stress testing, have come across some odd errors.

I'm attempting to run this code:

package main

import "code.google.com/p/go-sqlite/go1/sqlite3"
import "fmt"
import "sync"

func main() {
    var wg sync.WaitGroup
    wg.Add(100)
    for i := 0; i < 100; i++ {
        query(&wg, i)
    }
    wg.Wait()
}

func query(wg *sync.WaitGroup, id int) {
    defer wg.Done()
    c, _ := sqlite3.Open("sqlite.db")
    for i := 0; i < 100; i++{
        args := sqlite3.NamedArgs{"$bval": "demo"}
        sql := "SELECT rowid, * FROM x WHERE b = $bval"
        row := make(sqlite3.RowMap)
        for s, err := c.Query(sql, args); err == nil; err = s.Next() {
                var rowid int64
                s.Scan(&rowid, row)     // Assigns 1st column to rowid, the rest to row
                fmt.Println(rowid, row) // Prints "1 map[a:1 b:demo c:<nil>]"
        }
    }
    c.Close()
    fmt.Println("Worker ", id, " is done")
}

By default, there are no issues; But when I increase GOMAXPROCS to more than 1, the program crashes at arbitrary points, giving the error:

fatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0xc0000000d pc=0x4100b9]

runtime stack:
runtime.gothrow(0x5a1610, 0x2a)
    /usr/lib/go/src/runtime/panic.go:503 +0x8e fp=0x7f4bd5ffa830 sp=0x7f4bd5ffa818
runtime.sigpanic()
    /usr/lib/go/src/runtime/sigpanic_unix.go:14 +0x5e fp=0x7f4bd5ffa880 sp=0x7f4bd5ffa830

goroutine 1 [syscall, locked to thread]:
runtime.cgocall_errno(0x408043, 0xc20808fcf8, 0x0)
    /usr/lib/go/src/runtime/cgocall.go:130 +0xf5 fp=0xc20808fcd8 sp=0xc20808fcb0
code.google.com/p/go-sqlite/go1/sqlite3._Cfunc_sqlite3_prepare_v2(0x7f4bc00023b8, 0xc2080483f0, 0xffffffff, 0xc20808fd30, 0xc20808fd28, 0x0)
    /home/eumen/Programming/gocode/src/code.google.com/p/go-sqlite/go1/sqlite3/:534 +0x43 fp=0xc20808fcf8 sp=0xc20808fcd8
code.google.com/p/go-sqlite/go1/sqlite3.newStmt(0xc2080482d0, 0x599e90, 0x26, 0xc20804b8c8, 0x0, 0x0)
    /home/eumen/Programming/gocode/src/code.google.com/p/go-sqlite/go1/sqlite3/sqlite3.go:583 +0xe4 fp=0xc20808fdb0 sp=0xc20808fcf8
code.google.com/p/go-sqlite/go1/sqlite3.(*Conn).Query(0xc2080482d0, 0x599e90, 0x26, 0xc20808fee0, 0x1, 0x1, 0xc20807a000, 0x0, 0x0)
    /home/eumen/Programming/gocode/src/code.google.com/p/go-sqlite/go1/sqlite3/sqlite3.go:276 +0xc3 fp=0xc20808fe00 sp=0xc20808fdb0
main.query(0xc208022020, 0x16)
    /home/eumen/Programming/Go/practice/sqlite-sync.go:26 +0x261 fp=0xc20808ff70 sp=0xc20808fe00
main.main()
    /home/eumen/Programming/Go/practice/sqlite-sync.go:11 +0x66 fp=0xc20808ff98 sp=0xc20808ff70
runtime.main()
    /usr/lib/go/src/runtime/proc.go:63 +0xf3 fp=0xc20808ffe0 sp=0xc20808ff98
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc20808ffe8 sp=0xc20808ffe0

goroutine 2 [force gc (idle)]:
runtime.gopark(0x496870, 0x831d30, 0x583f70, 0xf)
    /usr/lib/go/src/runtime/proc.go:130 +0x105 fp=0xc20801e798 sp=0xc20801e768
runtime.goparkunlock(0x831d30, 0x583f70, 0xf)
    /usr/lib/go/src/runtime/proc.go:136 +0x48 fp=0xc20801e7c0 sp=0xc20801e798
runtime.forcegchelper()
    /usr/lib/go/src/runtime/proc.go:99 +0xce fp=0xc20801e7e0 sp=0xc20801e7c0
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc20801e7e8 sp=0xc20801e7e0
created by runtime.init·4
    /usr/lib/go/src/runtime/proc.go:87 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0x496870, 0x838ff8, 0x580b50, 0xd)
    /usr/lib/go/src/runtime/proc.go:130 +0x105 fp=0xc208021f98 sp=0xc208021f68
runtime.goparkunlock(0x838ff8, 0x580b50, 0xd)
    /usr/lib/go/src/runtime/proc.go:136 +0x48 fp=0xc208021fc0 sp=0xc208021f98
runtime.bgsweep()
    /usr/lib/go/src/runtime/mgc0.go:98 +0xbc fp=0xc208021fe0 sp=0xc208021fc0
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc208021fe8 sp=0xc208021fe0
created by gc
    /usr/lib/go/src/runtime/mgc0.c:1386

goroutine 4 [finalizer wait]:
runtime.gopark(0x496870, 0x838ff0, 0x583ad0, 0xe)
    /usr/lib/go/src/runtime/proc.go:130 +0x105 fp=0xc20801d730 sp=0xc20801d700
runtime.goparkunlock(0x838ff0, 0x583ad0, 0xe)
    /usr/lib/go/src/runtime/proc.go:136 +0x48 fp=0xc20801d758 sp=0xc20801d730
runtime.runfinq()
    /usr/lib/go/src/runtime/malloc.go:727 +0xba fp=0xc20801d7e0 sp=0xc20801d758
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc20801d7e8 sp=0xc20801d7e0
created by runtime.createfing
    /usr/lib/go/src/runtime/malloc.go:707 +0x5e

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc208032fe8 sp=0xc208032fe0
signal: aborted (core dumped)

This error only occurs when I increase GOMAXPROCS, and also occurs if I use go query(&wg, i). Oddly enough, the errors also do not occur if the loop isn't used, or if i is set to a small enough number. Would someone be able to shed some light about what is going on here?

Community · Accepted Answer · 2020-06-20 09:12:55Z

2

The library you are using doesn't support concurrency as per https://code.google.com/p/go-sqlite/source/browse/go1/sqlite3/doc.go#29 -

Concurrency

A single connection instance and all of its derived objects (prepared statements, backup operations, etc.) may NOT be used concurrently from multiple goroutines without external synchronization. The only exception is Conn.Interrupt(), which may be called from another goroutine to abort a long-running operation. It is safe to use separate connection instances concurrently, even if they are accessing the same database file.

As sqlite3 is a file-based database, the driver needs to manage concurrent access from multiple threads.

The https://github.com/mattn/go-sqlite3 library should support thread-safe operation provided your sqlite3 was compiled to do so.

edited Jun 20, 2020 at 9:12

CommunityBot

11 silver badge

answered Jun 29, 2015 at 3:36

elithrar

24.5k11 gold badges91 silver badges107 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Ezra Umen Over a year ago

I was under the impression that so long as I created a new Open on each function call, it wouldn't violate the libraries concurrency requirements, but I guess when the way Go multithreading works, it doesn't. Thanks for the info.

elithrar Over a year ago

You're often better creating a single *sql.DB - be it global or explicitly passed to your handlers/functions from main() and re-using that instead of Open(...); defer Close() in each function. Less overhead.

Ezra Umen Over a year ago

Yea, that's what I'm doing in the actual application I'm writing at the moment. In this question, I was just playing around to see whether or not SQLite would hold up under concurrency. After repeating similar tests with the library you recommended, it looks like it does, and quite well too.

Collectives™ on Stack Overflow

Go SQlite Concurrency Issue

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related