6

I am developing a web API using Go and there is a lot of redundant database query scan code.

func (m *ContractModel) WorkQuestions(cid int) ([]models.WorkQuestion, error) {
    results, err := m.DB.Query(queries.WORK_QUESTIONS, cid)
    if err != nil {
        return nil, err
    }

    var workQuestions []models.WorkQuestion
    for results.Next() {
        var wq models.WorkQuestion
        err = results.Scan(&wq.ContractStateID, &wq.QuestionID, &wq.Question, &wq.ID, &wq.Answer, &wq.Compulsory)
        if err != nil {
            return nil, err
        }
        workQuestions = append(workQuestions, wq)
    }

    return workQuestions, nil
}

func (m *ContractModel) Questions(cid int) ([]models.Question, error) {
    results, err := m.DB.Query(queries.QUESTIONS, cid)
    if err != nil {
        return nil, err
    }

    var questions []models.Question
    for results.Next() {
        var q models.Question
        err = results.Scan(&q.Question, &q.Answer)
        if err != nil {
            return nil, err
        }
        questions = append(questions, q)
    }

    return questions, nil
}

func (m *ContractModel) Documents(cid int) ([]models.Document, error) {
    results, err := m.DB.Query(queries.DOCUMENTS, cid)
    if err != nil {
        return nil, err
    }

    var documents []models.Document
    for results.Next() {
        var d models.Document
        err = results.Scan(&d.Document, &d.S3Region, &d.S3Bucket, &d.Source)
        if err != nil {
            return nil, err
        }
        documents = append(documents, d)
    }

    return documents, nil
}

I need to generalize this code so that I can pass in the result *sql.Rows to a function and obtain a struct slice containing the scanned rows. I know that there is a StructScan method in sqlx package but this cannot be used since I have a significant amount of code written using the go standard database/sql package.

Using the reflect package, I can create a generic StructScan function but reflect package cannot create a slice of struct from a passed interface{} type. What I need to achieve is something like as follows

func RowsToStructs(rows *sql.Rows, model interface{}) ([]interface{}, error) {
    // 1. Create a slice of structs from the passed struct type of model
    // 2. Loop through each row,
    // 3. Create a struct of passed mode interface{} type
    // 4. Scan the row results to a slice of interface{}
    // 5. Set the field values of struct created in step 3 using the slice in step 4
    // 6. Add the struct created in step 3 to slice created in step 1
    // 7. Return the struct slice
}

I cannot seem to find a way to scan the struct passed as the model parameter and create a slice of it using the reflect package. Is there any workaround to this or am I looking at the question in a wrong way?

Struct fields has the correct number of cols returned from the result and in correct order

5
  • What is the association between columns in the result set and the struct fields? Commented Jun 7, 2020 at 3:45
  • Struct fields has the correct number of cols returned from the result and in correct order Commented Jun 7, 2020 at 3:48
  • sqlx works with the standard database/sql package. And if you have to re-write your code anyway, to accomodate your new abstraction, why not just use sqlx, instead of re-inventing the wheel? Commented Jun 7, 2020 at 8:13
  • @Flimzy sqlx implements part of what I'm trying to achieve not the complete implementation Commented Jun 7, 2020 at 9:45
  • @S.Sandeep: Which part does sqlx not do? Commented Jun 8, 2020 at 6:49

3 Answers 3

9

You can avoid using a type assertion in the calling function by passing a pointer to the destination slice as an argument. Here's RowsToStructs with that modification:

// RowsToStructs scans rows to the slice pointed to by dest.
// The slice elements must be pointers to structs with exported
// fields corresponding to the the columns in the result set.
//
// The function panics if dest is not as described above.
func RowsToStructs(rows *sql.Rows, dest interface{}) error {

    // 1. Create a slice of structs from the passed struct type of model
    //
    // Not needed, the caller passes pointer to destination slice.
    // Elem() dereferences the pointer.
    //
    // If you do need to create the slice in this function
    // instead of using the argument, then use
    // destv := reflect.MakeSlice(reflect.TypeOf(model).

    destv := reflect.ValueOf(dest).Elem()

    // Allocate argument slice once before the loop.

    args := make([]interface{}, destv.Type().Elem().NumField())

    // 2. Loop through each row

    for rows.Next() {

        // 3. Create a struct of passed mode interface{} type
        rowp := reflect.New(destv.Type().Elem())
        rowv := rowp.Elem()

        // 4. Scan the row results to a slice of interface{}
        // 5. Set the field values of struct created in step 3 using the slice in step 4
        //
        // Scan directly to the struct fields so the database
        // package handles the conversion from database
        // types to a Go types.
        //
        // The slice args is filled with pointers to struct fields.

        for i := 0; i < rowv.NumField(); i++ {
            args[i] = rowv.Field(i).Addr().Interface()
        }

        if err := rows.Scan(args...); err != nil {
            return err
        }

        // 6. Add the struct created in step 3 to slice created in step 1

        destv.Set(reflect.Append(destv, rowv))

    }
    return nil
}

Call it like this:

func (m *ContractModel) Documents(cid int) ([]*models.Document, error) {
    results, err := m.DB.Query(queries.DOCUMENTS, cid)
    if err != nil {
        return nil, err
    }
    defer results.Close()
    var documents []*models.Document
    err := RowsToStruct(results, &documents)
    return documents, err
}

Eliminate more boilerplate by moving the query to a helper function:

func QueryToStructs(dest interface{}, db *sql.DB, q string, args ...interface{}) error {
    rows, err := db.Query(q, args...)
    if err != nil {
        return err
    }
    defer rows.Close()
    return RowsToStructs(rows, dest)
}

Call it like this:

func (m *ContractModel) Documents(cid int) ([]*models.Document, error) {
    var documents []*model.Document
    err := QueryToStructs(&documents, m.DB, queries.DOCUMENTS, cid)
    return documents, err
}
Sign up to request clarification or add additional context in comments.

3 Comments

Few compile error fixes, var args []interface{} should be args := make([]interface{}, rowv.NumField()) and destv.Set(reflect.Append(destv, rowp)) should be destv.Set(reflect.Append(destv, rowv)) I suggested an edit but it wasn't approved.
@S.Sandeep I whipped up the code in the edit box here, but never ran it. Thank you for reporting the errors. I edited the answer.
You saved me hundreds of hours. Many thanks!! This works flawlessly, as of Go 1.17.3.
0

My proposal to deal with "common code" is quite simple, relying mostly on the premise that retrieve/get operations most of the times are used to return those results to a web browser client. So, those results are, in turn, transformed into JSON leveraging the json package.

file: db_common.go

// Interface for things that performs an scan over a given row.
// Actually it is a common interface for https://pkg.go.dev/database/sql#Rows.Scan and https://pkg.go.dev/database/sql#Row.Scan
type RowScanner interface {
    Scan(dest ...interface{}) error
}

// Scans a single row from a given query
type RowScanFunc func(rows RowScanner) (interface{}, error)

// Scans multiples rows using a scanner function in order to build a new "scanable" struct
func ScanMultiples(rows *sql.Rows, rowScanFunc RowScanFunc) ([]interface{}, error) {
    scaneables := []interface{}{}
    for rows.Next() {
        scanable, err := rowScanFunc(rows)
        if scanable == nil {
            return nil, err
        }
        scaneables = append(scaneables, scanable)
    }
    err := rows.Err()
    if err != nil {
        return nil, err
    }
    return scaneables, nil
}

And then I used the above abstractions like this:

file: dao/crud_operations.go

// Type that models a row of a given db table
type TableRow struct {
    Id int
    Name string
    // more fields...
}

func GetAll() ([]interface{}, error) {
    rows, err := Db.Query(`
        SELECT id, name
        FROM a_table`
    if err != nil {
        return nil, err
    }
    defer rows.Close()
    return ScanMultiples(rows, scanRow)
}

// Gets an specific legal entity by id
func GetById(id int) (interface{}, error) {
    row := Db.QueryRow(`
        SELECT id, name
        FROM a_table
        WHERE id = $1`, id)
    return scanRow(row)
}


// Scans a row interpreting it as 'TableRow' struct
func scanRow(rows RowScanner) (*TableRow, error) { // DO NOTE that implements the RowScanner interface!
    var tableRow TableRow

    err := rows.Scan(tableRow.Id,tableRow.Name)
    if err != nil {
        return nil, err
    }
    return &tableRow, nil
}
  • The good thing is that you only must code a single function that scans a Row.
  • Another good thing is that is not code tangled with external dependencies.
  • Another good thing is that the code doesn't resort to reflect package or those complicated things.
  • The bad thing is that you return interface{} and you may has to cast it to the original data type, having to write, 3 lines (in the worst case) for that.
  • Again you don't have to worry if the resulting results are going to be Marshaled like bytes, err := json.Marshal(results), because the json package receives an interface{} (=any) as parameter.

Comments

-1

"... but reflect package cannot create a slice of struct from a passed interface{} type." -- You looking for this?

func sliceFromElemValue(v interface{}) (interface{}) {
    rt := reflect.TypeOf(v)
    rs := reflect.MakeSlice(reflect.SliceOf(rt), 0, 0)

    for i := 0; i < 3; i++ { // dummy loop
        rs = reflect.Append(rs, reflect.New(rt).Elem())
    }

    return rs.Interface()
}

https://play.golang.com/p/o4AJ-f71egw


"What I need to achieve is something like as follows"

func RowsToStructs(rows *sql.Rows, model interface{}) ([]interface{}, error) { ...

Or are you looking for this?

func sliceFromElemValue(v interface{}) ([]interface{}) {
    rt := reflect.TypeOf(v)
    s := []interface{}{}

    for i := 0; i < 3; i++ { // dummy loop
        s = append(s, reflect.New(rt).Elem().Interface())
    }

    return s
}

https://play.golang.com/p/i57Z8OO8n7G

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.