Tests Timing Out? Check Go Tests Parallelism

Go Tests Parallelism

The command go test ./... lists all packages in the current directory, builds tests executable for each package, and runs them. The run part is interesting.

By default, the number of packages executed in parallel is equal to the value of GOMAXPROCS variable. On my laptop, it is 12 which is the number of CPU cores available.

The other parallelism is inside a package. If a test uses t.Parallel, go test -parallel <num of tests> determines the parallelism.

Why is it important to tune the degree of parallelism

Too many tests running in parallel would time out/crash for want of resources or it could expose a bug. There might be a leakage in the socket, buffer, and so on. If your machine is busy and you see tests timing out often, do try to tune the parallelism.

Tip

Always use go test ./... because it caches tests result for a package. So you save time, CPU in upcoming runs given your changes are not spreading to all packages.

References

  • go help test
  • go help testflag
  • go help build

Credit to StackOverflow
GOMAXPROCS

package main

import (
    "runtime"
    "fmt"
)

func getGOMAXPROCS() int {
    return runtime.GOMAXPROCS(0)
}

func main() {
    fmt.Printf("GOMAXPROCS is %d\n", getGOMAXPROCS())
}

Written with StackEdit.

golang: How to Add Better Context to Errors

golang: How to Add Better Context to Errors

There are at least three ways to add more context to errors.

  1. fmt.Errorf
  2. errors package
  3. zap library
package main

import (
	"errors"
	"fmt"

	werr "github.com/pkg/errors"
	"go.uber.org/zap"
)

// 1. wraps with a message
func wrap(e error) error {
	return fmt.Errorf("another layer - %w", e)
}

// 2. wraps with callstack and message
func betterWrap(e error) error {
	return werr.Wrap(e, "a better layer")
}

// 3. wraps with the first callstack frame
func zapLogger(e error) {
	logger, _ := zap.NewProduction()
	defer logger.Sync()
	logger.Info("failed to fetch URL",
		// Structured context as strongly typed Field values.
		zap.Error(e))
}

func main() {
	e := errors.New("a test err")

	fmt.Printf("err= %+v\n", wrap(e))
	fmt.Printf("%+v\n", betterWrap(e))
	zapLogger(e)
}

Execution

 $ go run err_context.go

fmt.errorf

err= another layer - a test err

errors package

a test err
a better layer
main.betterWrap
        /Users/xxx/go-code/blog/err/err_context.go:18
main.main
        /Users/xxx/go-code/blog/err/err_context.go:35
runtime.main
        /Users/xxx/.goenv/versions/1.14.0/src/runtime/proc.go:203
runtime.goexit
        /Users/xxx/.goenv/versions/1.14.0/src/runtime/asm_amd64.s:1373

zap

{"level":"info","ts":1604374256.270431,"caller":"err/err_context.go:25",
"msg":"failed to fetch URL","error":"a test err"}

Conclusion

I like the errors package the most. It has a relevant context to troubleshoot a problem.

json: cannot unmarshal object into Go struct field

Problem

json: cannot unmarshal object into Go struct field

Code

package main

import (
	"encoding/json"
	"fmt"
)

// DTO object
type mould struct {
	Particle tiny
}

type tiny interface {
	GetName() string
}

// Implement the interface
type speck struct {
	Name string `json:"name"`
}

func (s *speck) GetName() string {
	return s.Name
}

// Constructor
func NewSpeck(name string) tiny {
	return &speck{
		Name: name,
	}
}

//
func main() {
	dirt := NewSpeck("space")

	brown := &mould{}
	brown.Particle = dirt
	brownBytes, err := json.Marshal(brown)

	if err != nil {
		panic("marshal error")
	}

	brownLike := &mould{} // unmarshal has no "idea" of how to unpack!
	err = json.Unmarshal(brownBytes, &brownLike)
	if err != nil {
		panic("unmarshal err=%+v\n", err)
		return
	}
}

Go Playground Link

The error is:

go run unmarshal_interface.go
panic: json: cannot unmarshal object into Go struct field mould.Particle of type main.tiny

The problem with code is the inability of Go runtime to get target struct for unmarshaling.

Unmarshal essentially copies the byte array to a struct type. However, in the case of the unmarshal of the type tiny, runtime could not find any destination struct. The interface in Go just has behavior, not types.

Solutions

  1. Easy! Don’t return interface in a function. 🙂
  2. Help the runtime and add some info on destination struct.
    // solution
    brownLike2 := &mould{}
    brownLike2.Particle = &speck{} // Now unmarshal "knows" how to unpack the byte array!
    
    err = json.Unmarshal(brownBytes, &brownLike2)
    fmt.Printf("brownLike2=%#v err=%+v\n", brownLike2.Particle.GetName(), err)
    

Refernces

Written with StackEdit.

HTTP Request Body for Form Values

HTTP Request Header

HTTP clients and servers largely depend on the “Content-Type” header to determine how to parse the request/response body. Common values for the “Content-Type” header are

  • application/json
  • application/atom+xml
  • application/pdf
  • application/octet-stream
  • application/x-www-form-urlencoded

and request/response body should be processed as JSON, XML, PDF, Raw Stream, or Form Data respectively.

How is Form Data Sent?

99.99% of forms in your web applications will be url-encoded and sent in the request body instead of the request URL for obvious reasons.

Parse Form Data with ParseForm()

func (r *Request) ParseForm() error
For all requests, ParseForm parses the raw query from the URL and updates r.Form.

For POST, PUT, and PATCH requests, it also reads the request body, parses it as
a form, and puts the results into both r.PostForm and r.Form. Request body 
parameters take precedence over URL query string values in r.Form.

If the request Body's size has not already been limited by MaxBytesReader,
the size is capped at 10MB.

For other HTTP methods, or when the Content-Type is not 
application/x-www-form-urlencoded, the request Body is not read, and r.PostForm
is initialized to a non-nil, empty value.

URL query parameters are also read in the r.Form map object.

References

Written with StackEdit.

Why be careful with defer in Golang

When does defer function executes?

The standard answer is: defer runs a function before the enclosing function returns.

It is a wrong explanation. The defer executes as follows:

  1. The enclosing function has processed returned values and stored them in registers
  2. All defer functions run in a LIFO order of definition.
  3. If the return parameters are named, the defer function can alter the return values.

Short Examples on Vulnerabilities in defer Code

defer Modifying Named Return Parameters

package main

import (
	"fmt"
	"sync"
)

func test4(x int) (y int) {
	defer func() {
		y = 100
	}()

	if x == 123 {
		return x
	}

	defer func() {
		y = 200
	}()

	return x
}

func main() {
	fmt.Printf("Hello x=%v\n", test4(1234))
}

The above code returns value 123 referenced by y. Now, defer functions start executing. Mind it, the function test4() is yet to return to main().

  • The last defer function modifies the return value y to 200.
  • The first defer function modifies the return value y to 100.
  • test4() returns 100.

Deadlocked Program with Improper Understanding of defer

func test5(abort bool) {
	var mutex = &sync.Mutex{}

	defer func() {
		fmt.Printf("locking the mutex-first\n")
		mutex.Unlock()
	}()

	if abort {
		mutex.Lock()
		fmt.Println("critical section")
		mutex.Unlock()
		return
	}

	defer func() {
		fmt.Printf("locking the mutex-second\n")
		mutex.Lock()
	}()

}

func main() {
	test5(true)
}
go run deferDetails/main.go
critical section
locking the mutex-first
fatal error: sync: unlock of unlocked mutex

goroutine 1 [running]:
runtime.throw(0x10d2d55, 0x1e)
        /Users/ovo/.goenv/versions/1.14.0/src/runtime/panic.go:1112 +0x72 fp=0xc000078e38 sp=0xc000078e08 pc=0x102e542
sync.throw(0x10d2d55, 0x1e)
        /Users/ovo/.goenv/versions/1.14.0/src/runtime/panic.go:1098 +0x35 fp=0xc000078e58 sp=0xc000078e38 pc=0x102e4c5
sync.(*Mutex).unlockSlow(0xc00018c008, 0xc0ffffffff)
        /Users/ovo/.goenv/versions/1.14.0/src/sync/mutex.go:196 +0xd6 fp=0xc000078e80 sp=0xc000078e58 pc=0x106a746
sync.(*Mutex).Unlock(...)
        /Users/ovo/.goenv/versions/1.14.0/src/sync/mutex.go:190
main.test5.func1(0xc00018c008)
        /Users/ovo/go/1.14.0/src/mygo/deferDetails/main.go:13 +0x8b fp=0xc000078ee0 sp=0xc000078e80 pc=0x109ec7b
main.test5(0xc00006c001)
        /Users/ovo/go/1.14.0/src/mygo/deferDetails/main.go:20 +0x11a fp=0xc000078f70 sp=0xc000078ee0 pc=0x109eb1a
main.main()
        /Users/ovo/go/1.14.0/src/mygo/deferDetails/main.go:31 +0x26 fp=0xc000078f88 sp=0xc000078f70 pc=0x109ebd6
runtime.main()
Explanation

ha! the defer caused a panic in the program. I am not quite sure why it failed. If I remove the return, the code works.

package main

import (
	"fmt"
	"sync"
)

func test5(abort bool) {
	var mutex = &sync.Mutex{}

	defer func() {
		fmt.Printf("locking the mutex-first\n")
		mutex.Unlock()
	}()

	if abort {
		mutex.Lock()
		fmt.Println("critical section")
		mutex.Unlock()
	}

	defer func() {
		fmt.Printf("locking the mutex-second\n")
		mutex.Lock()
	}()

}

func main() {
	test5(true)
}
go run deferDetails/main.go
critical section
locking the mutex-second
locking the mutex-first

Conclusion

  1. Use defer very carefully, especially around named return parameter
  2. Multiple defer statements are executed in reverse order of appearance.
  3. Need to root cause why the code with return panic.

References

Written with StackEdit.

Time Complexity of a Memoized Algorithm

The memoization eliminated duplicate calls to a function. A memoized implementation of Fibonacci code is as follows:

package complexity

import "fmt"

var mem map[int]int

func fibo(count int) int {
	if count == 0 || count == 1 {
		return count
	}

	if mem[count] != -1 {
		return mem[count]
	}

	s := fibo(count-1) + fibo(count-2)
	mem[count] = s
	return s
}

func Fibo(count int) {
	mem = make(map[int]int, count)

	for i := 0; i <= count; i++ {
		mem[i] = -1
	}

	fmt.Printf("fibonacci number for count=%+v is %v\n", count, fibo(count))
	fmt.Printf("fibonacci map is %v\n", mem)
}
$ go run main.go
fibonacci number for count=4 is 3
fibonacci map is map[0:-1 1:-1 2:1 3:2 4:3]

Since each function in the recursion tree is processed at most once, the worst-case complexity of the above code becomes O(N).

Reference
https://stackoverflow.com/questions/42617249/time-complexity-of-memoization-fibonacci

Written with StackEdit.

Summary: How to Write Go Code

Summary: How to Write Go Code

  1. The code organization follows repository -> module -> packages.
  2. Use go mod init and initialize module root as desired e.g. example.com/user/hello
  3. The Go binary goes to the path set in GOBIN
  4. All imported modules are stored in GOPATH/pkg/mod subdirectory
  5. The command go clean -modcache cleans downloaded packages.

Reference

Written with StackEdit.

Go, gorm, nested JSON and associations

Go, gorm, nested JSON and associations

gorm provides a clean way to store a nested JSON with the relation of associations among tables. The following code creates a DB with three levels of nesting.

package main

import (
	"encoding/json"
	"fmt"

	"github.com/jinzhu/gorm"
	_ "github.com/lib/pq"
)

const (
	host     = "localhost"
	port     = 5432
	user     = "postgres"
	password = ""
	dbname   = "postgres"
)

type Page struct {
	ID     int64  `sql:"auto_increment" json:"-"`
	Number int64  `json:"number"`
	Book   Book   `gorm:"foreignkey:book_id" json:"-"`
	BookID int64  `json:"book_id"`
	Text   string `json:"text"`
}

type Book struct {
	ID          int64  `sql:"auto_increment" json:"-"`
	ShelfPlace  int64  `json:"shelf_place"`
	Shelf       Shelf  `gorm:"foreignkey:shelf_id" json:"-"`
	ShelfID     int64  `json:"shelf_id"`
	Author      string `json:"author" gorm:"unique;not null"`
	Publisher   string `json:"publisher"`
	PagesAmount int64  `json:"pages_amount"`
	Pages       []Page `json:"pages"`
}

type Shelf struct {
	ID          int64  `sql:"auto_increment" json:"-"`
	Number      int64  `json:"number"`
	BooksAmount int64  `json:"books_amount"`
	Book        []Book `json:"books"`
}

func main() {
	psqlInfo := fmt.Sprintf("host=%s port=%d user=%s "+
		"password=%s dbname=%s sslmode=disable",
		host, port, user, password, dbname)

	db, err := gorm.Open("postgres", psqlInfo)
	if err != nil {
		panic(err)
	}
	defer db.Close()

	// Create
	//db.Create(&Shelf{
	record := `{
			"number": 1,
			"books": [
			  {
				"shelf_place": 5,
				"author": "Lewis Carroll",
				"publisher": "EA",
				"pages_amount": 2,
				"pages": [
				  {
					"number": 2,
					"text": "lorem ipsum"
				  },
				  {
					"number": 4,
					"text": "dolor sit amet"
				  }
				]
			  },
			  {
				"shelf_place": 7,
				"author": "Mark Twain",
				"publisher": "Activision",
				"pages_amount": 3,
				"pages": [
				  {
					"number": 1,
					"text": "this is"
				  },
				  {
					"number": 3,
					"text": "a test"
				  },
				  {
					"number": 6,
					"text": "of json"
				  }
				]
			  }
			]
		  }`
	var shelf Shelf

	err = json.Unmarshal([]byte(record), &shelf)
	fmt.Printf("err=%v\n", err)

	db.DropTableIfExists(&Shelf{})
	db.DropTableIfExists(&Page{})
	db.DropTableIfExists(&Book{})

	// Migrate the schema
	db.AutoMigrate(&Shelf{})
	db.AutoMigrate(&Page{})
	db.AutoMigrate(&Book{})

	db.Create(&shelf)
	
	// Fails because author is a unique attribute
	//db.Create(&shelf)

    // Preload is necessary to query nested structure.
	db.Preload("Book").Where("author = ?", "Mark Twain").Find(&shelf)
	fmt.Printf("shelf=%v", shelf)
}

References

Written with StackEdit.

go tool: no such tool “compile”

I started facing go tool: no such tool "compile" error after installing a local build of go.

The golang source was in /usr/local

$ cd /usr/local    
$ ls  
total 41240
drwxr-xr-x@  19 root             wheel   608B Oct 18 19:35 go
drwxr-xr-x   16 root             wheel   512B Oct 18 19:35 .
-rw-r--r--@   1 root             wheel    20M Oct 18 19:35 go1.11.13.src.tar.gz

Next was go installation as following:

cd src; ./all.bash

The final build output was

Installed Go for darwin/amd64 in /usr/local/go
Installed commands in /usr/local/go/bin

Now the PATH & GOROOT must have the following values:

$ export GOROOT=/usr/local/go
$export PATH=/usr/local/go/bin:$PATH

The correct GOPATH value fixed te problem.

What didn’t work

  • Setting GOTOOLDIR
  • Reinstalling GO

Reference