For file uploading, default behaviour will `ReadForm` and write data to new temporary file if oversize `maxMemory`.
If temporary directory size is too small, it will fail for uploading large file and even consume OS memory since copy to buffer.
Inspired by Flask, Flask `request.stream.read` will read underlying socket data directly if payload found. Refer to: https://werkzeug.palletsprojects.com/en/2.0.x/wsgi/#werkzeug.wsgi.LimitedStream
This method makes assumptions because the request payload exists for operating `POST/PATCH/PUT`.
A simple demo for uploading 5G large file in low memory based on docker:
```
[root@control-master tmp]# docker run -ti --rm --memory 1G --tmpfs /tmp:rw,size=1G,mode=1777 -v $(pwd)/go:/go golang:1.16.5 /bin/bash
root@b672a98e5314:/go/testmodules# cat main.go
package main
import (
"fmt"
"github.com/gin-gonic/gin"
"io"
"os"
"path/filepath"
)
func main() {
engine := gin.Default()
engine.POST("/streamupload", func(c *gin.Context) {
if c.GetHeader("Content-Type") != "application/octet-stream" {
err := fmt.Errorf("required octet-stream")
c.AbortWithStatusJSON(400, map[string]string{"message": err.Error()})
return
}
info, err := os.Create(filepath.Join("/home", "foo"))
if err != nil {
c.AbortWithStatusJSON(400, gin.H{"message": "Create: "+err.Error()})
return
}
defer info.Close()
_, err = io.Copy(info, c.Request.Body)
if err != nil {
c.AbortWithStatusJSON(400, gin.H{"message": "Copy: "+err.Error()})
return
}
c.JSON(200, map[string]string{"message": "ok stream"})
})
if err := engine.Run("0.0.0.0:19090"); err != nil {
panic(err)
}
}
root@b53810b3e294:/go/testmodules# GOTMPDIR=/opt/ go run main.go
[root@control-master ~]# dd if=/dev/zero of=5G bs=1M count=5170 status=progress
[root@control-master ~]# curl -vvv -H "Content-Type:application/octet-stream" -T 5G -X POST 172.17.0.2:19090/streamupload
```
Signed-off-by: Chenyang Yan <memory.yancy@gmail.com>
* feat(binding): support custom struct tag
Add function `binding.MapFormWithTag` (#2719)
* doc: add 'bind form-data with custom struct tag'
Add 'Bind form-data request with custom struct and custom tag' section (#2719)
* test(binding): add test for MapFromWithTag
Function setTimeField calls strconv.ParseInt with bit size 0 when
parsing Unix time, which means it is equivalent to specifying 32 on
32-bit architectures. This causes the function to suffer from the year
2038 problem. To fix it and keep the behavior the same on both 32-bit
and 64-bit architectures, explicitly specify bit size 64.
Co-authored-by: thinkerou <thinkerou@gmail.com>
* Relocate binding body tests
Every test file should be related to a tested file.
Remove useless tests.
* Add github.com/stretchr/testify/require package
* support bind http header param #1956
update #1956
```
package main
import (
"fmt"
"github.com/gin-gonic/gin"
)
type testHeader struct {
Rate int `header:"Rate"`
Domain string `header:"Domain"`
}
func main() {
r := gin.Default()
r.GET("/", func(c *gin.Context) {
h := testHeader{}
if err := c.ShouldBindHeader(&h); err != nil {
c.JSON(200, err)
}
fmt.Printf("%#v\n", h)
c.JSON(200, gin.H{"Rate": h.Rate, "Domain": h.Domain})
})
r.Run()
// client
// curl -H "rate:300" -H "domain:music" 127.0.0.1:8080/
// output
// {"Domain":"music","Rate":300}
}
```
* add unit test
* Modify the code to get the http header
When the http header is obtained in the standard library,
the key value will be modified by the CanonicalMIMEHeaderKey function,
and finally the value of the http header will be obtained from the map.
As follows.
```go
func (h MIMEHeader) Get(key string) string {
// ...
v := h[CanonicalMIMEHeaderKey(key)]
// ...
}
```
This pr also follows this modification
* Thanks to vkd for suggestions, modifying code
* Increase test coverage
env GOPATH=`pwd` go test github.com/gin-gonic/gin/binding -coverprofile=cover.prof
ok github.com/gin-gonic/gin/binding 0.015s coverage: 100.0% of statements
* Rollback check code
* add use case to README.md
* move tests of mapping to separate test file
make 100% coverage of form_mapping.go from form_mapping_test.go file
* fix tests for go 1.6
go 1.6 doesn't support `t.Run(...)` subtests
* refactor(form_mapping.go): mapping multipart request
* add checkers for a types to match with the setter interface
* form_mapping.go: rename method name on setter interface, add comments
* fix style of comments
* refactor(form_mapping.go): mapping ptr, struct and map
* fix#1672 correct work with ptr - not create value if field is not set
* avoid allocations on strings.Split() - change to strings.Index()
* fix#610 tag value "-" is mean ignoring field
* struct fields mapped like json.Unmarshal
* map fields mapped like json.Unmarshal
* fix after @thinkerou review
* support bind uri (1)
* uri binding successful run
* fix vet warning: github.com/gin-gonic/gin/internal.Param composite literal uses unkeyed fields
* fix code style
* update function name
* fix test function signature
* add test for CanSet
* update readme and add test case
* remove internal.Params
* add coverage
* fix warning
