For file uploading, default behaviour will `ReadForm` and write data to new temporary file if oversize `maxMemory`.
If temporary directory size is too small, it will fail for uploading large file and even consume OS memory since copy to buffer.
Inspired by Flask, Flask `request.stream.read` will read underlying socket data directly if payload found. Refer to: https://werkzeug.palletsprojects.com/en/2.0.x/wsgi/#werkzeug.wsgi.LimitedStream
This method makes assumptions because the request payload exists for operating `POST/PATCH/PUT`.
A simple demo for uploading 5G large file in low memory based on docker:
```
[root@control-master tmp]# docker run -ti --rm --memory 1G --tmpfs /tmp:rw,size=1G,mode=1777 -v $(pwd)/go:/go golang:1.16.5 /bin/bash
root@b672a98e5314:/go/testmodules# cat main.go
package main
import (
"fmt"
"github.com/gin-gonic/gin"
"io"
"os"
"path/filepath"
)
func main() {
engine := gin.Default()
engine.POST("/streamupload", func(c *gin.Context) {
if c.GetHeader("Content-Type") != "application/octet-stream" {
err := fmt.Errorf("required octet-stream")
c.AbortWithStatusJSON(400, map[string]string{"message": err.Error()})
return
}
info, err := os.Create(filepath.Join("/home", "foo"))
if err != nil {
c.AbortWithStatusJSON(400, gin.H{"message": "Create: "+err.Error()})
return
}
defer info.Close()
_, err = io.Copy(info, c.Request.Body)
if err != nil {
c.AbortWithStatusJSON(400, gin.H{"message": "Copy: "+err.Error()})
return
}
c.JSON(200, map[string]string{"message": "ok stream"})
})
if err := engine.Run("0.0.0.0:19090"); err != nil {
panic(err)
}
}
root@b53810b3e294:/go/testmodules# GOTMPDIR=/opt/ go run main.go
[root@control-master ~]# dd if=/dev/zero of=5G bs=1M count=5170 status=progress
[root@control-master ~]# curl -vvv -H "Content-Type:application/octet-stream" -T 5G -X POST 172.17.0.2:19090/streamupload
```
Signed-off-by: Chenyang Yan <memory.yancy@gmail.com>
* Update tree.go (#2659)
delete more "()"
* updated comments for Get function for params (#2756)
* ci: add github action workflows (#2596)
* ci: add github action workflows
* test: fixed the TestUnixSocket test on windows (#20)
* ci: add github action workflows (#18)
* Remove .travis.yml
* ci: replace GITTER_ROOM_ID and upload coverage every time you go test
* ci: update coverage using codecov/codecov-action@v1
* Merge branch 'master' into github-actions
* repo: replace travis ci to github actions
* ci: add go version 1.16
* fix: go install requires a specific version
* chore(ci): remove go 1.12 support
* chore(ci): remove os windows-latest
Co-authored-by: thinkerou <thinkerou@gmail.com>
Co-authored-by: Bo-Yi Wu <appleboy.tw@gmail.com>
* Setting trusted platform using an enum-like (#2739)
* gin.Context with fallback value from c.Request.Context()
* add test case
Co-authored-by: youzeliang <youzel@126.com>
Co-authored-by: Ashwani <ashwanisharma686@gmail.com>
Co-authored-by: Jeff <laojianzi1994@gmail.com>
Co-authored-by: thinkerou <thinkerou@gmail.com>
Co-authored-by: Bo-Yi Wu <appleboy.tw@gmail.com>
Co-authored-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com>
* support bind http header param #1956
update #1956
```
package main
import (
"fmt"
"github.com/gin-gonic/gin"
)
type testHeader struct {
Rate int `header:"Rate"`
Domain string `header:"Domain"`
}
func main() {
r := gin.Default()
r.GET("/", func(c *gin.Context) {
h := testHeader{}
if err := c.ShouldBindHeader(&h); err != nil {
c.JSON(200, err)
}
fmt.Printf("%#v\n", h)
c.JSON(200, gin.H{"Rate": h.Rate, "Domain": h.Domain})
})
r.Run()
// client
// curl -H "rate:300" -H "domain:music" 127.0.0.1:8080/
// output
// {"Domain":"music","Rate":300}
}
```
* add unit test
* Modify the code to get the http header
When the http header is obtained in the standard library,
the key value will be modified by the CanonicalMIMEHeaderKey function,
and finally the value of the http header will be obtained from the map.
As follows.
```go
func (h MIMEHeader) Get(key string) string {
// ...
v := h[CanonicalMIMEHeaderKey(key)]
// ...
}
```
This pr also follows this modification
* Thanks to vkd for suggestions, modifying code
* Increase test coverage
env GOPATH=`pwd` go test github.com/gin-gonic/gin/binding -coverprofile=cover.prof
ok github.com/gin-gonic/gin/binding 0.015s coverage: 100.0% of statements
* Rollback check code
* add use case to README.md
* Fix context.Params race condition on Copy()
Using context.Param(key) on a context.Copy inside a goroutine
may lead to incorrect value on a high load, where another request
overwrite a Param
* Using waitgroup to wait asynchronous test case
* Add context.HandlerNames()
This change adds a HandlerNames method that will return all registered handles in the context, in descending order
This is useful for debugging and troubleshooting purposes, especially in large apps
* Tests
Add tests for HandlerNames
* Fix HandlerNames test
* Simplify test
* fix Context.Next() - recheck len of handlers every iteration
* add tests when Context.reset() can be called inside of handler
TestEngineHandleContext
TestContextResetInHandler
TestRouterStaticFSFileNotFound
* Context.Next() - format to while style
When `gin.Context.FormFile("...")` is called the `engine.MaxMultipartMemory` is never used. This PR makes sure that the `MaxMultipartMemory` is passed and removes 2 calls to `http.Request.ParseForm` since they are called from `http.Request.ParseMultipartForm`
