sevenhill/QSfera
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial QSfera import

Browse Source
This commit is contained in:
Курнат Андрей
2026-06-07 10:20:04 +03:00
commit 2315f25754
16485 changed files with 4826827 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Server/vendor/github.com/minio/crc64nvme/LICENSE
Generated Vendored
+202
View File
@@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Server/vendor/github.com/minio/crc64nvme/README.md
Generated Vendored
+20
View File
@@ -0,0 +1,20 @@
## crc64nvme
This Golang package calculates CRC64 checksums using carryless-multiplication accelerated with SIMD instructions for both ARM and x86. It is based on the NVME polynomial as specified in the [NVM Express® NVM Command Set Specification](https://nvmexpress.org/wp-content/uploads/NVM-Express-NVM-Command-Set-Specification-1.0d-2023.12.28-Ratified.pdf).
The code is based on the [crc64fast-nvme](https://github.com/awesomized/crc64fast-nvme.git) package in Rust and is released under the Apache 2.0 license.
For more background on the exact technique used, see this [Fast CRC Computation for Generic Polynomials Using PCLMULQDQ Instruction](https://web.archive.org/web/20131224125630/https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/fast-crc-computation-generic-polynomials-pclmulqdq-paper.pdf) paper.
### Performance
To follow.
### Requirements
All Go versions >= 1.22 are supported.
### Contributing
Contributions are welcome, please send PRs for any enhancements.
Server/vendor/github.com/minio/crc64nvme/crc64.go
Generated Vendored
+185
View File
@@ -0,0 +1,185 @@
// Copyright (c) 2025 Minio Inc. All rights reserved.
// Use of this source code is governed by a license that can be
// found in the LICENSE file.
// Package crc64nvme implements the 64-bit cyclic redundancy check with NVME polynomial.
package crc64nvme
import (
"encoding/binary"
"errors"
"hash"
"sync"
"unsafe"
)
const (
// The size of a CRC-64 checksum in bytes.
Size = 8
// The NVME polynoimial (reversed, as used by Go)
NVME = 0x9a6c9329ac4bc9b5
)
var (
// precalculated table.
nvmeTable = makeTable(NVME)
)
// table is a 256-word table representing the polynomial for efficient processing.
type table [256]uint64
var (
slicing8TablesBuildOnce sync.Once
slicing8TableNVME *[8]table
)
func buildSlicing8TablesOnce() {
slicing8TablesBuildOnce.Do(buildSlicing8Tables)
}
func buildSlicing8Tables() {
slicing8TableNVME = makeSlicingBy8Table(makeTable(NVME))
}
func makeTable(poly uint64) *table {
t := new(table)
for i := 0; i < 256; i++ {
crc := uint64(i)
for j := 0; j < 8; j++ {
if crc&1 == 1 {
crc = (crc >> 1) ^ poly
} else {
crc >>= 1
}
}
t[i] = crc
}
return t
}
func makeSlicingBy8Table(t *table) *[8]table {
var helperTable [8]table
helperTable[0] = *t
for i := 0; i < 256; i++ {
crc := t[i]
for j := 1; j < 8; j++ {
crc = t[crc&0xff] ^ (crc >> 8)
helperTable[j][i] = crc
}
}
return &helperTable
}
// digest represents the partial evaluation of a checksum.
type digest struct {
crc uint64
}
// New creates a new hash.Hash64 computing the CRC-64 checksum using the
// NVME polynomial. Its Sum method will lay the
// value out in big-endian byte order. The returned Hash64 also
// implements [encoding.BinaryMarshaler] and [encoding.BinaryUnmarshaler] to
// marshal and unmarshal the internal state of the hash.
func New() hash.Hash64 { return &digest{0} }
func (d *digest) Size() int { return Size }
func (d *digest) BlockSize() int { return 1 }
func (d *digest) Reset() { d.crc = 0 }
const (
magic = "crc\x02"
marshaledSize = len(magic) + 8 + 8
)
func (d *digest) MarshalBinary() ([]byte, error) {
b := make([]byte, 0, marshaledSize)
b = append(b, magic...)
b = binary.BigEndian.AppendUint64(b, tableSum)
b = binary.BigEndian.AppendUint64(b, d.crc)
return b, nil
}
func (d *digest) UnmarshalBinary(b []byte) error {
if len(b) < len(magic) || string(b[:len(magic)]) != magic {
return errors.New("hash/crc64: invalid hash state identifier")
}
if len(b) != marshaledSize {
return errors.New("hash/crc64: invalid hash state size")
}
if tableSum != binary.BigEndian.Uint64(b[4:]) {
return errors.New("hash/crc64: tables do not match")
}
d.crc = binary.BigEndian.Uint64(b[12:])
return nil
}
func update(crc uint64, p []byte) uint64 {
if hasAsm && len(p) > 127 {
ptr := unsafe.Pointer(&p[0])
if align := (uintptr(ptr)+15)&^0xf - uintptr(ptr); align > 0 {
// Align to 16-byte boundary.
crc = update(crc, p[:align])
p = p[align:]
}
runs := len(p) / 128
if hasAsm512 && runs >= 8 {
// Use 512-bit wide instructions for >= 1KB.
crc = updateAsm512(crc, p[:128*runs])
} else if runs > 0 {
crc = updateAsm(crc, p[:128*runs])
}
return update(crc, p[128*runs:])
}
buildSlicing8TablesOnce()
crc = ^crc
// table comparison is somewhat expensive, so avoid it for small sizes
if len(p) >= 64 {
var helperTable = slicing8TableNVME
// Update using slicing-by-8
for len(p) > 8 {
crc ^= binary.LittleEndian.Uint64(p)
crc = helperTable[7][crc&0xff] ^
helperTable[6][(crc>>8)&0xff] ^
helperTable[5][(crc>>16)&0xff] ^
helperTable[4][(crc>>24)&0xff] ^
helperTable[3][(crc>>32)&0xff] ^
helperTable[2][(crc>>40)&0xff] ^
helperTable[1][(crc>>48)&0xff] ^
helperTable[0][crc>>56]
p = p[8:]
}
}
// For reminders or small sizes
for _, v := range p {
crc = nvmeTable[byte(crc)^v] ^ (crc >> 8)
}
return ^crc
}
// Update returns the result of adding the bytes in p to the crc.
func Update(crc uint64, p []byte) uint64 {
return update(crc, p)
}
func (d *digest) Write(p []byte) (n int, err error) {
d.crc = update(d.crc, p)
return len(p), nil
}
func (d *digest) Sum64() uint64 { return d.crc }
func (d *digest) Sum(in []byte) []byte {
s := d.Sum64()
return append(in, byte(s>>56), byte(s>>48), byte(s>>40), byte(s>>32), byte(s>>24), byte(s>>16), byte(s>>8), byte(s))
}
// Checksum returns the CRC-64 checksum of data
// using the NVME polynomial.
func Checksum(data []byte) uint64 { return update(0, data) }
// ISO tablesum of NVME poly
const tableSum = 0x8ddd9ee4402c7163
Server/vendor/github.com/minio/crc64nvme/crc64_amd64.go
Generated Vendored
+17
View File
@@ -0,0 +1,17 @@
// Copyright (c) 2025 Minio Inc. All rights reserved.
// Use of this source code is governed by a license that can be
// found in the LICENSE file.
//go:build !noasm && !appengine && !gccgo
package crc64nvme
import (
"github.com/klauspost/cpuid/v2"
)
var hasAsm = cpuid.CPU.Supports(cpuid.SSE2, cpuid.CLMUL, cpuid.SSE4)
var hasAsm512 = cpuid.CPU.Supports(cpuid.AVX512F, cpuid.VPCLMULQDQ, cpuid.AVX512VL, cpuid.CLMUL)
func updateAsm(crc uint64, p []byte) (checksum uint64)
func updateAsm512(crc uint64, p []byte) (checksum uint64)
Server/vendor/github.com/minio/crc64nvme/crc64_amd64.s
Generated Vendored
+309
View File
@@ -0,0 +1,309 @@
// Copyright (c) 2025 Minio Inc. All rights reserved.
// Use of this source code is governed by a license that can be
// found in the LICENSE file.
//go:build !noasm && !appengine && !gccgo
#include "textflag.h"
TEXT ·updateAsm(SB), $0-40
MOVQ crc+0(FP), AX // checksum
MOVQ p_base+8(FP), SI // start pointer
MOVQ p_len+16(FP), CX // length of buffer
NOTQ AX
SHRQ $7, CX
CMPQ CX, $1
JLT skip128
MOVOA 0x00(SI), X0
MOVOA 0x10(SI), X1
MOVOA 0x20(SI), X2
MOVOA 0x30(SI), X3
MOVOA 0x40(SI), X4
MOVOA 0x50(SI), X5
MOVOA 0x60(SI), X6
MOVOA 0x70(SI), X7
MOVQ AX, X8
PXOR X8, X0
CMPQ CX, $1
JE tail128
MOVQ $0xa1ca681e733f9c40, AX
MOVQ AX, X8
MOVQ $0x5f852fb61e8d92dc, AX
PINSRQ $0x1, AX, X9
loop128:
ADDQ $128, SI
SUBQ $1, CX
MOVOA X0, X10
PCLMULQDQ $0x00, X8, X10
PCLMULQDQ $0x11, X9, X0
PXOR X10, X0
PXOR 0(SI), X0
MOVOA X1, X10
PCLMULQDQ $0x00, X8, X10
PCLMULQDQ $0x11, X9, X1
PXOR X10, X1
PXOR 0x10(SI), X1
MOVOA X2, X10
PCLMULQDQ $0x00, X8, X10
PCLMULQDQ $0x11, X9, X2
PXOR X10, X2
PXOR 0x20(SI), X2
MOVOA X3, X10
PCLMULQDQ $0x00, X8, X10
PCLMULQDQ $0x11, X9, X3
PXOR X10, X3
PXOR 0x30(SI), X3
MOVOA X4, X10
PCLMULQDQ $0x00, X8, X10
PCLMULQDQ $0x11, X9, X4
PXOR X10, X4
PXOR 0x40(SI), X4
MOVOA X5, X10
PCLMULQDQ $0x00, X8, X10
PCLMULQDQ $0x11, X9, X5
PXOR X10, X5
PXOR 0x50(SI), X5
MOVOA X6, X10
PCLMULQDQ $0x00, X8, X10
PCLMULQDQ $0x11, X9, X6
PXOR X10, X6
PXOR 0x60(SI), X6
MOVOA X7, X10
PCLMULQDQ $0x00, X8, X10
PCLMULQDQ $0x11, X9, X7
PXOR X10, X7
PXOR 0x70(SI), X7
CMPQ CX, $1
JGT loop128
tail128:
MOVQ $0xd083dd594d96319d, AX
MOVQ AX, X11
PCLMULQDQ $0x00, X0, X11
MOVQ $0x946588403d4adcbc, AX
PINSRQ $0x1, AX, X12
PCLMULQDQ $0x11, X12, X0
PXOR X11, X7
PXOR X0, X7
MOVQ $0x3c255f5ebc414423, AX
MOVQ AX, X11
PCLMULQDQ $0x00, X1, X11
MOVQ $0x34f5a24e22d66e90, AX
PINSRQ $0x1, AX, X12
PCLMULQDQ $0x11, X12, X1
PXOR X11, X1
PXOR X7, X1
MOVQ $0x7b0ab10dd0f809fe, AX
MOVQ AX, X11
PCLMULQDQ $0x00, X2, X11
MOVQ $0x03363823e6e791e5, AX
PINSRQ $0x1, AX, X12
PCLMULQDQ $0x11, X12, X2
PXOR X11, X2
PXOR X1, X2
MOVQ $0x0c32cdb31e18a84a, AX
MOVQ AX, X11
PCLMULQDQ $0x00, X3, X11
MOVQ $0x62242240ace5045a, AX
PINSRQ $0x1, AX, X12
PCLMULQDQ $0x11, X12, X3
PXOR X11, X3
PXOR X2, X3
MOVQ $0xbdd7ac0ee1a4a0f0, AX
MOVQ AX, X11
PCLMULQDQ $0x00, X4, X11
MOVQ $0xa3ffdc1fe8e82a8b, AX
PINSRQ $0x1, AX, X12
PCLMULQDQ $0x11, X12, X4
PXOR X11, X4
PXOR X3, X4
MOVQ $0xb0bc2e589204f500, AX
MOVQ AX, X11
PCLMULQDQ $0x00, X5, X11
MOVQ $0xe1e0bb9d45d7a44c, AX
PINSRQ $0x1, AX, X12
PCLMULQDQ $0x11, X12, X5
PXOR X11, X5
PXOR X4, X5
MOVQ $0xeadc41fd2ba3d420, AX
MOVQ AX, X11
PCLMULQDQ $0x00, X6, X11
MOVQ $0x21e9761e252621ac, AX
PINSRQ $0x1, AX, X12
PCLMULQDQ $0x11, X12, X6
PXOR X11, X6
PXOR X5, X6
MOVQ AX, X5
PCLMULQDQ $0x00, X6, X5
PSHUFD $0xee, X6, X6
PXOR X5, X6
MOVQ $0x27ecfa329aef9f77, AX
MOVQ AX, X4
PCLMULQDQ $0x00, X4, X6
PEXTRQ $0, X6, BX
MOVQ $0x34d926535897936b, AX
MOVQ AX, X4
PCLMULQDQ $0x00, X4, X6
PXOR X5, X6
PEXTRQ $1, X6, AX
XORQ BX, AX
skip128:
NOTQ AX
MOVQ AX, checksum+32(FP)
RET
// Constants, pre-splatted.
DATA ·asmConstantsPoly<>+0x00(SB)/8, $0xa1ca681e733f9c40
DATA ·asmConstantsPoly<>+0x08(SB)/8, $0
DATA ·asmConstantsPoly<>+0x10(SB)/8, $0xa1ca681e733f9c40
DATA ·asmConstantsPoly<>+0x18(SB)/8, $0
DATA ·asmConstantsPoly<>+0x20(SB)/8, $0xa1ca681e733f9c40
DATA ·asmConstantsPoly<>+0x28(SB)/8, $0
DATA ·asmConstantsPoly<>+0x30(SB)/8, $0xa1ca681e733f9c40
DATA ·asmConstantsPoly<>+0x38(SB)/8, $0
// Upper
DATA ·asmConstantsPoly<>+0x40(SB)/8, $0
DATA ·asmConstantsPoly<>+0x48(SB)/8, $0x5f852fb61e8d92dc
DATA ·asmConstantsPoly<>+0x50(SB)/8, $0
DATA ·asmConstantsPoly<>+0x58(SB)/8, $0x5f852fb61e8d92dc
DATA ·asmConstantsPoly<>+0x60(SB)/8, $0
DATA ·asmConstantsPoly<>+0x68(SB)/8, $0x5f852fb61e8d92dc
DATA ·asmConstantsPoly<>+0x70(SB)/8, $0
DATA ·asmConstantsPoly<>+0x78(SB)/8, $0x5f852fb61e8d92dc
GLOBL ·asmConstantsPoly<>(SB), (NOPTR+RODATA), $128
TEXT ·updateAsm512(SB), $0-40
MOVQ crc+0(FP), AX // checksum
MOVQ p_base+8(FP), SI // start pointer
MOVQ p_len+16(FP), CX // length of buffer
NOTQ AX
SHRQ $7, CX
CMPQ CX, $1
VPXORQ Z8, Z8, Z8 // Initialize ZMM8 to zero
JLT skip128
VMOVDQU64 0x00(SI), Z0
VMOVDQU64 0x40(SI), Z4
MOVQ $·asmConstantsPoly<>(SB), BX
VMOVQ AX, X8
// XOR initialization value into lower 64 bits of ZMM0
VPXORQ Z8, Z0, Z0
CMPQ CX, $1
JE tail128
VMOVDQU64 0(BX), Z8
VMOVDQU64 64(BX), Z9
PCALIGN $16
loop128:
PREFETCHT0 512(SI)
VMOVDQU64 0x80(SI), Z1
VMOVDQU64 0xc0(SI), Z5
ADDQ $128, SI
SUBQ $1, CX
VPCLMULQDQ $0x00, Z8, Z0, Z10
VPCLMULQDQ $0x11, Z9, Z0, Z0
VPTERNLOGD $0x96, Z1, Z10, Z0 // Combine results with xor into Z0
PREFETCHT0 512-64(SI)
VPCLMULQDQ $0x00, Z8, Z4, Z10
VPCLMULQDQ $0x11, Z9, Z4, Z4
VPTERNLOGD $0x96, Z5, Z10, Z4 // Combine results with xor into Z4
CMPQ CX, $1
JGT loop128
tail128:
// Extract X0 to X3 from ZMM0
VEXTRACTF32X4 $1, Z0, X1 // X1: Second 128-bit lane
VEXTRACTF32X4 $2, Z0, X2 // X2: Third 128-bit lane
VEXTRACTF32X4 $3, Z0, X3 // X3: Fourth 128-bit lane
// Extract X4 to X7 from ZMM4
VEXTRACTF32X4 $1, Z4, X5 // X5: Second 128-bit lane
VEXTRACTF32X4 $2, Z4, X6 // X6: Third 128-bit lane
VEXTRACTF32X4 $3, Z4, X7 // X7: Fourth 128-bit lane
MOVQ $0xd083dd594d96319d, AX
MOVQ AX, X11
PCLMULQDQ $0x00, X0, X11
MOVQ $0x946588403d4adcbc, AX
PINSRQ $0x1, AX, X12
PCLMULQDQ $0x11, X12, X0
PXOR X11, X7
PXOR X0, X7
MOVQ $0x3c255f5ebc414423, AX
MOVQ AX, X11
PCLMULQDQ $0x00, X1, X11
MOVQ $0x34f5a24e22d66e90, AX
PINSRQ $0x1, AX, X12
PCLMULQDQ $0x11, X12, X1
PXOR X11, X1
PXOR X7, X1
MOVQ $0x7b0ab10dd0f809fe, AX
MOVQ AX, X11
PCLMULQDQ $0x00, X2, X11
MOVQ $0x03363823e6e791e5, AX
PINSRQ $0x1, AX, X12
PCLMULQDQ $0x11, X12, X2
PXOR X11, X2
PXOR X1, X2
MOVQ $0x0c32cdb31e18a84a, AX
MOVQ AX, X11
PCLMULQDQ $0x00, X3, X11
MOVQ $0x62242240ace5045a, AX
PINSRQ $0x1, AX, X12
PCLMULQDQ $0x11, X12, X3
PXOR X11, X3
PXOR X2, X3
MOVQ $0xbdd7ac0ee1a4a0f0, AX
MOVQ AX, X11
PCLMULQDQ $0x00, X4, X11
MOVQ $0xa3ffdc1fe8e82a8b, AX
PINSRQ $0x1, AX, X12
PCLMULQDQ $0x11, X12, X4
PXOR X11, X4
PXOR X3, X4
MOVQ $0xb0bc2e589204f500, AX
MOVQ AX, X11
PCLMULQDQ $0x00, X5, X11
MOVQ $0xe1e0bb9d45d7a44c, AX
PINSRQ $0x1, AX, X12
PCLMULQDQ $0x11, X12, X5
PXOR X11, X5
PXOR X4, X5
MOVQ $0xeadc41fd2ba3d420, AX
MOVQ AX, X11
PCLMULQDQ $0x00, X6, X11
MOVQ $0x21e9761e252621ac, AX
PINSRQ $0x1, AX, X12
PCLMULQDQ $0x11, X12, X6
PXOR X11, X6
PXOR X5, X6
MOVQ AX, X5
PCLMULQDQ $0x00, X6, X5
PSHUFD $0xee, X6, X6
PXOR X5, X6
MOVQ $0x27ecfa329aef9f77, AX
MOVQ AX, X4
PCLMULQDQ $0x00, X4, X6
PEXTRQ $0, X6, BX
MOVQ $0x34d926535897936b, AX
MOVQ AX, X4
PCLMULQDQ $0x00, X4, X6
PXOR X5, X6
PEXTRQ $1, X6, AX
XORQ BX, AX
skip128:
NOTQ AX
MOVQ AX, checksum+32(FP)
VZEROUPPER
RET
Server/vendor/github.com/minio/crc64nvme/crc64_arm64.go
Generated Vendored
+17
View File
@@ -0,0 +1,17 @@
// Copyright (c) 2025 Minio Inc. All rights reserved.
// Use of this source code is governed by a license that can be
// found in the LICENSE file.
//go:build !noasm && !appengine && !gccgo
package crc64nvme
import (
"github.com/klauspost/cpuid/v2"
)
var hasAsm = cpuid.CPU.Supports(cpuid.ASIMD, cpuid.PMULL, cpuid.SHA3)
var hasAsm512 = false
func updateAsm(crc uint64, p []byte) (checksum uint64)
func updateAsm512(crc uint64, p []byte) (checksum uint64) { panic("should not be reached") }
Server/vendor/github.com/minio/crc64nvme/crc64_arm64.s
Generated Vendored
+157
View File
@@ -0,0 +1,157 @@
// Copyright (c) 2025 Minio Inc. All rights reserved.
// Use of this source code is governed by a license that can be
// found in the LICENSE file.
//go:build !noasm && !appengine && !gccgo
#include "textflag.h"
TEXT ·updateAsm(SB), $0-40
MOVD crc+0(FP), R0 // checksum
MOVD p_base+8(FP), R1 // start pointer
MOVD p_len+16(FP), R2 // length of buffer
MOVD $·const(SB), R3 // constants
MVN R0, R0
LSR $7, R2, R2
CMP $1, R2
BLT skip128
FLDPQ (R1), (F0, F1)
FLDPQ 32(R1), (F2, F3)
FLDPQ 64(R1), (F4, F5)
FLDPQ 96(R1), (F6, F7)
FMOVD R0, F8
VMOVI $0, V9.B16
VMOV V9.D[0], V8.D[1]
VEOR V8.B16, V0.B16, V0.B16
CMP $1, R2
BEQ tail128
MOVD 112(R3), R4
MOVD 120(R3), R5
FMOVD R4, F8
VDUP R5, V9.D2
loop128:
ADD $128, R1, R1
SUB $1, R2, R2
VPMULL V0.D1, V8.D1, V10.Q1
VPMULL2 V0.D2, V9.D2, V0.Q1
FLDPQ (R1), (F11, F12)
VEOR3 V0.B16, V11.B16, V10.B16, V0.B16
VPMULL V1.D1, V8.D1, V10.Q1
VPMULL2 V1.D2, V9.D2, V1.Q1
VEOR3 V1.B16, V12.B16, V10.B16, V1.B16
VPMULL V2.D1, V8.D1, V10.Q1
VPMULL2 V2.D2, V9.D2, V2.Q1
FLDPQ 32(R1), (F11, F12)
VEOR3 V2.B16, V11.B16, V10.B16, V2.B16
VPMULL V3.D1, V8.D1, V10.Q1
VPMULL2 V3.D2, V9.D2, V3.Q1
VEOR3 V3.B16, V12.B16, V10.B16, V3.B16
VPMULL V4.D1, V8.D1, V10.Q1
VPMULL2 V4.D2, V9.D2, V4.Q1
FLDPQ 64(R1), (F11, F12)
VEOR3 V4.B16, V11.B16, V10.B16, V4.B16
VPMULL V5.D1, V8.D1, V10.Q1
VPMULL2 V5.D2, V9.D2, V5.Q1
VEOR3 V5.B16, V12.B16, V10.B16, V5.B16
VPMULL V6.D1, V8.D1, V10.Q1
VPMULL2 V6.D2, V9.D2, V6.Q1
FLDPQ 96(R1), (F11, F12)
VEOR3 V6.B16, V11.B16, V10.B16, V6.B16
VPMULL V7.D1, V8.D1, V10.Q1
VPMULL2 V7.D2, V9.D2, V7.Q1
VEOR3 V7.B16, V12.B16, V10.B16, V7.B16
CMP $1, R2
BHI loop128
tail128:
MOVD (R3), R4
FMOVD R4, F11
VPMULL V0.D1, V11.D1, V11.Q1
MOVD 8(R3), R4
VDUP R4, V12.D2
VPMULL2 V0.D2, V12.D2, V0.Q1
VEOR3 V0.B16, V7.B16, V11.B16, V7.B16
MOVD 16(R3), R4
FMOVD R4, F11
VPMULL V1.D1, V11.D1, V11.Q1
MOVD 24(R3), R4
VDUP R4, V12.D2
VPMULL2 V1.D2, V12.D2, V1.Q1
VEOR3 V1.B16, V11.B16, V7.B16, V1.B16
MOVD 32(R3), R4
FMOVD R4, F11
VPMULL V2.D1, V11.D1, V11.Q1
MOVD 40(R3), R4
VDUP R4, V12.D2
VPMULL2 V2.D2, V12.D2, V2.Q1
VEOR3 V2.B16, V11.B16, V1.B16, V2.B16
MOVD 48(R3), R4
FMOVD R4, F11
VPMULL V3.D1, V11.D1, V11.Q1
MOVD 56(R3), R4
VDUP R4, V12.D2
VPMULL2 V3.D2, V12.D2, V3.Q1
VEOR3 V3.B16, V11.B16, V2.B16, V3.B16
MOVD 64(R3), R4
FMOVD R4, F11
VPMULL V4.D1, V11.D1, V11.Q1
MOVD 72(R3), R4
VDUP R4, V12.D2
VPMULL2 V4.D2, V12.D2, V4.Q1
VEOR3 V4.B16, V11.B16, V3.B16, V4.B16
MOVD 80(R3), R4
FMOVD R4, F11
VPMULL V5.D1, V11.D1, V11.Q1
MOVD 88(R3), R4
VDUP R4, V12.D2
VPMULL2 V5.D2, V12.D2, V5.Q1
VEOR3 V5.B16, V11.B16, V4.B16, V5.B16
MOVD 96(R3), R4
FMOVD R4, F11
VPMULL V6.D1, V11.D1, V11.Q1
MOVD 104(R3), R4
VDUP R4, V12.D2
VPMULL2 V6.D2, V12.D2, V6.Q1
VEOR3 V6.B16, V11.B16, V5.B16, V6.B16
FMOVD R4, F5
VPMULL V6.D1, V5.D1, V5.Q1
VDUP V6.D[1], V6.D2
VEOR V5.B8, V6.B8, V6.B8
MOVD 128(R3), R4
FMOVD R4, F4
VPMULL V4.D1, V6.D1, V6.Q1
FMOVD F6, R4
MOVD 136(R3), R5
FMOVD R5, F4
VPMULL V4.D1, V6.D1, V6.Q1
VEOR V6.B16, V5.B16, V6.B16
VMOV V6.D[1], R5
EOR R4, R5, R0
skip128:
MVN R0, R0
MOVD R0, checksum+32(FP)
RET
DATA ·const+0x000(SB)/8, $0xd083dd594d96319d // K_959
DATA ·const+0x008(SB)/8, $0x946588403d4adcbc // K_895
DATA ·const+0x010(SB)/8, $0x3c255f5ebc414423 // K_831
DATA ·const+0x018(SB)/8, $0x34f5a24e22d66e90 // K_767
DATA ·const+0x020(SB)/8, $0x7b0ab10dd0f809fe // K_703
DATA ·const+0x028(SB)/8, $0x03363823e6e791e5 // K_639
DATA ·const+0x030(SB)/8, $0x0c32cdb31e18a84a // K_575
DATA ·const+0x038(SB)/8, $0x62242240ace5045a // K_511
DATA ·const+0x040(SB)/8, $0xbdd7ac0ee1a4a0f0 // K_447
DATA ·const+0x048(SB)/8, $0xa3ffdc1fe8e82a8b // K_383
DATA ·const+0x050(SB)/8, $0xb0bc2e589204f500 // K_319
DATA ·const+0x058(SB)/8, $0xe1e0bb9d45d7a44c // K_255
DATA ·const+0x060(SB)/8, $0xeadc41fd2ba3d420 // K_191
DATA ·const+0x068(SB)/8, $0x21e9761e252621ac // K_127
DATA ·const+0x070(SB)/8, $0xa1ca681e733f9c40 // K_1087
DATA ·const+0x078(SB)/8, $0x5f852fb61e8d92dc // K_1023
DATA ·const+0x080(SB)/8, $0x27ecfa329aef9f77 // MU
DATA ·const+0x088(SB)/8, $0x34d926535897936b // POLY
GLOBL ·const(SB), (NOPTR+RODATA), $144
Server/vendor/github.com/minio/crc64nvme/crc64_other.go
Generated Vendored
+13
View File
@@ -0,0 +1,13 @@
// Copyright (c) 2025 Minio Inc. All rights reserved.
// Use of this source code is governed by a license that can be
// found in the LICENSE file.
//go:build (!amd64 || noasm || appengine || gccgo) && (!arm64 || noasm || appengine || gccgo)
package crc64nvme
var hasAsm = false
var hasAsm512 = false
func updateAsm(crc uint64, p []byte) (checksum uint64) { panic("should not be reached") }
func updateAsm512(crc uint64, p []byte) (checksum uint64) { panic("should not be reached") }
Server/vendor/github.com/minio/highwayhash/.gitignore
Generated
+1
View File
@@ -0,0 +1 @@
*.test
Server/vendor/github.com/minio/highwayhash/.golangci.yml
Generated Vendored
+28
View File
@@ -0,0 +1,28 @@
linters-settings:
golint:
min-confidence: 0
misspell:
locale: US
linters:
disable-all: true
enable:
- typecheck
- goimports
- misspell
- govet
- revive
- ineffassign
- gosimple
- unparam
- unused
issues:
exclude-use-default: false
exclude:
- should have a package comment
- error strings should not be capitalized or end with punctuation or a newline
- should have comment # TODO(aead): Remove once all exported ident. have comments!
service:
golangci-lint-version: 1.51.2 # use the fixed version to not introduce new linters unexpectedly
Server/vendor/github.com/minio/highwayhash/LICENSE
Generated Vendored
+202
View File
@@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Server/vendor/github.com/minio/highwayhash/README.md
Generated Vendored
+99
View File
@@ -0,0 +1,99 @@
[![Godoc Reference](https://godoc.org/github.com/minio/highwayhash?status.svg)](https://godoc.org/github.com/minio/highwayhash)
[![Build Status](https://travis-ci.org/minio/highwayhash.svg?branch=master)](https://travis-ci.org/minio/highwayhash)
## HighwayHash
[HighwayHash](https://github.com/google/highwayhash) is a pseudo-random-function (PRF) developed by Jyrki Alakuijala, Bill Cox and Jan Wassenberg (Google research). HighwayHash takes a 256 bit key and computes 64, 128 or 256 bit hash values of given messages.
It can be used to prevent hash-flooding attacks or authenticate short-lived messages. Additionally it can be used as a fingerprinting function. HighwayHash is not a general purpose cryptographic hash function (such as Blake2b, SHA-3 or SHA-2) and should not be used if strong collision resistance is required.
This repository contains a native Go version and optimized assembly implementations for Intel, ARM and ppc64le architectures.
### High performance
HighwayHash is an approximately 5x faster SIMD hash function as compared to [SipHash](https://www.131002.net/siphash/siphash.pdf) which in itself is a fast and 'cryptographically strong' pseudo-random function designed by Aumasson and Bernstein.
HighwayHash uses a new way of mixing inputs with AVX2 multiply and permute instructions. The multiplications are 32x32 bit giving 64 bits-wide results and are therefore infeasible to reverse. Additionally permuting equalizes the distribution of the resulting bytes. The algorithm outputs digests ranging from 64 bits up to 256 bits at no extra cost.
### Stable
All three output sizes of HighwayHash have been declared [stable](https://github.com/google/highwayhash/#versioning-and-stability) as of January 2018. This means that the hash results for any given input message are guaranteed not to change.
### Installation
Install: `go get -u github.com/minio/highwayhash`
### Intel Performance
Below are the single core results on an Intel Core i7 (3.1 GHz) for 256 bit outputs:
```
BenchmarkSum256_16 204.17 MB/s
BenchmarkSum256_64 1040.63 MB/s
BenchmarkSum256_1K 8653.30 MB/s
BenchmarkSum256_8K 13476.07 MB/s
BenchmarkSum256_1M 14928.71 MB/s
BenchmarkSum256_5M 14180.04 MB/s
BenchmarkSum256_10M 12458.65 MB/s
BenchmarkSum256_25M 11927.25 MB/s
```
So for moderately sized messages it tops out at about 15 GB/sec. Also for small messages (1K) the performance is already at approximately 60% of the maximum throughput.
### ARM Performance
Below are the single core results on an EC2 c7g.4xlarge (Graviton3) instance for 256 bit outputs:
```
BenchmarkSum256_16 143.66 MB/s
BenchmarkSum256_64 628.75 MB/s
BenchmarkSum256_1K 3621.71 MB/s
BenchmarkSum256_8K 5039.64 MB/s
BenchmarkSum256_1M 5279.79 MB/s
BenchmarkSum256_5M 5474.60 MB/s
BenchmarkSum256_10M 5621.73 MB/s
BenchmarkSum256_25M 5250.47 MB/s
```
### ppc64le Performance
The ppc64le accelerated version is roughly 10x faster compared to the non-optimized version:
```
benchmark old MB/s new MB/s speedup
BenchmarkWrite_8K 531.19 5566.41 10.48x
BenchmarkSum64_8K 518.86 4971.88 9.58x
BenchmarkSum256_8K 502.45 4474.20 8.90x
```
### Performance compared to other hashing techniques
On a Skylake CPU (3.0 GHz Xeon Platinum 8124M) the table below shows how HighwayHash compares to other hashing techniques for 5 MB messages (single core performance, all Golang implementations, see [benchmark](https://github.com/fwessels/HashCompare/blob/master/benchmarks_test.go)).
```
BenchmarkHighwayHash 11986.98 MB/s
BenchmarkSHA256_AVX512 3552.74 MB/s
BenchmarkBlake2b 972.38 MB/s
BenchmarkSHA1 950.64 MB/s
BenchmarkMD5 684.18 MB/s
BenchmarkSHA512 562.04 MB/s
BenchmarkSHA256 383.07 MB/s
```
*Note: the AVX512 version of SHA256 uses the [multi-buffer crypto library](https://github.com/intel/intel-ipsec-mb) technique as developed by Intel, more details can be found in [sha256-simd](https://github.com/minio/sha256-simd/).*
### Qualitative assessment
We have performed a 'qualitative' assessment of how HighwayHash compares to Blake2b in terms of the distribution of the checksums for varying numbers of messages. It shows that HighwayHash behaves similarly according to the following graph:
![Hash Comparison Overview](https://s3.amazonaws.com/s3git-assets/hash-comparison-final.png)
More information can be found in [HashCompare](https://github.com/fwessels/HashCompare).
### Requirements
All Go versions >= 1.11 are supported (needed for required assembly support for the different platforms).
### Contributing
Contributions are welcome, please send PRs for any enhancements.
Server/vendor/github.com/minio/highwayhash/highwayhash.go
Generated Vendored
+250
View File
@@ -0,0 +1,250 @@
// Copyright (c) 2017 Minio Inc. All rights reserved.
// Use of this source code is governed by a license that can be
// found in the LICENSE file.
// Package highwayhash implements the pseudo-random-function (PRF) HighwayHash.
// HighwayHash is a fast hash function designed to defend hash-flooding attacks
// or to authenticate short-lived messages.
//
// HighwayHash is not a general purpose cryptographic hash function and does not
// provide (strong) collision resistance.
package highwayhash
import (
"encoding/binary"
"errors"
"hash"
)
const (
// Size is the size of HighwayHash-256 checksum in bytes.
Size = 32
// Size128 is the size of HighwayHash-128 checksum in bytes.
Size128 = 16
// Size64 is the size of HighwayHash-64 checksum in bytes.
Size64 = 8
)
// These will error at compile time if the interface is not conformant.
var _ hash.Hash = &Digest{}
var _ hash.Hash = &Digest64{}
var errKeySize = errors.New("highwayhash: invalid key size")
// New returns a hash.Hash computing the HighwayHash-256 checksum.
// It returns a non-nil error if the key is not 32 bytes long.
func New(key []byte) (hash.Hash, error) {
return NewDigest(key)
}
// New128 returns a hash.Hash computing the HighwayHash-128 checksum.
// It returns a non-nil error if the key is not 32 bytes long.
func New128(key []byte) (hash.Hash, error) {
return NewDigest128(key)
}
// New64 returns a hash.Hash64 computing the HighwayHash-64 checksum.
// It returns a non-nil error if the key is not 32 bytes long.
func New64(key []byte) (hash.Hash64, error) {
return NewDigest64(key)
}
// NewDigest returns a *Digest that conforms to hash.Hash computing
// the HighwayHash-256 checksum.
// It returns a non-nil error if the key is not 32 bytes long.
func NewDigest(key []byte) (*Digest, error) {
if len(key) != Size {
return nil, errKeySize
}
h := &Digest{size: Size}
copy(h.key[:], key)
h.Reset()
return h, nil
}
// NewDigest128 returns a *Digest that conforms to hash.Hash computing
// the HighwayHash-128 checksum.
// It returns a non-nil error if the key is not 32 bytes long.
func NewDigest128(key []byte) (*Digest, error) {
if len(key) != Size {
return nil, errKeySize
}
h := &Digest{size: Size128}
copy(h.key[:], key)
h.Reset()
return h, nil
}
// NewDigest64 returns a *Digest that conforms to hash.Hash computing
// the HighwayHash-64 checksum.
// It returns a non-nil error if the key is not 32 bytes long.
func NewDigest64(key []byte) (*Digest64, error) {
if len(key) != Size {
return nil, errKeySize
}
h := new(Digest64)
h.size = Size64
copy(h.key[:], key)
h.Reset()
return h, nil
}
// Sum computes the HighwayHash-256 checksum of data.
// It panics if the key is not 32 bytes long.
func Sum(data, key []byte) [Size]byte {
if len(key) != Size {
panic(errKeySize)
}
var state [16]uint64
initialize(&state, key)
if n := len(data) & (^(Size - 1)); n > 0 {
update(&state, data[:n])
data = data[n:]
}
if len(data) > 0 {
var block [Size]byte
offset := copy(block[:], data)
hashBuffer(&state, &block, offset)
}
var hash [Size]byte
finalize(hash[:], &state)
return hash
}
// Sum128 computes the HighwayHash-128 checksum of data.
// It panics if the key is not 32 bytes long.
func Sum128(data, key []byte) [Size128]byte {
if len(key) != Size {
panic(errKeySize)
}
var state [16]uint64
initialize(&state, key)
if n := len(data) & (^(Size - 1)); n > 0 {
update(&state, data[:n])
data = data[n:]
}
if len(data) > 0 {
var block [Size]byte
offset := copy(block[:], data)
hashBuffer(&state, &block, offset)
}
var hash [Size128]byte
finalize(hash[:], &state)
return hash
}
// Sum64 computes the HighwayHash-64 checksum of data.
// It panics if the key is not 32 bytes long.
func Sum64(data, key []byte) uint64 {
if len(key) != Size {
panic(errKeySize)
}
var state [16]uint64
initialize(&state, key)
if n := len(data) & (^(Size - 1)); n > 0 {
update(&state, data[:n])
data = data[n:]
}
if len(data) > 0 {
var block [Size]byte
offset := copy(block[:], data)
hashBuffer(&state, &block, offset)
}
var hash [Size64]byte
finalize(hash[:], &state)
return binary.LittleEndian.Uint64(hash[:])
}
type Digest64 struct{ Digest }
func (d *Digest64) Sum64() uint64 {
state := d.state
if d.offset > 0 {
hashBuffer(&state, &d.buffer, d.offset)
}
var hash [8]byte
finalize(hash[:], &state)
return binary.LittleEndian.Uint64(hash[:])
}
type Digest struct {
state [16]uint64 // v0 | v1 | mul0 | mul1
key, buffer [Size]byte
offset int
size int
}
func (d *Digest) Size() int { return d.size }
func (d *Digest) BlockSize() int { return Size }
func (d *Digest) Reset() {
initialize(&d.state, d.key[:])
d.offset = 0
}
func (d *Digest) Write(p []byte) (n int, err error) {
n = len(p)
if d.offset > 0 {
remaining := Size - d.offset
if n < remaining {
d.offset += copy(d.buffer[d.offset:], p)
return
}
copy(d.buffer[d.offset:], p[:remaining])
update(&d.state, d.buffer[:])
p = p[remaining:]
d.offset = 0
}
if nn := len(p) & (^(Size - 1)); nn > 0 {
update(&d.state, p[:nn])
p = p[nn:]
}
if len(p) > 0 {
d.offset = copy(d.buffer[d.offset:], p)
}
return
}
func (d *Digest) Sum(b []byte) []byte {
state := d.state
if d.offset > 0 {
hashBuffer(&state, &d.buffer, d.offset)
}
var hash [Size]byte
finalize(hash[:d.size], &state)
return append(b, hash[:d.size]...)
}
func hashBuffer(state *[16]uint64, buffer *[32]byte, offset int) {
var block [Size]byte
mod32 := (uint64(offset) << 32) + uint64(offset)
for i := range state[:4] {
state[i] += mod32
}
for i := range state[4:8] {
t0 := uint32(state[i+4])
t0 = (t0 << uint(offset)) | (t0 >> uint(32-offset))
t1 := uint32(state[i+4] >> 32)
t1 = (t1 << uint(offset)) | (t1 >> uint(32-offset))
state[i+4] = (uint64(t1) << 32) | uint64(t0)
}
mod4 := offset & 3
remain := offset - mod4
copy(block[:], buffer[:remain])
if offset >= 16 {
copy(block[28:], buffer[offset-4:])
} else if mod4 != 0 {
last := uint32(buffer[remain])
last += uint32(buffer[remain+mod4>>1]) << 8
last += uint32(buffer[offset-1]) << 16
binary.LittleEndian.PutUint32(block[16:], last)
}
update(state, block[:])
}
Server/vendor/github.com/minio/highwayhash/highwayhashAVX2_amd64.s
Generated Vendored
+248
View File
@@ -0,0 +1,248 @@
// Copyright (c) 2017 Minio Inc. All rights reserved.
// Use of this source code is governed by a license that can be
// found in the LICENSE file.
// +build amd64,!gccgo,!appengine,!nacl,!noasm
#include "textflag.h"
DATA ·consAVX2<>+0x00(SB)/8, $0xdbe6d5d5fe4cce2f
DATA ·consAVX2<>+0x08(SB)/8, $0xa4093822299f31d0
DATA ·consAVX2<>+0x10(SB)/8, $0x13198a2e03707344
DATA ·consAVX2<>+0x18(SB)/8, $0x243f6a8885a308d3
DATA ·consAVX2<>+0x20(SB)/8, $0x3bd39e10cb0ef593
DATA ·consAVX2<>+0x28(SB)/8, $0xc0acf169b5f18a8c
DATA ·consAVX2<>+0x30(SB)/8, $0xbe5466cf34e90c6c
DATA ·consAVX2<>+0x38(SB)/8, $0x452821e638d01377
GLOBL ·consAVX2<>(SB), (NOPTR+RODATA), $64
DATA ·zipperMergeAVX2<>+0x00(SB)/8, $0xf010e05020c03
DATA ·zipperMergeAVX2<>+0x08(SB)/8, $0x70806090d0a040b
DATA ·zipperMergeAVX2<>+0x10(SB)/8, $0xf010e05020c03
DATA ·zipperMergeAVX2<>+0x18(SB)/8, $0x70806090d0a040b
GLOBL ·zipperMergeAVX2<>(SB), (NOPTR+RODATA), $32
#define REDUCE_MOD(x0, x1, x2, x3, tmp0, tmp1, y0, y1) \
MOVQ $0x3FFFFFFFFFFFFFFF, tmp0 \
ANDQ tmp0, x3 \
MOVQ x2, y0 \
MOVQ x3, y1 \
\
MOVQ x2, tmp0 \
MOVQ x3, tmp1 \
SHLQ $1, tmp1 \
SHRQ $63, tmp0 \
MOVQ tmp1, x3 \
ORQ tmp0, x3 \
\
SHLQ $1, x2 \
\
MOVQ y0, tmp0 \
MOVQ y1, tmp1 \
SHLQ $2, tmp1 \
SHRQ $62, tmp0 \
MOVQ tmp1, y1 \
ORQ tmp0, y1 \
\
SHLQ $2, y0 \
\
XORQ x0, y0 \
XORQ x2, y0 \
XORQ x1, y1 \
XORQ x3, y1
#define UPDATE(msg) \
VPADDQ msg, Y2, Y2 \
VPADDQ Y3, Y2, Y2 \
\
VPSRLQ $32, Y1, Y0 \
BYTE $0xC5; BYTE $0xFD; BYTE $0xF4; BYTE $0xC2 \ // VPMULUDQ Y2, Y0, Y0
VPXOR Y0, Y3, Y3 \
\
VPADDQ Y4, Y1, Y1 \
\
VPSRLQ $32, Y2, Y0 \
BYTE $0xC5; BYTE $0xFD; BYTE $0xF4; BYTE $0xC1 \ // VPMULUDQ Y1, Y0, Y0
VPXOR Y0, Y4, Y4 \
\
VPSHUFB Y5, Y2, Y0 \
VPADDQ Y0, Y1, Y1 \
\
VPSHUFB Y5, Y1, Y0 \
VPADDQ Y0, Y2, Y2
// func initializeAVX2(state *[16]uint64, key []byte)
TEXT ·initializeAVX2(SB), 4, $0-32
MOVQ state+0(FP), AX
MOVQ key_base+8(FP), BX
MOVQ $·consAVX2<>(SB), CX
VMOVDQU 0(BX), Y1
VPSHUFD $177, Y1, Y2
VMOVDQU 0(CX), Y3
VMOVDQU 32(CX), Y4
VPXOR Y3, Y1, Y1
VPXOR Y4, Y2, Y2
VMOVDQU Y1, 0(AX)
VMOVDQU Y2, 32(AX)
VMOVDQU Y3, 64(AX)
VMOVDQU Y4, 96(AX)
VZEROUPPER
RET
// func updateAVX2(state *[16]uint64, msg []byte)
TEXT ·updateAVX2(SB), 4, $0-32
MOVQ state+0(FP), AX
MOVQ msg_base+8(FP), BX
MOVQ msg_len+16(FP), CX
CMPQ CX, $32
JB DONE
VMOVDQU 0(AX), Y1
VMOVDQU 32(AX), Y2
VMOVDQU 64(AX), Y3
VMOVDQU 96(AX), Y4
VMOVDQU ·zipperMergeAVX2<>(SB), Y5
LOOP:
VMOVDQU 0(BX), Y0
UPDATE(Y0)
ADDQ $32, BX
SUBQ $32, CX
JA LOOP
VMOVDQU Y1, 0(AX)
VMOVDQU Y2, 32(AX)
VMOVDQU Y3, 64(AX)
VMOVDQU Y4, 96(AX)
VZEROUPPER
DONE:
RET
// func finalizeAVX2(out []byte, state *[16]uint64)
TEXT ·finalizeAVX2(SB), 4, $0-32
MOVQ state+24(FP), AX
MOVQ out_base+0(FP), BX
MOVQ out_len+8(FP), CX
VMOVDQU 0(AX), Y1
VMOVDQU 32(AX), Y2
VMOVDQU 64(AX), Y3
VMOVDQU 96(AX), Y4
VMOVDQU ·zipperMergeAVX2<>(SB), Y5
VPERM2I128 $1, Y1, Y1, Y0
VPSHUFD $177, Y0, Y0
UPDATE(Y0)
VPERM2I128 $1, Y1, Y1, Y0
VPSHUFD $177, Y0, Y0
UPDATE(Y0)
VPERM2I128 $1, Y1, Y1, Y0
VPSHUFD $177, Y0, Y0
UPDATE(Y0)
VPERM2I128 $1, Y1, Y1, Y0
VPSHUFD $177, Y0, Y0
UPDATE(Y0)
CMPQ CX, $8
JE skipUpdate // Just 4 rounds for 64-bit checksum
VPERM2I128 $1, Y1, Y1, Y0
VPSHUFD $177, Y0, Y0
UPDATE(Y0)
VPERM2I128 $1, Y1, Y1, Y0
VPSHUFD $177, Y0, Y0
UPDATE(Y0)
CMPQ CX, $16
JE skipUpdate // 6 rounds for 128-bit checksum
VPERM2I128 $1, Y1, Y1, Y0
VPSHUFD $177, Y0, Y0
UPDATE(Y0)
VPERM2I128 $1, Y1, Y1, Y0
VPSHUFD $177, Y0, Y0
UPDATE(Y0)
VPERM2I128 $1, Y1, Y1, Y0
VPSHUFD $177, Y0, Y0
UPDATE(Y0)
VPERM2I128 $1, Y1, Y1, Y0
VPSHUFD $177, Y0, Y0
UPDATE(Y0)
skipUpdate:
VMOVDQU Y1, 0(AX)
VMOVDQU Y2, 32(AX)
VMOVDQU Y3, 64(AX)
VMOVDQU Y4, 96(AX)
VZEROUPPER
CMPQ CX, $8
JE hash64
CMPQ CX, $16
JE hash128
// 256-bit checksum
MOVQ 0*8(AX), R8
MOVQ 1*8(AX), R9
MOVQ 4*8(AX), R10
MOVQ 5*8(AX), R11
ADDQ 8*8(AX), R8
ADDQ 9*8(AX), R9
ADDQ 12*8(AX), R10
ADDQ 13*8(AX), R11
REDUCE_MOD(R8, R9, R10, R11, R12, R13, R14, R15)
MOVQ R14, 0(BX)
MOVQ R15, 8(BX)
MOVQ 2*8(AX), R8
MOVQ 3*8(AX), R9
MOVQ 6*8(AX), R10
MOVQ 7*8(AX), R11
ADDQ 10*8(AX), R8
ADDQ 11*8(AX), R9
ADDQ 14*8(AX), R10
ADDQ 15*8(AX), R11
REDUCE_MOD(R8, R9, R10, R11, R12, R13, R14, R15)
MOVQ R14, 16(BX)
MOVQ R15, 24(BX)
RET
hash128:
MOVQ 0*8(AX), R8
MOVQ 1*8(AX), R9
ADDQ 6*8(AX), R8
ADDQ 7*8(AX), R9
ADDQ 8*8(AX), R8
ADDQ 9*8(AX), R9
ADDQ 14*8(AX), R8
ADDQ 15*8(AX), R9
MOVQ R8, 0(BX)
MOVQ R9, 8(BX)
RET
hash64:
MOVQ 0*8(AX), DX
ADDQ 4*8(AX), DX
ADDQ 8*8(AX), DX
ADDQ 12*8(AX), DX
MOVQ DX, 0(BX)
RET
Server/vendor/github.com/minio/highwayhash/highwayhashSVE_arm64.s
Generated Vendored
+132
View File
@@ -0,0 +1,132 @@
//
// Copyright (c) 2024 Minio Inc. All rights reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
//+build !noasm,!appengine
#include "textflag.h"
TEXT ·getVectorLength(SB), NOSPLIT, $0
WORD $0xd2800002 // mov x2, #0
WORD $0x04225022 // addvl x2, x2, #1
WORD $0xd37df042 // lsl x2, x2, #3
WORD $0xd2800003 // mov x3, #0
WORD $0x04635023 // addpl x3, x3, #1
WORD $0xd37df063 // lsl x3, x3, #3
MOVD R2, vl+0(FP)
MOVD R3, pl+8(FP)
RET
TEXT ·updateArm64Sve(SB), NOSPLIT, $0
MOVD state+0(FP), R0
MOVD msg_base+8(FP), R1
MOVD msg_len+16(FP), R2 // length of message
SUBS $32, R2
BMI completeSve
WORD $0x2518e3e1 // ptrue p1.b
WORD $0xa5e0a401 // ld1d z1.d, p1/z, [x0]
WORD $0xa5e1a402 // ld1d z2.d, p1/z, [x0, #1, MUL VL]
WORD $0xa5e2a403 // ld1d z3.d, p1/z, [x0, #2, MUL VL]
WORD $0xa5e3a404 // ld1d z4.d, p1/z, [x0, #3, MUL VL]
// Load zipper merge constants table pointer
MOVD $·zipperMergeSve(SB), R3
WORD $0xa5e0a465 // ld1d z5.d, p1/z, [x3]
WORD $0x25b8c006 // mov z6.s, #0
WORD $0x25d8e3e2 // ptrue p2.d /* set every other lane for "s" type */
loopSve:
WORD $0xa5e0a420 // ld1d z0.d, p1/z, [x1]
ADD $32, R1
WORD $0x04e00042 // add z2.d, z2.d, z0.d
WORD $0x04e30042 // add z2.d, z2.d, z3.d
WORD $0x04e09420 // lsr z0.d, z1.d, #32
WORD $0x05a6c847 // sel z7.s, p2, z2.s, z6.s
WORD $0x04d004e0 // mul z0.d, p1/m, z0.d, z7.d
WORD $0x04a33003 // eor z3.d, z0.d, z3.d
WORD $0x04e10081 // add z1.d, z4.d, z1.d
WORD $0x04e09440 // lsr z0.d, z2.d, #32
WORD $0x05a6c827 // sel z7.s, p2, z1.s, z6.s
WORD $0x04d004e0 // mul z0.d, p1/m, z0.d, z7.d
WORD $0x04a43004 // eor z4.d, z0.d, z4.d
WORD $0x05253040 // tbl z0.b, z2.b, z5.b
WORD $0x04e00021 // add z1.d, z1.d, z0.d
WORD $0x05253020 // tbl z0.b, z1.b, z5.b
WORD $0x04e00042 // add z2.d, z2.d, z0.d
SUBS $32, R2
BPL loopSve
WORD $0xe5e0e401 // st1d z1.d, p1, [x0]
WORD $0xe5e1e402 // st1d z2.d, p1, [x0, #1, MUL VL]
WORD $0xe5e2e403 // st1d z3.d, p1, [x0, #2, MUL VL]
WORD $0xe5e3e404 // st1d z4.d, p1, [x0, #3, MUL VL]
completeSve:
RET
TEXT ·updateArm64Sve2(SB), NOSPLIT, $0
MOVD state+0(FP), R0
MOVD msg_base+8(FP), R1
MOVD msg_len+16(FP), R2 // length of message
SUBS $32, R2
BMI completeSve2
WORD $0x2518e3e1 // ptrue p1.b
WORD $0xa5e0a401 // ld1d z1.d, p1/z, [x0]
WORD $0xa5e1a402 // ld1d z2.d, p1/z, [x0, #1, MUL VL]
WORD $0xa5e2a403 // ld1d z3.d, p1/z, [x0, #2, MUL VL]
WORD $0xa5e3a404 // ld1d z4.d, p1/z, [x0, #3, MUL VL]
// Load zipper merge constants table pointer
MOVD $·zipperMergeSve(SB), R3
WORD $0xa5e0a465 // ld1d z5.d, p1/z, [x3]
loopSve2:
WORD $0xa5e0a420 // ld1d z0.d, p1/z, [x1]
ADD $32, R1
WORD $0x04e00042 // add z2.d, z2.d, z0.d
WORD $0x04e30042 // add z2.d, z2.d, z3.d
WORD $0x04e09420 // lsr z0.d, z1.d, #32
WORD $0x45c27800 // umullb z0.d, z0.s, z2.s
WORD $0x04a33003 // eor z3.d, z0.d, z3.d
WORD $0x04e10081 // add z1.d, z4.d, z1.d
WORD $0x04e09440 // lsr z0.d, z2.d, #32
WORD $0x45c17800 // umullb z0.d, z0.s, z1.s
WORD $0x04a43004 // eor z4.d, z0.d, z4.d
WORD $0x05253040 // tbl z0.b, z2.b, z5.b
WORD $0x04e00021 // add z1.d, z1.d, z0.d
WORD $0x05253020 // tbl z0.b, z1.b, z5.b
WORD $0x04e00042 // add z2.d, z2.d, z0.d
SUBS $32, R2
BPL loopSve2
WORD $0xe5e0e401 // st1d z1.d, p1, [x0]
WORD $0xe5e1e402 // st1d z2.d, p1, [x0, #1, MUL VL]
WORD $0xe5e2e403 // st1d z3.d, p1, [x0, #2, MUL VL]
WORD $0xe5e3e404 // st1d z4.d, p1, [x0, #3, MUL VL]
completeSve2:
RET
DATA ·zipperMergeSve+0x00(SB)/8, $0x000f010e05020c03
DATA ·zipperMergeSve+0x08(SB)/8, $0x070806090d0a040b
DATA ·zipperMergeSve+0x10(SB)/8, $0x101f111e15121c13
DATA ·zipperMergeSve+0x18(SB)/8, $0x171816191d1a141b
GLOBL ·zipperMergeSve(SB), (NOPTR+RODATA), $32
Server/vendor/github.com/minio/highwayhash/highwayhash_amd64.go
Generated Vendored
+70
View File
@@ -0,0 +1,70 @@
// Copyright (c) 2017 Minio Inc. All rights reserved.
// Use of this source code is governed by a license that can be
// found in the LICENSE file.
//go:build amd64 && !gccgo && !appengine && !nacl && !noasm
// +build amd64,!gccgo,!appengine,!nacl,!noasm
package highwayhash
import "golang.org/x/sys/cpu"
var (
useSSE4 = cpu.X86.HasSSE41
useAVX2 = cpu.X86.HasAVX2
useNEON = false
useSVE = false
useSVE2 = false
useVMX = false
)
//go:noescape
func initializeSSE4(state *[16]uint64, key []byte)
//go:noescape
func initializeAVX2(state *[16]uint64, key []byte)
//go:noescape
func updateSSE4(state *[16]uint64, msg []byte)
//go:noescape
func updateAVX2(state *[16]uint64, msg []byte)
//go:noescape
func finalizeSSE4(out []byte, state *[16]uint64)
//go:noescape
func finalizeAVX2(out []byte, state *[16]uint64)
func initialize(state *[16]uint64, key []byte) {
switch {
case useAVX2:
initializeAVX2(state, key)
case useSSE4:
initializeSSE4(state, key)
default:
initializeGeneric(state, key)
}
}
func update(state *[16]uint64, msg []byte) {
switch {
case useAVX2:
updateAVX2(state, msg)
case useSSE4:
updateSSE4(state, msg)
default:
updateGeneric(state, msg)
}
}
func finalize(out []byte, state *[16]uint64) {
switch {
case useAVX2:
finalizeAVX2(out, state)
case useSSE4:
finalizeSSE4(out, state)
default:
finalizeGeneric(out, state)
}
}
Server/vendor/github.com/minio/highwayhash/highwayhash_amd64.s
Generated Vendored
+294
View File
@@ -0,0 +1,294 @@
// Copyright (c) 2017 Minio Inc. All rights reserved.
// Use of this source code is governed by a license that can be
// found in the LICENSE file.
// +build amd64 !gccgo !appengine !nacl
#include "textflag.h"
DATA ·asmConstants<>+0x00(SB)/8, $0xdbe6d5d5fe4cce2f
DATA ·asmConstants<>+0x08(SB)/8, $0xa4093822299f31d0
DATA ·asmConstants<>+0x10(SB)/8, $0x13198a2e03707344
DATA ·asmConstants<>+0x18(SB)/8, $0x243f6a8885a308d3
DATA ·asmConstants<>+0x20(SB)/8, $0x3bd39e10cb0ef593
DATA ·asmConstants<>+0x28(SB)/8, $0xc0acf169b5f18a8c
DATA ·asmConstants<>+0x30(SB)/8, $0xbe5466cf34e90c6c
DATA ·asmConstants<>+0x38(SB)/8, $0x452821e638d01377
GLOBL ·asmConstants<>(SB), (NOPTR+RODATA), $64
DATA ·asmZipperMerge<>+0x00(SB)/8, $0xf010e05020c03
DATA ·asmZipperMerge<>+0x08(SB)/8, $0x70806090d0a040b
GLOBL ·asmZipperMerge<>(SB), (NOPTR+RODATA), $16
#define v00 X0
#define v01 X1
#define v10 X2
#define v11 X3
#define m00 X4
#define m01 X5
#define m10 X6
#define m11 X7
#define t0 X8
#define t1 X9
#define t2 X10
#define REDUCE_MOD(x0, x1, x2, x3, tmp0, tmp1, y0, y1) \
MOVQ $0x3FFFFFFFFFFFFFFF, tmp0 \
ANDQ tmp0, x3 \
MOVQ x2, y0 \
MOVQ x3, y1 \
\
MOVQ x2, tmp0 \
MOVQ x3, tmp1 \
SHLQ $1, tmp1 \
SHRQ $63, tmp0 \
MOVQ tmp1, x3 \
ORQ tmp0, x3 \
\
SHLQ $1, x2 \
\
MOVQ y0, tmp0 \
MOVQ y1, tmp1 \
SHLQ $2, tmp1 \
SHRQ $62, tmp0 \
MOVQ tmp1, y1 \
ORQ tmp0, y1 \
\
SHLQ $2, y0 \
\
XORQ x0, y0 \
XORQ x2, y0 \
XORQ x1, y1 \
XORQ x3, y1
#define UPDATE(msg0, msg1) \
PADDQ msg0, v10 \
PADDQ m00, v10 \
PADDQ msg1, v11 \
PADDQ m01, v11 \
\
MOVO v00, t0 \
MOVO v01, t1 \
PSRLQ $32, t0 \
PSRLQ $32, t1 \
PMULULQ v10, t0 \
PMULULQ v11, t1 \
PXOR t0, m00 \
PXOR t1, m01 \
\
PADDQ m10, v00 \
PADDQ m11, v01 \
\
MOVO v10, t0 \
MOVO v11, t1 \
PSRLQ $32, t0 \
PSRLQ $32, t1 \
PMULULQ v00, t0 \
PMULULQ v01, t1 \
PXOR t0, m10 \
PXOR t1, m11 \
\
MOVO v10, t0 \
PSHUFB t2, t0 \
MOVO v11, t1 \
PSHUFB t2, t1 \
PADDQ t0, v00 \
PADDQ t1, v01 \
\
MOVO v00, t0 \
PSHUFB t2, t0 \
MOVO v01, t1 \
PSHUFB t2, t1 \
PADDQ t0, v10 \
PADDQ t1, v11
// func initializeSSE4(state *[16]uint64, key []byte)
TEXT ·initializeSSE4(SB), NOSPLIT, $0-32
MOVQ state+0(FP), AX
MOVQ key_base+8(FP), BX
MOVQ $·asmConstants<>(SB), CX
MOVOU 0(BX), v00
MOVOU 16(BX), v01
PSHUFD $177, v00, v10
PSHUFD $177, v01, v11
MOVOU 0(CX), m00
MOVOU 16(CX), m01
MOVOU 32(CX), m10
MOVOU 48(CX), m11
PXOR m00, v00
PXOR m01, v01
PXOR m10, v10
PXOR m11, v11
MOVOU v00, 0(AX)
MOVOU v01, 16(AX)
MOVOU v10, 32(AX)
MOVOU v11, 48(AX)
MOVOU m00, 64(AX)
MOVOU m01, 80(AX)
MOVOU m10, 96(AX)
MOVOU m11, 112(AX)
RET
// func updateSSE4(state *[16]uint64, msg []byte)
TEXT ·updateSSE4(SB), NOSPLIT, $0-32
MOVQ state+0(FP), AX
MOVQ msg_base+8(FP), BX
MOVQ msg_len+16(FP), CX
CMPQ CX, $32
JB DONE
MOVOU 0(AX), v00
MOVOU 16(AX), v01
MOVOU 32(AX), v10
MOVOU 48(AX), v11
MOVOU 64(AX), m00
MOVOU 80(AX), m01
MOVOU 96(AX), m10
MOVOU 112(AX), m11
MOVOU ·asmZipperMerge<>(SB), t2
LOOP:
MOVOU 0(BX), t0
MOVOU 16(BX), t1
UPDATE(t0, t1)
ADDQ $32, BX
SUBQ $32, CX
JA LOOP
MOVOU v00, 0(AX)
MOVOU v01, 16(AX)
MOVOU v10, 32(AX)
MOVOU v11, 48(AX)
MOVOU m00, 64(AX)
MOVOU m01, 80(AX)
MOVOU m10, 96(AX)
MOVOU m11, 112(AX)
DONE:
RET
// func finalizeSSE4(out []byte, state *[16]uint64)
TEXT ·finalizeSSE4(SB), NOSPLIT, $0-32
MOVQ state+24(FP), AX
MOVQ out_base+0(FP), BX
MOVQ out_len+8(FP), CX
MOVOU 0(AX), v00
MOVOU 16(AX), v01
MOVOU 32(AX), v10
MOVOU 48(AX), v11
MOVOU 64(AX), m00
MOVOU 80(AX), m01
MOVOU 96(AX), m10
MOVOU 112(AX), m11
MOVOU ·asmZipperMerge<>(SB), t2
PSHUFD $177, v01, t0
PSHUFD $177, v00, t1
UPDATE(t0, t1)
PSHUFD $177, v01, t0
PSHUFD $177, v00, t1
UPDATE(t0, t1)
PSHUFD $177, v01, t0
PSHUFD $177, v00, t1
UPDATE(t0, t1)
PSHUFD $177, v01, t0
PSHUFD $177, v00, t1
UPDATE(t0, t1)
CMPQ CX, $8
JE skipUpdate // Just 4 rounds for 64-bit checksum
PSHUFD $177, v01, t0
PSHUFD $177, v00, t1
UPDATE(t0, t1)
PSHUFD $177, v01, t0
PSHUFD $177, v00, t1
UPDATE(t0, t1)
CMPQ CX, $16
JE skipUpdate // 6 rounds for 128-bit checksum
PSHUFD $177, v01, t0
PSHUFD $177, v00, t1
UPDATE(t0, t1)
PSHUFD $177, v01, t0
PSHUFD $177, v00, t1
UPDATE(t0, t1)
PSHUFD $177, v01, t0
PSHUFD $177, v00, t1
UPDATE(t0, t1)
PSHUFD $177, v01, t0
PSHUFD $177, v00, t1
UPDATE(t0, t1)
skipUpdate:
MOVOU v00, 0(AX)
MOVOU v01, 16(AX)
MOVOU v10, 32(AX)
MOVOU v11, 48(AX)
MOVOU m00, 64(AX)
MOVOU m01, 80(AX)
MOVOU m10, 96(AX)
MOVOU m11, 112(AX)
CMPQ CX, $8
JE hash64
CMPQ CX, $16
JE hash128
// 256-bit checksum
PADDQ v00, m00
PADDQ v10, m10
PADDQ v01, m01
PADDQ v11, m11
MOVQ m00, R8
PEXTRQ $1, m00, R9
MOVQ m10, R10
PEXTRQ $1, m10, R11
REDUCE_MOD(R8, R9, R10, R11, R12, R13, R14, R15)
MOVQ R14, 0(BX)
MOVQ R15, 8(BX)
MOVQ m01, R8
PEXTRQ $1, m01, R9
MOVQ m11, R10
PEXTRQ $1, m11, R11
REDUCE_MOD(R8, R9, R10, R11, R12, R13, R14, R15)
MOVQ R14, 16(BX)
MOVQ R15, 24(BX)
RET
hash128:
PADDQ v00, v11
PADDQ m00, m11
PADDQ v11, m11
MOVOU m11, 0(BX)
RET
hash64:
PADDQ v00, v10
PADDQ m00, m10
PADDQ v10, m10
MOVQ m10, DX
MOVQ DX, 0(BX)
RET
Server/vendor/github.com/minio/highwayhash/highwayhash_arm64.go
Generated Vendored
+81
View File
@@ -0,0 +1,81 @@
// Copyright (c) 2017-2024 Minio Inc. All rights reserved.
// Use of this source code is governed by a license that can be
// found in the LICENSE file.
//go:build !noasm && !appengine
// +build !noasm,!appengine
package highwayhash
import (
"golang.org/x/sys/cpu"
)
var (
useSSE4 = false
useAVX2 = false
useNEON = cpu.ARM64.HasASIMD
useSVE = cpu.ARM64.HasSVE
useSVE2 = false // cpu.ARM64.HasSVE2 -- disable until tested on real hardware
useVMX = false
)
func init() {
if useSVE {
if vl, _ := getVectorLength(); vl != 256 {
//
// Since HighwayHash is designed for AVX2,
// SVE/SVE2 instructions only run correctly
// for vector length of 256
//
useSVE2 = false
useSVE = false
}
}
}
//go:noescape
func initializeArm64(state *[16]uint64, key []byte)
//go:noescape
func updateArm64(state *[16]uint64, msg []byte)
//go:noescape
func getVectorLength() (vl, pl uint64)
//go:noescape
func updateArm64Sve(state *[16]uint64, msg []byte)
//go:noescape
func updateArm64Sve2(state *[16]uint64, msg []byte)
//go:noescape
func finalizeArm64(out []byte, state *[16]uint64)
func initialize(state *[16]uint64, key []byte) {
if useNEON {
initializeArm64(state, key)
} else {
initializeGeneric(state, key)
}
}
func update(state *[16]uint64, msg []byte) {
if useSVE2 {
updateArm64Sve2(state, msg)
} else if useSVE {
updateArm64Sve(state, msg)
} else if useNEON {
updateArm64(state, msg)
} else {
updateGeneric(state, msg)
}
}
func finalize(out []byte, state *[16]uint64) {
if useNEON {
finalizeArm64(out, state)
} else {
finalizeGeneric(out, state)
}
}
Server/vendor/github.com/minio/highwayhash/highwayhash_arm64.s
Generated Vendored
+324
View File
@@ -0,0 +1,324 @@
//
// Minio Cloud Storage, (C) 2017 Minio, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
//+build !noasm,!appengine
// Use github.com/minio/asm2plan9s on this file to assemble ARM instructions to
// the opcodes of their Plan9 equivalents
#include "textflag.h"
#define REDUCE_MOD(x0, x1, x2, x3, tmp0, tmp1, y0, y1) \
MOVD $0x3FFFFFFFFFFFFFFF, tmp0 \
AND tmp0, x3 \
MOVD x2, y0 \
MOVD x3, y1 \
\
MOVD x2, tmp0 \
MOVD x3, tmp1 \
LSL $1, tmp1 \
LSR $63, tmp0 \
MOVD tmp1, x3 \
ORR tmp0, x3 \
\
LSL $1, x2 \
\
MOVD y0, tmp0 \
MOVD y1, tmp1 \
LSL $2, tmp1 \
LSR $62, tmp0 \
MOVD tmp1, y1 \
ORR tmp0, y1 \
\
LSL $2, y0 \
\
EOR x0, y0 \
EOR x2, y0 \
EOR x1, y1 \
EOR x3, y1
#define UPDATE(MSG1, MSG2) \
\ // Add message
VADD MSG1.D2, V2.D2, V2.D2 \
VADD MSG2.D2, V3.D2, V3.D2 \
\
\ // v1 += mul0
VADD V4.D2, V2.D2, V2.D2 \
VADD V5.D2, V3.D2, V3.D2 \
\
\ // First pair of multiplies
VTBL V29.B16, [V0.B16, V1.B16], V10.B16 \
VTBL V30.B16, [V2.B16, V3.B16], V11.B16 \
\
\ // VUMULL V10.S2, V11.S2, V12.D2 /* assembler support missing */
\ // VUMULL2 V10.S4, V11.S4, V13.D2 /* assembler support missing */
WORD $0x2eaac16c \ // umull v12.2d, v11.2s, v10.2s
WORD $0x6eaac16d \ // umull2 v13.2d, v11.4s, v10.4s
\
\ // v0 += mul1
VADD V6.D2, V0.D2, V0.D2 \
VADD V7.D2, V1.D2, V1.D2 \
\
\ // Second pair of multiplies
VTBL V29.B16, [V2.B16, V3.B16], V15.B16 \
VTBL V30.B16, [V0.B16, V1.B16], V14.B16 \
\
\ // EOR multiplication result in
VEOR V12.B16, V4.B16, V4.B16 \
VEOR V13.B16, V5.B16, V5.B16 \
\
\ // VUMULL V14.S2, V15.S2, V16.D2 /* assembler support missing */
\ // VUMULL2 V14.S4, V15.S4, V17.D2 /* assembler support missing */
WORD $0x2eaec1f0 \ // umull v16.2d, v15.2s, v14.2s
WORD $0x6eaec1f1 \ // umull2 v17.2d, v15.4s, v14.4s
\
\ // First pair of zipper-merges
VTBL V28.B16, [V2.B16], V18.B16 \
VADD V18.D2, V0.D2, V0.D2 \
VTBL V28.B16, [V3.B16], V19.B16 \
VADD V19.D2, V1.D2, V1.D2 \
\
\ // Second pair of zipper-merges
VTBL V28.B16, [V0.B16], V20.B16 \
VADD V20.D2, V2.D2, V2.D2 \
VTBL V28.B16, [V1.B16], V21.B16 \
VADD V21.D2, V3.D2, V3.D2 \
\
\ // EOR multiplication result in
VEOR V16.B16, V6.B16, V6.B16 \
VEOR V17.B16, V7.B16, V7.B16
// func initializeArm64(state *[16]uint64, key []byte)
TEXT ·initializeArm64(SB), NOSPLIT, $0
MOVD state+0(FP), R0
MOVD key_base+8(FP), R1
VLD1 (R1), [V1.S4, V2.S4]
VREV64 V1.S4, V3.S4
VREV64 V2.S4, V4.S4
MOVD $·asmConstants(SB), R3
VLD1 (R3), [V5.S4, V6.S4, V7.S4, V8.S4]
VEOR V5.B16, V1.B16, V1.B16
VEOR V6.B16, V2.B16, V2.B16
VEOR V7.B16, V3.B16, V3.B16
VEOR V8.B16, V4.B16, V4.B16
VST1.P [V1.D2, V2.D2, V3.D2, V4.D2], 64(R0)
VST1 [V5.D2, V6.D2, V7.D2, V8.D2], (R0)
RET
TEXT ·updateArm64(SB), NOSPLIT, $0
MOVD state+0(FP), R0
MOVD msg_base+8(FP), R1
MOVD msg_len+16(FP), R2 // length of message
SUBS $32, R2
BMI complete
// Definition of registers
// v0 = v0.lo
// v1 = v0.hi
// v2 = v1.lo
// v3 = v1.hi
// v4 = mul0.lo
// v5 = mul0.hi
// v6 = mul1.lo
// v7 = mul1.hi
// Load zipper merge constants table pointer
MOVD $·asmZipperMerge(SB), R3
// and load zipper merge constants into v28, v29, and v30
VLD1 (R3), [V28.B16, V29.B16, V30.B16]
VLD1.P 64(R0), [V0.D2, V1.D2, V2.D2, V3.D2]
VLD1 (R0), [V4.D2, V5.D2, V6.D2, V7.D2]
SUBS $64, R0
loop:
// Main loop
VLD1.P 32(R1), [V26.S4, V27.S4]
UPDATE(V26, V27)
SUBS $32, R2
BPL loop
// Store result
VST1.P [V0.D2, V1.D2, V2.D2, V3.D2], 64(R0)
VST1 [V4.D2, V5.D2, V6.D2, V7.D2], (R0)
complete:
RET
// func finalizeArm64(out []byte, state *[16]uint64)
TEXT ·finalizeArm64(SB), NOSPLIT, $0-32
MOVD state+24(FP), R0
MOVD out_base+0(FP), R1
MOVD out_len+8(FP), R2
// Load zipper merge constants table pointer
MOVD $·asmZipperMerge(SB), R3
// and load zipper merge constants into v28, v29, and v30
VLD1 (R3), [V28.B16, V29.B16, V30.B16]
VLD1.P 64(R0), [V0.D2, V1.D2, V2.D2, V3.D2]
VLD1 (R0), [V4.D2, V5.D2, V6.D2, V7.D2]
SUB $64, R0
VREV64 V1.S4, V26.S4
VREV64 V0.S4, V27.S4
UPDATE(V26, V27)
VREV64 V1.S4, V26.S4
VREV64 V0.S4, V27.S4
UPDATE(V26, V27)
VREV64 V1.S4, V26.S4
VREV64 V0.S4, V27.S4
UPDATE(V26, V27)
VREV64 V1.S4, V26.S4
VREV64 V0.S4, V27.S4
UPDATE(V26, V27)
CMP $8, R2
BEQ skipUpdate // Just 4 rounds for 64-bit checksum
VREV64 V1.S4, V26.S4
VREV64 V0.S4, V27.S4
UPDATE(V26, V27)
VREV64 V1.S4, V26.S4
VREV64 V0.S4, V27.S4
UPDATE(V26, V27)
CMP $16, R2
BEQ skipUpdate // 6 rounds for 128-bit checksum
VREV64 V1.S4, V26.S4
VREV64 V0.S4, V27.S4
UPDATE(V26, V27)
VREV64 V1.S4, V26.S4
VREV64 V0.S4, V27.S4
UPDATE(V26, V27)
VREV64 V1.S4, V26.S4
VREV64 V0.S4, V27.S4
UPDATE(V26, V27)
VREV64 V1.S4, V26.S4
VREV64 V0.S4, V27.S4
UPDATE(V26, V27)
skipUpdate:
// Store result
VST1.P [V0.D2, V1.D2, V2.D2, V3.D2], 64(R0)
VST1 [V4.D2, V5.D2, V6.D2, V7.D2], (R0)
SUB $64, R0
CMP $8, R2
BEQ hash64
CMP $16, R2
BEQ hash128
// 256-bit checksum
MOVD 0*8(R0), R8
MOVD 1*8(R0), R9
MOVD 4*8(R0), R10
MOVD 5*8(R0), R11
MOVD 8*8(R0), R4
MOVD 9*8(R0), R5
MOVD 12*8(R0), R6
MOVD 13*8(R0), R7
ADD R4, R8
ADD R5, R9
ADD R6, R10
ADD R7, R11
REDUCE_MOD(R8, R9, R10, R11, R4, R5, R6, R7)
MOVD R6, 0(R1)
MOVD R7, 8(R1)
MOVD 2*8(R0), R8
MOVD 3*8(R0), R9
MOVD 6*8(R0), R10
MOVD 7*8(R0), R11
MOVD 10*8(R0), R4
MOVD 11*8(R0), R5
MOVD 14*8(R0), R6
MOVD 15*8(R0), R7
ADD R4, R8
ADD R5, R9
ADD R6, R10
ADD R7, R11
REDUCE_MOD(R8, R9, R10, R11, R4, R5, R6, R7)
MOVD R6, 16(R1)
MOVD R7, 24(R1)
RET
hash128:
MOVD 0*8(R0), R8
MOVD 1*8(R0), R9
MOVD 6*8(R0), R10
MOVD 7*8(R0), R11
ADD R10, R8
ADD R11, R9
MOVD 8*8(R0), R10
MOVD 9*8(R0), R11
ADD R10, R8
ADD R11, R9
MOVD 14*8(R0), R10
MOVD 15*8(R0), R11
ADD R10, R8
ADD R11, R9
MOVD R8, 0(R1)
MOVD R9, 8(R1)
RET
hash64:
MOVD 0*8(R0), R4
MOVD 4*8(R0), R5
MOVD 8*8(R0), R6
MOVD 12*8(R0), R7
ADD R5, R4
ADD R7, R6
ADD R6, R4
MOVD R4, (R1)
RET
DATA ·asmConstants+0x00(SB)/8, $0xdbe6d5d5fe4cce2f
DATA ·asmConstants+0x08(SB)/8, $0xa4093822299f31d0
DATA ·asmConstants+0x10(SB)/8, $0x13198a2e03707344
DATA ·asmConstants+0x18(SB)/8, $0x243f6a8885a308d3
DATA ·asmConstants+0x20(SB)/8, $0x3bd39e10cb0ef593
DATA ·asmConstants+0x28(SB)/8, $0xc0acf169b5f18a8c
DATA ·asmConstants+0x30(SB)/8, $0xbe5466cf34e90c6c
DATA ·asmConstants+0x38(SB)/8, $0x452821e638d01377
GLOBL ·asmConstants(SB), 8, $64
// Constants for TBL instructions
DATA ·asmZipperMerge+0x0(SB)/8, $0x000f010e05020c03 // zipper merge constant
DATA ·asmZipperMerge+0x8(SB)/8, $0x070806090d0a040b
DATA ·asmZipperMerge+0x10(SB)/8, $0x0f0e0d0c07060504 // setup first register for multiply
DATA ·asmZipperMerge+0x18(SB)/8, $0x1f1e1d1c17161514
DATA ·asmZipperMerge+0x20(SB)/8, $0x0b0a090803020100 // setup second register for multiply
DATA ·asmZipperMerge+0x28(SB)/8, $0x1b1a191813121110
GLOBL ·asmZipperMerge(SB), 8, $48
Server/vendor/github.com/minio/highwayhash/highwayhash_generic.go
Generated Vendored
+338
View File
@@ -0,0 +1,338 @@
// Copyright (c) 2017 Minio Inc. All rights reserved.
// Use of this source code is governed by a license that can be
// found in the LICENSE file.
package highwayhash
import (
"encoding/binary"
)
const (
v0 = 0
v1 = 4
mul0 = 8
mul1 = 12
)
var (
init0 = [4]uint64{0xdbe6d5d5fe4cce2f, 0xa4093822299f31d0, 0x13198a2e03707344, 0x243f6a8885a308d3}
init1 = [4]uint64{0x3bd39e10cb0ef593, 0xc0acf169b5f18a8c, 0xbe5466cf34e90c6c, 0x452821e638d01377}
)
func initializeGeneric(state *[16]uint64, k []byte) {
var key [4]uint64
key[0] = binary.LittleEndian.Uint64(k[0:])
key[1] = binary.LittleEndian.Uint64(k[8:])
key[2] = binary.LittleEndian.Uint64(k[16:])
key[3] = binary.LittleEndian.Uint64(k[24:])
copy(state[mul0:], init0[:])
copy(state[mul1:], init1[:])
for i, k := range key {
state[v0+i] = init0[i] ^ k
}
key[0] = key[0]>>32 | key[0]<<32
key[1] = key[1]>>32 | key[1]<<32
key[2] = key[2]>>32 | key[2]<<32
key[3] = key[3]>>32 | key[3]<<32
for i, k := range key {
state[v1+i] = init1[i] ^ k
}
}
func updateGeneric(state *[16]uint64, msg []byte) {
for len(msg) >= 32 {
m := msg[:32]
// add message + mul0
// Interleave operations to hide multiplication
state[v1+0] += binary.LittleEndian.Uint64(m) + state[mul0+0]
state[mul0+0] ^= uint64(uint32(state[v1+0])) * (state[v0+0] >> 32)
state[v0+0] += state[mul1+0]
state[mul1+0] ^= uint64(uint32(state[v0+0])) * (state[v1+0] >> 32)
state[v1+1] += binary.LittleEndian.Uint64(m[8:]) + state[mul0+1]
state[mul0+1] ^= uint64(uint32(state[v1+1])) * (state[v0+1] >> 32)
state[v0+1] += state[mul1+1]
state[mul1+1] ^= uint64(uint32(state[v0+1])) * (state[v1+1] >> 32)
state[v1+2] += binary.LittleEndian.Uint64(m[16:]) + state[mul0+2]
state[mul0+2] ^= uint64(uint32(state[v1+2])) * (state[v0+2] >> 32)
state[v0+2] += state[mul1+2]
state[mul1+2] ^= uint64(uint32(state[v0+2])) * (state[v1+2] >> 32)
state[v1+3] += binary.LittleEndian.Uint64(m[24:]) + state[mul0+3]
state[mul0+3] ^= uint64(uint32(state[v1+3])) * (state[v0+3] >> 32)
state[v0+3] += state[mul1+3]
state[mul1+3] ^= uint64(uint32(state[v0+3])) * (state[v1+3] >> 32)
// inlined: zipperMerge(state[v1+0], state[v1+1], &state[v0+0], &state[v0+1])
{
val0 := state[v1+0]
val1 := state[v1+1]
res := val0 & (0xff << (2 * 8))
res2 := (val0 & (0xff << (7 * 8))) + (val1 & (0xff << (2 * 8)))
res += (val1 & (0xff << (7 * 8))) >> 8
res2 += (val0 & (0xff << (6 * 8))) >> 8
res += ((val0 & (0xff << (5 * 8))) + (val1 & (0xff << (6 * 8)))) >> 16
res2 += (val1 & (0xff << (5 * 8))) >> 16
res += ((val0 & (0xff << (3 * 8))) + (val1 & (0xff << (4 * 8)))) >> 24
res2 += ((val1 & (0xff << (3 * 8))) + (val0 & (0xff << (4 * 8)))) >> 24
res += (val0 & (0xff << (1 * 8))) << 32
res2 += (val1 & 0xff) << 48
res += val0 << 56
res2 += (val1 & (0xff << (1 * 8))) << 24
state[v0+0] += res
state[v0+1] += res2
}
// zipperMerge(state[v1+2], state[v1+3], &state[v0+2], &state[v0+3])
{
val0 := state[v1+2]
val1 := state[v1+3]
res := val0 & (0xff << (2 * 8))
res2 := (val0 & (0xff << (7 * 8))) + (val1 & (0xff << (2 * 8)))
res += (val1 & (0xff << (7 * 8))) >> 8
res2 += (val0 & (0xff << (6 * 8))) >> 8
res += ((val0 & (0xff << (5 * 8))) + (val1 & (0xff << (6 * 8)))) >> 16
res2 += (val1 & (0xff << (5 * 8))) >> 16
res += ((val0 & (0xff << (3 * 8))) + (val1 & (0xff << (4 * 8)))) >> 24
res2 += ((val1 & (0xff << (3 * 8))) + (val0 & (0xff << (4 * 8)))) >> 24
res += (val0 & (0xff << (1 * 8))) << 32
res2 += (val1 & 0xff) << 48
res += val0 << 56
res2 += (val1 & (0xff << (1 * 8))) << 24
state[v0+2] += res
state[v0+3] += res2
}
// inlined: zipperMerge(state[v0+0], state[v0+1], &state[v1+0], &state[v1+1])
{
val0 := state[v0+0]
val1 := state[v0+1]
res := val0 & (0xff << (2 * 8))
res2 := (val0 & (0xff << (7 * 8))) + (val1 & (0xff << (2 * 8)))
res += (val1 & (0xff << (7 * 8))) >> 8
res2 += (val0 & (0xff << (6 * 8))) >> 8
res += ((val0 & (0xff << (5 * 8))) + (val1 & (0xff << (6 * 8)))) >> 16
res2 += (val1 & (0xff << (5 * 8))) >> 16
res += ((val0 & (0xff << (3 * 8))) + (val1 & (0xff << (4 * 8)))) >> 24
res2 += ((val1 & (0xff << (3 * 8))) + (val0 & (0xff << (4 * 8)))) >> 24
res += (val0 & (0xff << (1 * 8))) << 32
res2 += (val1 & 0xff) << 48
res += val0 << 56
res2 += (val1 & (0xff << (1 * 8))) << 24
state[v1+0] += res
state[v1+1] += res2
}
//inlined: zipperMerge(state[v0+2], state[v0+3], &state[v1+2], &state[v1+3])
{
val0 := state[v0+2]
val1 := state[v0+3]
res := val0 & (0xff << (2 * 8))
res2 := (val0 & (0xff << (7 * 8))) + (val1 & (0xff << (2 * 8)))
res += (val1 & (0xff << (7 * 8))) >> 8
res2 += (val0 & (0xff << (6 * 8))) >> 8
res += ((val0 & (0xff << (5 * 8))) + (val1 & (0xff << (6 * 8)))) >> 16
res2 += (val1 & (0xff << (5 * 8))) >> 16
res += ((val0 & (0xff << (3 * 8))) + (val1 & (0xff << (4 * 8)))) >> 24
res2 += ((val1 & (0xff << (3 * 8))) + (val0 & (0xff << (4 * 8)))) >> 24
res += (val0 & (0xff << (1 * 8))) << 32
res2 += (val1 & 0xff) << 48
res += val0 << 56
res2 += (val1 & (0xff << (1 * 8))) << 24
state[v1+2] += res
state[v1+3] += res2
}
msg = msg[32:]
}
}
func finalizeGeneric(out []byte, state *[16]uint64) {
var perm [4]uint64
var tmp [32]byte
runs := 4
if len(out) == 16 {
runs = 6
} else if len(out) == 32 {
runs = 10
}
for i := 0; i < runs; i++ {
perm[0] = state[v0+2]>>32 | state[v0+2]<<32
perm[1] = state[v0+3]>>32 | state[v0+3]<<32
perm[2] = state[v0+0]>>32 | state[v0+0]<<32
perm[3] = state[v0+1]>>32 | state[v0+1]<<32
binary.LittleEndian.PutUint64(tmp[0:], perm[0])
binary.LittleEndian.PutUint64(tmp[8:], perm[1])
binary.LittleEndian.PutUint64(tmp[16:], perm[2])
binary.LittleEndian.PutUint64(tmp[24:], perm[3])
update(state, tmp[:])
}
switch len(out) {
case 8:
binary.LittleEndian.PutUint64(out, state[v0+0]+state[v1+0]+state[mul0+0]+state[mul1+0])
case 16:
binary.LittleEndian.PutUint64(out, state[v0+0]+state[v1+2]+state[mul0+0]+state[mul1+2])
binary.LittleEndian.PutUint64(out[8:], state[v0+1]+state[v1+3]+state[mul0+1]+state[mul1+3])
case 32:
h0, h1 := reduceMod(state[v0+0]+state[mul0+0], state[v0+1]+state[mul0+1], state[v1+0]+state[mul1+0], state[v1+1]+state[mul1+1])
binary.LittleEndian.PutUint64(out[0:], h0)
binary.LittleEndian.PutUint64(out[8:], h1)
h0, h1 = reduceMod(state[v0+2]+state[mul0+2], state[v0+3]+state[mul0+3], state[v1+2]+state[mul1+2], state[v1+3]+state[mul1+3])
binary.LittleEndian.PutUint64(out[16:], h0)
binary.LittleEndian.PutUint64(out[24:], h1)
}
}
// Experiments on variations left for future reference...
/*
func zipperMerge(v0, v1 uint64, d0, d1 *uint64) {
if true {
// fastest. original interleaved...
res := v0 & (0xff << (2 * 8))
res2 := (v0 & (0xff << (7 * 8))) + (v1 & (0xff << (2 * 8)))
res += (v1 & (0xff << (7 * 8))) >> 8
res2 += (v0 & (0xff << (6 * 8))) >> 8
res += ((v0 & (0xff << (5 * 8))) + (v1 & (0xff << (6 * 8)))) >> 16
res2 += (v1 & (0xff << (5 * 8))) >> 16
res += ((v0 & (0xff << (3 * 8))) + (v1 & (0xff << (4 * 8)))) >> 24
res2 += ((v1 & (0xff << (3 * 8))) + (v0 & (0xff << (4 * 8)))) >> 24
res += (v0 & (0xff << (1 * 8))) << 32
res2 += (v1 & 0xff) << 48
res += v0 << 56
res2 += (v1 & (0xff << (1 * 8))) << 24
*d0 += res
*d1 += res2
} else if false {
// Reading bytes and combining into uint64
var v0b [8]byte
binary.LittleEndian.PutUint64(v0b[:], v0)
var v1b [8]byte
binary.LittleEndian.PutUint64(v1b[:], v1)
var res, res2 uint64
res = uint64(v0b[0]) << (7 * 8)
res2 = uint64(v1b[0]) << (6 * 8)
res |= uint64(v0b[1]) << (5 * 8)
res2 |= uint64(v1b[1]) << (4 * 8)
res |= uint64(v0b[2]) << (2 * 8)
res2 |= uint64(v1b[2]) << (2 * 8)
res |= uint64(v0b[3])
res2 |= uint64(v0b[4]) << (1 * 8)
res |= uint64(v0b[5]) << (3 * 8)
res2 |= uint64(v0b[6]) << (5 * 8)
res |= uint64(v1b[4]) << (1 * 8)
res2 |= uint64(v0b[7]) << (7 * 8)
res |= uint64(v1b[6]) << (4 * 8)
res2 |= uint64(v1b[3])
res |= uint64(v1b[7]) << (6 * 8)
res2 |= uint64(v1b[5]) << (3 * 8)
*d0 += res
*d1 += res2
} else if false {
// bytes to bytes shuffle
var v0b [8]byte
binary.LittleEndian.PutUint64(v0b[:], v0)
var v1b [8]byte
binary.LittleEndian.PutUint64(v1b[:], v1)
var res [8]byte
//res += ((v0 & (0xff << (3 * 8))) + (v1 & (0xff << (4 * 8)))) >> 24
res[0] = v0b[3]
res[1] = v1b[4]
// res := v0 & (0xff << (2 * 8))
res[2] = v0b[2]
//res += ((v0 & (0xff << (5 * 8))) + (v1 & (0xff << (6 * 8)))) >> 16
res[3] = v0b[5]
res[4] = v1b[6]
//res += (v0 & (0xff << (1 * 8))) << 32
res[5] = v0b[1]
//res += (v1 & (0xff << (7 * 8))) >> 8
res[6] += v1b[7]
//res += v0 << 56
res[7] = v0b[0]
v0 = binary.LittleEndian.Uint64(res[:])
*d0 += v0
//res += ((v1 & (0xff << (3 * 8))) + (v0 & (0xff << (4 * 8)))) >> 24
res[0] = v1b[3]
res[1] = v0b[4]
res[2] = v1b[2]
// res += (v1 & (0xff << (5 * 8))) >> 16
res[3] = v1b[5]
//res += (v1 & (0xff << (1 * 8))) << 24
res[4] = v1b[1]
// res += (v0 & (0xff << (6 * 8))) >> 8
res[5] = v0b[6]
//res := (v0 & (0xff << (7 * 8))) + (v1 & (0xff << (2 * 8)))
res[7] = v0b[7]
//res += (v1 & 0xff) << 48
res[6] = v1b[0]
v0 = binary.LittleEndian.Uint64(res[:])
*d1 += v0
} else {
// original.
res := v0 & (0xff << (2 * 8))
res += (v1 & (0xff << (7 * 8))) >> 8
res += ((v0 & (0xff << (5 * 8))) + (v1 & (0xff << (6 * 8)))) >> 16
res += ((v0 & (0xff << (3 * 8))) + (v1 & (0xff << (4 * 8)))) >> 24
res += (v0 & (0xff << (1 * 8))) << 32
res += v0 << 56
*d0 += res
res = (v0 & (0xff << (7 * 8))) + (v1 & (0xff << (2 * 8)))
res += (v0 & (0xff << (6 * 8))) >> 8
res += (v1 & (0xff << (5 * 8))) >> 16
res += ((v1 & (0xff << (3 * 8))) + (v0 & (0xff << (4 * 8)))) >> 24
res += (v1 & 0xff) << 48
res += (v1 & (0xff << (1 * 8))) << 24
*d1 += res
}
}
*/
// reduce v = [v0, v1, v2, v3] mod the irreducible polynomial x^128 + x^2 + x
func reduceMod(v0, v1, v2, v3 uint64) (r0, r1 uint64) {
v3 &= 0x3FFFFFFFFFFFFFFF
r0, r1 = v2, v3
v3 = (v3 << 1) | (v2 >> (64 - 1))
v2 <<= 1
r1 = (r1 << 2) | (r0 >> (64 - 2))
r0 <<= 2
r0 ^= v0 ^ v2
r1 ^= v1 ^ v3
return
}
Server/vendor/github.com/minio/highwayhash/highwayhash_ppc64le.go
Generated Vendored
+36
View File
@@ -0,0 +1,36 @@
// Copyright (c) 2017 Minio Inc. All rights reserved.
// Use of this source code is governed by a license that can be
// found in the LICENSE file.
//go:build !noasm && !appengine
// +build !noasm,!appengine
package highwayhash
var (
useSSE4 = false
useAVX2 = false
useNEON = false
useSVE = false
useSVE2 = false
useVMX = true
)
//go:noescape
func updatePpc64Le(state *[16]uint64, msg []byte)
func initialize(state *[16]uint64, key []byte) {
initializeGeneric(state, key)
}
func update(state *[16]uint64, msg []byte) {
if useVMX {
updatePpc64Le(state, msg)
} else {
updateGeneric(state, msg)
}
}
func finalize(out []byte, state *[16]uint64) {
finalizeGeneric(out, state)
}
Server/vendor/github.com/minio/highwayhash/highwayhash_ppc64le.s
Generated Vendored
+182
View File
@@ -0,0 +1,182 @@
//
// Minio Cloud Storage, (C) 2018 Minio, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
//+build !noasm,!appengine
#include "textflag.h"
// Definition of registers
#define V0_LO VS32
#define V0_LO_ V0
#define V0_HI VS33
#define V0_HI_ V1
#define V1_LO VS34
#define V1_LO_ V2
#define V1_HI VS35
#define V1_HI_ V3
#define MUL0_LO VS36
#define MUL0_LO_ V4
#define MUL0_HI VS37
#define MUL0_HI_ V5
#define MUL1_LO VS38
#define MUL1_LO_ V6
#define MUL1_HI VS39
#define MUL1_HI_ V7
// Message
#define MSG_LO VS40
#define MSG_LO_ V8
#define MSG_HI VS41
// Constants
#define ROTATE VS42
#define ROTATE_ V10
#define MASK VS43
#define MASK_ V11
// Temps
#define TEMP1 VS44
#define TEMP1_ V12
#define TEMP2 VS45
#define TEMP2_ V13
#define TEMP3 VS46
#define TEMP3_ V14
#define TEMP4_ V15
#define TEMP5_ V16
#define TEMP6_ V17
#define TEMP7_ V18
// Regular registers
#define STATE R3
#define MSG_BASE R4
#define MSG_LEN R5
#define CONSTANTS R6
#define P1 R7
#define P2 R8
#define P3 R9
#define P4 R10
#define P5 R11
#define P6 R12
#define P7 R14 // avoid using R13
TEXT ·updatePpc64Le(SB), NOFRAME|NOSPLIT, $0-32
MOVD state+0(FP), STATE
MOVD msg_base+8(FP), MSG_BASE
MOVD msg_len+16(FP), MSG_LEN // length of message
// Sanity check for length
CMPU MSG_LEN, $31
BLE complete
// Setup offsets
MOVD $16, P1
MOVD $32, P2
MOVD $48, P3
MOVD $64, P4
MOVD $80, P5
MOVD $96, P6
MOVD $112, P7
// Load state
LXVD2X (STATE)(R0), V0_LO
LXVD2X (STATE)(P1), V0_HI
LXVD2X (STATE)(P2), V1_LO
LXVD2X (STATE)(P3), V1_HI
LXVD2X (STATE)(P4), MUL0_LO
LXVD2X (STATE)(P5), MUL0_HI
LXVD2X (STATE)(P6), MUL1_LO
LXVD2X (STATE)(P7), MUL1_HI
XXPERMDI V0_LO, V0_LO, $2, V0_LO
XXPERMDI V0_HI, V0_HI, $2, V0_HI
XXPERMDI V1_LO, V1_LO, $2, V1_LO
XXPERMDI V1_HI, V1_HI, $2, V1_HI
XXPERMDI MUL0_LO, MUL0_LO, $2, MUL0_LO
XXPERMDI MUL0_HI, MUL0_HI, $2, MUL0_HI
XXPERMDI MUL1_LO, MUL1_LO, $2, MUL1_LO
XXPERMDI MUL1_HI, MUL1_HI, $2, MUL1_HI
// Load asmConstants table pointer
MOVD $·asmConstants(SB), CONSTANTS
LXVD2X (CONSTANTS)(R0), ROTATE
LXVD2X (CONSTANTS)(P1), MASK
XXLNAND MASK, MASK, MASK
loop:
// Main highwayhash update loop
LXVD2X (MSG_BASE)(R0), MSG_LO
VADDUDM V0_LO_, MUL1_LO_, TEMP1_
VRLD V0_LO_, ROTATE_, TEMP2_
VADDUDM MUL1_HI_, V0_HI_, TEMP3_
LXVD2X (MSG_BASE)(P1), MSG_HI
ADD $32, MSG_BASE, MSG_BASE
XXPERMDI MSG_LO, MSG_LO, $2, MSG_LO
XXPERMDI MSG_HI, MSG_HI, $2, V0_LO
VADDUDM MSG_LO_, MUL0_LO_, MSG_LO_
VADDUDM V0_LO_, MUL0_HI_, V0_LO_
VADDUDM MSG_LO_, V1_LO_, V1_LO_
VSRD V0_HI_, ROTATE_, MSG_LO_
VADDUDM V0_LO_, V1_HI_, V1_HI_
VPERM V1_LO_, V1_LO_, MASK_, V0_LO_
VMULOUW V1_LO_, TEMP2_, TEMP2_
VPERM V1_HI_, V1_HI_, MASK_, TEMP7_
VADDUDM V0_LO_, TEMP1_, V0_LO_
VMULOUW V1_HI_, MSG_LO_, MSG_LO_
VADDUDM TEMP7_, TEMP3_, V0_HI_
VPERM V0_LO_, V0_LO_, MASK_, TEMP6_
VRLD V1_LO_, ROTATE_, TEMP4_
VSRD V1_HI_, ROTATE_, TEMP5_
VPERM V0_HI_, V0_HI_, MASK_, TEMP7_
XXLXOR MUL0_LO, TEMP2, MUL0_LO
VMULOUW TEMP1_, TEMP4_, TEMP1_
VMULOUW TEMP3_, TEMP5_, TEMP3_
XXLXOR MUL0_HI, MSG_LO, MUL0_HI
XXLXOR MUL1_LO, TEMP1, MUL1_LO
XXLXOR MUL1_HI, TEMP3, MUL1_HI
VADDUDM TEMP6_, V1_LO_, V1_LO_
VADDUDM TEMP7_, V1_HI_, V1_HI_
SUB $32, MSG_LEN, MSG_LEN
CMPU MSG_LEN, $32
BGE loop
// Save state
XXPERMDI V0_LO, V0_LO, $2, V0_LO
XXPERMDI V0_HI, V0_HI, $2, V0_HI
XXPERMDI V1_LO, V1_LO, $2, V1_LO
XXPERMDI V1_HI, V1_HI, $2, V1_HI
XXPERMDI MUL0_LO, MUL0_LO, $2, MUL0_LO
XXPERMDI MUL0_HI, MUL0_HI, $2, MUL0_HI
XXPERMDI MUL1_LO, MUL1_LO, $2, MUL1_LO
XXPERMDI MUL1_HI, MUL1_HI, $2, MUL1_HI
STXVD2X V0_LO, (STATE)(R0)
STXVD2X V0_HI, (STATE)(P1)
STXVD2X V1_LO, (STATE)(P2)
STXVD2X V1_HI, (STATE)(P3)
STXVD2X MUL0_LO, (STATE)(P4)
STXVD2X MUL0_HI, (STATE)(P5)
STXVD2X MUL1_LO, (STATE)(P6)
STXVD2X MUL1_HI, (STATE)(P7)
complete:
RET
// Constants table
DATA ·asmConstants+0x0(SB)/8, $0x0000000000000020
DATA ·asmConstants+0x8(SB)/8, $0x0000000000000020
DATA ·asmConstants+0x10(SB)/8, $0x070806090d0a040b // zipper merge constant
DATA ·asmConstants+0x18(SB)/8, $0x000f010e05020c03 // zipper merge constant
GLOBL ·asmConstants(SB), 8, $32
Server/vendor/github.com/minio/highwayhash/highwayhash_ref.go
Generated Vendored
+29
View File
@@ -0,0 +1,29 @@
// Copyright (c) 2017 Minio Inc. All rights reserved.
// Use of this source code is governed by a license that can be
// found in the LICENSE file.
//go:build noasm || (!amd64 && !arm64 && !ppc64le)
// +build noasm !amd64,!arm64,!ppc64le
package highwayhash
var (
useSSE4 = false
useAVX2 = false
useNEON = false
useSVE = false
useSVE2 = false
useVMX = false
)
func initialize(state *[16]uint64, k []byte) {
initializeGeneric(state, k)
}
func update(state *[16]uint64, msg []byte) {
updateGeneric(state, msg)
}
func finalize(out []byte, state *[16]uint64) {
finalizeGeneric(out, state)
}
Server/vendor/github.com/minio/md5-simd/LICENSE
Generated Vendored
+202
View File
@@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Server/vendor/github.com/minio/md5-simd/LICENSE.Golang
Generated Vendored
+27
View File
@@ -0,0 +1,27 @@
Copyright (c) 2009 The Go Authors. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Server/vendor/github.com/minio/md5-simd/README.md
Generated Vendored
+198
View File
@@ -0,0 +1,198 @@
# md5-simd
This is a SIMD accelerated MD5 package, allowing up to either 8 (AVX2) or 16 (AVX512) independent MD5 sums to be calculated on a single CPU core.
It was originally based on the [md5vec](https://github.com/igneous-systems/md5vec) repository by Igneous Systems, but has been made more flexible by amongst others supporting different message sizes per lane and adding AVX512.
`md5-simd` integrates a similar mechanism as described in [minio/sha256-simd](https://github.com/minio/sha256-simd#support-for-avx512) for making it easy for clients to take advantages of the parallel nature of the MD5 calculation. This will result in reduced overall CPU load.
It is important to understand that `md5-simd` **does not speed up** a single threaded MD5 hash sum.
Rather it allows multiple __independent__ MD5 sums to be computed in parallel on the same CPU core,
thereby making more efficient usage of the computing resources.
## Usage
[![Documentation](https://godoc.org/github.com/minio/md5-simd?status.svg)](https://pkg.go.dev/github.com/minio/md5-simd?tab=doc)
In order to use `md5-simd`, you must first create an `Server` which can be
used to instantiate one or more objects for MD5 hashing.
These objects conform to the regular [`hash.Hash`](https://pkg.go.dev/hash?tab=doc#Hash) interface
and as such the normal Write/Reset/Sum functionality works as expected.
As an example:
```
// Create server
server := md5simd.NewServer()
defer server.Close()
// Create hashing object (conforming to hash.Hash)
md5Hash := server.NewHash()
defer md5Hash.Close()
// Write one (or more) blocks
md5Hash.Write(block)
// Return digest
digest := md5Hash.Sum([]byte{})
```
To keep performance both a [Server](https://pkg.go.dev/github.com/minio/md5-simd?tab=doc#Server)
and individual [Hasher](https://pkg.go.dev/github.com/minio/md5-simd?tab=doc#Hasher) should
be closed using the `Close()` function when no longer needed.
A Hasher can efficiently be re-used by using [`Reset()`](https://pkg.go.dev/hash?tab=doc#Hash) functionality.
In case your system does not support the instructions required it will fall back to using `crypto/md5` for hashing.
## Limitations
As explained above `md5-simd` does not speed up an individual MD5 hash sum computation,
unless some hierarchical tree construct is used but this will result in different outcomes.
Running a single hash on a server results in approximately half the throughput.
Instead, it allows running multiple MD5 calculations in parallel on a single CPU core.
This can be beneficial in e.g. multi-threaded server applications where many go-routines
are dealing with many requests and multiple MD5 calculations can be packed/scheduled for parallel execution on a single core.
This will result in a lower overall CPU usage as compared to using the standard `crypto/md5`
functionality where each MD5 hash computation will consume a single thread (core).
It is best to test and measure the overall CPU usage in a representative usage scenario in your application
to get an overall understanding of the benefits of `md5-simd` as compared to `crypto/md5`, ideally under heavy CPU load.
Also note that `md5-simd` is best meant to work with large objects,
so if your application only hashes small objects of a few kilobytes
you may be better of by using `crypto/md5`.
## Performance
For the best performance writes should be a multiple of 64 bytes, ideally a multiple of 32KB.
To help with that a [`buffered := bufio.NewWriterSize(hasher, 32<<10)`](https://golang.org/pkg/bufio/#NewWriterSize)
can be inserted if you are unsure of the sizes of the writes.
Remember to [flush](https://golang.org/pkg/bufio/#Writer.Flush) `buffered` before reading the hash.
A single 'server' can process 16 streams concurrently with 1 core (AVX-512) or 2 cores (AVX2).
In situations where it is likely that more than 16 streams are fully loaded it may be beneficial
to use multiple servers.
The following chart compares the multi-core performance between `crypto/md5` vs the AVX2 vs the AVX512 code:
![md5-performance-overview](chart/Multi-core-MD5-Aggregated-Hashing-Performance.png)
Compared to `crypto/md5`, the AVX2 version is up to 4x faster:
```
$ benchcmp crypto-md5.txt avx2.txt
benchmark old MB/s new MB/s speedup
BenchmarkParallel/32KB-4 2229.22 7370.50 3.31x
BenchmarkParallel/64KB-4 2233.61 8248.46 3.69x
BenchmarkParallel/128KB-4 2235.43 8660.74 3.87x
BenchmarkParallel/256KB-4 2236.39 8863.87 3.96x
BenchmarkParallel/512KB-4 2238.05 8985.39 4.01x
BenchmarkParallel/1MB-4 2233.56 9042.62 4.05x
BenchmarkParallel/2MB-4 2224.11 9014.46 4.05x
BenchmarkParallel/4MB-4 2199.78 8993.61 4.09x
BenchmarkParallel/8MB-4 2182.48 8748.22 4.01x
```
Compared to `crypto/md5`, the AVX512 is up to 8x faster (for larger block sizes):
```
$ benchcmp crypto-md5.txt avx512.txt
benchmark old MB/s new MB/s speedup
BenchmarkParallel/32KB-4 2229.22 11605.78 5.21x
BenchmarkParallel/64KB-4 2233.61 14329.65 6.42x
BenchmarkParallel/128KB-4 2235.43 16166.39 7.23x
BenchmarkParallel/256KB-4 2236.39 15570.09 6.96x
BenchmarkParallel/512KB-4 2238.05 16705.83 7.46x
BenchmarkParallel/1MB-4 2233.56 16941.95 7.59x
BenchmarkParallel/2MB-4 2224.11 17136.01 7.70x
BenchmarkParallel/4MB-4 2199.78 17218.61 7.83x
BenchmarkParallel/8MB-4 2182.48 17252.88 7.91x
```
These measurements were performed on AWS EC2 instance of type `c5.xlarge` equipped with a Xeon Platinum 8124M CPU at 3.0 GHz.
If only one or two inputs are available the scalar calculation method will be used for the
optimal speed in these cases.
## Operation
To make operation as easy as possible there is a “Server” coordinating everything. The server keeps track of individual hash states and updates them as new data comes in. This can be visualized as follows:
![server-architecture](chart/server-architecture.png)
The data is sent to the server from each hash input in blocks of up to 32KB per round. In our testing we found this to be the block size that yielded the best results.
Whenever there is data available the server will collect data for up to 16 hashes and process all 16 lanes in parallel. This means that if 16 hashes have data available all the lanes will be filled. However since that may not be the case, the server will fill less lanes and do a round anyway. Lanes can also be partially filled if less than 32KB of data is written.
![server-lanes-example](chart/server-lanes-example.png)
In this example 4 lanes are fully filled and 2 lanes are partially filled. In this case the black areas will simply be masked out from the results and ignored. This is also why calculating a single hash on a server will not result in any speedup and hash writes should be a multiple of 32KB for the best performance.
For AVX512 all 16 calculations will be done on a single core, on AVX2 on 2 cores if there is data for more than 8 lanes.
So for optimal usage there should be data available for all 16 hashes. It may be perfectly reasonable to use more than 16 concurrent hashes.
## Design & Tech
md5-simd has both an AVX2 (8-lane parallel), and an AVX512 (16-lane parallel version) algorithm to accelerate the computation with the following function definitions:
```
//go:noescape
func block8(state *uint32, base uintptr, bufs *int32, cache *byte, n int)
//go:noescape
func block16(state *uint32, ptrs *int64, mask uint64, n int)
```
The AVX2 version is based on the [md5vec](https://github.com/igneous-systems/md5vec) repository and is essentially unchanged except for minor (cosmetic) changes.
The AVX512 version is derived from the AVX2 version but adds some further optimizations and simplifications.
### Caching in upper ZMM registers
The AVX2 version passes in a `cache8` block of memory (about 0.5 KB) for temporary storage of intermediate results during `ROUND1` which are subsequently used during `ROUND2` through to `ROUND4`.
Since AVX512 has double the amount of registers (32 ZMM registers as compared to 16 YMM registers), it is possible to use the upper 16 ZMM registers for keeping the intermediate states on the CPU. As such, there is no need to pass in a corresponding `cache16` into the AVX512 block function.
### Direct loading using 64-bit pointers
The AVX2 uses the `VPGATHERDD` instruction (for YMM) to do a parallel load of 8 lanes using (8 independent) 32-bit offets. Since there is no control over how the 8 slices that are passed into the (Golang) `blockMd5` function are laid out into memory, it is not possible to derive a "base" address and corresponding offsets (all within 32-bits) for all 8 slices.
As such the AVX2 version uses an interim buffer to collect the byte slices to be hashed from all 8 inut slices and passed this buffer along with (fixed) 32-bit offsets into the assembly code.
For the AVX512 version this interim buffer is not needed since the AVX512 code uses a pair of `VPGATHERQD` instructions to directly dereference 64-bit pointers (from a base register address that is initialized to zero).
Note that two load (gather) instructions are needed because the AVX512 version processes 16-lanes in parallel, requiring 16 times 64-bit = 1024 bits in total to be loaded. A simple `VALIGND` and `VPORD` are subsequently used to merge the lower and upper halves together into a single ZMM register (that contains 16 lanes of 32-bit DWORDS).
### Masking support
Due to the fact that pointers are passed directly from the Golang slices, we need to protect against NULL pointers.
For this a 16-bit mask is passed in the AVX512 assembly code which is used during the `VPGATHERQD` instructions to mask out lanes that could otherwise result in segment violations.
### Minor optimizations
The `roll` macro (three instructions on AVX2) is no longer needed for AVX512 and is replaced by a single `VPROLD` instruction.
Also several logical operations from the various ROUNDS of the AVX2 version could be combined into a single instruction using ternary logic (with the `VPTERMLOGD` instruction), resulting in a further simplification and speed-up.
## Low level block function performance
The benchmark below shows the (single thread) maximum performance of the `block()` function for AVX2 (having 8 lanes) and AVX512 (having 16 lanes). Also the baseline single-core performance from the standard `crypto/md5` package is shown for comparison.
```
BenchmarkCryptoMd5-4 687.66 MB/s 0 B/op 0 allocs/op
BenchmarkBlock8-4 4144.80 MB/s 0 B/op 0 allocs/op
BenchmarkBlock16-4 8228.88 MB/s 0 B/op 0 allocs/op
```
## License
`md5-simd` is released under the Apache License v2.0. You can find the complete text in the file LICENSE.
## Contributing
Contributions are welcome, please send PRs for any enhancements.
Server/vendor/github.com/minio/md5-simd/block16_amd64.s
Generated Vendored
+228
View File
@@ -0,0 +1,228 @@
// Copyright (c) 2020 MinIO Inc. All rights reserved.
// Use of this source code is governed by a license that can be
// found in the LICENSE file.
//+build !noasm,!appengine,gc
// This is the AVX512 implementation of the MD5 block function (16-way parallel)
#define prep(index) \
KMOVQ kmask, ktmp \
VPGATHERDD index*4(base)(ptrs*1), ktmp, mem
#define ROUND1(a, b, c, d, index, const, shift) \
VPXORQ c, tmp, tmp \
VPADDD 64*const(consts), a, a \
VPADDD mem, a, a \
VPTERNLOGD $0x6C, b, d, tmp \
prep(index) \
VPADDD tmp, a, a \
VPROLD $shift, a, a \
VMOVAPD c, tmp \
VPADDD b, a, a
#define ROUND1noload(a, b, c, d, const, shift) \
VPXORQ c, tmp, tmp \
VPADDD 64*const(consts), a, a \
VPADDD mem, a, a \
VPTERNLOGD $0x6C, b, d, tmp \
VPADDD tmp, a, a \
VPROLD $shift, a, a \
VMOVAPD c, tmp \
VPADDD b, a, a
#define ROUND2(a, b, c, d, zreg, const, shift) \
VPADDD 64*const(consts), a, a \
VPADDD zreg, a, a \
VANDNPD c, tmp, tmp \
VPTERNLOGD $0xEC, b, tmp, tmp2 \
VMOVAPD c, tmp \
VPADDD tmp2, a, a \
VMOVAPD c, tmp2 \
VPROLD $shift, a, a \
VPADDD b, a, a
#define ROUND3(a, b, c, d, zreg, const, shift) \
VPADDD 64*const(consts), a, a \
VPADDD zreg, a, a \
VPTERNLOGD $0x96, b, d, tmp \
VPADDD tmp, a, a \
VPROLD $shift, a, a \
VMOVAPD b, tmp \
VPADDD b, a, a
#define ROUND4(a, b, c, d, zreg, const, shift) \
VPADDD 64*const(consts), a, a \
VPADDD zreg, a, a \
VPTERNLOGD $0x36, b, c, tmp \
VPADDD tmp, a, a \
VPROLD $shift, a, a \
VPXORQ c, ones, tmp \
VPADDD b, a, a
TEXT ·block16(SB), 4, $0-40
MOVQ state+0(FP), BX
MOVQ base+8(FP), SI
MOVQ ptrs+16(FP), AX
KMOVQ mask+24(FP), K1
MOVQ n+32(FP), DX
MOVQ ·avx512md5consts+0(SB), DI
#define a Z0
#define b Z1
#define c Z2
#define d Z3
#define sa Z4
#define sb Z5
#define sc Z6
#define sd Z7
#define tmp Z8
#define tmp2 Z9
#define ptrs Z10
#define ones Z12
#define mem Z15
#define kmask K1
#define ktmp K3
// ----------------------------------------------------------
// Registers Z16 through to Z31 are used for caching purposes
// ----------------------------------------------------------
#define dig BX
#define count DX
#define base SI
#define consts DI
// load digest into state registers
VMOVUPD (dig), a
VMOVUPD 0x40(dig), b
VMOVUPD 0x80(dig), c
VMOVUPD 0xc0(dig), d
// load source pointers
VMOVUPD 0x00(AX), ptrs
MOVQ $-1, AX
VPBROADCASTQ AX, ones
loop:
VMOVAPD a, sa
VMOVAPD b, sb
VMOVAPD c, sc
VMOVAPD d, sd
prep(0)
VMOVAPD d, tmp
VMOVAPD mem, Z16
ROUND1(a,b,c,d, 1,0x00, 7)
VMOVAPD mem, Z17
ROUND1(d,a,b,c, 2,0x01,12)
VMOVAPD mem, Z18
ROUND1(c,d,a,b, 3,0x02,17)
VMOVAPD mem, Z19
ROUND1(b,c,d,a, 4,0x03,22)
VMOVAPD mem, Z20
ROUND1(a,b,c,d, 5,0x04, 7)
VMOVAPD mem, Z21
ROUND1(d,a,b,c, 6,0x05,12)
VMOVAPD mem, Z22
ROUND1(c,d,a,b, 7,0x06,17)
VMOVAPD mem, Z23
ROUND1(b,c,d,a, 8,0x07,22)
VMOVAPD mem, Z24
ROUND1(a,b,c,d, 9,0x08, 7)
VMOVAPD mem, Z25
ROUND1(d,a,b,c,10,0x09,12)
VMOVAPD mem, Z26
ROUND1(c,d,a,b,11,0x0a,17)
VMOVAPD mem, Z27
ROUND1(b,c,d,a,12,0x0b,22)
VMOVAPD mem, Z28
ROUND1(a,b,c,d,13,0x0c, 7)
VMOVAPD mem, Z29
ROUND1(d,a,b,c,14,0x0d,12)
VMOVAPD mem, Z30
ROUND1(c,d,a,b,15,0x0e,17)
VMOVAPD mem, Z31
ROUND1noload(b,c,d,a, 0x0f,22)
VMOVAPD d, tmp
VMOVAPD d, tmp2
ROUND2(a,b,c,d, Z17,0x10, 5)
ROUND2(d,a,b,c, Z22,0x11, 9)
ROUND2(c,d,a,b, Z27,0x12,14)
ROUND2(b,c,d,a, Z16,0x13,20)
ROUND2(a,b,c,d, Z21,0x14, 5)
ROUND2(d,a,b,c, Z26,0x15, 9)
ROUND2(c,d,a,b, Z31,0x16,14)
ROUND2(b,c,d,a, Z20,0x17,20)
ROUND2(a,b,c,d, Z25,0x18, 5)
ROUND2(d,a,b,c, Z30,0x19, 9)
ROUND2(c,d,a,b, Z19,0x1a,14)
ROUND2(b,c,d,a, Z24,0x1b,20)
ROUND2(a,b,c,d, Z29,0x1c, 5)
ROUND2(d,a,b,c, Z18,0x1d, 9)
ROUND2(c,d,a,b, Z23,0x1e,14)
ROUND2(b,c,d,a, Z28,0x1f,20)
VMOVAPD c, tmp
ROUND3(a,b,c,d, Z21,0x20, 4)
ROUND3(d,a,b,c, Z24,0x21,11)
ROUND3(c,d,a,b, Z27,0x22,16)
ROUND3(b,c,d,a, Z30,0x23,23)
ROUND3(a,b,c,d, Z17,0x24, 4)
ROUND3(d,a,b,c, Z20,0x25,11)
ROUND3(c,d,a,b, Z23,0x26,16)
ROUND3(b,c,d,a, Z26,0x27,23)
ROUND3(a,b,c,d, Z29,0x28, 4)
ROUND3(d,a,b,c, Z16,0x29,11)
ROUND3(c,d,a,b, Z19,0x2a,16)
ROUND3(b,c,d,a, Z22,0x2b,23)
ROUND3(a,b,c,d, Z25,0x2c, 4)
ROUND3(d,a,b,c, Z28,0x2d,11)
ROUND3(c,d,a,b, Z31,0x2e,16)
ROUND3(b,c,d,a, Z18,0x2f,23)
VPXORQ d, ones, tmp
ROUND4(a,b,c,d, Z16,0x30, 6)
ROUND4(d,a,b,c, Z23,0x31,10)
ROUND4(c,d,a,b, Z30,0x32,15)
ROUND4(b,c,d,a, Z21,0x33,21)
ROUND4(a,b,c,d, Z28,0x34, 6)
ROUND4(d,a,b,c, Z19,0x35,10)
ROUND4(c,d,a,b, Z26,0x36,15)
ROUND4(b,c,d,a, Z17,0x37,21)
ROUND4(a,b,c,d, Z24,0x38, 6)
ROUND4(d,a,b,c, Z31,0x39,10)
ROUND4(c,d,a,b, Z22,0x3a,15)
ROUND4(b,c,d,a, Z29,0x3b,21)
ROUND4(a,b,c,d, Z20,0x3c, 6)
ROUND4(d,a,b,c, Z27,0x3d,10)
ROUND4(c,d,a,b, Z18,0x3e,15)
ROUND4(b,c,d,a, Z25,0x3f,21)
VPADDD sa, a, a
VPADDD sb, b, b
VPADDD sc, c, c
VPADDD sd, d, d
LEAQ 64(base), base
SUBQ $64, count
JNE loop
VMOVUPD a, (dig)
VMOVUPD b, 0x40(dig)
VMOVUPD c, 0x80(dig)
VMOVUPD d, 0xc0(dig)
VZEROUPPER
RET
Server/vendor/github.com/minio/md5-simd/block8_amd64.s
Generated Vendored
+281
View File
@@ -0,0 +1,281 @@
//+build !noasm,!appengine,gc
// Copyright (c) 2018 Igneous Systems
// MIT License
//
// Permission is hereby granted, free of charge, to any person obtaining a copy
// of this software and associated documentation files (the "Software"), to deal
// in the Software without restriction, including without limitation the rights
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
// copies of the Software, and to permit persons to whom the Software is
// furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in all
// copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
// SOFTWARE.
// Copyright (c) 2020 MinIO Inc. All rights reserved.
// Use of this source code is governed by a license that can be
// found in the LICENSE file.
// This is the AVX2 implementation of the MD5 block function (8-way parallel)
// block8(state *uint64, base uintptr, bufs *int32, cache *byte, n int)
TEXT ·block8(SB), 4, $0-40
MOVQ state+0(FP), BX
MOVQ base+8(FP), SI
MOVQ bufs+16(FP), AX
MOVQ cache+24(FP), CX
MOVQ n+32(FP), DX
MOVQ ·avx256md5consts+0(SB), DI
// Align cache (which is stack allocated by the compiler)
// to a 256 bit boundary (ymm register alignment)
// The cache8 type is deliberately oversized to permit this.
ADDQ $31, CX
ANDB $-32, CL
#define a Y0
#define b Y1
#define c Y2
#define d Y3
#define sa Y4
#define sb Y5
#define sc Y6
#define sd Y7
#define tmp Y8
#define tmp2 Y9
#define mask Y10
#define off Y11
#define ones Y12
#define rtmp1 Y13
#define rtmp2 Y14
#define mem Y15
#define dig BX
#define cache CX
#define count DX
#define base SI
#define consts DI
#define prepmask \
VPXOR mask, mask, mask \
VPCMPGTD mask, off, mask
#define prep(index) \
VMOVAPD mask, rtmp2 \
VPGATHERDD rtmp2, index*4(base)(off*1), mem
#define load(index) \
VMOVAPD index*32(cache), mem
#define store(index) \
VMOVAPD mem, index*32(cache)
#define roll(shift, a) \
VPSLLD $shift, a, rtmp1 \
VPSRLD $32-shift, a, a \
VPOR rtmp1, a, a
#define ROUND1(a, b, c, d, index, const, shift) \
VPXOR c, tmp, tmp \
VPADDD 32*const(consts), a, a \
VPADDD mem, a, a \
VPAND b, tmp, tmp \
VPXOR d, tmp, tmp \
prep(index) \
VPADDD tmp, a, a \
roll(shift,a) \
VMOVAPD c, tmp \
VPADDD b, a, a
#define ROUND1load(a, b, c, d, index, const, shift) \
VXORPD c, tmp, tmp \
VPADDD 32*const(consts), a, a \
VPADDD mem, a, a \
VPAND b, tmp, tmp \
VPXOR d, tmp, tmp \
load(index) \
VPADDD tmp, a, a \
roll(shift,a) \
VMOVAPD c, tmp \
VPADDD b, a, a
#define ROUND2(a, b, c, d, index, const, shift) \
VPADDD 32*const(consts), a, a \
VPADDD mem, a, a \
VPAND b, tmp2, tmp2 \
VANDNPD c, tmp, tmp \
load(index) \
VPOR tmp, tmp2, tmp2 \
VMOVAPD c, tmp \
VPADDD tmp2, a, a \
VMOVAPD c, tmp2 \
roll(shift,a) \
VPADDD b, a, a
#define ROUND3(a, b, c, d, index, const, shift) \
VPADDD 32*const(consts), a, a \
VPADDD mem, a, a \
load(index) \
VPXOR d, tmp, tmp \
VPXOR b, tmp, tmp \
VPADDD tmp, a, a \
roll(shift,a) \
VMOVAPD b, tmp \
VPADDD b, a, a
#define ROUND4(a, b, c, d, index, const, shift) \
VPADDD 32*const(consts), a, a \
VPADDD mem, a, a \
VPOR b, tmp, tmp \
VPXOR c, tmp, tmp \
VPADDD tmp, a, a \
load(index) \
roll(shift,a) \
VPXOR c, ones, tmp \
VPADDD b, a, a
// load digest into state registers
VMOVUPD (dig), a
VMOVUPD 32(dig), b
VMOVUPD 64(dig), c
VMOVUPD 96(dig), d
// load source buffer offsets
VMOVUPD (AX), off
prepmask
VPCMPEQD ones, ones, ones
loop:
VMOVAPD a, sa
VMOVAPD b, sb
VMOVAPD c, sc
VMOVAPD d, sd
prep(0)
VMOVAPD d, tmp
store(0)
ROUND1(a,b,c,d, 1,0x00, 7)
store(1)
ROUND1(d,a,b,c, 2,0x01,12)
store(2)
ROUND1(c,d,a,b, 3,0x02,17)
store(3)
ROUND1(b,c,d,a, 4,0x03,22)
store(4)
ROUND1(a,b,c,d, 5,0x04, 7)
store(5)
ROUND1(d,a,b,c, 6,0x05,12)
store(6)
ROUND1(c,d,a,b, 7,0x06,17)
store(7)
ROUND1(b,c,d,a, 8,0x07,22)
store(8)
ROUND1(a,b,c,d, 9,0x08, 7)
store(9)
ROUND1(d,a,b,c,10,0x09,12)
store(10)
ROUND1(c,d,a,b,11,0x0a,17)
store(11)
ROUND1(b,c,d,a,12,0x0b,22)
store(12)
ROUND1(a,b,c,d,13,0x0c, 7)
store(13)
ROUND1(d,a,b,c,14,0x0d,12)
store(14)
ROUND1(c,d,a,b,15,0x0e,17)
store(15)
ROUND1load(b,c,d,a, 1,0x0f,22)
VMOVAPD d, tmp
VMOVAPD d, tmp2
ROUND2(a,b,c,d, 6,0x10, 5)
ROUND2(d,a,b,c,11,0x11, 9)
ROUND2(c,d,a,b, 0,0x12,14)
ROUND2(b,c,d,a, 5,0x13,20)
ROUND2(a,b,c,d,10,0x14, 5)
ROUND2(d,a,b,c,15,0x15, 9)
ROUND2(c,d,a,b, 4,0x16,14)
ROUND2(b,c,d,a, 9,0x17,20)
ROUND2(a,b,c,d,14,0x18, 5)
ROUND2(d,a,b,c, 3,0x19, 9)
ROUND2(c,d,a,b, 8,0x1a,14)
ROUND2(b,c,d,a,13,0x1b,20)
ROUND2(a,b,c,d, 2,0x1c, 5)
ROUND2(d,a,b,c, 7,0x1d, 9)
ROUND2(c,d,a,b,12,0x1e,14)
ROUND2(b,c,d,a, 0,0x1f,20)
load(5)
VMOVAPD c, tmp
ROUND3(a,b,c,d, 8,0x20, 4)
ROUND3(d,a,b,c,11,0x21,11)
ROUND3(c,d,a,b,14,0x22,16)
ROUND3(b,c,d,a, 1,0x23,23)
ROUND3(a,b,c,d, 4,0x24, 4)
ROUND3(d,a,b,c, 7,0x25,11)
ROUND3(c,d,a,b,10,0x26,16)
ROUND3(b,c,d,a,13,0x27,23)
ROUND3(a,b,c,d, 0,0x28, 4)
ROUND3(d,a,b,c, 3,0x29,11)
ROUND3(c,d,a,b, 6,0x2a,16)
ROUND3(b,c,d,a, 9,0x2b,23)
ROUND3(a,b,c,d,12,0x2c, 4)
ROUND3(d,a,b,c,15,0x2d,11)
ROUND3(c,d,a,b, 2,0x2e,16)
ROUND3(b,c,d,a, 0,0x2f,23)
load(0)
VPXOR d, ones, tmp
ROUND4(a,b,c,d, 7,0x30, 6)
ROUND4(d,a,b,c,14,0x31,10)
ROUND4(c,d,a,b, 5,0x32,15)
ROUND4(b,c,d,a,12,0x33,21)
ROUND4(a,b,c,d, 3,0x34, 6)
ROUND4(d,a,b,c,10,0x35,10)
ROUND4(c,d,a,b, 1,0x36,15)
ROUND4(b,c,d,a, 8,0x37,21)
ROUND4(a,b,c,d,15,0x38, 6)
ROUND4(d,a,b,c, 6,0x39,10)
ROUND4(c,d,a,b,13,0x3a,15)
ROUND4(b,c,d,a, 4,0x3b,21)
ROUND4(a,b,c,d,11,0x3c, 6)
ROUND4(d,a,b,c, 2,0x3d,10)
ROUND4(c,d,a,b, 9,0x3e,15)
ROUND4(b,c,d,a, 0,0x3f,21)
VPADDD sa, a, a
VPADDD sb, b, b
VPADDD sc, c, c
VPADDD sd, d, d
LEAQ 64(base), base
SUBQ $64, count
JNE loop
VMOVUPD a, (dig)
VMOVUPD b, 32(dig)
VMOVUPD c, 64(dig)
VMOVUPD d, 96(dig)
VZEROUPPER
RET
Server/vendor/github.com/minio/md5-simd/block_amd64.go
Generated Vendored
+210
View File
@@ -0,0 +1,210 @@
//+build !noasm,!appengine,gc
// Copyright (c) 2020 MinIO Inc. All rights reserved.
// Use of this source code is governed by a license that can be
// found in the LICENSE file.
package md5simd
import (
"fmt"
"math"
"unsafe"
"github.com/klauspost/cpuid/v2"
)
var hasAVX512 bool
func init() {
// VANDNPD requires AVX512DQ. Technically it could be VPTERNLOGQ which is AVX512F.
hasAVX512 = cpuid.CPU.Supports(cpuid.AVX512F, cpuid.AVX512DQ)
}
//go:noescape
func block8(state *uint32, base uintptr, bufs *int32, cache *byte, n int)
//go:noescape
func block16(state *uint32, base uintptr, ptrs *int32, mask uint64, n int)
// 8-way 4x uint32 digests in 4 ymm registers
// (ymm0, ymm1, ymm2, ymm3)
type digest8 struct {
v0, v1, v2, v3 [8]uint32
}
// Stack cache for 8x64 byte md5.BlockSize bytes.
// Must be 32-byte aligned, so allocate 512+32 and
// align upwards at runtime.
type cache8 [512 + 32]byte
// MD5 magic numbers for one lane of hashing; inflated
// 8x below at init time.
var md5consts = [64]uint32{
0xd76aa478, 0xe8c7b756, 0x242070db, 0xc1bdceee,
0xf57c0faf, 0x4787c62a, 0xa8304613, 0xfd469501,
0x698098d8, 0x8b44f7af, 0xffff5bb1, 0x895cd7be,
0x6b901122, 0xfd987193, 0xa679438e, 0x49b40821,
0xf61e2562, 0xc040b340, 0x265e5a51, 0xe9b6c7aa,
0xd62f105d, 0x02441453, 0xd8a1e681, 0xe7d3fbc8,
0x21e1cde6, 0xc33707d6, 0xf4d50d87, 0x455a14ed,
0xa9e3e905, 0xfcefa3f8, 0x676f02d9, 0x8d2a4c8a,
0xfffa3942, 0x8771f681, 0x6d9d6122, 0xfde5380c,
0xa4beea44, 0x4bdecfa9, 0xf6bb4b60, 0xbebfbc70,
0x289b7ec6, 0xeaa127fa, 0xd4ef3085, 0x04881d05,
0xd9d4d039, 0xe6db99e5, 0x1fa27cf8, 0xc4ac5665,
0xf4292244, 0x432aff97, 0xab9423a7, 0xfc93a039,
0x655b59c3, 0x8f0ccc92, 0xffeff47d, 0x85845dd1,
0x6fa87e4f, 0xfe2ce6e0, 0xa3014314, 0x4e0811a1,
0xf7537e82, 0xbd3af235, 0x2ad7d2bb, 0xeb86d391,
}
// inflate the consts 8-way for 8x md5 (256 bit ymm registers)
var avx256md5consts = func(c []uint32) []uint32 {
inf := make([]uint32, 8*len(c))
for i := range c {
for j := 0; j < 8; j++ {
inf[(i*8)+j] = c[i]
}
}
return inf
}(md5consts[:])
// 16-way 4x uint32 digests in 4 zmm registers
type digest16 struct {
v0, v1, v2, v3 [16]uint32
}
// inflate the consts 16-way for 16x md5 (512 bit zmm registers)
var avx512md5consts = func(c []uint32) []uint32 {
inf := make([]uint32, 16*len(c))
for i := range c {
for j := 0; j < 16; j++ {
inf[(i*16)+j] = c[i]
}
}
return inf
}(md5consts[:])
// Interface function to assembly code
func (s *md5Server) blockMd5_x16(d *digest16, input [16][]byte, half bool) {
if hasAVX512 {
blockMd5_avx512(d, input, s.allBufs, &s.maskRounds16)
return
}
// Preparing data using copy is slower since copies aren't inlined.
// Calculate on this goroutine
if half {
for i := range s.i8[0][:] {
s.i8[0][i] = input[i]
}
for i := range s.d8a.v0[:] {
s.d8a.v0[i], s.d8a.v1[i], s.d8a.v2[i], s.d8a.v3[i] = d.v0[i], d.v1[i], d.v2[i], d.v3[i]
}
blockMd5_avx2(&s.d8a, s.i8[0], s.allBufs, &s.maskRounds8a)
for i := range s.d8a.v0[:] {
d.v0[i], d.v1[i], d.v2[i], d.v3[i] = s.d8a.v0[i], s.d8a.v1[i], s.d8a.v2[i], s.d8a.v3[i]
}
return
}
for i := range s.i8[0][:] {
s.i8[0][i], s.i8[1][i] = input[i], input[8+i]
}
for i := range s.d8a.v0[:] {
j := (i + 8) & 15
s.d8a.v0[i], s.d8a.v1[i], s.d8a.v2[i], s.d8a.v3[i] = d.v0[i], d.v1[i], d.v2[i], d.v3[i]
s.d8b.v0[i], s.d8b.v1[i], s.d8b.v2[i], s.d8b.v3[i] = d.v0[j], d.v1[j], d.v2[j], d.v3[j]
}
// Benchmarks appears to be slightly faster when spinning up 2 goroutines instead
// of using the current for one of the blocks.
s.wg.Add(2)
go func() { blockMd5_avx2(&s.d8a, s.i8[0], s.allBufs, &s.maskRounds8a); s.wg.Done() }()
go func() { blockMd5_avx2(&s.d8b, s.i8[1], s.allBufs, &s.maskRounds8b); s.wg.Done() }()
s.wg.Wait()
for i := range s.d8a.v0[:] {
d.v0[i], d.v1[i], d.v2[i], d.v3[i] = s.d8a.v0[i], s.d8a.v1[i], s.d8a.v2[i], s.d8a.v3[i]
}
for i := range s.d8b.v0[:] {
j := (i + 8) & 15
d.v0[j], d.v1[j], d.v2[j], d.v3[j] = s.d8b.v0[i], s.d8b.v1[i], s.d8b.v2[i], s.d8b.v3[i]
}
}
// Interface function to AVX512 assembly code
func blockMd5_avx512(s *digest16, input [16][]byte, base []byte, maskRounds *[16]maskRounds) {
baseMin := uint64(uintptr(unsafe.Pointer(&(base[0]))))
ptrs := [16]int32{}
for i := range ptrs {
if len(input[i]) > 0 {
if len(input[i]) > internalBlockSize {
panic(fmt.Sprintf("Sanity check fails for lane %d: maximum input length cannot exceed internalBlockSize", i))
}
off := uint64(uintptr(unsafe.Pointer(&(input[i][0])))) - baseMin
if off > math.MaxUint32 {
panic(fmt.Sprintf("invalid buffer sent with offset %x", off))
}
ptrs[i] = int32(off)
}
}
sdup := *s // create copy of initial states to receive intermediate updates
rounds := generateMaskAndRounds16(input, maskRounds)
for r := 0; r < rounds; r++ {
m := maskRounds[r]
block16(&sdup.v0[0], uintptr(baseMin), &ptrs[0], m.mask, int(64*m.rounds))
for j := 0; j < len(ptrs); j++ {
ptrs[j] += int32(64 * m.rounds) // update pointers for next round
if m.mask&(1<<j) != 0 { // update digest if still masked as active
(*s).v0[j], (*s).v1[j], (*s).v2[j], (*s).v3[j] = sdup.v0[j], sdup.v1[j], sdup.v2[j], sdup.v3[j]
}
}
}
}
// Interface function to AVX2 assembly code
func blockMd5_avx2(s *digest8, input [8][]byte, base []byte, maskRounds *[8]maskRounds) {
baseMin := uint64(uintptr(unsafe.Pointer(&(base[0])))) - 4
ptrs := [8]int32{}
for i := range ptrs {
if len(input[i]) > 0 {
if len(input[i]) > internalBlockSize {
panic(fmt.Sprintf("Sanity check fails for lane %d: maximum input length cannot exceed internalBlockSize", i))
}
off := uint64(uintptr(unsafe.Pointer(&(input[i][0])))) - baseMin
if off > math.MaxUint32 {
panic(fmt.Sprintf("invalid buffer sent with offset %x", off))
}
ptrs[i] = int32(off)
}
}
sdup := *s // create copy of initial states to receive intermediate updates
rounds := generateMaskAndRounds8(input, maskRounds)
for r := 0; r < rounds; r++ {
m := maskRounds[r]
var cache cache8 // stack storage for block8 tmp state
block8(&sdup.v0[0], uintptr(baseMin), &ptrs[0], &cache[0], int(64*m.rounds))
for j := 0; j < len(ptrs); j++ {
ptrs[j] += int32(64 * m.rounds) // update pointers for next round
if m.mask&(1<<j) != 0 { // update digest if still masked as active
(*s).v0[j], (*s).v1[j], (*s).v2[j], (*s).v3[j] = sdup.v0[j], sdup.v1[j], sdup.v2[j], sdup.v3[j]
}
}
}
}
Server/vendor/github.com/minio/md5-simd/md5-digest_amd64.go
Generated Vendored
+188
View File
@@ -0,0 +1,188 @@
//+build !noasm,!appengine,gc
// Copyright (c) 2020 MinIO Inc. All rights reserved.
// Use of this source code is governed by a license that can be
// found in the LICENSE file.
package md5simd
import (
"encoding/binary"
"errors"
"fmt"
"sync"
"sync/atomic"
)
// md5Digest - Type for computing MD5 using either AVX2 or AVX512
type md5Digest struct {
uid uint64
blocksCh chan blockInput
cycleServer chan uint64
x [BlockSize]byte
nx int
len uint64
buffers <-chan []byte
}
// NewHash - initialize instance for Md5 implementation.
func (s *md5Server) NewHash() Hasher {
uid := atomic.AddUint64(&s.uidCounter, 1)
blockCh := make(chan blockInput, buffersPerLane)
s.newInput <- newClient{
uid: uid,
input: blockCh,
}
return &md5Digest{
uid: uid,
buffers: s.buffers,
blocksCh: blockCh,
cycleServer: s.cycle,
}
}
// Size - Return size of checksum
func (d *md5Digest) Size() int { return Size }
// BlockSize - Return blocksize of checksum
func (d md5Digest) BlockSize() int { return BlockSize }
func (d *md5Digest) Reset() {
if d.blocksCh == nil {
panic("reset after close")
}
d.nx = 0
d.len = 0
d.sendBlock(blockInput{uid: d.uid, reset: true}, false)
}
// write to digest
func (d *md5Digest) Write(p []byte) (nn int, err error) {
if d.blocksCh == nil {
return 0, errors.New("md5Digest closed")
}
// break input into chunks of maximum internalBlockSize size
for {
l := len(p)
if l > internalBlockSize {
l = internalBlockSize
}
nnn, err := d.write(p[:l])
if err != nil {
return nn, err
}
nn += nnn
p = p[l:]
if len(p) == 0 {
break
}
}
return
}
func (d *md5Digest) write(p []byte) (nn int, err error) {
nn = len(p)
d.len += uint64(nn)
if d.nx > 0 {
n := copy(d.x[d.nx:], p)
d.nx += n
if d.nx == BlockSize {
// Create a copy of the overflow buffer in order to send it async over the channel
// (since we will modify the overflow buffer down below with any access beyond multiples of 64)
tmp := <-d.buffers
tmp = tmp[:BlockSize]
copy(tmp, d.x[:])
d.sendBlock(blockInput{uid: d.uid, msg: tmp}, len(p)-n < BlockSize)
d.nx = 0
}
p = p[n:]
}
if len(p) >= BlockSize {
n := len(p) &^ (BlockSize - 1)
buf := <-d.buffers
buf = buf[:n]
copy(buf, p)
d.sendBlock(blockInput{uid: d.uid, msg: buf}, len(p)-n < BlockSize)
p = p[n:]
}
if len(p) > 0 {
d.nx = copy(d.x[:], p)
}
return
}
func (d *md5Digest) Close() {
if d.blocksCh != nil {
close(d.blocksCh)
d.blocksCh = nil
}
}
var sumChPool sync.Pool
func init() {
sumChPool.New = func() interface{} {
return make(chan sumResult, 1)
}
}
// Sum - Return MD5 sum in bytes
func (d *md5Digest) Sum(in []byte) (result []byte) {
if d.blocksCh == nil {
panic("sum after close")
}
trail := <-d.buffers
trail = append(trail[:0], d.x[:d.nx]...)
length := d.len
// Padding. Add a 1 bit and 0 bits until 56 bytes mod 64.
var tmp [64]byte
tmp[0] = 0x80
if length%64 < 56 {
trail = append(trail, tmp[0:56-length%64]...)
} else {
trail = append(trail, tmp[0:64+56-length%64]...)
}
// Length in bits.
length <<= 3
binary.LittleEndian.PutUint64(tmp[:], length) // append length in bits
trail = append(trail, tmp[0:8]...)
if len(trail)%BlockSize != 0 {
panic(fmt.Errorf("internal error: sum block was not aligned. len=%d, nx=%d", len(trail), d.nx))
}
sumCh := sumChPool.Get().(chan sumResult)
d.sendBlock(blockInput{uid: d.uid, msg: trail, sumCh: sumCh}, true)
sum := <-sumCh
sumChPool.Put(sumCh)
return append(in, sum.digest[:]...)
}
// sendBlock will send a block for processing.
// If cycle is true we will block on cycle, otherwise we will only block
// if the block channel is full.
func (d *md5Digest) sendBlock(bi blockInput, cycle bool) {
if cycle {
select {
case d.blocksCh <- bi:
d.cycleServer <- d.uid
}
return
}
// Only block on cycle if we filled the buffer
select {
case d.blocksCh <- bi:
return
default:
d.cycleServer <- d.uid
d.blocksCh <- bi
}
}
Server/vendor/github.com/minio/md5-simd/md5-server_amd64.go
Generated Vendored
+397
View File
@@ -0,0 +1,397 @@
//+build !noasm,!appengine,gc
// Copyright (c) 2020 MinIO Inc. All rights reserved.
// Use of this source code is governed by a license that can be
// found in the LICENSE file.
package md5simd
import (
"encoding/binary"
"fmt"
"runtime"
"sync"
"github.com/klauspost/cpuid/v2"
)
// MD5 initialization constants
const (
// Lanes is the number of concurrently calculated hashes.
Lanes = 16
init0 = 0x67452301
init1 = 0xefcdab89
init2 = 0x98badcfe
init3 = 0x10325476
// Use scalar routine when below this many lanes
useScalarBelow = 3
)
// md5ServerUID - Does not start at 0 but next multiple of 16 so as to be able to
// differentiate with default initialisation value of 0
const md5ServerUID = Lanes
const buffersPerLane = 3
// Message to send across input channel
type blockInput struct {
uid uint64
msg []byte
sumCh chan sumResult
reset bool
}
type sumResult struct {
digest [Size]byte
}
type lanesInfo [Lanes]blockInput
// md5Server - Type to implement parallel handling of MD5 invocations
type md5Server struct {
uidCounter uint64
cycle chan uint64 // client with uid has update.
newInput chan newClient // Add new client.
digests map[uint64][Size]byte // Map of uids to (interim) digest results
maskRounds16 [16]maskRounds // Pre-allocated static array for max 16 rounds
maskRounds8a [8]maskRounds // Pre-allocated static array for max 8 rounds (1st AVX2 core)
maskRounds8b [8]maskRounds // Pre-allocated static array for max 8 rounds (2nd AVX2 core)
allBufs []byte // Preallocated buffer.
buffers chan []byte // Preallocated buffers, sliced from allBufs.
i8 [2][8][]byte // avx2 temporary vars
d8a, d8b digest8
wg sync.WaitGroup
}
// NewServer - Create new object for parallel processing handling
func NewServer() Server {
if !cpuid.CPU.Supports(cpuid.AVX2) {
return &fallbackServer{}
}
md5srv := &md5Server{}
md5srv.digests = make(map[uint64][Size]byte)
md5srv.newInput = make(chan newClient, Lanes)
md5srv.cycle = make(chan uint64, Lanes*10)
md5srv.uidCounter = md5ServerUID - 1
md5srv.allBufs = make([]byte, 32+buffersPerLane*Lanes*internalBlockSize)
md5srv.buffers = make(chan []byte, buffersPerLane*Lanes)
// Fill buffers.
for i := 0; i < buffersPerLane*Lanes; i++ {
s := 32 + i*internalBlockSize
md5srv.buffers <- md5srv.allBufs[s : s+internalBlockSize : s+internalBlockSize]
}
// Start a single thread for reading from the input channel
go md5srv.process(md5srv.newInput)
return md5srv
}
type newClient struct {
uid uint64
input chan blockInput
}
// process - Sole handler for reading from the input channel.
func (s *md5Server) process(newClients chan newClient) {
// To fill up as many lanes as possible:
//
// 1. Wait for a cycle id.
// 2. If not already in a lane, add, otherwise leave on channel
// 3. Start timer
// 4. Check if lanes is full, if so, goto 10 (process).
// 5. If timeout, goto 10.
// 6. Wait for new id (goto 2) or timeout (goto 10).
// 10. Process.
// 11. Check all input if there is already input, if so add to lanes.
// 12. Goto 1
// lanes contains the lanes.
var lanes lanesInfo
// lanesFilled contains the number of filled lanes for current cycle.
var lanesFilled int
// clients contains active clients
var clients = make(map[uint64]chan blockInput, Lanes)
addToLane := func(uid uint64) {
cl, ok := clients[uid]
if !ok {
// Unknown client. Maybe it was already removed.
return
}
// Check if we already have it.
for _, lane := range lanes[:lanesFilled] {
if lane.uid == uid {
return
}
}
// Continue until we get a block or there is nothing on channel
for {
select {
case block, ok := <-cl:
if !ok {
// Client disconnected
delete(clients, block.uid)
return
}
if block.uid != uid {
panic(fmt.Errorf("uid mismatch, %d (block) != %d (client)", block.uid, uid))
}
// If reset message, reset and we're done
if block.reset {
delete(s.digests, uid)
continue
}
// If requesting sum, we will need to maintain state.
if block.sumCh != nil {
var dig digest
d, ok := s.digests[uid]
if ok {
dig.s[0] = binary.LittleEndian.Uint32(d[0:4])
dig.s[1] = binary.LittleEndian.Uint32(d[4:8])
dig.s[2] = binary.LittleEndian.Uint32(d[8:12])
dig.s[3] = binary.LittleEndian.Uint32(d[12:16])
} else {
dig.s[0], dig.s[1], dig.s[2], dig.s[3] = init0, init1, init2, init3
}
sum := sumResult{}
// Add end block to current digest.
blockScalar(&dig.s, block.msg)
binary.LittleEndian.PutUint32(sum.digest[0:], dig.s[0])
binary.LittleEndian.PutUint32(sum.digest[4:], dig.s[1])
binary.LittleEndian.PutUint32(sum.digest[8:], dig.s[2])
binary.LittleEndian.PutUint32(sum.digest[12:], dig.s[3])
block.sumCh <- sum
if block.msg != nil {
s.buffers <- block.msg
}
continue
}
if len(block.msg) == 0 {
continue
}
lanes[lanesFilled] = block
lanesFilled++
return
default:
return
}
}
}
addNewClient := func(cl newClient) {
if _, ok := clients[cl.uid]; ok {
panic("internal error: duplicate client registration")
}
clients[cl.uid] = cl.input
}
allLanesFilled := func() bool {
return lanesFilled == Lanes || lanesFilled >= len(clients)
}
for {
// Step 1.
for lanesFilled == 0 {
select {
case cl, ok := <-newClients:
if !ok {
return
}
addNewClient(cl)
// Check if it already sent a payload.
addToLane(cl.uid)
continue
case uid := <-s.cycle:
addToLane(uid)
}
}
fillLanes:
for !allLanesFilled() {
select {
case cl, ok := <-newClients:
if !ok {
return
}
addNewClient(cl)
case uid := <-s.cycle:
addToLane(uid)
default:
// Nothing more queued...
break fillLanes
}
}
// If we did not fill all lanes, check if there is more waiting
if !allLanesFilled() {
runtime.Gosched()
for uid := range clients {
addToLane(uid)
if allLanesFilled() {
break
}
}
}
if false {
if !allLanesFilled() {
fmt.Println("Not all lanes filled", lanesFilled, "of", len(clients))
//pprof.Lookup("goroutine").WriteTo(os.Stdout, 1)
} else if true {
fmt.Println("all lanes filled")
}
}
// Process the lanes we could collect
s.blocks(lanes[:lanesFilled])
// Clear lanes...
lanesFilled = 0
// Add all current queued
for uid := range clients {
addToLane(uid)
if allLanesFilled() {
break
}
}
}
}
func (s *md5Server) Close() {
if s.newInput != nil {
close(s.newInput)
s.newInput = nil
}
}
// Invoke assembly and send results back
func (s *md5Server) blocks(lanes []blockInput) {
if len(lanes) < useScalarBelow {
// Use scalar routine when below this many lanes
switch len(lanes) {
case 0:
case 1:
lane := lanes[0]
var d digest
a, ok := s.digests[lane.uid]
if ok {
d.s[0] = binary.LittleEndian.Uint32(a[0:4])
d.s[1] = binary.LittleEndian.Uint32(a[4:8])
d.s[2] = binary.LittleEndian.Uint32(a[8:12])
d.s[3] = binary.LittleEndian.Uint32(a[12:16])
} else {
d.s[0] = init0
d.s[1] = init1
d.s[2] = init2
d.s[3] = init3
}
if len(lane.msg) > 0 {
// Update...
blockScalar(&d.s, lane.msg)
}
dig := [Size]byte{}
binary.LittleEndian.PutUint32(dig[0:], d.s[0])
binary.LittleEndian.PutUint32(dig[4:], d.s[1])
binary.LittleEndian.PutUint32(dig[8:], d.s[2])
binary.LittleEndian.PutUint32(dig[12:], d.s[3])
s.digests[lane.uid] = dig
if lane.msg != nil {
s.buffers <- lane.msg
}
lanes[0] = blockInput{}
default:
s.wg.Add(len(lanes))
var results [useScalarBelow]digest
for i := range lanes {
lane := lanes[i]
go func(i int) {
var d digest
defer s.wg.Done()
a, ok := s.digests[lane.uid]
if ok {
d.s[0] = binary.LittleEndian.Uint32(a[0:4])
d.s[1] = binary.LittleEndian.Uint32(a[4:8])
d.s[2] = binary.LittleEndian.Uint32(a[8:12])
d.s[3] = binary.LittleEndian.Uint32(a[12:16])
} else {
d.s[0] = init0
d.s[1] = init1
d.s[2] = init2
d.s[3] = init3
}
if len(lane.msg) == 0 {
results[i] = d
return
}
// Update...
blockScalar(&d.s, lane.msg)
results[i] = d
}(i)
}
s.wg.Wait()
for i, lane := range lanes {
dig := [Size]byte{}
binary.LittleEndian.PutUint32(dig[0:], results[i].s[0])
binary.LittleEndian.PutUint32(dig[4:], results[i].s[1])
binary.LittleEndian.PutUint32(dig[8:], results[i].s[2])
binary.LittleEndian.PutUint32(dig[12:], results[i].s[3])
s.digests[lane.uid] = dig
if lane.msg != nil {
s.buffers <- lane.msg
}
lanes[i] = blockInput{}
}
}
return
}
inputs := [16][]byte{}
for i := range lanes {
inputs[i] = lanes[i].msg
}
// Collect active digests...
state := s.getDigests(lanes)
// Process all lanes...
s.blockMd5_x16(&state, inputs, len(lanes) <= 8)
for i, lane := range lanes {
uid := lane.uid
dig := [Size]byte{}
binary.LittleEndian.PutUint32(dig[0:], state.v0[i])
binary.LittleEndian.PutUint32(dig[4:], state.v1[i])
binary.LittleEndian.PutUint32(dig[8:], state.v2[i])
binary.LittleEndian.PutUint32(dig[12:], state.v3[i])
s.digests[uid] = dig
if lane.msg != nil {
s.buffers <- lane.msg
}
lanes[i] = blockInput{}
}
}
func (s *md5Server) getDigests(lanes []blockInput) (d digest16) {
for i, lane := range lanes {
a, ok := s.digests[lane.uid]
if ok {
d.v0[i] = binary.LittleEndian.Uint32(a[0:4])
d.v1[i] = binary.LittleEndian.Uint32(a[4:8])
d.v2[i] = binary.LittleEndian.Uint32(a[8:12])
d.v3[i] = binary.LittleEndian.Uint32(a[12:16])
} else {
d.v0[i] = init0
d.v1[i] = init1
d.v2[i] = init2
d.v3[i] = init3
}
}
return
}
Server/vendor/github.com/minio/md5-simd/md5-server_fallback.go
Generated Vendored
+12
View File
@@ -0,0 +1,12 @@
//+build !amd64 appengine !gc noasm
// Copyright (c) 2020 MinIO Inc. All rights reserved.
// Use of this source code is governed by a license that can be
// found in the LICENSE file.
package md5simd
// NewServer - Create new object for parallel processing handling
func NewServer() *fallbackServer {
return &fallbackServer{}
}
Server/vendor/github.com/minio/md5-simd/md5-util_amd64.go
Generated Vendored
+85
View File
@@ -0,0 +1,85 @@
//+build !noasm,!appengine,gc
// Copyright (c) 2020 MinIO Inc. All rights reserved.
// Use of this source code is governed by a license that can be
// found in the LICENSE file.
package md5simd
// Helper struct for sorting blocks based on length
type lane struct {
len uint
pos uint
}
type digest struct {
s [4]uint32
}
// Helper struct for generating number of rounds in combination with mask for valid lanes
type maskRounds struct {
mask uint64
rounds uint64
}
func generateMaskAndRounds8(input [8][]byte, mr *[8]maskRounds) (rounds int) {
// Sort on blocks length small to large
var sorted [8]lane
for c, inpt := range input[:] {
sorted[c] = lane{uint(len(inpt)), uint(c)}
for i := c - 1; i >= 0; i-- {
// swap so largest is at the end...
if sorted[i].len > sorted[i+1].len {
sorted[i], sorted[i+1] = sorted[i+1], sorted[i]
continue
}
break
}
}
// Create mask array including 'rounds' (of processing blocks of 64 bytes) between masks
m, round := uint64(0xff), uint64(0)
for _, s := range sorted[:] {
if s.len > 0 {
if uint64(s.len)>>6 > round {
mr[rounds] = maskRounds{m, (uint64(s.len) >> 6) - round}
rounds++
}
round = uint64(s.len) >> 6
}
m = m & ^(1 << uint(s.pos))
}
return
}
func generateMaskAndRounds16(input [16][]byte, mr *[16]maskRounds) (rounds int) {
// Sort on blocks length small to large
var sorted [16]lane
for c, inpt := range input[:] {
sorted[c] = lane{uint(len(inpt)), uint(c)}
for i := c - 1; i >= 0; i-- {
// swap so largest is at the end...
if sorted[i].len > sorted[i+1].len {
sorted[i], sorted[i+1] = sorted[i+1], sorted[i]
continue
}
break
}
}
// Create mask array including 'rounds' (of processing blocks of 64 bytes) between masks
m, round := uint64(0xffff), uint64(0)
for _, s := range sorted[:] {
if s.len > 0 {
if uint64(s.len)>>6 > round {
mr[rounds] = maskRounds{m, (uint64(s.len) >> 6) - round}
rounds++
}
round = uint64(s.len) >> 6
}
m = m & ^(1 << uint(s.pos))
}
return
}
Server/vendor/github.com/minio/md5-simd/md5.go
Generated Vendored
+63
View File
@@ -0,0 +1,63 @@
package md5simd
import (
"crypto/md5"
"hash"
"sync"
)
const (
// The blocksize of MD5 in bytes.
BlockSize = 64
// The size of an MD5 checksum in bytes.
Size = 16
// internalBlockSize is the internal block size.
internalBlockSize = 32 << 10
)
type Server interface {
NewHash() Hasher
Close()
}
type Hasher interface {
hash.Hash
Close()
}
// StdlibHasher returns a Hasher that uses the stdlib for hashing.
// Used hashers are stored in a pool for fast reuse.
func StdlibHasher() Hasher {
return &md5Wrapper{Hash: md5Pool.New().(hash.Hash)}
}
// md5Wrapper is a wrapper around the builtin hasher.
type md5Wrapper struct {
hash.Hash
}
var md5Pool = sync.Pool{New: func() interface{} {
return md5.New()
}}
// fallbackServer - Fallback when no assembly is available.
type fallbackServer struct {
}
// NewHash -- return regular Golang md5 hashing from crypto
func (s *fallbackServer) NewHash() Hasher {
return &md5Wrapper{Hash: md5Pool.New().(hash.Hash)}
}
func (s *fallbackServer) Close() {
}
func (m *md5Wrapper) Close() {
if m.Hash != nil {
m.Reset()
md5Pool.Put(m.Hash)
m.Hash = nil
}
}
Server/vendor/github.com/minio/md5-simd/md5block_amd64.go
Generated Vendored
+11
View File
@@ -0,0 +1,11 @@
// Code generated by command: go run gen.go -out ../md5block_amd64.s -stubs ../md5block_amd64.go -pkg=md5simd. DO NOT EDIT.
// +build !appengine
// +build !noasm
// +build gc
package md5simd
// Encode p to digest
//go:noescape
func blockScalar(dig *[4]uint32, p []byte)
Server/vendor/github.com/minio/md5-simd/md5block_amd64.s
Generated Vendored
+714
View File
@@ -0,0 +1,714 @@
// Code generated by command: go run gen.go -out ../md5block_amd64.s -stubs ../md5block_amd64.go -pkg=md5simd. DO NOT EDIT.
// +build !appengine
// +build !noasm
// +build gc
// func blockScalar(dig *[4]uint32, p []byte)
TEXT ·blockScalar(SB), $0-32
MOVQ p_len+16(FP), AX
MOVQ dig+0(FP), CX
MOVQ p_base+8(FP), DX
SHRQ $0x06, AX
SHLQ $0x06, AX
LEAQ (DX)(AX*1), AX
CMPQ DX, AX
JEQ end
MOVL (CX), BX
MOVL 4(CX), BP
MOVL 8(CX), SI
MOVL 12(CX), CX
MOVL $0xffffffff, DI
loop:
MOVL (DX), R8
MOVL CX, R9
MOVL BX, R10
MOVL BP, R11
MOVL SI, R12
MOVL CX, R13
// ROUND1
XORL SI, R9
ADDL $0xd76aa478, BX
ADDL R8, BX
ANDL BP, R9
XORL CX, R9
MOVL 4(DX), R8
ADDL R9, BX
ROLL $0x07, BX
MOVL SI, R9
ADDL BP, BX
XORL BP, R9
ADDL $0xe8c7b756, CX
ADDL R8, CX
ANDL BX, R9
XORL SI, R9
MOVL 8(DX), R8
ADDL R9, CX
ROLL $0x0c, CX
MOVL BP, R9
ADDL BX, CX
XORL BX, R9
ADDL $0x242070db, SI
ADDL R8, SI
ANDL CX, R9
XORL BP, R9
MOVL 12(DX), R8
ADDL R9, SI
ROLL $0x11, SI
MOVL BX, R9
ADDL CX, SI
XORL CX, R9
ADDL $0xc1bdceee, BP
ADDL R8, BP
ANDL SI, R9
XORL BX, R9
MOVL 16(DX), R8
ADDL R9, BP
ROLL $0x16, BP
MOVL CX, R9
ADDL SI, BP
XORL SI, R9
ADDL $0xf57c0faf, BX
ADDL R8, BX
ANDL BP, R9
XORL CX, R9
MOVL 20(DX), R8
ADDL R9, BX
ROLL $0x07, BX
MOVL SI, R9
ADDL BP, BX
XORL BP, R9
ADDL $0x4787c62a, CX
ADDL R8, CX
ANDL BX, R9
XORL SI, R9
MOVL 24(DX), R8
ADDL R9, CX
ROLL $0x0c, CX
MOVL BP, R9
ADDL BX, CX
XORL BX, R9
ADDL $0xa8304613, SI
ADDL R8, SI
ANDL CX, R9
XORL BP, R9
MOVL 28(DX), R8
ADDL R9, SI
ROLL $0x11, SI
MOVL BX, R9
ADDL CX, SI
XORL CX, R9
ADDL $0xfd469501, BP
ADDL R8, BP
ANDL SI, R9
XORL BX, R9
MOVL 32(DX), R8
ADDL R9, BP
ROLL $0x16, BP
MOVL CX, R9
ADDL SI, BP
XORL SI, R9
ADDL $0x698098d8, BX
ADDL R8, BX
ANDL BP, R9
XORL CX, R9
MOVL 36(DX), R8
ADDL R9, BX
ROLL $0x07, BX
MOVL SI, R9
ADDL BP, BX
XORL BP, R9
ADDL $0x8b44f7af, CX
ADDL R8, CX
ANDL BX, R9
XORL SI, R9
MOVL 40(DX), R8
ADDL R9, CX
ROLL $0x0c, CX
MOVL BP, R9
ADDL BX, CX
XORL BX, R9
ADDL $0xffff5bb1, SI
ADDL R8, SI
ANDL CX, R9
XORL BP, R9
MOVL 44(DX), R8
ADDL R9, SI
ROLL $0x11, SI
MOVL BX, R9
ADDL CX, SI
XORL CX, R9
ADDL $0x895cd7be, BP
ADDL R8, BP
ANDL SI, R9
XORL BX, R9
MOVL 48(DX), R8
ADDL R9, BP
ROLL $0x16, BP
MOVL CX, R9
ADDL SI, BP
XORL SI, R9
ADDL $0x6b901122, BX
ADDL R8, BX
ANDL BP, R9
XORL CX, R9
MOVL 52(DX), R8
ADDL R9, BX
ROLL $0x07, BX
MOVL SI, R9
ADDL BP, BX
XORL BP, R9
ADDL $0xfd987193, CX
ADDL R8, CX
ANDL BX, R9
XORL SI, R9
MOVL 56(DX), R8
ADDL R9, CX
ROLL $0x0c, CX
MOVL BP, R9
ADDL BX, CX
XORL BX, R9
ADDL $0xa679438e, SI
ADDL R8, SI
ANDL CX, R9
XORL BP, R9
MOVL 60(DX), R8
ADDL R9, SI
ROLL $0x11, SI
MOVL BX, R9
ADDL CX, SI
XORL CX, R9
ADDL $0x49b40821, BP
ADDL R8, BP
ANDL SI, R9
XORL BX, R9
MOVL 4(DX), R8
ADDL R9, BP
ROLL $0x16, BP
MOVL CX, R9
ADDL SI, BP
// ROUND2
MOVL CX, R9
MOVL CX, R14
XORL DI, R9
ADDL $0xf61e2562, BX
ADDL R8, BX
ANDL BP, R14
ANDL SI, R9
MOVL 24(DX), R8
ORL R9, R14
MOVL SI, R9
ADDL R14, BX
MOVL SI, R14
ROLL $0x05, BX
ADDL BP, BX
XORL DI, R9
ADDL $0xc040b340, CX
ADDL R8, CX
ANDL BX, R14
ANDL BP, R9
MOVL 44(DX), R8
ORL R9, R14
MOVL BP, R9
ADDL R14, CX
MOVL BP, R14
ROLL $0x09, CX
ADDL BX, CX
XORL DI, R9
ADDL $0x265e5a51, SI
ADDL R8, SI
ANDL CX, R14
ANDL BX, R9
MOVL (DX), R8
ORL R9, R14
MOVL BX, R9
ADDL R14, SI
MOVL BX, R14
ROLL $0x0e, SI
ADDL CX, SI
XORL DI, R9
ADDL $0xe9b6c7aa, BP
ADDL R8, BP
ANDL SI, R14
ANDL CX, R9
MOVL 20(DX), R8
ORL R9, R14
MOVL CX, R9
ADDL R14, BP
MOVL CX, R14
ROLL $0x14, BP
ADDL SI, BP
XORL DI, R9
ADDL $0xd62f105d, BX
ADDL R8, BX
ANDL BP, R14
ANDL SI, R9
MOVL 40(DX), R8
ORL R9, R14
MOVL SI, R9
ADDL R14, BX
MOVL SI, R14
ROLL $0x05, BX
ADDL BP, BX
XORL DI, R9
ADDL $0x02441453, CX
ADDL R8, CX
ANDL BX, R14
ANDL BP, R9
MOVL 60(DX), R8
ORL R9, R14
MOVL BP, R9
ADDL R14, CX
MOVL BP, R14
ROLL $0x09, CX
ADDL BX, CX
XORL DI, R9
ADDL $0xd8a1e681, SI
ADDL R8, SI
ANDL CX, R14
ANDL BX, R9
MOVL 16(DX), R8
ORL R9, R14
MOVL BX, R9
ADDL R14, SI
MOVL BX, R14
ROLL $0x0e, SI
ADDL CX, SI
XORL DI, R9
ADDL $0xe7d3fbc8, BP
ADDL R8, BP
ANDL SI, R14
ANDL CX, R9
MOVL 36(DX), R8
ORL R9, R14
MOVL CX, R9
ADDL R14, BP
MOVL CX, R14
ROLL $0x14, BP
ADDL SI, BP
XORL DI, R9
ADDL $0x21e1cde6, BX
ADDL R8, BX
ANDL BP, R14
ANDL SI, R9
MOVL 56(DX), R8
ORL R9, R14
MOVL SI, R9
ADDL R14, BX
MOVL SI, R14
ROLL $0x05, BX
ADDL BP, BX
XORL DI, R9
ADDL $0xc33707d6, CX
ADDL R8, CX
ANDL BX, R14
ANDL BP, R9
MOVL 12(DX), R8
ORL R9, R14
MOVL BP, R9
ADDL R14, CX
MOVL BP, R14
ROLL $0x09, CX
ADDL BX, CX
XORL DI, R9
ADDL $0xf4d50d87, SI
ADDL R8, SI
ANDL CX, R14
ANDL BX, R9
MOVL 32(DX), R8
ORL R9, R14
MOVL BX, R9
ADDL R14, SI
MOVL BX, R14
ROLL $0x0e, SI
ADDL CX, SI
XORL DI, R9
ADDL $0x455a14ed, BP
ADDL R8, BP
ANDL SI, R14
ANDL CX, R9
MOVL 52(DX), R8
ORL R9, R14
MOVL CX, R9
ADDL R14, BP
MOVL CX, R14
ROLL $0x14, BP
ADDL SI, BP
XORL DI, R9
ADDL $0xa9e3e905, BX
ADDL R8, BX
ANDL BP, R14
ANDL SI, R9
MOVL 8(DX), R8
ORL R9, R14
MOVL SI, R9
ADDL R14, BX
MOVL SI, R14
ROLL $0x05, BX
ADDL BP, BX
XORL DI, R9
ADDL $0xfcefa3f8, CX
ADDL R8, CX
ANDL BX, R14
ANDL BP, R9
MOVL 28(DX), R8
ORL R9, R14
MOVL BP, R9
ADDL R14, CX
MOVL BP, R14
ROLL $0x09, CX
ADDL BX, CX
XORL DI, R9
ADDL $0x676f02d9, SI
ADDL R8, SI
ANDL CX, R14
ANDL BX, R9
MOVL 48(DX), R8
ORL R9, R14
MOVL BX, R9
ADDL R14, SI
MOVL BX, R14
ROLL $0x0e, SI
ADDL CX, SI
XORL DI, R9
ADDL $0x8d2a4c8a, BP
ADDL R8, BP
ANDL SI, R14
ANDL CX, R9
MOVL 20(DX), R8
ORL R9, R14
MOVL CX, R9
ADDL R14, BP
MOVL CX, R14
ROLL $0x14, BP
ADDL SI, BP
// ROUND3
MOVL SI, R9
ADDL $0xfffa3942, BX
ADDL R8, BX
MOVL 32(DX), R8
XORL CX, R9
XORL BP, R9
ADDL R9, BX
ROLL $0x04, BX
MOVL BP, R9
ADDL BP, BX
ADDL $0x8771f681, CX
ADDL R8, CX
MOVL 44(DX), R8
XORL SI, R9
XORL BX, R9
ADDL R9, CX
ROLL $0x0b, CX
MOVL BX, R9
ADDL BX, CX
ADDL $0x6d9d6122, SI
ADDL R8, SI
MOVL 56(DX), R8
XORL BP, R9
XORL CX, R9
ADDL R9, SI
ROLL $0x10, SI
MOVL CX, R9
ADDL CX, SI
ADDL $0xfde5380c, BP
ADDL R8, BP
MOVL 4(DX), R8
XORL BX, R9
XORL SI, R9
ADDL R9, BP
ROLL $0x17, BP
MOVL SI, R9
ADDL SI, BP
ADDL $0xa4beea44, BX
ADDL R8, BX
MOVL 16(DX), R8
XORL CX, R9
XORL BP, R9
ADDL R9, BX
ROLL $0x04, BX
MOVL BP, R9
ADDL BP, BX
ADDL $0x4bdecfa9, CX
ADDL R8, CX
MOVL 28(DX), R8
XORL SI, R9
XORL BX, R9
ADDL R9, CX
ROLL $0x0b, CX
MOVL BX, R9
ADDL BX, CX
ADDL $0xf6bb4b60, SI
ADDL R8, SI
MOVL 40(DX), R8
XORL BP, R9
XORL CX, R9
ADDL R9, SI
ROLL $0x10, SI
MOVL CX, R9
ADDL CX, SI
ADDL $0xbebfbc70, BP
ADDL R8, BP
MOVL 52(DX), R8
XORL BX, R9
XORL SI, R9
ADDL R9, BP
ROLL $0x17, BP
MOVL SI, R9
ADDL SI, BP
ADDL $0x289b7ec6, BX
ADDL R8, BX
MOVL (DX), R8
XORL CX, R9
XORL BP, R9
ADDL R9, BX
ROLL $0x04, BX
MOVL BP, R9
ADDL BP, BX
ADDL $0xeaa127fa, CX
ADDL R8, CX
MOVL 12(DX), R8
XORL SI, R9
XORL BX, R9
ADDL R9, CX
ROLL $0x0b, CX
MOVL BX, R9
ADDL BX, CX
ADDL $0xd4ef3085, SI
ADDL R8, SI
MOVL 24(DX), R8
XORL BP, R9
XORL CX, R9
ADDL R9, SI
ROLL $0x10, SI
MOVL CX, R9
ADDL CX, SI
ADDL $0x04881d05, BP
ADDL R8, BP
MOVL 36(DX), R8
XORL BX, R9
XORL SI, R9
ADDL R9, BP
ROLL $0x17, BP
MOVL SI, R9
ADDL SI, BP
ADDL $0xd9d4d039, BX
ADDL R8, BX
MOVL 48(DX), R8
XORL CX, R9
XORL BP, R9
ADDL R9, BX
ROLL $0x04, BX
MOVL BP, R9
ADDL BP, BX
ADDL $0xe6db99e5, CX
ADDL R8, CX
MOVL 60(DX), R8
XORL SI, R9
XORL BX, R9
ADDL R9, CX
ROLL $0x0b, CX
MOVL BX, R9
ADDL BX, CX
ADDL $0x1fa27cf8, SI
ADDL R8, SI
MOVL 8(DX), R8
XORL BP, R9
XORL CX, R9
ADDL R9, SI
ROLL $0x10, SI
MOVL CX, R9
ADDL CX, SI
ADDL $0xc4ac5665, BP
ADDL R8, BP
MOVL (DX), R8
XORL BX, R9
XORL SI, R9
ADDL R9, BP
ROLL $0x17, BP
MOVL SI, R9
ADDL SI, BP
// ROUND4
MOVL DI, R9
XORL CX, R9
ADDL $0xf4292244, BX
ADDL R8, BX
ORL BP, R9
XORL SI, R9
ADDL R9, BX
MOVL 28(DX), R8
MOVL DI, R9
ROLL $0x06, BX
XORL SI, R9
ADDL BP, BX
ADDL $0x432aff97, CX
ADDL R8, CX
ORL BX, R9
XORL BP, R9
ADDL R9, CX
MOVL 56(DX), R8
MOVL DI, R9
ROLL $0x0a, CX
XORL BP, R9
ADDL BX, CX
ADDL $0xab9423a7, SI
ADDL R8, SI
ORL CX, R9
XORL BX, R9
ADDL R9, SI
MOVL 20(DX), R8
MOVL DI, R9
ROLL $0x0f, SI
XORL BX, R9
ADDL CX, SI
ADDL $0xfc93a039, BP
ADDL R8, BP
ORL SI, R9
XORL CX, R9
ADDL R9, BP
MOVL 48(DX), R8
MOVL DI, R9
ROLL $0x15, BP
XORL CX, R9
ADDL SI, BP
ADDL $0x655b59c3, BX
ADDL R8, BX
ORL BP, R9
XORL SI, R9
ADDL R9, BX
MOVL 12(DX), R8
MOVL DI, R9
ROLL $0x06, BX
XORL SI, R9
ADDL BP, BX
ADDL $0x8f0ccc92, CX
ADDL R8, CX
ORL BX, R9
XORL BP, R9
ADDL R9, CX
MOVL 40(DX), R8
MOVL DI, R9
ROLL $0x0a, CX
XORL BP, R9
ADDL BX, CX
ADDL $0xffeff47d, SI
ADDL R8, SI
ORL CX, R9
XORL BX, R9
ADDL R9, SI
MOVL 4(DX), R8
MOVL DI, R9
ROLL $0x0f, SI
XORL BX, R9
ADDL CX, SI
ADDL $0x85845dd1, BP
ADDL R8, BP
ORL SI, R9
XORL CX, R9
ADDL R9, BP
MOVL 32(DX), R8
MOVL DI, R9
ROLL $0x15, BP
XORL CX, R9
ADDL SI, BP
ADDL $0x6fa87e4f, BX
ADDL R8, BX
ORL BP, R9
XORL SI, R9
ADDL R9, BX
MOVL 60(DX), R8
MOVL DI, R9
ROLL $0x06, BX
XORL SI, R9
ADDL BP, BX
ADDL $0xfe2ce6e0, CX
ADDL R8, CX
ORL BX, R9
XORL BP, R9
ADDL R9, CX
MOVL 24(DX), R8
MOVL DI, R9
ROLL $0x0a, CX
XORL BP, R9
ADDL BX, CX
ADDL $0xa3014314, SI
ADDL R8, SI
ORL CX, R9
XORL BX, R9
ADDL R9, SI
MOVL 52(DX), R8
MOVL DI, R9
ROLL $0x0f, SI
XORL BX, R9
ADDL CX, SI
ADDL $0x4e0811a1, BP
ADDL R8, BP
ORL SI, R9
XORL CX, R9
ADDL R9, BP
MOVL 16(DX), R8
MOVL DI, R9
ROLL $0x15, BP
XORL CX, R9
ADDL SI, BP
ADDL $0xf7537e82, BX
ADDL R8, BX
ORL BP, R9
XORL SI, R9
ADDL R9, BX
MOVL 44(DX), R8
MOVL DI, R9
ROLL $0x06, BX
XORL SI, R9
ADDL BP, BX
ADDL $0xbd3af235, CX
ADDL R8, CX
ORL BX, R9
XORL BP, R9
ADDL R9, CX
MOVL 8(DX), R8
MOVL DI, R9
ROLL $0x0a, CX
XORL BP, R9
ADDL BX, CX
ADDL $0x2ad7d2bb, SI
ADDL R8, SI
ORL CX, R9
XORL BX, R9
ADDL R9, SI
MOVL 36(DX), R8
MOVL DI, R9
ROLL $0x0f, SI
XORL BX, R9
ADDL CX, SI
ADDL $0xeb86d391, BP
ADDL R8, BP
ORL SI, R9
XORL CX, R9
ADDL R9, BP
ROLL $0x15, BP
ADDL SI, BP
ADDL R10, BX
ADDL R11, BP
ADDL R12, SI
ADDL R13, CX
// Prepare next loop
ADDQ $0x40, DX
CMPQ DX, AX
JB loop
// Write output
MOVQ dig+0(FP), AX
MOVL BX, (AX)
MOVL BP, 4(AX)
MOVL SI, 8(AX)
MOVL CX, 12(AX)
end:
RET
Server/vendor/github.com/minio/minio-go/v7/.gitignore
Generated
+7
View File
@@ -0,0 +1,7 @@
*~
*.test
validator
golangci-lint
functional_tests
.idea
vendor/
Server/vendor/github.com/minio/minio-go/v7/.golangci.yml
Generated Vendored
+72
View File
@@ -0,0 +1,72 @@
version: "2"
linters:
disable-all: true
enable:
- durationcheck
- gocritic
- gomodguard
- govet
- ineffassign
- misspell
- revive
- staticcheck
- unconvert
- unused
- usetesting
- whitespace
settings:
misspell:
locale: US
staticcheck:
checks:
- all
- -SA1008
- -SA1019
- -SA4000
- -SA9004
- -ST1000
- -ST1005
- -ST1016
- -ST1021
- -ST1020
- -U1000
exclusions:
generated: lax
rules:
- path: (.+)\.go$
text: "empty-block:"
- path: (.+)\.go$
text: "unused-parameter:"
- path: (.+)\.go$
text: "dot-imports:"
- path: (.+)\.go$
text: "singleCaseSwitch: should rewrite switch statement to if statement"
- path: (.+)\.go$
text: "unlambda: replace"
- path: (.+)\.go$
text: "captLocal:"
- path: (.+)\.go$
text: "should have a package comment"
- path: (.+)\.go$
text: "ifElseChain:"
- path: (.+)\.go$
text: "elseif:"
- path: (.+)\.go$
text: "Error return value of"
- path: (.+)\.go$
text: "unnecessary conversion"
- path: (.+)\.go$
text: "Error return value is not checked"
issues:
max-issues-per-linter: 100
max-same-issues: 100
formatters:
enable:
- gofumpt
- goimports
exclusions:
generated: lax
paths:
- third_party$
- builtin$
- examples$
Server/vendor/github.com/minio/minio-go/v7/CLAUDE.md
Generated Vendored
+124
View File
@@ -0,0 +1,124 @@
CLAUDE.md
=========
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
Commands
--------
### Testing
```bash
# Run all tests with race detection (requires MinIO server at localhost:9000)
SERVER_ENDPOINT=localhost:9000 ACCESS_KEY=minioadmin SECRET_KEY=minioadmin ENABLE_HTTPS=1 MINT_MODE=full go test -race -v ./...
# Run tests without race detection
go test ./...
# Run short tests only (no functional tests)
go test -short -race ./...
# Run functional tests
go build -race functional_tests.go
SERVER_ENDPOINT=localhost:9000 ACCESS_KEY=minioadmin SECRET_KEY=minioadmin ENABLE_HTTPS=1 MINT_MODE=full ./functional_tests
# Run functional tests without TLS
SERVER_ENDPOINT=localhost:9000 ACCESS_KEY=minioadmin SECRET_KEY=minioadmin ENABLE_HTTPS=0 MINT_MODE=full ./functional_tests
```
### Linting and Code Quality
```bash
# Run all checks (lint, test, examples, functional tests)
make checks
# Run linter only (includes govet, staticcheck, and other linters)
make lint
# Run golangci-lint directly
golangci-lint run --timeout=5m --config ./.golangci.yml
# Note: 'make vet' is now an alias for 'make lint' for backwards compatibility
```
### Building Examples
```bash
# Build all examples
make examples
# Build a specific example
cd examples/s3 && go build -mod=mod putobject.go
```
Architecture
------------
### Core Client Structure
The MinIO Go SDK is organized around a central `Client` struct (api.go:52) that implements Amazon S3 compatible methods. Key architectural patterns:
1. **Modular API Organization**: API methods are split into logical files:
- `api-bucket-*.go`: Bucket operations (lifecycle, encryption, versioning, etc.)
- `api-object-*.go`: Object operations (legal hold, retention, tagging, etc.)
- `api-get-*.go`, `api-put-*.go`: GET and PUT operations
- `api-list.go`: Listing operations
- `api-stat.go`: Status/info operations
2. **Credential Management**: The `pkg/credentials/` package provides various credential providers:
- Static credentials
- Environment variables (AWS/MinIO)
- IAM roles
- STS (Security Token Service) variants
- File-based credentials
- Chain provider for fallback mechanisms
3. **Request Signing**: The `pkg/signer/` package handles AWS signature versions:
- V2 signatures (legacy)
- V4 signatures (standard)
- Streaming signatures for large uploads
4. **Transport Layer**: Custom HTTP transport with:
- Retry logic with configurable max retries
- Health status monitoring
- Tracing support via httptrace
- Bucket location caching (`bucketLocCache`\)
- Session caching for credentials
5. **Helper Packages**:
- `pkg/encrypt/`: Server-side encryption utilities
- `pkg/notification/`: Event notification handling
- `pkg/policy/`: Bucket policy management
- `pkg/lifecycle/`: Object lifecycle rules
- `pkg/tags/`: Object and bucket tagging
- `pkg/s3utils/`: S3 utility functions
- `pkg/kvcache/`: Key-value caching
- `pkg/singleflight/`: Deduplication of concurrent requests
### Testing Strategy
- Unit tests alongside implementation files (`*_test.go`\)
- Comprehensive functional tests in `functional_tests.go` requiring a live MinIO server
- Example programs in `examples/` directory demonstrating API usage
- Build tag `//go:build mint` for integration tests
### Error Handling
- Custom error types in `api-error-response.go`
- HTTP status code mapping
- Retry logic for transient failures
- Detailed error context preservation
Important Patterns
------------------
1. **Context Usage**: All API methods accept `context.Context` for cancellation and timeout control
2. **Options Pattern**: Methods use Options structs for optional parameters (e.g., `PutObjectOptions`, `GetObjectOptions`\)
3. **Streaming Support**: Large file operations use io.Reader/Writer interfaces for memory efficiency
4. **Bucket Lookup Types**: Supports both path-style and virtual-host-style S3 URLs
5. **MD5/SHA256 Hashing**: Configurable hash functions for integrity checks via `md5Hasher` and `sha256Hasher`
Server/vendor/github.com/minio/minio-go/v7/CNAME
Generated Vendored
+1
View File
@@ -0,0 +1 @@
minio-go.min.io
Server/vendor/github.com/minio/minio-go/v7/CONTRIBUTING.md
Generated Vendored
+23
View File
@@ -0,0 +1,23 @@
### Developer Guidelines
`minio-go` welcomes your contribution. To make the process as seamless as possible, we ask for the following:
- Go ahead and fork the project and make your changes. We encourage pull requests to discuss code changes.
- Fork it
- Create your feature branch (git checkout -b my-new-feature)
- Commit your changes (git commit -am 'Add some feature')
- Push to the branch (git push origin my-new-feature)
- Create new Pull Request
- When you're ready to create a pull request, be sure to:
- Have test cases for the new code. If you have questions about how to do it, please ask in your pull request.
- Run `go fmt`
- Squash your commits into a single commit. `git rebase -i`. It's okay to force update your pull request.
- Make sure `go test -race ./...` and `go build` completes. NOTE: go test runs functional tests and requires you to have a AWS S3 account. Set them as environment variables`ACCESS_KEY` and `SECRET_KEY`. To run shorter version of the tests please use `go test -short -race ./...`
- Read [Effective Go](https://github.com/golang/go/wiki/CodeReviewComments) article from Golang project
- `minio-go` project is strictly conformant with Golang style
- if you happen to observe offending code, please feel free to send a pull request
Server/vendor/github.com/minio/minio-go/v7/CREDITS
Generated Vendored
+1101
View File
File diff suppressed because it is too large Load Diff
Server/vendor/github.com/minio/minio-go/v7/LICENSE
Generated Vendored
+202
View File
@@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Server/vendor/github.com/minio/minio-go/v7/MAINTAINERS.md
Generated Vendored
+43
View File
@@ -0,0 +1,43 @@
For maintainers only
====================
Responsibilities
----------------
Please go through this link [Maintainer Responsibility](https://gist.github.com/abperiasamy/f4d9b31d3186bbd26522)
### Making new releases
Tag and sign your release commit, additionally this step requires you to have access to MinIO's trusted private key.
```sh
$ export GNUPGHOME=/media/${USER}/minio/trusted
$ git tag -s 4.0.0
$ git push
$ git push --tags
```
### Update version
Once release has been made update `libraryVersion` constant in `api.go` to next to be released version.
```sh
$ grep libraryVersion api.go
libraryVersion = "4.0.1"
```
Commit your changes
```
$ git commit -a -m "Update version for next release" --author "MinIO Trusted <trusted@min.io>"
```
### Announce
Announce new release by adding release notes at https://github.com/minio/minio-go/releases from `trusted@min.io` account. Release notes requires two sections `highlights` and `changelog`. Highlights is a bulleted list of salient features in this release and Changelog contains list of all commits since the last release.
To generate `changelog`
```sh
$ git log --no-color --pretty=format:'-%d %s (%cr) <%an>' <last_release_tag>..<latest_release_tag>
```
Server/vendor/github.com/minio/minio-go/v7/Makefile
Generated Vendored
+39
View File
@@ -0,0 +1,39 @@
GOPATH := $(shell go env GOPATH)
TMPDIR := $(shell mktemp -d)
all: checks
.PHONY: examples docs
checks: lint test examples functional-test
lint:
@mkdir -p ${GOPATH}/bin
@echo "Installing golangci-lint" && curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(GOPATH)/bin
@echo "Running $@ check"
@GO111MODULE=on ${GOPATH}/bin/golangci-lint cache clean
@GO111MODULE=on ${GOPATH}/bin/golangci-lint run --timeout=5m --config ./.golangci.yml
vet: lint
test:
@GO111MODULE=on SERVER_ENDPOINT=localhost:9000 ACCESS_KEY=minioadmin SECRET_KEY=minioadmin ENABLE_HTTPS=1 MINT_MODE=full go test -race -v ./...
examples:
@echo "Building s3 examples"
@cd ./examples/s3 && $(foreach v,$(wildcard examples/s3/*.go),go build -mod=mod -o ${TMPDIR}/$(basename $(v)) $(notdir $(v)) || exit 1;)
@echo "Building minio examples"
@cd ./examples/minio && $(foreach v,$(wildcard examples/minio/*.go),go build -mod=mod -o ${TMPDIR}/$(basename $(v)) $(notdir $(v)) || exit 1;)
functional-test:
@GO111MODULE=on go build -race functional_tests.go
@SERVER_ENDPOINT=localhost:9000 ACCESS_KEY=minioadmin SECRET_KEY=minioadmin ENABLE_HTTPS=1 MINT_MODE=full ./functional_tests
functional-test-notls:
@GO111MODULE=on go build -race functional_tests.go
@SERVER_ENDPOINT=localhost:9000 ACCESS_KEY=minioadmin SECRET_KEY=minioadmin ENABLE_HTTPS=0 MINT_MODE=full ./functional_tests
clean:
@echo "Cleaning up all the generated files"
@find . -name '*.test' | xargs rm -fv
@find . -name '*~' | xargs rm -fv
Server/vendor/github.com/minio/minio-go/v7/NOTICE
Generated Vendored
+9
View File
@@ -0,0 +1,9 @@
MinIO Cloud Storage, (C) 2014-2020 MinIO, Inc.
This product includes software developed at MinIO, Inc.
(https://min.io/).
The MinIO project contains unmodified/modified subcomponents too with
separate copyright notices and license terms. Your use of the source
code for these subcomponents is subject to the terms and conditions
of Apache License Version 2.0
Server/vendor/github.com/minio/minio-go/v7/README.md
Generated Vendored
+318
View File
@@ -0,0 +1,318 @@
MinIO Go Client SDK for Amazon S3 Compatible Cloud Storage [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Sourcegraph](https://sourcegraph.com/github.com/minio/minio-go/-/badge.svg)](https://sourcegraph.com/github.com/minio/minio-go?badge) [![Apache V2 License](https://img.shields.io/badge/license-Apache%20V2-blue.svg)](https://github.com/minio/minio-go/blob/master/LICENSE)
==================================================================================================================================================================================================================================================================================================================================================================================================================
The MinIO Go Client SDK provides straightforward APIs to access any Amazon S3 compatible object storage.
This Quickstart Guide covers how to install the MinIO client SDK, connect to MinIO, and create a sample file uploader. For a complete list of APIs and examples, see the [godoc documentation](https://pkg.go.dev/github.com/minio/minio-go/v7) or [Go Client API Reference](https://min.io/docs/minio/linux/developers/go/API.html).
These examples presume a working [Go development environment](https://golang.org/doc/install) and the [MinIO `mc` command line tool](https://min.io/docs/minio/linux/reference/minio-mc.html).
Download from Github
--------------------
From your project directory:
```sh
go get github.com/minio/minio-go/v7
```
Initialize a MinIO Client Object
--------------------------------
The MinIO client requires the following parameters to connect to an Amazon S3 compatible object storage:
| Parameter | Description |
|-------------------|------------------------------------------------------------|
| `endpoint` | URL to object storage service. |
| `_minio.Options_` | All the options such as credentials, custom transport etc. |
```go
package main
import (
"log"
"github.com/minio/minio-go/v7"
"github.com/minio/minio-go/v7/pkg/credentials"
)
func main() {
endpoint := "play.min.io"
accessKeyID := "Q3AM3UQ867SPQQA43P2F"
secretAccessKey := "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG"
useSSL := true
// Initialize minio client object.
minioClient, err := minio.New(endpoint, &minio.Options{
Creds: credentials.NewStaticV4(accessKeyID, secretAccessKey, ""),
Secure: useSSL,
})
if err != nil {
log.Fatalln(err)
}
log.Printf("%#v\n", minioClient) // minioClient is now set up
}
```
Example - File Uploader
-----------------------
This sample code connects to an object storage server, creates a bucket, and uploads a file to the bucket. It uses the MinIO `play` server, a public MinIO cluster located at [https://play.min.io](https://play.min.io).
The `play` server runs the latest stable version of MinIO and may be used for testing and development. The access credentials shown in this example are open to the public and all data uploaded to `play` should be considered public and non-protected.
### FileUploader.go
This example does the following:
- Connects to the MinIO `play` server using the provided credentials.
- Creates a bucket named `testbucket`.
- Uploads a file named `testdata` from `/tmp`.
- Verifies the file was created using `mc ls`.
```go
// FileUploader.go MinIO example
package main
import (
"context"
"log"
"github.com/minio/minio-go/v7"
"github.com/minio/minio-go/v7/pkg/credentials"
)
func main() {
ctx := context.Background()
endpoint := "play.min.io"
accessKeyID := "Q3AM3UQ867SPQQA43P2F"
secretAccessKey := "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG"
useSSL := true
// Initialize minio client object.
minioClient, err := minio.New(endpoint, &minio.Options{
Creds: credentials.NewStaticV4(accessKeyID, secretAccessKey, ""),
Secure: useSSL,
})
if err != nil {
log.Fatalln(err)
}
// Make a new bucket called testbucket.
bucketName := "testbucket"
location := "us-east-1"
err = minioClient.MakeBucket(ctx, bucketName, minio.MakeBucketOptions{Region: location})
if err != nil {
// Check to see if we already own this bucket (which happens if you run this twice)
exists, errBucketExists := minioClient.BucketExists(ctx, bucketName)
if errBucketExists == nil && exists {
log.Printf("We already own %s\n", bucketName)
} else {
log.Fatalln(err)
}
} else {
log.Printf("Successfully created %s\n", bucketName)
}
// Upload the test file
// Change the value of filePath if the file is in another location
objectName := "testdata"
filePath := "/tmp/testdata"
contentType := "application/octet-stream"
// Upload the test file with FPutObject
info, err := minioClient.FPutObject(ctx, bucketName, objectName, filePath, minio.PutObjectOptions{ContentType: contentType})
if err != nil {
log.Fatalln(err)
}
log.Printf("Successfully uploaded %s of size %d\n", objectName, info.Size)
}
```
**1. Create a test file containing data:**
You can do this with `dd` on Linux or macOS systems:
```sh
dd if=/dev/urandom of=/tmp/testdata bs=2048 count=10
```
or `fsutil` on Windows:
```sh
fsutil file createnew "C:\Users\<username>\Desktop\sample.txt" 20480
```
**2. Run FileUploader with the following commands:**
```sh
go mod init example/FileUploader
go get github.com/minio/minio-go/v7
go get github.com/minio/minio-go/v7/pkg/credentials
go run FileUploader.go
```
The output resembles the following:
```sh
2023/11/01 14:27:55 Successfully created testbucket
2023/11/01 14:27:55 Successfully uploaded testdata of size 20480
```
**3. Verify the Uploaded File With `mc ls`:**
```sh
mc ls play/testbucket
[2023-11-01 14:27:55 UTC] 20KiB STANDARD TestDataFile
```
API Reference
-------------
The full API Reference is available here.
- [Complete API Reference](https://min.io/docs/minio/linux/developers/go/API.html)
### API Reference : Bucket Operations
- [`MakeBucket`](https://min.io/docs/minio/linux/developers/go/API.html#MakeBucket)
- [`ListBuckets`](https://min.io/docs/minio/linux/developers/go/API.html#ListBuckets)
- [`BucketExists`](https://min.io/docs/minio/linux/developers/go/API.html#BucketExists)
- [`RemoveBucket`](https://min.io/docs/minio/linux/developers/go/API.html#RemoveBucket)
- [`ListObjects`](https://min.io/docs/minio/linux/developers/go/API.html#ListObjects)
- [`ListIncompleteUploads`](https://min.io/docs/minio/linux/developers/go/API.html#ListIncompleteUploads)
### API Reference : Bucket policy Operations
- [`SetBucketPolicy`](https://min.io/docs/minio/linux/developers/go/API.html#SetBucketPolicy)
- [`GetBucketPolicy`](https://min.io/docs/minio/linux/developers/go/API.html#GetBucketPolicy)
### API Reference : Bucket notification Operations
- [`SetBucketNotification`](https://min.io/docs/minio/linux/developers/go/API.html#SetBucketNotification)
- [`GetBucketNotification`](https://min.io/docs/minio/linux/developers/go/API.html#GetBucketNotification)
- [`RemoveAllBucketNotification`](https://min.io/docs/minio/linux/developers/go/API.html#RemoveAllBucketNotification)
- [`ListenBucketNotification`](https://min.io/docs/minio/linux/developers/go/API.html#ListenBucketNotification) (MinIO Extension)
- [`ListenNotification`](https://min.io/docs/minio/linux/developers/go/API.html#ListenNotification) (MinIO Extension)
### API Reference : File Object Operations
- [`FPutObject`](https://min.io/docs/minio/linux/developers/go/API.html#FPutObject)
- [`FGetObject`](https://min.io/docs/minio/linux/developers/go/API.html#FGetObject)
### API Reference : Object Operations
- [`GetObject`](https://min.io/docs/minio/linux/developers/go/API.html#GetObject)
- [`PutObject`](https://min.io/docs/minio/linux/developers/go/API.html#PutObject)
- [`PutObjectStreaming`](https://min.io/docs/minio/linux/developers/go/API.html#PutObjectStreaming)
- [`StatObject`](https://min.io/docs/minio/linux/developers/go/API.html#StatObject)
- [`CopyObject`](https://min.io/docs/minio/linux/developers/go/API.html#CopyObject)
- [`RemoveObject`](https://min.io/docs/minio/linux/developers/go/API.html#RemoveObject)
- [`RemoveObjects`](https://min.io/docs/minio/linux/developers/go/API.html#RemoveObjects)
- [`RemoveIncompleteUpload`](https://min.io/docs/minio/linux/developers/go/API.html#RemoveIncompleteUpload)
- [`SelectObjectContent`](https://min.io/docs/minio/linux/developers/go/API.html#SelectObjectContent)
### API Reference : Presigned Operations
- [`PresignedGetObject`](https://min.io/docs/minio/linux/developers/go/API.html#PresignedGetObject)
- [`PresignedPutObject`](https://min.io/docs/minio/linux/developers/go/API.html#PresignedPutObject)
- [`PresignedHeadObject`](https://min.io/docs/minio/linux/developers/go/API.html#PresignedHeadObject)
- [`PresignedPostPolicy`](https://min.io/docs/minio/linux/developers/go/API.html#PresignedPostPolicy)
### API Reference : Client custom settings
- [`SetAppInfo`](https://min.io/docs/minio/linux/developers/go/API.html#SetAppInfo)
- [`TraceOn`](https://min.io/docs/minio/linux/developers/go/API.html#TraceOn)
- [`TraceOff`](https://min.io/docs/minio/linux/developers/go/API.html#TraceOff)
Full Examples
-------------
### Full Examples : Bucket Operations
- [makebucket.go](https://github.com/minio/minio-go/blob/master/examples/s3/makebucket.go)
- [listbuckets.go](https://github.com/minio/minio-go/blob/master/examples/s3/listbuckets.go)
- [bucketexists.go](https://github.com/minio/minio-go/blob/master/examples/s3/bucketexists.go)
- [removebucket.go](https://github.com/minio/minio-go/blob/master/examples/s3/removebucket.go)
- [listobjects.go](https://github.com/minio/minio-go/blob/master/examples/s3/listobjects.go)
- [listobjectsV2.go](https://github.com/minio/minio-go/blob/master/examples/s3/listobjectsV2.go)
- [listincompleteuploads.go](https://github.com/minio/minio-go/blob/master/examples/s3/listincompleteuploads.go)
### Full Examples : Bucket policy Operations
- [setbucketpolicy.go](https://github.com/minio/minio-go/blob/master/examples/s3/setbucketpolicy.go)
- [getbucketpolicy.go](https://github.com/minio/minio-go/blob/master/examples/s3/getbucketpolicy.go)
- [listbucketpolicies.go](https://github.com/minio/minio-go/blob/master/examples/s3/listbucketpolicies.go)
### Full Examples : Bucket lifecycle Operations
- [setbucketlifecycle.go](https://github.com/minio/minio-go/blob/master/examples/s3/setbucketlifecycle.go)
- [getbucketlifecycle.go](https://github.com/minio/minio-go/blob/master/examples/s3/getbucketlifecycle.go)
### Full Examples : Bucket encryption Operations
- [setbucketencryption.go](https://github.com/minio/minio-go/blob/master/examples/s3/setbucketencryption.go)
- [getbucketencryption.go](https://github.com/minio/minio-go/blob/master/examples/s3/getbucketencryption.go)
- [removebucketencryption.go](https://github.com/minio/minio-go/blob/master/examples/s3/removebucketencryption.go)
### Full Examples : Bucket replication Operations
- [setbucketreplication.go](https://github.com/minio/minio-go/blob/master/examples/s3/setbucketreplication.go)
- [getbucketreplication.go](https://github.com/minio/minio-go/blob/master/examples/s3/getbucketreplication.go)
- [removebucketreplication.go](https://github.com/minio/minio-go/blob/master/examples/s3/removebucketreplication.go)
### Full Examples : Bucket notification Operations
- [setbucketnotification.go](https://github.com/minio/minio-go/blob/master/examples/s3/setbucketnotification.go)
- [getbucketnotification.go](https://github.com/minio/minio-go/blob/master/examples/s3/getbucketnotification.go)
- [removeallbucketnotification.go](https://github.com/minio/minio-go/blob/master/examples/s3/removeallbucketnotification.go)
- [listenbucketnotification.go](https://github.com/minio/minio-go/blob/master/examples/minio/listenbucketnotification.go) (MinIO Extension)
- [listennotification.go](https://github.com/minio/minio-go/blob/master/examples/minio/listen-notification.go) (MinIO Extension)
### Full Examples : File Object Operations
- [fputobject.go](https://github.com/minio/minio-go/blob/master/examples/s3/fputobject.go)
- [fgetobject.go](https://github.com/minio/minio-go/blob/master/examples/s3/fgetobject.go)
### Full Examples : Object Operations
- [putobject.go](https://github.com/minio/minio-go/blob/master/examples/s3/putobject.go)
- [getobject.go](https://github.com/minio/minio-go/blob/master/examples/s3/getobject.go)
- [statobject.go](https://github.com/minio/minio-go/blob/master/examples/s3/statobject.go)
- [copyobject.go](https://github.com/minio/minio-go/blob/master/examples/s3/copyobject.go)
- [removeobject.go](https://github.com/minio/minio-go/blob/master/examples/s3/removeobject.go)
- [removeincompleteupload.go](https://github.com/minio/minio-go/blob/master/examples/s3/removeincompleteupload.go)
- [removeobjects.go](https://github.com/minio/minio-go/blob/master/examples/s3/removeobjects.go)
### Full Examples : Encrypted Object Operations
- [put-encrypted-object.go](https://github.com/minio/minio-go/blob/master/examples/s3/put-encrypted-object.go)
- [get-encrypted-object.go](https://github.com/minio/minio-go/blob/master/examples/s3/get-encrypted-object.go)
- [fput-encrypted-object.go](https://github.com/minio/minio-go/blob/master/examples/s3/fputencrypted-object.go)
### Full Examples : Presigned Operations
- [presignedgetobject.go](https://github.com/minio/minio-go/blob/master/examples/s3/presignedgetobject.go)
- [presignedputobject.go](https://github.com/minio/minio-go/blob/master/examples/s3/presignedputobject.go)
- [presignedheadobject.go](https://github.com/minio/minio-go/blob/master/examples/s3/presignedheadobject.go)
- [presignedpostpolicy.go](https://github.com/minio/minio-go/blob/master/examples/s3/presignedpostpolicy.go)
Explore Further
---------------
- [Godoc Documentation](https://pkg.go.dev/github.com/minio/minio-go/v7)
- [Complete Documentation](https://min.io/docs/minio/kubernetes/upstream/index.html)
- [MinIO Go Client SDK API Reference](https://min.io/docs/minio/linux/developers/go/API.html)
Contribute
----------
[Contributors Guide](https://github.com/minio/minio-go/blob/master/CONTRIBUTING.md)
License
-------
This SDK is distributed under the [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0), see [LICENSE](https://github.com/minio/minio-go/blob/master/LICENSE) and [NOTICE](https://github.com/minio/minio-go/blob/master/NOTICE) for more information.
Server/vendor/github.com/minio/minio-go/v7/api-append-object.go
Generated Vendored
+244
View File
@@ -0,0 +1,244 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2015-2025 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"net/http"
"strconv"
"github.com/minio/minio-go/v7/pkg/s3utils"
)
// AppendObjectOptions https://docs.aws.amazon.com/AmazonS3/latest/userguide/directory-buckets-objects-append.html
type AppendObjectOptions struct {
// Provide a progress reader to indicate the current append() progress.
Progress io.Reader
// ChunkSize indicates the maximum append() size,
// it is useful when you want to control how much data
// per append() you are interested in sending to server
// while keeping the input io.Reader of a longer length.
ChunkSize uint64
// Aggressively disable sha256 payload, it is automatically
// turned-off for TLS supporting endpoints, useful in benchmarks
// where you are interested in the peak() numbers.
DisableContentSha256 bool
customHeaders http.Header
checksumType ChecksumType
offset int64
}
// Header returns the custom header for AppendObject API
func (opts AppendObjectOptions) Header() (header http.Header) {
header = make(http.Header)
for k, v := range opts.customHeaders {
header[k] = v
}
return header
}
func (opts *AppendObjectOptions) setWriteOffset(offset int64) {
if len(opts.customHeaders) == 0 {
opts.customHeaders = make(http.Header)
}
opts.customHeaders["x-amz-write-offset-bytes"] = []string{strconv.FormatInt(offset, 10)}
opts.offset = offset
}
func (opts *AppendObjectOptions) setChecksumParams(info ObjectInfo) {
if len(opts.customHeaders) == 0 {
opts.customHeaders = make(http.Header)
}
fullObject := info.ChecksumMode == ChecksumFullObjectMode.String()
switch {
case info.ChecksumCRC32 != "":
if fullObject {
opts.checksumType = ChecksumFullObjectCRC32
}
case info.ChecksumCRC32C != "":
if fullObject {
opts.checksumType = ChecksumFullObjectCRC32C
}
case info.ChecksumCRC64NVME != "":
// CRC64NVME only has a full object variant
// so it does not carry any special full object
// modifier
opts.checksumType = ChecksumCRC64NVME
}
}
func (opts AppendObjectOptions) validate(c *Client) (err error) {
if opts.ChunkSize > maxPartSize {
return errInvalidArgument("Append chunkSize cannot be larger than max part size allowed")
}
switch {
case !c.trailingHeaderSupport:
return errInvalidArgument("AppendObject() requires Client with TrailingHeaders enabled")
case c.overrideSignerType.IsV2():
return errInvalidArgument("AppendObject() cannot be used with v2 signatures")
case s3utils.IsGoogleEndpoint(*c.endpointURL):
return errInvalidArgument("AppendObject() cannot be used with GCS endpoints")
}
return nil
}
// appendObjectDo - executes the append object http operation.
// NOTE: You must have WRITE permissions on a bucket to add an object to it.
func (c *Client) appendObjectDo(ctx context.Context, bucketName, objectName string, reader io.Reader, size int64, opts AppendObjectOptions) (UploadInfo, error) {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return UploadInfo{}, err
}
if err := s3utils.CheckValidObjectName(objectName); err != nil {
return UploadInfo{}, err
}
// Set headers.
customHeader := opts.Header()
// Populate request metadata.
reqMetadata := requestMetadata{
bucketName: bucketName,
objectName: objectName,
customHeader: customHeader,
contentBody: reader,
contentLength: size,
streamSha256: !opts.DisableContentSha256,
}
if opts.checksumType.IsSet() {
reqMetadata.addCrc = &opts.checksumType
reqMetadata.customHeader.Set(amzChecksumAlgo, opts.checksumType.String())
if opts.checksumType.FullObjectRequested() {
reqMetadata.customHeader.Set(amzChecksumMode, ChecksumFullObjectMode.String())
}
}
// Execute PUT an objectName.
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
defer closeResponse(resp)
if err != nil {
return UploadInfo{}, err
}
if resp != nil {
if resp.StatusCode != http.StatusOK {
return UploadInfo{}, httpRespToErrorResponse(resp, bucketName, objectName)
}
}
h := resp.Header
// When AppendObject() is used, S3 Express will return final object size as x-amz-object-size
if amzSize := h.Get("x-amz-object-size"); amzSize != "" {
oSize := size
size, err = strconv.ParseInt(amzSize, 10, 64)
if err != nil {
return UploadInfo{}, err
}
if size != opts.offset+oSize {
return UploadInfo{}, errors.New("server returned incorrect object size")
}
} else {
return UploadInfo{}, errors.New("server does not support appends. Object has been overwritten")
}
return UploadInfo{
Bucket: bucketName,
Key: objectName,
ETag: trimEtag(h.Get("ETag")),
Size: size,
// Checksum values
ChecksumCRC32: h.Get(ChecksumCRC32.Key()),
ChecksumCRC32C: h.Get(ChecksumCRC32C.Key()),
ChecksumSHA1: h.Get(ChecksumSHA1.Key()),
ChecksumSHA256: h.Get(ChecksumSHA256.Key()),
ChecksumCRC64NVME: h.Get(ChecksumCRC64NVME.Key()),
ChecksumMD5: h.Get(ChecksumMD5.Key()),
ChecksumSHA512: h.Get(ChecksumSHA512.Key()),
ChecksumXXHash64: h.Get(ChecksumXXHash64.Key()),
ChecksumXXHash3: h.Get(ChecksumXXHash3.Key()),
ChecksumXXHash128: h.Get(ChecksumXXHash128.Key()),
ChecksumMode: h.Get(ChecksumFullObjectMode.Key()),
}, nil
}
// AppendObject - S3 Express Zone https://docs.aws.amazon.com/AmazonS3/latest/userguide/directory-buckets-objects-append.html
// Note that appending on a server without append support may overwrite the object.
func (c *Client) AppendObject(ctx context.Context, bucketName, objectName string, reader io.Reader, objectSize int64,
opts AppendObjectOptions,
) (info UploadInfo, err error) {
if objectSize < 0 && opts.ChunkSize == 0 {
return UploadInfo{}, errors.New("object size must be provided when no chunk size is provided")
}
if err = opts.validate(c); err != nil {
return UploadInfo{}, err
}
oinfo, err := c.StatObject(ctx, bucketName, objectName, StatObjectOptions{Checksum: true})
if err != nil {
return UploadInfo{}, err
}
if oinfo.ChecksumMode != "" && oinfo.ChecksumMode != ChecksumFullObjectMode.String() {
return UploadInfo{}, fmt.Errorf("Append() is not allowed on objects that are not of FULL_OBJECT checksum type: %s", oinfo.ChecksumMode)
}
opts.setChecksumParams(oinfo) // set the appropriate checksum params based on the existing object checksum metadata.
opts.setWriteOffset(oinfo.Size) // First append must set the current object size as the offset.
if opts.ChunkSize > 0 {
finalObjSize := int64(-1)
if objectSize > 0 {
finalObjSize = info.Size + objectSize
}
totalPartsCount, partSize, lastPartSize, err := OptimalPartInfo(finalObjSize, opts.ChunkSize)
if err != nil {
return UploadInfo{}, err
}
buf := make([]byte, partSize)
var partNumber int
for partNumber = 1; partNumber <= totalPartsCount; partNumber++ {
// Proceed to upload the part.
if partNumber == totalPartsCount {
partSize = lastPartSize
}
n, err := readFull(reader, buf)
if err != nil {
return info, err
}
if n != int(partSize) {
return info, io.ErrUnexpectedEOF
}
rd := newHook(bytes.NewReader(buf[:n]), opts.Progress)
uinfo, err := c.appendObjectDo(ctx, bucketName, objectName, rd, partSize, opts)
if err != nil {
return info, err
}
opts.setWriteOffset(uinfo.Size)
}
}
rd := newHook(reader, opts.Progress)
return c.appendObjectDo(ctx, bucketName, objectName, rd, objectSize, opts)
}
Server/vendor/github.com/minio/minio-go/v7/api-bucket-cors.go
Generated Vendored
+151
View File
@@ -0,0 +1,151 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2024 MinIO, Inc.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"bytes"
"context"
"net/http"
"net/url"
"github.com/minio/minio-go/v7/pkg/cors"
"github.com/minio/minio-go/v7/pkg/s3utils"
)
// SetBucketCors sets the Cross-Origin Resource Sharing (CORS) configuration for the bucket.
// If corsConfig is nil, the existing CORS configuration will be removed.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucketName: Name of the bucket
// - corsConfig: CORS configuration to apply (nil to remove existing configuration)
//
// Returns an error if the operation fails.
func (c *Client) SetBucketCors(ctx context.Context, bucketName string, corsConfig *cors.Config) error {
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return err
}
if corsConfig == nil {
return c.removeBucketCors(ctx, bucketName)
}
return c.putBucketCors(ctx, bucketName, corsConfig)
}
func (c *Client) putBucketCors(ctx context.Context, bucketName string, corsConfig *cors.Config) error {
urlValues := make(url.Values)
urlValues.Set("cors", "")
corsStr, err := corsConfig.ToXML()
if err != nil {
return err
}
reqMetadata := requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
contentBody: bytes.NewReader(corsStr),
contentLength: int64(len(corsStr)),
contentMD5Base64: sumMD5Base64([]byte(corsStr)),
}
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
defer closeResponse(resp)
if err != nil {
return err
}
if resp != nil {
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusNoContent {
return httpRespToErrorResponse(resp, bucketName, "")
}
}
return nil
}
func (c *Client) removeBucketCors(ctx context.Context, bucketName string) error {
urlValues := make(url.Values)
urlValues.Set("cors", "")
resp, err := c.executeMethod(ctx, http.MethodDelete, requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
contentSHA256Hex: emptySHA256Hex,
})
defer closeResponse(resp)
if err != nil {
return err
}
if resp.StatusCode != http.StatusNoContent {
return httpRespToErrorResponse(resp, bucketName, "")
}
return nil
}
// GetBucketCors retrieves the Cross-Origin Resource Sharing (CORS) configuration from the bucket.
// If no CORS configuration exists, returns nil with no error.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucketName: Name of the bucket
//
// Returns the CORS configuration or an error if the operation fails.
func (c *Client) GetBucketCors(ctx context.Context, bucketName string) (*cors.Config, error) {
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return nil, err
}
bucketCors, err := c.getBucketCors(ctx, bucketName)
if err != nil {
errResponse := ToErrorResponse(err)
if errResponse.Code == NoSuchCORSConfiguration {
return nil, nil
}
return nil, err
}
return bucketCors, nil
}
func (c *Client) getBucketCors(ctx context.Context, bucketName string) (*cors.Config, error) {
urlValues := make(url.Values)
urlValues.Set("cors", "")
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
contentSHA256Hex: emptySHA256Hex, // TODO: needed? copied over from other example, but not spec'd in API.
})
defer closeResponse(resp)
if err != nil {
return nil, err
}
if resp != nil {
if resp.StatusCode != http.StatusOK {
return nil, httpRespToErrorResponse(resp, bucketName, "")
}
}
corsConfig, err := cors.ParseBucketCorsConfig(resp.Body)
if err != nil {
return nil, err
}
return corsConfig, nil
}
Server/vendor/github.com/minio/minio-go/v7/api-bucket-encryption.go
Generated Vendored
+156
View File
@@ -0,0 +1,156 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2020 MinIO, Inc.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"bytes"
"context"
"encoding/xml"
"net/http"
"net/url"
"github.com/minio/minio-go/v7/pkg/s3utils"
"github.com/minio/minio-go/v7/pkg/sse"
)
// SetBucketEncryption sets the default encryption configuration on an existing bucket.
// The encryption configuration specifies the default encryption behavior for objects uploaded to the bucket.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucketName: Name of the bucket
// - config: Server-side encryption configuration to apply
//
// Returns an error if the operation fails or if config is nil.
func (c *Client) SetBucketEncryption(ctx context.Context, bucketName string, config *sse.Configuration) error {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return err
}
if config == nil {
return errInvalidArgument("configuration cannot be empty")
}
buf, err := xml.Marshal(config)
if err != nil {
return err
}
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("encryption", "")
// Content-length is mandatory to set a default encryption configuration
reqMetadata := requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
contentBody: bytes.NewReader(buf),
contentLength: int64(len(buf)),
contentMD5Base64: sumMD5Base64(buf),
}
// Execute PUT to upload a new bucket default encryption configuration.
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
defer closeResponse(resp)
if err != nil {
return err
}
if resp.StatusCode != http.StatusOK {
return httpRespToErrorResponse(resp, bucketName, "")
}
return nil
}
// RemoveBucketEncryption removes the default encryption configuration from a bucket.
// After removal, the bucket will no longer apply default encryption to new objects.
// It uses the provided context to control cancellations and timeouts.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucketName: Name of the bucket
//
// Returns an error if the operation fails.
func (c *Client) RemoveBucketEncryption(ctx context.Context, bucketName string) error {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return err
}
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("encryption", "")
// DELETE default encryption configuration on a bucket.
resp, err := c.executeMethod(ctx, http.MethodDelete, requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
contentSHA256Hex: emptySHA256Hex,
})
defer closeResponse(resp)
if err != nil {
return err
}
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusNoContent {
return httpRespToErrorResponse(resp, bucketName, "")
}
return nil
}
// GetBucketEncryption retrieves the default encryption configuration from a bucket.
// It uses the provided context to control cancellations and timeouts.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucketName: Name of the bucket
//
// Returns the bucket's encryption configuration or an error if the operation fails.
func (c *Client) GetBucketEncryption(ctx context.Context, bucketName string) (*sse.Configuration, error) {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return nil, err
}
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("encryption", "")
// Execute GET on bucket to get the default encryption configuration.
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
})
defer closeResponse(resp)
if err != nil {
return nil, err
}
if resp.StatusCode != http.StatusOK {
return nil, httpRespToErrorResponse(resp, bucketName, "")
}
encryptionConfig := &sse.Configuration{}
if err = xmlDecoder(resp.Body, encryptionConfig); err != nil {
return nil, err
}
return encryptionConfig, nil
}
Server/vendor/github.com/minio/minio-go/v7/api-bucket-lifecycle.go
Generated Vendored
+169
View File
@@ -0,0 +1,169 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2020 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"bytes"
"context"
"encoding/xml"
"io"
"net/http"
"net/url"
"time"
"github.com/minio/minio-go/v7/pkg/lifecycle"
"github.com/minio/minio-go/v7/pkg/s3utils"
)
// SetBucketLifecycle set the lifecycle on an existing bucket.
func (c *Client) SetBucketLifecycle(ctx context.Context, bucketName string, config *lifecycle.Configuration) error {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return err
}
// If lifecycle is empty then delete it.
if config.Empty() {
return c.removeBucketLifecycle(ctx, bucketName)
}
buf, err := xml.Marshal(config)
if err != nil {
return err
}
// Save the updated lifecycle.
return c.putBucketLifecycle(ctx, bucketName, buf)
}
// Saves a new bucket lifecycle.
func (c *Client) putBucketLifecycle(ctx context.Context, bucketName string, buf []byte) error {
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("lifecycle", "")
// Content-length is mandatory for put lifecycle request
reqMetadata := requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
contentBody: bytes.NewReader(buf),
contentLength: int64(len(buf)),
contentMD5Base64: sumMD5Base64(buf),
}
// Execute PUT to upload a new bucket lifecycle.
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
defer closeResponse(resp)
if err != nil {
return err
}
if resp != nil {
if resp.StatusCode != http.StatusOK {
return httpRespToErrorResponse(resp, bucketName, "")
}
}
return nil
}
// Remove lifecycle from a bucket.
func (c *Client) removeBucketLifecycle(ctx context.Context, bucketName string) error {
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("lifecycle", "")
// Execute DELETE on objectName.
resp, err := c.executeMethod(ctx, http.MethodDelete, requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
contentSHA256Hex: emptySHA256Hex,
})
defer closeResponse(resp)
if err != nil {
return err
}
return nil
}
// GetBucketLifecycle fetch bucket lifecycle configuration
func (c *Client) GetBucketLifecycle(ctx context.Context, bucketName string) (*lifecycle.Configuration, error) {
lc, _, err := c.GetBucketLifecycleWithInfo(ctx, bucketName)
return lc, err
}
// GetBucketLifecycleWithInfo fetch bucket lifecycle configuration along with when it was last updated
func (c *Client) GetBucketLifecycleWithInfo(ctx context.Context, bucketName string) (*lifecycle.Configuration, time.Time, error) {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return nil, time.Time{}, err
}
bucketLifecycle, updatedAt, err := c.getBucketLifecycle(ctx, bucketName)
if err != nil {
return nil, time.Time{}, err
}
config := lifecycle.NewConfiguration()
if err = xml.Unmarshal(bucketLifecycle, config); err != nil {
return nil, time.Time{}, err
}
return config, updatedAt, nil
}
// Request server for current bucket lifecycle.
func (c *Client) getBucketLifecycle(ctx context.Context, bucketName string) ([]byte, time.Time, error) {
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("lifecycle", "")
urlValues.Set("withUpdatedAt", "true")
// Execute GET on bucket to get lifecycle.
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
})
defer closeResponse(resp)
if err != nil {
return nil, time.Time{}, err
}
if resp != nil {
if resp.StatusCode != http.StatusOK {
return nil, time.Time{}, httpRespToErrorResponse(resp, bucketName, "")
}
}
lcBytes, err := io.ReadAll(resp.Body)
if err != nil {
return nil, time.Time{}, err
}
const minIOLifecycleCfgUpdatedAt = "X-Minio-LifecycleConfig-UpdatedAt"
var updatedAt time.Time
if timeStr := resp.Header.Get(minIOLifecycleCfgUpdatedAt); timeStr != "" {
updatedAt, err = time.Parse(iso8601DateFormat, timeStr)
if err != nil {
return nil, time.Time{}, err
}
}
return lcBytes, updatedAt, nil
}
Server/vendor/github.com/minio/minio-go/v7/api-bucket-notification.go
Generated Vendored
+252
View File
@@ -0,0 +1,252 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2017-2020 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"bufio"
"bytes"
"context"
"encoding/json"
"encoding/xml"
"net/http"
"net/url"
"time"
"github.com/minio/minio-go/v7/pkg/notification"
"github.com/minio/minio-go/v7/pkg/s3utils"
)
// SetBucketNotification saves a new bucket notification with a context to control cancellations and timeouts.
func (c *Client) SetBucketNotification(ctx context.Context, bucketName string, config notification.Configuration) error {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return err
}
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("notification", "")
notifBytes, err := xml.Marshal(&config)
if err != nil {
return err
}
notifBuffer := bytes.NewReader(notifBytes)
reqMetadata := requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
contentBody: notifBuffer,
contentLength: int64(len(notifBytes)),
contentMD5Base64: sumMD5Base64(notifBytes),
contentSHA256Hex: sum256Hex(notifBytes),
}
// Execute PUT to upload a new bucket notification.
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
defer closeResponse(resp)
if err != nil {
return err
}
if resp != nil {
if resp.StatusCode != http.StatusOK {
return httpRespToErrorResponse(resp, bucketName, "")
}
}
return nil
}
// RemoveAllBucketNotification - Remove bucket notification clears all previously specified config
func (c *Client) RemoveAllBucketNotification(ctx context.Context, bucketName string) error {
return c.SetBucketNotification(ctx, bucketName, notification.Configuration{})
}
// GetBucketNotification returns current bucket notification configuration
func (c *Client) GetBucketNotification(ctx context.Context, bucketName string) (bucketNotification notification.Configuration, err error) {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return notification.Configuration{}, err
}
return c.getBucketNotification(ctx, bucketName)
}
// Request server for notification rules.
func (c *Client) getBucketNotification(ctx context.Context, bucketName string) (notification.Configuration, error) {
urlValues := make(url.Values)
urlValues.Set("notification", "")
// Execute GET on bucket to list objects.
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
contentSHA256Hex: emptySHA256Hex,
})
defer closeResponse(resp)
if err != nil {
return notification.Configuration{}, err
}
return processBucketNotificationResponse(bucketName, resp)
}
// processes the GetNotification http response from the server.
func processBucketNotificationResponse(bucketName string, resp *http.Response) (notification.Configuration, error) {
if resp.StatusCode != http.StatusOK {
errResponse := httpRespToErrorResponse(resp, bucketName, "")
return notification.Configuration{}, errResponse
}
var bucketNotification notification.Configuration
err := xmlDecoder(resp.Body, &bucketNotification)
if err != nil {
return notification.Configuration{}, err
}
return bucketNotification, nil
}
// ListenNotification listen for all events, this is a MinIO specific API
func (c *Client) ListenNotification(ctx context.Context, prefix, suffix string, events []string) <-chan notification.Info {
return c.ListenBucketNotification(ctx, "", prefix, suffix, events)
}
// ListenBucketNotification listen for bucket events, this is a MinIO specific API
func (c *Client) ListenBucketNotification(ctx context.Context, bucketName, prefix, suffix string, events []string) <-chan notification.Info {
notificationInfoCh := make(chan notification.Info, 1)
const notificationCapacity = 4 * 1024 * 1024
notificationEventBuffer := make([]byte, notificationCapacity)
// Only success, start a routine to start reading line by line.
go func(notificationInfoCh chan<- notification.Info) {
defer close(notificationInfoCh)
// Validate the bucket name.
if bucketName != "" {
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
select {
case notificationInfoCh <- notification.Info{
Err: err,
}:
case <-ctx.Done():
}
return
}
}
// Check ARN partition to verify if listening bucket is supported
if s3utils.IsAmazonEndpoint(*c.endpointURL) || s3utils.IsGoogleEndpoint(*c.endpointURL) {
select {
case notificationInfoCh <- notification.Info{
Err: errAPINotSupported("Listening for bucket notification is specific only to `minio` server endpoints"),
}:
case <-ctx.Done():
}
return
}
// Prepare urlValues to pass into the request on every loop
urlValues := make(url.Values)
urlValues.Set("ping", "10")
urlValues.Set("prefix", prefix)
urlValues.Set("suffix", suffix)
urlValues["events"] = events
// Wait on the jitter retry loop.
for range c.newRetryTimerContinous(time.Second, time.Second*30, MaxJitter) {
// Execute GET on bucket to list objects.
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
contentSHA256Hex: emptySHA256Hex,
})
if err != nil {
select {
case notificationInfoCh <- notification.Info{
Err: err,
}:
case <-ctx.Done():
}
return
}
// Validate http response, upon error return quickly.
if resp.StatusCode != http.StatusOK {
errResponse := httpRespToErrorResponse(resp, bucketName, "")
select {
case notificationInfoCh <- notification.Info{
Err: errResponse,
}:
case <-ctx.Done():
}
return
}
// Initialize a new bufio scanner, to read line by line.
bio := bufio.NewScanner(resp.Body)
// Use a higher buffer to support unexpected
// caching done by proxies
bio.Buffer(notificationEventBuffer, notificationCapacity)
// Unmarshal each line, returns marshaled values.
for bio.Scan() {
var notificationInfo notification.Info
if err = json.Unmarshal(bio.Bytes(), &notificationInfo); err != nil {
// Unexpected error during json unmarshal, send
// the error to caller for actionable as needed.
select {
case notificationInfoCh <- notification.Info{
Err: err,
}:
case <-ctx.Done():
return
}
closeResponse(resp)
continue
}
// Empty events pinged from the server
if len(notificationInfo.Records) == 0 && notificationInfo.Err == nil {
continue
}
// Send notificationInfo
select {
case notificationInfoCh <- notificationInfo:
case <-ctx.Done():
closeResponse(resp)
return
}
}
if err = bio.Err(); err != nil {
select {
case notificationInfoCh <- notification.Info{
Err: err,
}:
case <-ctx.Done():
return
}
}
// Close current connection before looping further.
closeResponse(resp)
}
}(notificationInfoCh)
// Returns the notification info channel, for caller to start reading from.
return notificationInfoCh
}
Server/vendor/github.com/minio/minio-go/v7/api-bucket-policy.go
Generated Vendored
+163
View File
@@ -0,0 +1,163 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2020 MinIO, Inc.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"context"
"io"
"net/http"
"net/url"
"strings"
"github.com/minio/minio-go/v7/pkg/s3utils"
)
// SetBucketPolicy sets the access permissions policy on an existing bucket.
// The policy should be a valid JSON string that conforms to the IAM policy format.
// If policy is an empty string, the existing bucket policy will be removed.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucketName: Name of the bucket
// - policy: JSON policy string (empty string to remove existing policy)
//
// Returns an error if the operation fails.
func (c *Client) SetBucketPolicy(ctx context.Context, bucketName, policy string) error {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return err
}
// If policy is empty then delete the bucket policy.
if policy == "" {
return c.removeBucketPolicy(ctx, bucketName)
}
// Save the updated policies.
return c.putBucketPolicy(ctx, bucketName, policy)
}
// Saves a new bucket policy.
func (c *Client) putBucketPolicy(ctx context.Context, bucketName, policy string) error {
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("policy", "")
reqMetadata := requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
contentBody: strings.NewReader(policy),
contentLength: int64(len(policy)),
}
// Execute PUT to upload a new bucket policy.
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
defer closeResponse(resp)
if err != nil {
return err
}
if resp != nil {
if resp.StatusCode != http.StatusNoContent && resp.StatusCode != http.StatusOK {
return httpRespToErrorResponse(resp, bucketName, "")
}
}
return nil
}
// Removes all policies on a bucket.
func (c *Client) removeBucketPolicy(ctx context.Context, bucketName string) error {
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("policy", "")
// Execute DELETE on objectName.
resp, err := c.executeMethod(ctx, http.MethodDelete, requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
contentSHA256Hex: emptySHA256Hex,
})
defer closeResponse(resp)
if err != nil {
return err
}
if resp.StatusCode != http.StatusNoContent {
return httpRespToErrorResponse(resp, bucketName, "")
}
return nil
}
// GetBucketPolicy retrieves the access permissions policy for the bucket.
// If no bucket policy exists, returns an empty string with no error.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucketName: Name of the bucket
//
// Returns the policy as a JSON string or an error if the operation fails.
func (c *Client) GetBucketPolicy(ctx context.Context, bucketName string) (string, error) {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return "", err
}
bucketPolicy, err := c.getBucketPolicy(ctx, bucketName)
if err != nil {
errResponse := ToErrorResponse(err)
if errResponse.Code == NoSuchBucketPolicy {
return "", nil
}
return "", err
}
return bucketPolicy, nil
}
// Request server for current bucket policy.
func (c *Client) getBucketPolicy(ctx context.Context, bucketName string) (string, error) {
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("policy", "")
// Execute GET on bucket to list objects.
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
contentSHA256Hex: emptySHA256Hex,
})
defer closeResponse(resp)
if err != nil {
return "", err
}
if resp != nil {
if resp.StatusCode != http.StatusOK {
return "", httpRespToErrorResponse(resp, bucketName, "")
}
}
bucketPolicyBuf, err := io.ReadAll(resp.Body)
if err != nil {
return "", err
}
policy := string(bucketPolicyBuf)
return policy, err
}
Server/vendor/github.com/minio/minio-go/v7/api-bucket-qos.go
Generated Vendored
+212
View File
@@ -0,0 +1,212 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2025 MinIO, Inc.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"context"
"encoding/json"
"io"
"net/http"
"net/url"
"strings"
"github.com/minio/minio-go/v7/pkg/s3utils"
"go.yaml.in/yaml/v3"
)
// QOSConfigVersionCurrent is the current version of the QoS configuration.
const QOSConfigVersionCurrent = "v1"
// QOSConfig represents the QoS configuration for a bucket.
type QOSConfig struct {
Version string `yaml:"version"`
Rules []QOSRule `yaml:"rules"`
}
// QOSRule represents a single QoS rule.
type QOSRule struct {
ID string `yaml:"id"`
Label string `yaml:"label,omitempty"`
Priority int `yaml:"priority"`
ObjectPrefix string `yaml:"objectPrefix"`
API string `yaml:"api"`
Rate int64 `yaml:"rate"`
Burst int64 `yaml:"burst"` // not required for concurrency limit
Limit string `yaml:"limit"` // "concurrency" or "rps"
}
// NewQOSConfig creates a new empty QoS configuration.
func NewQOSConfig() *QOSConfig {
return &QOSConfig{
Version: "v1",
Rules: []QOSRule{},
}
}
// GetBucketQOS retrieves the Quality of Service (QoS) configuration for the bucket.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucket: Name of the bucket
//
// Returns the QoS configuration or an error if the operation fails.
func (c *Client) GetBucketQOS(ctx context.Context, bucket string) (*QOSConfig, error) {
var qosCfg QOSConfig
// Input validation.
if err := s3utils.CheckValidBucketName(bucket); err != nil {
return nil, err
}
urlValues := make(url.Values)
urlValues.Set("qos", "")
// Execute GET on bucket to fetch qos.
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
bucketName: bucket,
queryValues: urlValues,
contentSHA256Hex: emptySHA256Hex,
})
defer closeResponse(resp)
if err != nil {
return nil, err
}
if resp.StatusCode != http.StatusOK {
return nil, httpRespToErrorResponse(resp, bucket, "")
}
b, err := io.ReadAll(resp.Body)
if err != nil {
return nil, err
}
if err = yaml.Unmarshal(b, &qosCfg); err != nil {
return nil, err
}
return &qosCfg, nil
}
// SetBucketQOS sets the Quality of Service (QoS) configuration for a bucket.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucket: Name of the bucket
// - qosCfg: QoS configuration to apply
//
// Returns an error if the operation fails.
func (c *Client) SetBucketQOS(ctx context.Context, bucket string, qosCfg *QOSConfig) error {
// Input validation.
if err := s3utils.CheckValidBucketName(bucket); err != nil {
return err
}
data, err := yaml.Marshal(qosCfg)
if err != nil {
return err
}
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("qos", "")
reqMetadata := requestMetadata{
bucketName: bucket,
queryValues: urlValues,
contentBody: strings.NewReader(string(data)),
contentLength: int64(len(data)),
}
// Execute PUT to upload a new bucket QoS configuration.
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
defer closeResponse(resp)
if err != nil {
return err
}
if resp != nil {
if resp.StatusCode != http.StatusNoContent && resp.StatusCode != http.StatusOK {
return httpRespToErrorResponse(resp, bucket, "")
}
}
return nil
}
// CounterMetric returns stats for a counter
type CounterMetric struct {
Last1m uint64 `json:"last1m"`
Last1hr uint64 `json:"last1hr"`
Total uint64 `json:"total"`
}
// QOSMetric - metric for a qos rule per bucket
type QOSMetric struct {
APIName string `json:"apiName"`
Rule QOSRule `json:"rule"`
Totals CounterMetric `json:"totals"`
Throttled CounterMetric `json:"throttleCount"`
ExceededRateLimit CounterMetric `json:"exceededRateLimitCount"`
ClientDisconnCount CounterMetric `json:"clientDisconnectCount"`
ReqTimeoutCount CounterMetric `json:"reqTimeoutCount"`
}
// QOSNodeStats represents stats for a bucket on a single node
type QOSNodeStats struct {
Stats []QOSMetric `json:"stats"`
NodeName string `json:"node"`
}
// GetBucketQOSMetrics retrieves Quality of Service (QoS) metrics for a bucket.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucketName: Name of the bucket
// - nodeName: Name of the node (empty string for all nodes)
//
// Returns QoS metrics per node or an error if the operation fails.
func (c *Client) GetBucketQOSMetrics(ctx context.Context, bucketName, nodeName string) (qs []QOSNodeStats, err error) {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return qs, err
}
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("qos-metrics", "")
if nodeName != "" {
urlValues.Set("node", nodeName)
}
// Execute GET on bucket to get qos metrics.
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
})
defer closeResponse(resp)
if err != nil {
return qs, err
}
if resp.StatusCode != http.StatusOK {
return qs, httpRespToErrorResponse(resp, bucketName, "")
}
respBytes, err := io.ReadAll(resp.Body)
if err != nil {
return qs, err
}
if err := json.Unmarshal(respBytes, &qs); err != nil {
return qs, err
}
return qs, nil
}
Server/vendor/github.com/minio/minio-go/v7/api-bucket-replication.go
Generated Vendored
+458
View File
@@ -0,0 +1,458 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2020 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"bytes"
"context"
"encoding/json"
"encoding/xml"
"io"
"net/http"
"net/url"
"time"
"github.com/google/uuid"
"github.com/minio/minio-go/v7/pkg/replication"
"github.com/minio/minio-go/v7/pkg/s3utils"
)
// RemoveBucketReplication removes the replication configuration from an existing bucket.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucketName: Name of the bucket
//
// Returns an error if the operation fails.
func (c *Client) RemoveBucketReplication(ctx context.Context, bucketName string) error {
return c.removeBucketReplication(ctx, bucketName)
}
// SetBucketReplication sets the replication configuration on an existing bucket.
// If the provided configuration is empty, this method removes the existing replication configuration.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucketName: Name of the bucket
// - cfg: Replication configuration to apply
//
// Returns an error if the operation fails.
func (c *Client) SetBucketReplication(ctx context.Context, bucketName string, cfg replication.Config) error {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return err
}
// If replication is empty then delete it.
if cfg.Empty() {
return c.removeBucketReplication(ctx, bucketName)
}
// Save the updated replication.
return c.putBucketReplication(ctx, bucketName, cfg)
}
// Saves a new bucket replication.
func (c *Client) putBucketReplication(ctx context.Context, bucketName string, cfg replication.Config) error {
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("replication", "")
replication, err := xml.Marshal(cfg)
if err != nil {
return err
}
reqMetadata := requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
contentBody: bytes.NewReader(replication),
contentLength: int64(len(replication)),
contentMD5Base64: sumMD5Base64(replication),
}
// Execute PUT to upload a new bucket replication config.
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
defer closeResponse(resp)
if err != nil {
return err
}
if resp.StatusCode != http.StatusOK {
return httpRespToErrorResponse(resp, bucketName, "")
}
return nil
}
// Remove replication from a bucket.
func (c *Client) removeBucketReplication(ctx context.Context, bucketName string) error {
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("replication", "")
// Execute DELETE on objectName.
resp, err := c.executeMethod(ctx, http.MethodDelete, requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
contentSHA256Hex: emptySHA256Hex,
})
defer closeResponse(resp)
if err != nil {
return err
}
if resp.StatusCode != http.StatusOK {
return httpRespToErrorResponse(resp, bucketName, "")
}
return nil
}
// GetBucketReplication retrieves the bucket replication configuration.
// If no replication configuration is found, returns an empty config with nil error.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucketName: Name of the bucket
//
// Returns the replication configuration or an error if the operation fails.
func (c *Client) GetBucketReplication(ctx context.Context, bucketName string) (cfg replication.Config, err error) {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return cfg, err
}
bucketReplicationCfg, err := c.getBucketReplication(ctx, bucketName)
if err != nil {
errResponse := ToErrorResponse(err)
if errResponse.Code == "ReplicationConfigurationNotFoundError" {
return cfg, nil
}
return cfg, err
}
return bucketReplicationCfg, nil
}
// Request server for current bucket replication config.
func (c *Client) getBucketReplication(ctx context.Context, bucketName string) (cfg replication.Config, err error) {
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("replication", "")
// Execute GET on bucket to get replication config.
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
})
defer closeResponse(resp)
if err != nil {
return cfg, err
}
if resp.StatusCode != http.StatusOK {
return cfg, httpRespToErrorResponse(resp, bucketName, "")
}
if err = xmlDecoder(resp.Body, &cfg); err != nil {
return cfg, err
}
return cfg, nil
}
// GetBucketReplicationMetrics retrieves bucket replication status metrics.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucketName: Name of the bucket
//
// Returns the replication metrics or an error if the operation fails.
func (c *Client) GetBucketReplicationMetrics(ctx context.Context, bucketName string) (s replication.Metrics, err error) {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return s, err
}
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("replication-metrics", "")
// Execute GET on bucket to get replication config.
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
})
defer closeResponse(resp)
if err != nil {
return s, err
}
if resp.StatusCode != http.StatusOK {
return s, httpRespToErrorResponse(resp, bucketName, "")
}
respBytes, err := io.ReadAll(resp.Body)
if err != nil {
return s, err
}
if err := json.Unmarshal(respBytes, &s); err != nil {
return s, err
}
return s, nil
}
// mustGetUUID - get a random UUID.
func mustGetUUID() string {
u, err := uuid.NewRandom()
if err != nil {
return ""
}
return u.String()
}
// ResetBucketReplication initiates replication of previously replicated objects.
// This requires ExistingObjectReplication to be enabled in the replication configuration.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucketName: Name of the bucket
// - olderThan: Only replicate objects older than this duration (0 for all objects)
//
// Returns a reset ID that can be used to track the operation, or an error if the operation fails.
func (c *Client) ResetBucketReplication(ctx context.Context, bucketName string, olderThan time.Duration) (rID string, err error) {
rID = mustGetUUID()
_, err = c.resetBucketReplicationOnTarget(ctx, bucketName, olderThan, "", rID)
if err != nil {
return rID, err
}
return rID, nil
}
// ResetBucketReplicationOnTarget initiates replication of previously replicated objects to a specific target.
// This requires ExistingObjectReplication to be enabled in the replication configuration.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucketName: Name of the bucket
// - olderThan: Only replicate objects older than this duration (0 for all objects)
// - tgtArn: ARN of the target to reset replication for
//
// Returns resync target information or an error if the operation fails.
func (c *Client) ResetBucketReplicationOnTarget(ctx context.Context, bucketName string, olderThan time.Duration, tgtArn string) (replication.ResyncTargetsInfo, error) {
return c.resetBucketReplicationOnTarget(ctx, bucketName, olderThan, tgtArn, mustGetUUID())
}
// ResetBucketReplication kicks off replication of previously replicated objects if ExistingObjectReplication
// is enabled in the replication config
func (c *Client) resetBucketReplicationOnTarget(ctx context.Context, bucketName string, olderThan time.Duration, tgtArn, resetID string) (rinfo replication.ResyncTargetsInfo, err error) {
// Input validation.
if err = s3utils.CheckValidBucketName(bucketName); err != nil {
return rinfo, err
}
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("replication-reset", "")
if olderThan > 0 {
urlValues.Set("older-than", olderThan.String())
}
if tgtArn != "" {
urlValues.Set("arn", tgtArn)
}
urlValues.Set("reset-id", resetID)
// Execute GET on bucket to get replication config.
resp, err := c.executeMethod(ctx, http.MethodPut, requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
})
defer closeResponse(resp)
if err != nil {
return rinfo, err
}
if resp.StatusCode != http.StatusOK {
return rinfo, httpRespToErrorResponse(resp, bucketName, "")
}
if err = json.NewDecoder(resp.Body).Decode(&rinfo); err != nil {
return rinfo, err
}
return rinfo, nil
}
// GetBucketReplicationResyncStatus retrieves the status of a replication resync operation.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucketName: Name of the bucket
// - arn: ARN of the replication target (empty string for all targets)
//
// Returns resync status information or an error if the operation fails.
func (c *Client) GetBucketReplicationResyncStatus(ctx context.Context, bucketName, arn string) (rinfo replication.ResyncTargetsInfo, err error) {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return rinfo, err
}
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("replication-reset-status", "")
if arn != "" {
urlValues.Set("arn", arn)
}
// Execute GET on bucket to get replication config.
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
})
defer closeResponse(resp)
if err != nil {
return rinfo, err
}
if resp.StatusCode != http.StatusOK {
return rinfo, httpRespToErrorResponse(resp, bucketName, "")
}
if err = json.NewDecoder(resp.Body).Decode(&rinfo); err != nil {
return rinfo, err
}
return rinfo, nil
}
// CancelBucketReplicationResync cancels an in-progress replication resync operation.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucketName: Name of the bucket
// - tgtArn: ARN of the replication target (empty string for all targets)
//
// Returns the ID of the canceled resync operation or an error if the operation fails.
func (c *Client) CancelBucketReplicationResync(ctx context.Context, bucketName string, tgtArn string) (id string, err error) {
// Input validation.
if err = s3utils.CheckValidBucketName(bucketName); err != nil {
return id, err
}
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("replication-reset-cancel", "")
if tgtArn != "" {
urlValues.Set("arn", tgtArn)
}
// Execute GET on bucket to get replication config.
resp, err := c.executeMethod(ctx, http.MethodPut, requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
})
defer closeResponse(resp)
if err != nil {
return id, err
}
if resp.StatusCode != http.StatusOK {
return id, httpRespToErrorResponse(resp, bucketName, "")
}
strBuf, err := io.ReadAll(resp.Body)
if err != nil {
return "", err
}
id = string(strBuf)
return id, nil
}
// GetBucketReplicationMetricsV2 retrieves bucket replication status metrics using the V2 API.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucketName: Name of the bucket
//
// Returns the V2 replication metrics or an error if the operation fails.
func (c *Client) GetBucketReplicationMetricsV2(ctx context.Context, bucketName string) (s replication.MetricsV2, err error) {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return s, err
}
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("replication-metrics", "2")
// Execute GET on bucket to get replication metrics.
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
})
defer closeResponse(resp)
if err != nil {
return s, err
}
if resp.StatusCode != http.StatusOK {
return s, httpRespToErrorResponse(resp, bucketName, "")
}
respBytes, err := io.ReadAll(resp.Body)
if err != nil {
return s, err
}
if err := json.Unmarshal(respBytes, &s); err != nil {
return s, err
}
return s, nil
}
// CheckBucketReplication validates whether replication is properly configured for a bucket.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucketName: Name of the bucket
//
// Returns nil if replication is valid, or an error describing the validation failure.
func (c *Client) CheckBucketReplication(ctx context.Context, bucketName string) (err error) {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return err
}
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("replication-check", "")
// Execute GET on bucket to get replication config.
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
})
defer closeResponse(resp)
if err != nil {
return err
}
if resp.StatusCode != http.StatusOK {
return httpRespToErrorResponse(resp, bucketName, "")
}
return nil
}
Server/vendor/github.com/minio/minio-go/v7/api-bucket-tagging.go
Generated Vendored
+153
View File
@@ -0,0 +1,153 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2020 MinIO, Inc.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"bytes"
"context"
"encoding/xml"
"errors"
"io"
"net/http"
"net/url"
"github.com/minio/minio-go/v7/pkg/s3utils"
"github.com/minio/minio-go/v7/pkg/tags"
)
// GetBucketTagging fetches the tagging configuration for a bucket.
// It uses the provided context to control cancellations and timeouts.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucketName: Name of the bucket
//
// Returns the bucket's tags or an error if the operation fails.
func (c *Client) GetBucketTagging(ctx context.Context, bucketName string) (*tags.Tags, error) {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return nil, err
}
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("tagging", "")
// Execute GET on bucket to get tagging configuration.
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
})
defer closeResponse(resp)
if err != nil {
return nil, err
}
if resp.StatusCode != http.StatusOK {
return nil, httpRespToErrorResponse(resp, bucketName, "")
}
defer io.Copy(io.Discard, resp.Body)
return tags.ParseBucketXML(resp.Body)
}
// SetBucketTagging sets the tagging configuration for a bucket.
// It uses the provided context to control cancellations and timeouts.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucketName: Name of the bucket
// - tags: Tag set to apply to the bucket
//
// Returns an error if the operation fails or if tags is nil.
func (c *Client) SetBucketTagging(ctx context.Context, bucketName string, tags *tags.Tags) error {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return err
}
if tags == nil {
return errors.New("nil tags passed")
}
buf, err := xml.Marshal(tags)
if err != nil {
return err
}
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("tagging", "")
// Content-length is mandatory to set a default encryption configuration
reqMetadata := requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
contentBody: bytes.NewReader(buf),
contentLength: int64(len(buf)),
contentMD5Base64: sumMD5Base64(buf),
}
// Execute PUT on bucket to put tagging configuration.
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
defer closeResponse(resp)
if err != nil {
return err
}
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusNoContent {
return httpRespToErrorResponse(resp, bucketName, "")
}
return nil
}
// RemoveBucketTagging removes the tagging configuration from a bucket.
// It uses the provided context to control cancellations and timeouts.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucketName: Name of the bucket
//
// Returns an error if the operation fails.
func (c *Client) RemoveBucketTagging(ctx context.Context, bucketName string) error {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return err
}
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("tagging", "")
// Execute DELETE on bucket to remove tagging configuration.
resp, err := c.executeMethod(ctx, http.MethodDelete, requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
contentSHA256Hex: emptySHA256Hex,
})
defer closeResponse(resp)
if err != nil {
return err
}
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusNoContent {
return httpRespToErrorResponse(resp, bucketName, "")
}
return nil
}
Server/vendor/github.com/minio/minio-go/v7/api-bucket-versioning.go
Generated Vendored
+147
View File
@@ -0,0 +1,147 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2020 MinIO, Inc.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"bytes"
"context"
"encoding/xml"
"net/http"
"net/url"
"github.com/minio/minio-go/v7/pkg/s3utils"
)
// SetBucketVersioning sets a bucket versioning configuration
func (c *Client) SetBucketVersioning(ctx context.Context, bucketName string, config BucketVersioningConfiguration) error {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return err
}
buf, err := xml.Marshal(config)
if err != nil {
return err
}
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("versioning", "")
reqMetadata := requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
contentBody: bytes.NewReader(buf),
contentLength: int64(len(buf)),
contentMD5Base64: sumMD5Base64(buf),
contentSHA256Hex: sum256Hex(buf),
}
// Execute PUT to set a bucket versioning.
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
defer closeResponse(resp)
if err != nil {
return err
}
if resp != nil {
if resp.StatusCode != http.StatusOK {
return httpRespToErrorResponse(resp, bucketName, "")
}
}
return nil
}
// EnableVersioning - enable object versioning in given bucket.
func (c *Client) EnableVersioning(ctx context.Context, bucketName string) error {
return c.SetBucketVersioning(ctx, bucketName, BucketVersioningConfiguration{Status: "Enabled"})
}
// SuspendVersioning - suspend object versioning in given bucket.
func (c *Client) SuspendVersioning(ctx context.Context, bucketName string) error {
return c.SetBucketVersioning(ctx, bucketName, BucketVersioningConfiguration{Status: "Suspended"})
}
// ExcludedPrefix - holds individual prefixes excluded from being versioned.
type ExcludedPrefix struct {
Prefix string
}
// BucketVersioningConfiguration is the versioning configuration structure
type BucketVersioningConfiguration struct {
XMLName xml.Name `xml:"VersioningConfiguration"`
Status string `xml:"Status"`
MFADelete string `xml:"MfaDelete,omitempty"`
// MinIO extension - allows selective, prefix-level versioning exclusion.
// Requires versioning to be enabled
ExcludedPrefixes []ExcludedPrefix `xml:",omitempty"`
ExcludeFolders bool `xml:",omitempty"`
PurgeOnDelete string `xml:",omitempty"`
}
// Various supported states
const (
Enabled = "Enabled"
// Disabled State = "Disabled" only used by MFA Delete not supported yet.
Suspended = "Suspended"
)
// Enabled returns true if bucket versioning is enabled
func (b BucketVersioningConfiguration) Enabled() bool {
return b.Status == Enabled
}
// Suspended returns true if bucket versioning is suspended
func (b BucketVersioningConfiguration) Suspended() bool {
return b.Status == Suspended
}
// GetBucketVersioning gets the versioning configuration on
// an existing bucket with a context to control cancellations and timeouts.
func (c *Client) GetBucketVersioning(ctx context.Context, bucketName string) (BucketVersioningConfiguration, error) {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return BucketVersioningConfiguration{}, err
}
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("versioning", "")
// Execute GET on bucket to get the versioning configuration.
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
})
defer closeResponse(resp)
if err != nil {
return BucketVersioningConfiguration{}, err
}
if resp.StatusCode != http.StatusOK {
return BucketVersioningConfiguration{}, httpRespToErrorResponse(resp, bucketName, "")
}
versioningConfig := BucketVersioningConfiguration{}
if err = xmlDecoder(resp.Body, &versioningConfig); err != nil {
return versioningConfig, err
}
return versioningConfig, nil
}
Server/vendor/github.com/minio/minio-go/v7/api-compose-object.go
Generated Vendored
+631
View File
@@ -0,0 +1,631 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2017, 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"context"
"fmt"
"io"
"net/http"
"net/url"
"strconv"
"strings"
"time"
"github.com/google/uuid"
"github.com/minio/minio-go/v7/pkg/encrypt"
"github.com/minio/minio-go/v7/pkg/s3utils"
"github.com/minio/minio-go/v7/pkg/tags"
)
// CopyDestOptions represents options specified by user for CopyObject/ComposeObject APIs
type CopyDestOptions struct {
Bucket string // points to destination bucket
Object string // points to destination object
// `Encryption` is the key info for server-side-encryption with customer
// provided key. If it is nil, no encryption is performed.
Encryption encrypt.ServerSide
ChecksumType ChecksumType
// `userMeta` is the user-metadata key-value pairs to be set on the
// destination. The keys are automatically prefixed with `x-amz-meta-`
// if needed. If nil is passed, and if only a single source (of any
// size) is provided in the ComposeObject call, then metadata from the
// source is copied to the destination.
// if no user-metadata is provided, it is copied from source
// (when there is only one source object in the compose
// request)
UserMetadata map[string]string
// UserMetadata is only set to destination if ReplaceMetadata is true
// other value is UserMetadata is ignored and we preserve src.UserMetadata
// NOTE: if you set this value to true and now metadata is present
// in UserMetadata your destination object will not have any metadata
// set.
ReplaceMetadata bool
// `userTags` is the user defined object tags to be set on destination.
// This will be set only if the `replaceTags` field is set to true.
// Otherwise this field is ignored
UserTags map[string]string
ReplaceTags bool
// Specifies whether you want to apply a Legal Hold to the copied object.
LegalHold LegalHoldStatus
// Object Retention related fields
Mode RetentionMode
RetainUntilDate time.Time
Expires time.Time
ContentType string
ContentEncoding string
ContentDisposition string
ContentLanguage string
CacheControl string
Size int64 // Needs to be specified if progress bar is specified.
// Progress of the entire copy operation will be sent here.
Progress io.Reader
// PartSize specifies the part size for multipart copy operations.
// If not specified, defaults to maxPartSize (5 GiB).
PartSize uint64
}
// Process custom-metadata to remove a `x-amz-meta-` prefix if
// present and validate that keys are distinct (after this
// prefix removal).
func filterCustomMeta(userMeta map[string]string) map[string]string {
m := make(map[string]string)
for k, v := range userMeta {
if strings.HasPrefix(strings.ToLower(k), "x-amz-meta-") {
k = k[len("x-amz-meta-"):]
}
if _, ok := m[k]; ok {
continue
}
m[k] = v
}
return m
}
// Marshal converts all the CopyDestOptions into their
// equivalent HTTP header representation
func (opts CopyDestOptions) Marshal(header http.Header) {
const replaceDirective = "REPLACE"
if opts.ReplaceTags {
header.Set(amzTaggingHeaderDirective, replaceDirective)
if tags, _ := tags.NewTags(opts.UserTags, true); tags != nil {
header.Set(amzTaggingHeader, tags.String())
}
}
if opts.LegalHold != LegalHoldStatus("") {
header.Set(amzLegalHoldHeader, opts.LegalHold.String())
}
if opts.Mode != RetentionMode("") && !opts.RetainUntilDate.IsZero() {
header.Set(amzLockMode, opts.Mode.String())
header.Set(amzLockRetainUntil, opts.RetainUntilDate.Format(time.RFC3339))
}
if opts.Encryption != nil {
opts.Encryption.Marshal(header)
}
if opts.ContentType != "" {
header.Set("Content-Type", opts.ContentType)
}
if opts.ContentEncoding != "" {
header.Set("Content-Encoding", opts.ContentEncoding)
}
if opts.ContentDisposition != "" {
header.Set("Content-Disposition", opts.ContentDisposition)
}
if opts.ContentLanguage != "" {
header.Set("Content-Language", opts.ContentLanguage)
}
if opts.CacheControl != "" {
header.Set("Cache-Control", opts.CacheControl)
}
if !opts.Expires.IsZero() {
header.Set("Expires", opts.Expires.UTC().Format(http.TimeFormat))
}
if opts.ChecksumType.IsSet() {
header.Set(amzChecksumAlgo, opts.ChecksumType.String())
}
if opts.ReplaceMetadata {
header.Set("x-amz-metadata-directive", replaceDirective)
for k, v := range filterCustomMeta(opts.UserMetadata) {
if isAmzHeader(k) || isStandardHeader(k) || isStorageClassHeader(k) || isMinioHeader(k) {
header.Set(k, v)
} else {
header.Set("x-amz-meta-"+k, v)
}
}
}
}
// toDestinationInfo returns a validated copyOptions object.
func (opts CopyDestOptions) validate() (err error) {
// Input validation.
if err = s3utils.CheckValidBucketName(opts.Bucket); err != nil {
return err
}
if err = s3utils.CheckValidObjectName(opts.Object); err != nil {
return err
}
if opts.Progress != nil && opts.Size < 0 {
return errInvalidArgument("For progress bar effective size needs to be specified")
}
return nil
}
// CopySrcOptions represents a source object to be copied, using
// server-side copying APIs.
type CopySrcOptions struct {
Bucket, Object string
VersionID string
MatchETag string
NoMatchETag string
MatchModifiedSince time.Time
MatchUnmodifiedSince time.Time
MatchRange bool
Start, End int64
Encryption encrypt.ServerSide
}
// Marshal converts all the CopySrcOptions into their
// equivalent HTTP header representation
func (opts CopySrcOptions) Marshal(header http.Header) {
// Set the source header
header.Set("x-amz-copy-source", s3utils.EncodePath(opts.Bucket+"/"+opts.Object))
if opts.VersionID != "" {
header.Set("x-amz-copy-source", s3utils.EncodePath(opts.Bucket+"/"+opts.Object)+"?versionId="+opts.VersionID)
}
if opts.MatchETag != "" {
header.Set("x-amz-copy-source-if-match", opts.MatchETag)
}
if opts.NoMatchETag != "" {
header.Set("x-amz-copy-source-if-none-match", opts.NoMatchETag)
}
if !opts.MatchModifiedSince.IsZero() {
header.Set("x-amz-copy-source-if-modified-since", opts.MatchModifiedSince.Format(http.TimeFormat))
}
if !opts.MatchUnmodifiedSince.IsZero() {
header.Set("x-amz-copy-source-if-unmodified-since", opts.MatchUnmodifiedSince.Format(http.TimeFormat))
}
if opts.Encryption != nil {
encrypt.SSECopy(opts.Encryption).Marshal(header)
}
}
func (opts CopySrcOptions) validate() (err error) {
// Input validation.
if err = s3utils.CheckValidBucketName(opts.Bucket); err != nil {
return err
}
if err = s3utils.CheckValidObjectName(opts.Object); err != nil {
return err
}
if opts.Start > opts.End || opts.Start < 0 {
return errInvalidArgument("start must be non-negative, and start must be at most end.")
}
return nil
}
// Low level implementation of CopyObject API, supports only upto 5GiB worth of copy.
func (c *Client) copyObjectDo(ctx context.Context, srcBucket, srcObject, destBucket, destObject string,
metadata map[string]string, srcOpts CopySrcOptions, dstOpts PutObjectOptions,
) (ObjectInfo, error) {
// Build headers.
headers := make(http.Header)
// Set all the metadata headers.
for k, v := range metadata {
headers.Set(k, v)
}
if !dstOpts.Internal.ReplicationStatus.Empty() {
headers.Set(amzBucketReplicationStatus, string(dstOpts.Internal.ReplicationStatus))
}
if !dstOpts.Internal.SourceMTime.IsZero() {
headers.Set(minIOBucketSourceMTime, dstOpts.Internal.SourceMTime.Format(time.RFC3339Nano))
}
if dstOpts.Internal.SourceETag != "" {
headers.Set(minIOBucketSourceETag, dstOpts.Internal.SourceETag)
}
if dstOpts.Internal.ReplicationRequest {
headers.Set(minIOBucketReplicationRequest, "true")
}
if dstOpts.Internal.ReplicationValidityCheck {
headers.Set(minIOBucketReplicationCheck, "true")
}
if !dstOpts.Internal.LegalholdTimestamp.IsZero() {
headers.Set(minIOBucketReplicationObjectLegalHoldTimestamp, dstOpts.Internal.LegalholdTimestamp.Format(time.RFC3339Nano))
}
if !dstOpts.Internal.RetentionTimestamp.IsZero() {
headers.Set(minIOBucketReplicationObjectRetentionTimestamp, dstOpts.Internal.RetentionTimestamp.Format(time.RFC3339Nano))
}
if !dstOpts.Internal.TaggingTimestamp.IsZero() {
headers.Set(minIOBucketReplicationTaggingTimestamp, dstOpts.Internal.TaggingTimestamp.Format(time.RFC3339Nano))
}
if len(dstOpts.UserTags) != 0 {
if tags, _ := tags.NewTags(dstOpts.UserTags, true); tags != nil {
headers.Set(amzTaggingHeader, tags.String())
}
}
reqMetadata := requestMetadata{
bucketName: destBucket,
objectName: destObject,
customHeader: headers,
}
if dstOpts.Internal.SourceVersionID != "" {
if dstOpts.Internal.SourceVersionID != nullVersionID {
if _, err := uuid.Parse(dstOpts.Internal.SourceVersionID); err != nil {
return ObjectInfo{}, errInvalidArgument(err.Error())
}
}
urlValues := make(url.Values)
urlValues.Set("versionId", dstOpts.Internal.SourceVersionID)
reqMetadata.queryValues = urlValues
}
// Set the source header
headers.Set("x-amz-copy-source", s3utils.EncodePath(srcBucket+"/"+srcObject))
if srcOpts.VersionID != "" {
headers.Set("x-amz-copy-source", s3utils.EncodePath(srcBucket+"/"+srcObject)+"?versionId="+srcOpts.VersionID)
}
// Send upload-part-copy request
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
defer closeResponse(resp)
if err != nil {
return ObjectInfo{}, err
}
// Check if we got an error response.
if resp.StatusCode != http.StatusOK {
return ObjectInfo{}, httpRespToErrorResponse(resp, srcBucket, srcObject)
}
cpObjRes := copyObjectResult{}
err = xmlDecoder(resp.Body, &cpObjRes)
if err != nil {
return ObjectInfo{}, err
}
objInfo := ObjectInfo{
Key: destObject,
ETag: strings.Trim(cpObjRes.ETag, "\""),
LastModified: cpObjRes.LastModified,
}
return objInfo, nil
}
func (c *Client) copyObjectPartDo(ctx context.Context, srcBucket, srcObject, destBucket, destObject, uploadID string,
partID int, startOffset, length int64, metadata map[string]string,
) (p CompletePart, err error) {
headers := make(http.Header)
// Set source
headers.Set("x-amz-copy-source", s3utils.EncodePath(srcBucket+"/"+srcObject))
if startOffset < 0 {
return p, errInvalidArgument("startOffset must be non-negative")
}
if length >= 0 {
headers.Set("x-amz-copy-source-range", fmt.Sprintf("bytes=%d-%d", startOffset, startOffset+length-1))
}
for k, v := range metadata {
headers.Set(k, v)
}
queryValues := make(url.Values)
queryValues.Set("partNumber", strconv.Itoa(partID))
queryValues.Set("uploadId", uploadID)
resp, err := c.executeMethod(ctx, http.MethodPut, requestMetadata{
bucketName: destBucket,
objectName: destObject,
customHeader: headers,
queryValues: queryValues,
})
defer closeResponse(resp)
if err != nil {
return p, err
}
// Check if we got an error response.
if resp.StatusCode != http.StatusOK {
return p, httpRespToErrorResponse(resp, destBucket, destObject)
}
// Decode copy-part response on success.
cpObjRes := copyObjectResult{}
err = xmlDecoder(resp.Body, &cpObjRes)
if err != nil {
return p, err
}
p.PartNumber, p.ETag = partID, cpObjRes.ETag
return p, nil
}
// uploadPartCopy - helper function to create a part in a multipart
// upload via an upload-part-copy request
// https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPartCopy.html
func (c *Client) uploadPartCopy(ctx context.Context, bucket, object, uploadID string, partNumber int,
headers http.Header,
) (p CompletePart, err error) {
// Build query parameters
urlValues := make(url.Values)
urlValues.Set("partNumber", strconv.Itoa(partNumber))
urlValues.Set("uploadId", uploadID)
// Send upload-part-copy request
resp, err := c.executeMethod(ctx, http.MethodPut, requestMetadata{
bucketName: bucket,
objectName: object,
customHeader: headers,
queryValues: urlValues,
})
defer closeResponse(resp)
if err != nil {
return p, err
}
// Check if we got an error response.
if resp.StatusCode != http.StatusOK {
return p, httpRespToErrorResponse(resp, bucket, object)
}
// Decode copy-part response on success.
cpObjRes := copyObjectResult{}
err = xmlDecoder(resp.Body, &cpObjRes)
if err != nil {
return p, err
}
p.PartNumber, p.ETag = partNumber, cpObjRes.ETag
return p, nil
}
// ComposeObject - creates an object using server-side copying
// of existing objects. It takes a list of source objects (with optional offsets)
// and concatenates them into a new object using only server-side copying
// operations. Optionally takes progress reader hook for applications to
// look at current progress.
func (c *Client) ComposeObject(ctx context.Context, dst CopyDestOptions, srcs ...CopySrcOptions) (UploadInfo, error) {
if len(srcs) < 1 || len(srcs) > maxPartsCount {
return UploadInfo{}, errInvalidArgument("There must be as least one and up to 10000 source objects.")
}
for _, src := range srcs {
if err := src.validate(); err != nil {
return UploadInfo{}, err
}
}
if err := dst.validate(); err != nil {
return UploadInfo{}, err
}
srcObjectInfos := make([]ObjectInfo, len(srcs))
srcObjectSizes := make([]int64, len(srcs))
var totalSize, totalParts int64
var err error
for i, src := range srcs {
opts := StatObjectOptions{ServerSideEncryption: encrypt.SSE(src.Encryption), VersionID: src.VersionID}
srcObjectInfos[i], err = c.StatObject(context.Background(), src.Bucket, src.Object, opts)
if err != nil {
return UploadInfo{}, err
}
srcCopySize := srcObjectInfos[i].Size
// Check if a segment is specified, and if so, is the
// segment within object bounds?
if src.MatchRange {
// Since range is specified,
// 0 <= src.start <= src.end
// so only invalid case to check is:
if src.End >= srcCopySize || src.Start < 0 {
return UploadInfo{}, errInvalidArgument(
fmt.Sprintf("CopySrcOptions %d has invalid segment-to-copy [%d, %d] (size is %d)",
i, src.Start, src.End, srcCopySize))
}
srcCopySize = src.End - src.Start + 1
}
// Only the last source may be less than `absMinPartSize`
if srcCopySize < absMinPartSize && i < len(srcs)-1 {
return UploadInfo{}, errInvalidArgument(
fmt.Sprintf("CopySrcOptions %d is too small (%d) and it is not the last part", i, srcCopySize))
}
// Is data to copy too large?
totalSize += srcCopySize
if totalSize > maxObjectSize {
return UploadInfo{}, errInvalidArgument(fmt.Sprintf("Cannot compose an object of size %d (> 5GiB * 10000)", totalSize))
}
// record source size
srcObjectSizes[i] = srcCopySize
// calculate parts needed for current source
totalParts += partsRequired(srcCopySize, int64(dst.PartSize))
// Do we need more parts than we are allowed?
if totalParts > maxPartsCount {
return UploadInfo{}, errInvalidArgument(fmt.Sprintf(
"Your proposed compose object requires more than %d parts", maxPartsCount))
}
}
// Single source object case (i.e. when only one source is
// involved, it is being copied wholly and at most 5GiB in
// size, emptyfiles are also supported).
if (totalParts == 1 && srcs[0].Start == -1 && totalSize <= maxPartSize) || (totalSize == 0) {
return c.CopyObject(ctx, dst, srcs[0])
}
// Now, handle multipart-copy cases.
// 1. Ensure that the object has not been changed while
// we are copying data.
for i, src := range srcs {
src.MatchETag = srcObjectInfos[i].ETag
}
// 2. Initiate a new multipart upload.
// Set user-metadata on the destination object. If no
// user-metadata is specified, and there is only one source,
// (only) then metadata from source is copied.
var userMeta map[string]string
if dst.ReplaceMetadata {
userMeta = dst.UserMetadata
} else {
userMeta = srcObjectInfos[0].UserMetadata
}
var userTags map[string]string
if dst.ReplaceTags {
userTags = dst.UserTags
} else {
userTags = srcObjectInfos[0].UserTags
}
uploadID, err := c.newUploadID(ctx, dst.Bucket, dst.Object, PutObjectOptions{
ServerSideEncryption: dst.Encryption,
UserMetadata: userMeta,
UserTags: userTags,
Mode: dst.Mode,
RetainUntilDate: dst.RetainUntilDate,
LegalHold: dst.LegalHold,
})
if err != nil {
return UploadInfo{}, err
}
// 3. Perform copy part uploads
objParts := []CompletePart{}
partIndex := 1
for i, src := range srcs {
h := make(http.Header)
src.Marshal(h)
if dst.Encryption != nil && dst.Encryption.Type() == encrypt.SSEC {
dst.Encryption.Marshal(h)
}
// calculate start/end indices of parts after
// splitting.
startIdx, endIdx := calculateEvenSplits(srcObjectSizes[i], src, int64(dst.PartSize))
for j, start := range startIdx {
end := endIdx[j]
// Add (or reset) source range header for
// upload part copy request.
h.Set("x-amz-copy-source-range",
fmt.Sprintf("bytes=%d-%d", start, end))
// make upload-part-copy request
complPart, err := c.uploadPartCopy(ctx, dst.Bucket,
dst.Object, uploadID, partIndex, h)
if err != nil {
return UploadInfo{}, err
}
if dst.Progress != nil {
io.CopyN(io.Discard, dst.Progress, end-start+1)
}
objParts = append(objParts, complPart)
partIndex++
}
}
// 4. Make final complete-multipart request.
uploadInfo, err := c.completeMultipartUpload(ctx, dst.Bucket, dst.Object, uploadID,
completeMultipartUpload{Parts: objParts}, PutObjectOptions{ServerSideEncryption: dst.Encryption})
if err != nil {
return UploadInfo{}, err
}
uploadInfo.Size = totalSize
return uploadInfo, nil
}
// partsRequired calculates the number of parts needed for a given size
// using the specified part size. If partSize is 0, defaults to maxPartSize (5 GiB).
func partsRequired(size int64, partSize int64) int64 {
if partSize == 0 {
partSize = maxPartSize
}
r := size / partSize
if size%partSize > 0 {
r++
}
return r
}
// calculateEvenSplits - computes splits for a source and returns
// start and end index slices. Splits happen evenly to be sure that no
// part is less than 5MiB, as that could fail the multipart request if
// it is not the last part. If partSize is 0, defaults to maxPartSize (5 GiB).
func calculateEvenSplits(size int64, src CopySrcOptions, partSize int64) (startIndex, endIndex []int64) {
if size == 0 {
return startIndex, endIndex
}
reqParts := partsRequired(size, partSize)
startIndex = make([]int64, reqParts)
endIndex = make([]int64, reqParts)
// Compute number of required parts `k`, as:
//
// k = ceiling(size / copyPartSize)
//
// Now, distribute the `size` bytes in the source into
// k parts as evenly as possible:
//
// r parts sized (q+1) bytes, and
// (k - r) parts sized q bytes, where
//
// size = q * k + r (by simple division of size by k,
// so that 0 <= r < k)
//
start := src.Start
if start == -1 {
start = 0
}
quot, rem := size/reqParts, size%reqParts
nextStart := start
for j := int64(0); j < reqParts; j++ {
curPartSize := quot
if j < rem {
curPartSize++
}
cStart := nextStart
cEnd := cStart + curPartSize - 1
nextStart = cEnd + 1
startIndex[j], endIndex[j] = cStart, cEnd
}
return startIndex, endIndex
}
Server/vendor/github.com/minio/minio-go/v7/api-copy-object.go
Generated Vendored
+76
View File
@@ -0,0 +1,76 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2017, 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"context"
"io"
"net/http"
)
// CopyObject - copy a source object into a new object
func (c *Client) CopyObject(ctx context.Context, dst CopyDestOptions, src CopySrcOptions) (UploadInfo, error) {
if err := src.validate(); err != nil {
return UploadInfo{}, err
}
if err := dst.validate(); err != nil {
return UploadInfo{}, err
}
header := make(http.Header)
dst.Marshal(header)
src.Marshal(header)
resp, err := c.executeMethod(ctx, http.MethodPut, requestMetadata{
bucketName: dst.Bucket,
objectName: dst.Object,
customHeader: header,
})
if err != nil {
return UploadInfo{}, err
}
defer closeResponse(resp)
if resp.StatusCode != http.StatusOK {
return UploadInfo{}, httpRespToErrorResponse(resp, dst.Bucket, dst.Object)
}
// Update the progress properly after successful copy.
if dst.Progress != nil {
io.Copy(io.Discard, io.LimitReader(dst.Progress, dst.Size))
}
cpObjRes := copyObjectResult{}
if err = xmlDecoder(resp.Body, &cpObjRes); err != nil {
return UploadInfo{}, err
}
// extract lifecycle expiry date and rule ID
expTime, ruleID := amzExpirationToExpiryDateRuleID(resp.Header.Get(amzExpiration))
return UploadInfo{
Bucket: dst.Bucket,
Key: dst.Object,
LastModified: cpObjRes.LastModified,
ETag: trimEtag(cpObjRes.ETag),
VersionID: resp.Header.Get(amzVersionID),
Expiration: expTime,
ExpirationRuleID: ruleID,
}, nil
}
Server/vendor/github.com/minio/minio-go/v7/api-datatypes.go
Generated Vendored
+274
View File
@@ -0,0 +1,274 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2015-2020 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"encoding/xml"
"io"
"net/http"
"net/url"
"strings"
"time"
)
// BucketInfo container for bucket metadata.
type BucketInfo struct {
// The name of the bucket.
Name string `json:"name"`
// Date the bucket was created.
CreationDate time.Time `json:"creationDate"`
// BucketRegion region where the bucket is present
BucketRegion string `json:"bucketRegion"`
}
// StringMap represents map with custom UnmarshalXML
type StringMap map[string]string
// UnmarshalXML unmarshals the XML into a map of string to strings,
// creating a key in the map for each tag and setting it's value to the
// tags contents.
//
// The fact this function is on the pointer of Map is important, so that
// if m is nil it can be initialized, which is often the case if m is
// nested in another xml structural. This is also why the first thing done
// on the first line is initialize it.
func (m *StringMap) UnmarshalXML(d *xml.Decoder, _ xml.StartElement) error {
*m = StringMap{}
for {
// Format is <key>value</key>
var e struct {
XMLName xml.Name
Value string `xml:",chardata"`
}
err := d.Decode(&e)
if err == io.EOF {
break
}
if err != nil {
return err
}
(*m)[e.XMLName.Local] = e.Value
}
return nil
}
// URLMap represents map with custom UnmarshalXML
type URLMap map[string]string
// UnmarshalXML unmarshals the XML into a map of string to strings,
// creating a key in the map for each tag and setting it's value to the
// tags contents.
//
// The fact this function is on the pointer of Map is important, so that
// if m is nil it can be initialized, which is often the case if m is
// nested in another xml structural. This is also why the first thing done
// on the first line is initialize it.
func (m *URLMap) UnmarshalXML(d *xml.Decoder, se xml.StartElement) error {
*m = URLMap{}
var tgs string
if err := d.DecodeElement(&tgs, &se); err != nil {
if err == io.EOF {
return nil
}
return err
}
for tgs != "" {
var key string
key, tgs, _ = stringsCut(tgs, "&")
if key == "" {
continue
}
key, value, _ := stringsCut(key, "=")
key, err := url.QueryUnescape(key)
if err != nil {
return err
}
value, err = url.QueryUnescape(value)
if err != nil {
return err
}
(*m)[key] = value
}
return nil
}
// stringsCut slices s around the first instance of sep,
// returning the text before and after sep.
// The found result reports whether sep appears in s.
// If sep does not appear in s, cut returns s, "", false.
func stringsCut(s, sep string) (before, after string, found bool) {
if i := strings.Index(s, sep); i >= 0 {
return s[:i], s[i+len(sep):], true
}
return s, "", false
}
// Owner name.
type Owner struct {
XMLName xml.Name `xml:"Owner" json:"owner"`
DisplayName string `xml:"ID" json:"name"`
ID string `xml:"DisplayName" json:"id"`
}
// UploadInfo contains information about the
// newly uploaded or copied object.
type UploadInfo struct {
Bucket string
Key string
ETag string
Size int64
LastModified time.Time
Location string
VersionID string
// Lifecycle expiry-date and ruleID associated with the expiry
// not to be confused with `Expires` HTTP header.
Expiration time.Time
ExpirationRuleID string
// Verified checksum values, if any.
// Values are base64 (standard) encoded.
// For multipart objects this is a checksum of the checksum of each part.
ChecksumCRC32 string
ChecksumCRC32C string
ChecksumSHA1 string
ChecksumSHA256 string
ChecksumCRC64NVME string
ChecksumMD5 string
ChecksumSHA512 string
ChecksumXXHash64 string
ChecksumXXHash3 string
ChecksumXXHash128 string
ChecksumMode string
}
// RestoreInfo contains information of the restore operation of an archived object
type RestoreInfo struct {
// Is the restoring operation is still ongoing
OngoingRestore bool
// When the restored copy of the archived object will be removed
ExpiryTime time.Time
}
// ObjectInfo container for object metadata.
type ObjectInfo struct {
// An ETag is optionally set to md5sum of an object. In case of multipart objects,
// ETag is of the form MD5SUM-N where MD5SUM is md5sum of all individual md5sums of
// each parts concatenated into one string.
ETag string `json:"etag"`
Key string `json:"name"` // Name of the object
LastModified time.Time `json:"lastModified"` // Date and time the object was last modified.
Size int64 `json:"size"` // Size in bytes of the object.
ContentType string `json:"contentType"` // A standard MIME type describing the format of the object data.
ContentEncoding string `json:"contentEncoding"` // A standard MIME type describing encoding of the object data.
Expires time.Time `json:"expires"` // The date and time at which the object is no longer able to be cached.
// Collection of additional metadata on the object.
// eg: x-amz-meta-*, content-encoding etc.
Metadata http.Header `json:"metadata" xml:"-"`
// x-amz-meta-* headers stripped "x-amz-meta-" prefix containing the first value.
// Only returned by MinIO servers.
UserMetadata StringMap `json:"userMetadata,omitempty"`
// x-amz-tagging values in their k/v values.
// Only returned by MinIO servers.
UserTags URLMap `json:"userTags,omitempty" xml:"UserTags"`
// x-amz-tagging-count value
UserTagCount int
// Owner name.
Owner Owner
// ACL grant.
Grant []Grant
// The class of storage used to store the object.
StorageClass string `json:"storageClass"`
// Versioning related information
IsLatest bool
IsDeleteMarker bool
VersionID string `xml:"VersionId"`
// x-amz-replication-status value is either in one of the following states
// - COMPLETED
// - PENDING
// - FAILED
// - REPLICA (on the destination)
ReplicationStatus string `xml:"ReplicationStatus"`
// set to true if delete marker has backing object version on target, and eligible to replicate
ReplicationReady bool
// Lifecycle expiry-date and ruleID associated with the expiry
// not to be confused with `Expires` HTTP header.
Expiration time.Time
ExpirationRuleID string
// NumVersions is the number of versions of the object.
NumVersions int
Restore *RestoreInfo
// Checksum values
ChecksumCRC32 string
ChecksumCRC32C string
ChecksumSHA1 string
ChecksumSHA256 string
ChecksumCRC64NVME string
ChecksumMD5 string
ChecksumSHA512 string
ChecksumXXHash64 string
ChecksumXXHash3 string
ChecksumXXHash128 string
ChecksumAlgorithm string
ChecksumMode string `xml:"ChecksumType"`
Internal *struct {
K int // Data blocks
M int // Parity blocks
} `xml:"Internal"`
// Error
Err error `json:"-"`
}
// ObjectMultipartInfo container for multipart object metadata.
type ObjectMultipartInfo struct {
// Date and time at which the multipart upload was initiated.
Initiated time.Time `type:"timestamp" timestampFormat:"iso8601"`
Initiator initiator
Owner owner
// The type of storage to use for the object. Defaults to 'STANDARD'.
StorageClass string
// Key of the object for which the multipart upload was initiated.
Key string
// Size in bytes of the object.
Size int64
// Upload ID that identifies the multipart upload.
UploadID string `xml:"UploadId"`
// Error
Err error
}
Server/vendor/github.com/minio/minio-go/v7/api-error-response.go
Generated Vendored
+295
View File
@@ -0,0 +1,295 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2015-2020 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"bytes"
"encoding/xml"
"errors"
"fmt"
"io"
"net/http"
"strings"
)
/* **** SAMPLE ERROR RESPONSE ****
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<BucketName>bucketName</BucketName>
<Key>objectName</Key>
<RequestId>F19772218238A85A</RequestId>
<HostId>GuWkjyviSiGHizehqpmsD1ndz5NClSP19DOT+s2mv7gXGQ8/X1lhbDGiIJEXpGFD</HostId>
</Error>
*/
// ErrorResponse - Is the typed error returned by all API operations.
// ErrorResponse struct should be comparable since it is compared inside
// golang http API (https://github.com/golang/go/issues/29768)
type ErrorResponse struct {
XMLName xml.Name `xml:"Error" json:"-"`
Code string
Message string
BucketName string
Key string
Resource string
RequestID string `xml:"RequestId"`
HostID string `xml:"HostId"`
// Region where the bucket is located. This header is returned
// only in HEAD bucket and ListObjects response.
Region string
// Captures the server string returned in response header.
Server string
// Underlying HTTP status code for the returned error
StatusCode int `xml:"-" json:"-"`
}
// ToErrorResponse - Returns parsed ErrorResponse struct from body and
// http headers.
//
// For example:
//
// import s3 "github.com/minio/minio-go/v7"
// ...
// ...
// reader, stat, err := s3.GetObject(...)
// if err != nil {
// resp := s3.ToErrorResponse(err)
// }
// ...
func ToErrorResponse(err error) ErrorResponse {
switch err := err.(type) {
case ErrorResponse:
return err
default:
return ErrorResponse{}
}
}
// Error - Returns S3 error string.
func (e ErrorResponse) Error() string {
if e.Message == "" {
msg, ok := s3ErrorResponseMap[e.Code]
if !ok {
msg = fmt.Sprintf("Error response code %s.", e.Code)
}
return msg
}
return e.Message
}
// Common string for errors to report issue location in unexpected
// cases.
const (
reportIssue = "Please report this issue at https://github.com/minio/minio-go/issues."
)
// xmlDecodeAndBody reads the whole body up to 1MB and
// tries to XML decode it into v.
// The body that was read and any error from reading or decoding is returned.
func xmlDecodeAndBody(bodyReader io.Reader, v interface{}) ([]byte, error) {
// read the whole body (up to 1MB)
const maxBodyLength = 1 << 20
body, err := io.ReadAll(io.LimitReader(bodyReader, maxBodyLength))
if err != nil {
return nil, err
}
return bytes.TrimSpace(body), xmlDecoder(bytes.NewReader(body), v)
}
// httpRespToErrorResponse returns a new encoded ErrorResponse
// structure as error.
func httpRespToErrorResponse(resp *http.Response, bucketName, objectName string) error {
if resp == nil {
msg := "Empty http response. " + reportIssue
return errInvalidArgument(msg)
}
errResp := ErrorResponse{
StatusCode: resp.StatusCode,
Server: resp.Header.Get("Server"),
}
success := successStatus.Contains(resp.StatusCode)
errBody, err := xmlDecodeAndBody(resp.Body, &errResp)
// Xml decoding failed with no body, fall back to HTTP headers.
if err != nil {
var unmarshalErr xml.UnmarshalError
if success && errors.As(err, &unmarshalErr) {
// This is a successful message so not an error response
// return nil,
return nil
}
switch resp.StatusCode {
case http.StatusNotFound:
if objectName == "" {
errResp = ErrorResponse{
StatusCode: resp.StatusCode,
Code: NoSuchBucket,
Message: s3ErrorResponseMap[NoSuchBucket],
BucketName: bucketName,
}
} else {
errResp = ErrorResponse{
StatusCode: resp.StatusCode,
Code: NoSuchKey,
Message: s3ErrorResponseMap[NoSuchKey],
BucketName: bucketName,
Key: objectName,
}
}
case http.StatusForbidden:
errResp = ErrorResponse{
StatusCode: resp.StatusCode,
Code: AccessDenied,
Message: s3ErrorResponseMap[AccessDenied],
BucketName: bucketName,
Key: objectName,
}
case http.StatusConflict:
errResp = ErrorResponse{
StatusCode: resp.StatusCode,
Code: Conflict,
Message: s3ErrorResponseMap[Conflict],
BucketName: bucketName,
}
case http.StatusPreconditionFailed:
errResp = ErrorResponse{
StatusCode: resp.StatusCode,
Code: PreconditionFailed,
Message: s3ErrorResponseMap[PreconditionFailed],
BucketName: bucketName,
Key: objectName,
}
default:
msg := resp.Status
if len(errBody) > 0 {
msg = string(errBody)
if len(msg) > 1024 {
msg = msg[:1024] + "..."
}
}
errResp = ErrorResponse{
StatusCode: resp.StatusCode,
Code: resp.Status,
Message: msg,
BucketName: bucketName,
}
}
}
code := resp.Header.Get("x-minio-error-code")
if code != "" {
errResp.Code = code
}
desc := resp.Header.Get("x-minio-error-desc")
if desc != "" {
errResp.Message = strings.Trim(desc, `"`)
}
// Save hostID, requestID and region information
// from headers if not available through error XML.
if errResp.RequestID == "" {
errResp.RequestID = resp.Header.Get("x-amz-request-id")
}
if errResp.HostID == "" {
errResp.HostID = resp.Header.Get("x-amz-id-2")
}
if errResp.Region == "" {
errResp.Region = resp.Header.Get("x-amz-bucket-region")
}
if errResp.Code == InvalidRegion && errResp.Region != "" {
errResp.Message = fmt.Sprintf("Region does not match, expecting region ‘%s’.", errResp.Region)
}
return errResp
}
// errTransferAccelerationBucket - bucket name is invalid to be used with transfer acceleration.
func errTransferAccelerationBucket(bucketName string) error {
msg := "The name of the bucket used for Transfer Acceleration must be DNS-compliant and must not contain periods ‘.’."
return ErrorResponse{
StatusCode: http.StatusBadRequest,
Code: InvalidArgument,
Message: msg,
BucketName: bucketName,
}
}
// errEntityTooLarge - Input size is larger than supported maximum.
func errEntityTooLarge(totalSize, maxObjectSize int64, bucketName, objectName string) error {
msg := fmt.Sprintf("Your proposed upload size ‘%d’ exceeds the maximum allowed object size ‘%d’ for single PUT operation.", totalSize, maxObjectSize)
return ErrorResponse{
StatusCode: http.StatusBadRequest,
Code: EntityTooLarge,
Message: msg,
BucketName: bucketName,
Key: objectName,
}
}
// errEntityTooSmall - Input size is smaller than supported minimum.
func errEntityTooSmall(totalSize int64, bucketName, objectName string) error {
msg := fmt.Sprintf("Your proposed upload size ‘%d’ is below the minimum allowed object size ‘0B’ for single PUT operation.", totalSize)
return ErrorResponse{
StatusCode: http.StatusBadRequest,
Code: EntityTooSmall,
Message: msg,
BucketName: bucketName,
Key: objectName,
}
}
// errUnexpectedEOF - Unexpected end of file reached.
func errUnexpectedEOF(totalRead, totalSize int64, bucketName, objectName string) error {
msg := fmt.Sprintf("Data read ‘%d’ is not equal to the size ‘%d’ of the input Reader.", totalRead, totalSize)
return ErrorResponse{
StatusCode: http.StatusBadRequest,
Code: UnexpectedEOF,
Message: msg,
BucketName: bucketName,
Key: objectName,
}
}
// errInvalidArgument - Invalid argument response.
func errInvalidArgument(message string) error {
return ErrorResponse{
StatusCode: http.StatusBadRequest,
Code: InvalidArgument,
Message: message,
RequestID: "minio",
}
}
// errAPINotSupported - API not supported response
// The specified API call is not supported
func errAPINotSupported(message string) error {
return ErrorResponse{
StatusCode: http.StatusNotImplemented,
Code: APINotSupported,
Message: message,
RequestID: "minio",
}
}
Server/vendor/github.com/minio/minio-go/v7/api-get-object-acl.go
Generated Vendored
+152
View File
@@ -0,0 +1,152 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"context"
"encoding/xml"
"net/http"
"net/url"
)
// Grantee represents the person being granted permissions.
type Grantee struct {
XMLName xml.Name `xml:"Grantee"`
ID string `xml:"ID"`
DisplayName string `xml:"DisplayName"`
URI string `xml:"URI"`
}
// Grant holds grant information
type Grant struct {
XMLName xml.Name `xml:"Grant"`
Grantee Grantee
Permission string `xml:"Permission"`
}
// AccessControlList contains the set of grantees and the permissions assigned to each grantee.
type AccessControlList struct {
XMLName xml.Name `xml:"AccessControlList"`
Grant []Grant
Permission string `xml:"Permission"`
}
type accessControlPolicy struct {
XMLName xml.Name `xml:"AccessControlPolicy"`
Owner Owner
AccessControlList AccessControlList
}
// GetObjectACL get object ACLs
func (c *Client) GetObjectACL(ctx context.Context, bucketName, objectName string) (*ObjectInfo, error) {
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
bucketName: bucketName,
objectName: objectName,
queryValues: url.Values{
"acl": []string{""},
},
})
if err != nil {
return nil, err
}
defer closeResponse(resp)
if resp.StatusCode != http.StatusOK {
return nil, httpRespToErrorResponse(resp, bucketName, objectName)
}
res := &accessControlPolicy{}
if err := xmlDecoder(resp.Body, res); err != nil {
return nil, err
}
objInfo, err := c.StatObject(ctx, bucketName, objectName, StatObjectOptions{})
if err != nil {
return nil, err
}
objInfo.Owner.DisplayName = res.Owner.DisplayName
objInfo.Owner.ID = res.Owner.ID
objInfo.Grant = append(objInfo.Grant, res.AccessControlList.Grant...)
cannedACL := getCannedACL(res)
if cannedACL != "" {
objInfo.Metadata.Add("X-Amz-Acl", cannedACL)
return &objInfo, nil
}
grantACL := getAmzGrantACL(res)
for k, v := range grantACL {
objInfo.Metadata[k] = v
}
return &objInfo, nil
}
func getCannedACL(aCPolicy *accessControlPolicy) string {
grants := aCPolicy.AccessControlList.Grant
switch {
case len(grants) == 1:
if grants[0].Grantee.URI == "" && grants[0].Permission == "FULL_CONTROL" {
return "private"
}
case len(grants) == 2:
for _, g := range grants {
if g.Grantee.URI == "http://acs.amazonaws.com/groups/global/AuthenticatedUsers" && g.Permission == "READ" {
return "authenticated-read"
}
if g.Grantee.URI == "http://acs.amazonaws.com/groups/global/AllUsers" && g.Permission == "READ" {
return "public-read"
}
if g.Permission == "READ" && g.Grantee.ID == aCPolicy.Owner.ID {
return "bucket-owner-read"
}
}
case len(grants) == 3:
for _, g := range grants {
if g.Grantee.URI == "http://acs.amazonaws.com/groups/global/AllUsers" && g.Permission == "WRITE" {
return "public-read-write"
}
}
}
return ""
}
func getAmzGrantACL(aCPolicy *accessControlPolicy) map[string][]string {
grants := aCPolicy.AccessControlList.Grant
res := map[string][]string{}
for _, g := range grants {
switch g.Permission {
case "READ":
res["X-Amz-Grant-Read"] = append(res["X-Amz-Grant-Read"], "id="+g.Grantee.ID)
case "WRITE":
res["X-Amz-Grant-Write"] = append(res["X-Amz-Grant-Write"], "id="+g.Grantee.ID)
case "READ_ACP":
res["X-Amz-Grant-Read-Acp"] = append(res["X-Amz-Grant-Read-Acp"], "id="+g.Grantee.ID)
case "WRITE_ACP":
res["X-Amz-Grant-Write-Acp"] = append(res["X-Amz-Grant-Write-Acp"], "id="+g.Grantee.ID)
case "FULL_CONTROL":
res["X-Amz-Grant-Full-Control"] = append(res["X-Amz-Grant-Full-Control"], "id="+g.Grantee.ID)
}
}
return res
}
Server/vendor/github.com/minio/minio-go/v7/api-get-object-attributes.go
Generated Vendored
+287
View File
@@ -0,0 +1,287 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2020 MinIO, Inc.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"context"
"encoding/xml"
"errors"
"net/http"
"net/url"
"strconv"
"strings"
"time"
"github.com/minio/minio-go/v7/pkg/encrypt"
"github.com/minio/minio-go/v7/pkg/s3utils"
)
// ObjectAttributesOptions are options used for the GetObjectAttributes API
//
// - MaxParts
// How many parts the caller wants to be returned (default: 1000)
//
// - VersionID
// The object version you want to attributes for
//
// - PartNumberMarker
// the listing will start AFTER the part matching PartNumberMarker
//
// - ServerSideEncryption
// The server-side encryption algorithm used when storing this object in Minio
type ObjectAttributesOptions struct {
MaxParts int
VersionID string
PartNumberMarker int
ServerSideEncryption encrypt.ServerSide
}
// ObjectAttributes is the response object returned by the GetObjectAttributes API
//
// - VersionID
// The object version
//
// - LastModified
// The last time the object was modified
//
// - ObjectAttributesResponse
// Contains more information about the object
type ObjectAttributes struct {
VersionID string
LastModified time.Time
ObjectAttributesResponse
}
// ObjectAttributesResponse contains details returned by the GetObjectAttributes API
//
// Noteworthy fields:
//
// - ObjectParts.PartsCount
// Contains the total part count for the object (not the current response)
//
// - ObjectParts.PartNumberMarker
// Pagination of parts will begin at (but not include) PartNumberMarker
//
// - ObjectParts.NextPartNumberMarket
// The next PartNumberMarker to be used in order to continue pagination
//
// - ObjectParts.IsTruncated
// Indicates if the last part is included in the request (does not check if parts are missing from the start of the list, ONLY the end)
//
// - ObjectParts.MaxParts
// Reflects the MaxParts used by the caller or the default MaxParts value of the API
type ObjectAttributesResponse struct {
ETag string `xml:",omitempty"`
StorageClass string
ObjectSize int
Checksum struct {
ChecksumCRC32 string `xml:",omitempty"`
ChecksumCRC32C string `xml:",omitempty"`
ChecksumCRC64NVME string `xml:",omitempty"`
ChecksumSHA1 string `xml:",omitempty"`
ChecksumSHA256 string `xml:",omitempty"`
ChecksumMD5 string `xml:",omitempty"`
ChecksumSHA512 string `xml:",omitempty"`
ChecksumXXHash64 string `xml:"ChecksumXXHASH64,omitempty"`
ChecksumXXHash3 string `xml:"ChecksumXXHASH3,omitempty"`
ChecksumXXHash128 string `xml:"ChecksumXXHASH128,omitempty"`
ChecksumType string `xml:",omitempty"`
}
ObjectParts struct {
PartsCount int
PartNumberMarker int
NextPartNumberMarker int
MaxParts int
IsTruncated bool
Parts []*ObjectAttributePart `xml:"Part"`
}
}
// ObjectAttributePart is used by ObjectAttributesResponse to describe an object part
type ObjectAttributePart struct {
ChecksumCRC32 string `xml:",omitempty"`
ChecksumCRC32C string `xml:",omitempty"`
ChecksumCRC64NVME string `xml:",omitempty"`
ChecksumSHA1 string `xml:",omitempty"`
ChecksumSHA256 string `xml:",omitempty"`
ChecksumMD5 string `xml:",omitempty"`
ChecksumSHA512 string `xml:",omitempty"`
ChecksumXXHash64 string `xml:"ChecksumXXHASH64,omitempty"`
ChecksumXXHash3 string `xml:"ChecksumXXHASH3,omitempty"`
ChecksumXXHash128 string `xml:"ChecksumXXHASH128,omitempty"`
PartNumber int
Size int
}
// ChecksumMap returns a map of checksums for the object.
func (o *ObjectAttributesResponse) ChecksumMap() map[string]string {
res := make(map[string]string)
setif := func(typ ChecksumType, value string) {
if value != "" {
res[typ.Key()] = value
}
}
setif(ChecksumCRC32C, o.Checksum.ChecksumCRC32C)
setif(ChecksumCRC32, o.Checksum.ChecksumCRC32)
setif(ChecksumCRC64NVME, o.Checksum.ChecksumCRC64NVME)
setif(ChecksumSHA1, o.Checksum.ChecksumSHA1)
setif(ChecksumSHA256, o.Checksum.ChecksumSHA256)
setif(ChecksumMD5, o.Checksum.ChecksumMD5)
setif(ChecksumSHA512, o.Checksum.ChecksumSHA512)
setif(ChecksumXXHash64, o.Checksum.ChecksumXXHash64)
setif(ChecksumXXHash3, o.Checksum.ChecksumXXHash3)
setif(ChecksumXXHash128, o.Checksum.ChecksumXXHash128)
return res
}
// ChecksumMode returns the checksum mode of the object.
// If unable to determine, returns ChecksumUnknownMode.
func (o *ObjectAttributesResponse) ChecksumMode() ChecksumMode {
t := o.ChecksumType()
if !t.IsSet() {
return ChecksumUnknownMode
}
switch o.Checksum.ChecksumType {
case amzChecksumModeComposite:
return ChecksumCompositeMode
case amzChecksumModeFullObject:
return ChecksumFullObjectMode
case "":
// Likely not supported by the server.
if o.Checksum.ChecksumCRC64NVME != "" || !strings.ContainsRune(o.ETag, '-') {
// Always full object.
return ChecksumFullObjectMode
}
if !t.CanMergeCRC() {
// Only composite possible.
return ChecksumCompositeMode
}
}
return ChecksumUnknownMode
}
// ChecksumType returns the checksum type of the object.
// If none is set, returns ChecksumNone.
func (o *ObjectAttributesResponse) ChecksumType() ChecksumType {
t := ChecksumNone
setif := func(typ ChecksumType, value string) {
if value != "" {
t = typ
}
}
setif(ChecksumCRC32C, o.Checksum.ChecksumCRC32C)
setif(ChecksumCRC32, o.Checksum.ChecksumCRC32)
setif(ChecksumCRC64NVME, o.Checksum.ChecksumCRC64NVME)
setif(ChecksumSHA1, o.Checksum.ChecksumSHA1)
setif(ChecksumSHA256, o.Checksum.ChecksumSHA256)
setif(ChecksumMD5, o.Checksum.ChecksumMD5)
setif(ChecksumSHA512, o.Checksum.ChecksumSHA512)
setif(ChecksumXXHash64, o.Checksum.ChecksumXXHash64)
setif(ChecksumXXHash3, o.Checksum.ChecksumXXHash3)
setif(ChecksumXXHash128, o.Checksum.ChecksumXXHash128)
if t.IsSet() && o.Checksum.ChecksumType == amzChecksumModeFullObject {
t |= ChecksumFullObject
}
return t
}
func (o *ObjectAttributes) parseResponse(resp *http.Response) (err error) {
mod, err := parseRFC7231Time(resp.Header.Get("Last-Modified"))
if err != nil {
return err
}
o.LastModified = mod
o.VersionID = resp.Header.Get(amzVersionID)
response := new(ObjectAttributesResponse)
if err := xml.NewDecoder(resp.Body).Decode(response); err != nil {
return err
}
o.ObjectAttributesResponse = *response
return err
}
// GetObjectAttributes API combines HeadObject and ListParts.
// More details on usage can be found in the documentation for ObjectAttributesOptions{}
func (c *Client) GetObjectAttributes(ctx context.Context, bucketName, objectName string, opts ObjectAttributesOptions) (*ObjectAttributes, error) {
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return nil, err
}
if err := s3utils.CheckValidObjectName(objectName); err != nil {
return nil, err
}
urlValues := make(url.Values)
urlValues.Add("attributes", "")
if opts.VersionID != "" {
urlValues.Add("versionId", opts.VersionID)
}
headers := make(http.Header)
headers.Set(amzObjectAttributes, GetObjectAttributesTags)
if opts.PartNumberMarker > 0 {
headers.Set(amzPartNumberMarker, strconv.Itoa(opts.PartNumberMarker))
}
if opts.MaxParts > 0 {
headers.Set(amzMaxParts, strconv.Itoa(opts.MaxParts))
} else {
headers.Set(amzMaxParts, strconv.Itoa(GetObjectAttributesMaxParts))
}
if opts.ServerSideEncryption != nil {
opts.ServerSideEncryption.Marshal(headers)
}
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
bucketName: bucketName,
objectName: objectName,
queryValues: urlValues,
contentSHA256Hex: emptySHA256Hex,
customHeader: headers,
})
if err != nil {
return nil, err
}
defer closeResponse(resp)
hasEtag := resp.Header.Get(ETag)
if hasEtag != "" {
return nil, errors.New("getObjectAttributes is not supported by the current endpoint version")
}
if resp.StatusCode != http.StatusOK {
ER := new(ErrorResponse)
if err := xml.NewDecoder(resp.Body).Decode(ER); err != nil {
return nil, err
}
return nil, *ER
}
OA := new(ObjectAttributes)
err = OA.parseResponse(resp)
if err != nil {
return nil, err
}
return OA, nil
}
Server/vendor/github.com/minio/minio-go/v7/api-get-object-file.go
Generated Vendored
+127
View File
@@ -0,0 +1,127 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2015-2017 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"context"
"io"
"os"
"path/filepath"
"github.com/minio/minio-go/v7/pkg/s3utils"
)
// FGetObject - download contents of an object to a local file.
// The options can be used to specify the GET request further.
func (c *Client) FGetObject(ctx context.Context, bucketName, objectName, filePath string, opts GetObjectOptions) error {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return err
}
if err := s3utils.CheckValidObjectName(objectName); err != nil {
return err
}
// Verify if destination already exists.
st, err := os.Stat(filePath)
if err == nil {
// If the destination exists and is a directory.
if st.IsDir() {
return errInvalidArgument("fileName is a directory.")
}
}
// Proceed if file does not exist. return for all other errors.
if err != nil {
if !os.IsNotExist(err) {
return err
}
}
// Extract top level directory.
objectDir, _ := filepath.Split(filePath)
if objectDir != "" {
// Create any missing top level directories.
if err := os.MkdirAll(objectDir, 0o700); err != nil {
return err
}
}
// Gather md5sum.
objectStat, err := c.StatObject(ctx, bucketName, objectName, StatObjectOptions(opts))
if err != nil {
return err
}
// Write to a temporary file "fileName.part.minio" before saving.
filePartPath := filepath.Join(filepath.Dir(filePath), sum256Hex([]byte(filepath.Base(filePath)+objectStat.ETag))+".part.minio")
// If exists, open in append mode. If not create it as a part file.
filePart, err := os.OpenFile(filePartPath, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0o600)
if err != nil {
return err
}
// If we return early with an error, be sure to close and delete
// filePart. If we have an error along the way there is a chance
// that filePart is somehow damaged, and we should discard it.
closeAndRemove := true
defer func() {
if closeAndRemove {
_ = filePart.Close()
_ = os.Remove(filePartPath)
}
}()
// Issue Stat to get the current offset.
st, err = filePart.Stat()
if err != nil {
return err
}
// Initialize get object request headers to set the
// appropriate range offsets to read from.
if st.Size() > 0 {
opts.SetRange(st.Size(), 0)
}
// Seek to current position for incoming reader.
objectReader, objectStat, _, err := c.getObject(ctx, bucketName, objectName, opts)
if err != nil {
return err
}
// Write to the part file.
if _, err = io.CopyN(filePart, objectReader, objectStat.Size); err != nil {
return err
}
// Close the file before rename, this is specifically needed for Windows users.
closeAndRemove = false
if err = filePart.Close(); err != nil {
return err
}
// Safely completed. Now commit by renaming to actual filename.
if err = os.Rename(filePartPath, filePath); err != nil {
return err
}
// Return.
return nil
}
Server/vendor/github.com/minio/minio-go/v7/api-get-object.go
Generated Vendored
+706
View File
@@ -0,0 +1,706 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2015-2020 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"context"
"errors"
"fmt"
"io"
"net/http"
"sync"
"github.com/minio/minio-go/v7/pkg/s3utils"
)
// GetObject wrapper function that accepts a request context
func (c *Client) GetObject(ctx context.Context, bucketName, objectName string, opts GetObjectOptions) (*Object, error) {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return nil, ErrorResponse{
StatusCode: http.StatusBadRequest,
Code: InvalidBucketName,
Message: err.Error(),
}
}
if err := s3utils.CheckValidObjectName(objectName); err != nil {
return nil, ErrorResponse{
StatusCode: http.StatusBadRequest,
Code: XMinioInvalidObjectName,
Message: err.Error(),
}
}
gctx, cancel := context.WithCancel(ctx)
// Detect if snowball is server location we are talking to.
var snowball bool
if location, ok := c.bucketLocCache.Get(bucketName); ok {
snowball = location == "snowball"
}
var (
err error
httpReader io.ReadCloser
objectInfo ObjectInfo
totalRead int
)
// Create request channel.
reqCh := make(chan getRequest)
// Create response channel.
resCh := make(chan getResponse)
// This routine feeds partial object data as and when the caller reads.
go func() {
defer close(resCh)
defer func() {
// Close the http response body before returning.
// This ends the connection with the server.
if httpReader != nil {
httpReader.Close()
}
}()
defer cancel()
// Used to verify if etag of object has changed since last read.
var etag string
for req := range reqCh {
// If this is the first request we may not need to do a getObject request yet.
if req.isFirstReq {
// First request is a Read/ReadAt.
if req.isReadOp {
// Differentiate between wanting the whole object and just a range.
if req.isReadAt {
// If this is a ReadAt request only get the specified range.
// Range is set with respect to the offset and length of the buffer requested.
// Do not set objectInfo from the first readAt request because it will not get
// the whole object.
opts.SetRange(req.Offset, req.Offset+int64(len(req.Buffer))-1)
} else if req.Offset > 0 {
opts.SetRange(req.Offset, 0)
}
httpReader, objectInfo, _, err = c.getObject(gctx, bucketName, objectName, opts)
if err != nil {
resCh <- getResponse{Error: err}
return
}
etag = objectInfo.ETag
// Read at least firstReq.Buffer bytes, if not we have
// reached our EOF.
size, err := readFull(httpReader, req.Buffer)
totalRead += size
if size > 0 && err == io.ErrUnexpectedEOF {
if int64(size) < objectInfo.Size {
// In situations when returned size
// is less than the expected content
// length set by the server, make sure
// we return io.ErrUnexpectedEOF
err = io.ErrUnexpectedEOF
} else {
// If an EOF happens after reading some but not
// all the bytes ReadFull returns ErrUnexpectedEOF
err = io.EOF
}
} else if size == 0 && err == io.EOF && objectInfo.Size > 0 {
// Special cases when server writes more data
// than the content-length, net/http response
// body returns an error, instead of converting
// it to io.EOF - return unexpected EOF.
err = io.ErrUnexpectedEOF
}
// Send back the first response.
resCh <- getResponse{
objectInfo: objectInfo,
Size: size,
Error: err,
didRead: true,
}
} else {
// First request is a Stat or Seek call.
// Only need to run a StatObject until an actual Read or ReadAt request comes through.
// Remove range header if already set, for stat Operations to get original file size.
delete(opts.headers, "Range")
objectInfo, err = c.StatObject(gctx, bucketName, objectName, StatObjectOptions(opts))
if err != nil {
resCh <- getResponse{
Error: err,
}
// Exit the go-routine.
return
}
etag = objectInfo.ETag
// Send back the first response.
resCh <- getResponse{
objectInfo: objectInfo,
}
}
} else if req.settingObjectInfo { // Request is just to get objectInfo.
// Remove range header if already set, for stat Operations to get original file size.
delete(opts.headers, "Range")
// Check whether this is snowball
// if yes do not use If-Match feature
// it doesn't work.
if etag != "" && !snowball {
opts.SetMatchETag(etag)
}
objectInfo, err := c.StatObject(gctx, bucketName, objectName, StatObjectOptions(opts))
if err != nil {
resCh <- getResponse{
Error: err,
}
// Exit the goroutine.
return
}
// Send back the objectInfo.
resCh <- getResponse{
objectInfo: objectInfo,
}
} else {
// Offset changes fetch the new object at an Offset.
// Because the httpReader may not be set by the first
// request if it was a stat or seek it must be checked
// if the object has been read or not to only initialize
// new ones when they haven't been already.
// All readAt requests are new requests.
if req.DidOffsetChange || !req.beenRead {
// Check whether this is snowball
// if yes do not use If-Match feature
// it doesn't work.
if etag != "" && !snowball {
opts.SetMatchETag(etag)
}
if httpReader != nil {
// Close previously opened http reader.
httpReader.Close()
}
// If this request is a readAt only get the specified range.
if req.isReadAt {
// Range is set with respect to the offset and length of the buffer requested.
opts.SetRange(req.Offset, req.Offset+int64(len(req.Buffer))-1)
} else if req.Offset > 0 { // Range is set with respect to the offset.
opts.SetRange(req.Offset, 0)
} else {
// Remove range header if already set
delete(opts.headers, "Range")
}
httpReader, objectInfo, _, err = c.getObject(gctx, bucketName, objectName, opts)
if err != nil {
resCh <- getResponse{
Error: err,
}
return
}
totalRead = 0
}
// Read at least req.Buffer bytes, if not we have
// reached our EOF.
size, err := readFull(httpReader, req.Buffer)
totalRead += size
if size > 0 && err == io.ErrUnexpectedEOF {
if int64(totalRead) < objectInfo.Size {
// In situations when returned size
// is less than the expected content
// length set by the server, make sure
// we return io.ErrUnexpectedEOF
err = io.ErrUnexpectedEOF
} else {
// If an EOF happens after reading some but not
// all the bytes ReadFull returns ErrUnexpectedEOF
err = io.EOF
}
} else if size == 0 && err == io.EOF && objectInfo.Size > 0 {
// Special cases when server writes more data
// than the content-length, net/http response
// body returns an error, instead of converting
// it to io.EOF - return unexpected EOF.
err = io.ErrUnexpectedEOF
}
// Reply back how much was read.
resCh <- getResponse{
Size: size,
Error: err,
didRead: true,
objectInfo: objectInfo,
}
}
}
}()
// Create a newObject through the information sent back by reqCh.
return newObject(gctx, cancel, reqCh, resCh), nil
}
// get request message container to communicate with internal
// go-routine.
type getRequest struct {
Buffer []byte
Offset int64 // readAt offset.
DidOffsetChange bool // Tracks the offset changes for Seek requests.
beenRead bool // Determines if this is the first time an object is being read.
isReadAt bool // Determines if this request is a request to a specific range
isReadOp bool // Determines if this request is a Read or Read/At request.
isFirstReq bool // Determines if this request is the first time an object is being accessed.
settingObjectInfo bool // Determines if this request is to set the objectInfo of an object.
}
// get response message container to reply back for the request.
type getResponse struct {
Size int
Error error
didRead bool // Lets subsequent calls know whether or not httpReader has been initiated.
objectInfo ObjectInfo // Used for the first request.
}
// Object represents an open object. It implements
// Reader, ReaderAt, Seeker, Closer for a HTTP stream.
type Object struct {
// Mutex.
mutex *sync.Mutex
// User allocated and defined.
reqCh chan<- getRequest
resCh <-chan getResponse
ctx context.Context
cancel context.CancelFunc
currOffset int64
objectInfo ObjectInfo
// Ask lower level to initiate data fetching based on currOffset
seekData bool
// Keeps track of closed call.
isClosed bool
// Keeps track of if this is the first call.
isStarted bool
// Previous error saved for future calls.
prevErr error
// Keeps track of if this object has been read yet.
beenRead bool
// Keeps track of if objectInfo has been set yet.
objectInfoSet bool
}
// doGetRequest - sends and blocks on the firstReqCh and reqCh of an object.
// Returns back the size of the buffer read, if anything was read, as well
// as any error encountered. For all first requests sent on the object
// it is also responsible for sending back the objectInfo.
func (o *Object) doGetRequest(request getRequest) (getResponse, error) {
select {
case <-o.ctx.Done():
return getResponse{}, o.ctx.Err()
case o.reqCh <- request:
}
response := <-o.resCh
// Return any error to the top level.
if response.Error != nil && response.Error != io.EOF {
return response, response.Error
}
// This was the first request.
if !o.isStarted {
// The object has been operated on.
o.isStarted = true
}
// Set the objectInfo if the request was not readAt
// and it hasn't been set before.
if !o.objectInfoSet && !request.isReadAt {
o.objectInfo = response.objectInfo
o.objectInfoSet = true
}
// Set beenRead only if it has not been set before.
if !o.beenRead {
o.beenRead = response.didRead
}
// Data are ready on the wire, no need to reinitiate connection in lower level
o.seekData = false
return response, response.Error
}
// setOffset - handles the setting of offsets for
// Read/ReadAt/Seek requests.
func (o *Object) setOffset(bytesRead int64) error {
// Update the currentOffset.
o.currOffset += bytesRead
if o.objectInfo.Size > -1 && o.currOffset >= o.objectInfo.Size {
return io.EOF
}
return nil
}
// Read reads up to len(b) bytes into b. It returns the number of
// bytes read (0 <= n <= len(b)) and any error encountered. Returns
// io.EOF upon end of file.
func (o *Object) Read(b []byte) (n int, err error) {
if o == nil {
return 0, errInvalidArgument("Object is nil")
}
// Locking.
o.mutex.Lock()
defer o.mutex.Unlock()
// prevErr is previous error saved from previous operation.
if o.prevErr != nil || o.isClosed {
return 0, o.prevErr
}
// Create a new request.
readReq := getRequest{
isReadOp: true,
beenRead: o.beenRead,
Buffer: b,
}
// Alert that this is the first request.
if !o.isStarted {
readReq.isFirstReq = true
}
// Ask to establish a new data fetch routine based on seekData flag
readReq.DidOffsetChange = o.seekData
readReq.Offset = o.currOffset
// Send and receive from the first request.
response, err := o.doGetRequest(readReq)
if err != nil && err != io.EOF {
// Save the error for future calls.
o.prevErr = err
return response.Size, err
}
// Bytes read.
bytesRead := int64(response.Size)
// Set the new offset.
oerr := o.setOffset(bytesRead)
if oerr != nil {
// Save the error for future calls.
o.prevErr = oerr
return response.Size, oerr
}
// Return the response.
return response.Size, err
}
// Stat returns the ObjectInfo structure describing Object.
func (o *Object) Stat() (ObjectInfo, error) {
if o == nil {
return ObjectInfo{}, errInvalidArgument("Object is nil")
}
// Locking.
o.mutex.Lock()
defer o.mutex.Unlock()
if o.prevErr != nil && o.prevErr != io.EOF || o.isClosed {
return ObjectInfo{}, o.prevErr
}
// This is the first request.
if !o.isStarted || !o.objectInfoSet {
// Send the request and get the response.
_, err := o.doGetRequest(getRequest{
isFirstReq: !o.isStarted,
settingObjectInfo: !o.objectInfoSet,
})
if err != nil {
o.prevErr = err
return ObjectInfo{}, err
}
}
return o.objectInfo, nil
}
// ReadAt reads len(b) bytes from the File starting at byte offset
// off. It returns the number of bytes read and the error, if any.
// ReadAt always returns a non-nil error when n < len(b). At end of
// file, that error is io.EOF.
func (o *Object) ReadAt(b []byte, offset int64) (n int, err error) {
if o == nil {
return 0, errInvalidArgument("Object is nil")
}
// Locking.
o.mutex.Lock()
defer o.mutex.Unlock()
// prevErr is error which was saved in previous operation.
if o.prevErr != nil && o.prevErr != io.EOF || o.isClosed {
return 0, o.prevErr
}
// Save and restore currOffset so ReadAt doesn't affect sequential Read operations.
// Per io.ReaderAt: "ReadAt should not affect nor be affected by the underlying seek offset."
savedOffset := o.currOffset
defer func() {
o.currOffset = savedOffset
o.seekData = true // Force next Read to re-establish stream at correct position
}()
o.currOffset = offset
// Can only compare offsets to size when size has been set.
if o.objectInfoSet {
// If offset is negative than we return io.EOF.
// If offset is greater than or equal to object size we return io.EOF.
if (o.objectInfo.Size > -1 && offset >= o.objectInfo.Size) || offset < 0 {
return 0, io.EOF
}
}
// Create the new readAt request.
readAtReq := getRequest{
isReadOp: true,
isReadAt: true,
DidOffsetChange: true, // Offset always changes.
beenRead: o.beenRead, // Set if this is the first request to try and read.
Offset: offset, // Set the offset.
Buffer: b,
}
// Alert that this is the first request.
if !o.isStarted {
readAtReq.isFirstReq = true
}
// Send and receive from the first request.
response, err := o.doGetRequest(readAtReq)
if err != nil && err != io.EOF {
// Save the error.
o.prevErr = err
return response.Size, err
}
// Bytes read.
bytesRead := int64(response.Size)
// There is no valid objectInfo yet
// to compare against for EOF.
if !o.objectInfoSet {
// Update the currentOffset.
o.currOffset += bytesRead
} else {
// If this was not the first request update
// the offsets and compare against objectInfo
// for EOF.
oerr := o.setOffset(bytesRead)
if oerr != nil {
o.prevErr = oerr
return response.Size, oerr
}
}
return response.Size, err
}
// Seek sets the offset for the next Read or Write to offset,
// interpreted according to whence: 0 means relative to the
// origin of the file, 1 means relative to the current offset,
// and 2 means relative to the end.
// Seek returns the new offset and an error, if any.
//
// Seeking to a negative offset is an error. Seeking to any positive
// offset is legal, subsequent io operations succeed until the
// underlying object is not closed.
func (o *Object) Seek(offset int64, whence int) (n int64, err error) {
if o == nil {
return 0, errInvalidArgument("Object is nil")
}
// Locking.
o.mutex.Lock()
defer o.mutex.Unlock()
// At EOF seeking is legal allow only io.EOF, for any other errors we return.
if o.prevErr != nil && o.prevErr != io.EOF {
return 0, o.prevErr
}
// Negative offset is valid for whence of '2'.
if offset < 0 && whence != 2 {
return 0, errInvalidArgument(fmt.Sprintf("Negative position not allowed for %d", whence))
}
// This is the first request. So before anything else
// get the ObjectInfo.
if !o.isStarted || !o.objectInfoSet {
// Create the new Seek request.
seekReq := getRequest{
isReadOp: false,
Offset: offset,
isFirstReq: true,
}
// Send and receive from the seek request.
_, err := o.doGetRequest(seekReq)
if err != nil {
// Save the error.
o.prevErr = err
return 0, err
}
}
newOffset := o.currOffset
// Switch through whence.
switch whence {
default:
return 0, errInvalidArgument(fmt.Sprintf("Invalid whence %d", whence))
case 0:
if o.objectInfo.Size > -1 && offset > o.objectInfo.Size {
return 0, io.EOF
}
newOffset = offset
case 1:
if o.objectInfo.Size > -1 && o.currOffset+offset > o.objectInfo.Size {
return 0, io.EOF
}
newOffset += offset
case 2:
// If we don't know the object size return an error for io.SeekEnd
if o.objectInfo.Size < 0 {
return 0, errInvalidArgument("Whence END is not supported when the object size is unknown")
}
// Seeking to positive offset is valid for whence '2', but
// since we are backing a Reader we have reached 'EOF' if
// offset is positive.
if offset > 0 {
return 0, io.EOF
}
// Seeking to negative position not allowed for whence.
if o.objectInfo.Size+offset < 0 {
return 0, errInvalidArgument(fmt.Sprintf("Seeking at negative offset not allowed for %d", whence))
}
newOffset = o.objectInfo.Size + offset
}
// Reset the saved error since we successfully seeked, let the Read
// and ReadAt decide.
if o.prevErr == io.EOF {
o.prevErr = nil
}
// Ask lower level to fetch again from source when necessary
o.seekData = (newOffset != o.currOffset) || o.seekData
o.currOffset = newOffset
// Return the effective offset.
return o.currOffset, nil
}
// Close - The behavior of Close after the first call returns error
// for subsequent Close() calls.
func (o *Object) Close() (err error) {
if o == nil {
return errInvalidArgument("Object is nil")
}
// Locking.
o.mutex.Lock()
defer o.mutex.Unlock()
// if already closed return an error.
if o.isClosed {
return o.prevErr
}
// Close successfully.
o.cancel()
// Close the request channel to indicate the internal go-routine to exit.
close(o.reqCh)
// Save for future operations.
errMsg := "Object is already closed. Bad file descriptor."
o.prevErr = errors.New(errMsg)
// Save here that we closed done channel successfully.
o.isClosed = true
return nil
}
// newObject instantiates a new *minio.Object*
// ObjectInfo will be set by setObjectInfo
func newObject(ctx context.Context, cancel context.CancelFunc, reqCh chan<- getRequest, resCh <-chan getResponse) *Object {
return &Object{
ctx: ctx,
cancel: cancel,
mutex: &sync.Mutex{},
reqCh: reqCh,
resCh: resCh,
}
}
// getObject - retrieve object from Object Storage.
//
// Additionally this function also takes range arguments to download the specified
// range bytes of an object. Setting offset and length = 0 will download the full object.
//
// For more information about the HTTP Range header.
// go to http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35.
func (c *Client) getObject(ctx context.Context, bucketName, objectName string, opts GetObjectOptions) (io.ReadCloser, ObjectInfo, http.Header, error) {
// Validate input arguments.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return nil, ObjectInfo{}, nil, ErrorResponse{
StatusCode: http.StatusBadRequest,
Code: InvalidBucketName,
Message: err.Error(),
}
}
if err := s3utils.CheckValidObjectName(objectName); err != nil {
return nil, ObjectInfo{}, nil, ErrorResponse{
StatusCode: http.StatusBadRequest,
Code: XMinioInvalidObjectName,
Message: err.Error(),
}
}
// Execute GET on objectName.
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
bucketName: bucketName,
objectName: objectName,
queryValues: opts.toQueryValues(),
customHeader: opts.Header(),
contentSHA256Hex: emptySHA256Hex,
})
if err != nil {
return nil, ObjectInfo{}, nil, err
}
if resp != nil {
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusPartialContent {
return nil, ObjectInfo{}, nil, httpRespToErrorResponse(resp, bucketName, objectName)
}
}
objectStat, err := ToObjectInfo(bucketName, objectName, resp.Header)
if err != nil {
closeResponse(resp)
return nil, ObjectInfo{}, nil, err
}
// do not close body here, caller will close
return resp.Body, objectStat, resp.Header, nil
}
Server/vendor/github.com/minio/minio-go/v7/api-get-options.go
Generated Vendored
+203
View File
@@ -0,0 +1,203 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2015-2020 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"fmt"
"net/http"
"net/url"
"strconv"
"time"
"github.com/minio/minio-go/v7/pkg/encrypt"
)
// AdvancedGetOptions for internal use by MinIO server - not intended for client use.
type AdvancedGetOptions struct {
ReplicationDeleteMarker bool
IsReplicationReadyForDeleteMarker bool
ReplicationProxyRequest string
}
// GetObjectOptions are used to specify additional headers or options
// during GET requests.
type GetObjectOptions struct {
headers map[string]string
reqParams url.Values
ServerSideEncryption encrypt.ServerSide
VersionID string
PartNumber int
// Include any checksums, if object was uploaded with checksum.
// For multipart objects this is a checksum of part checksums.
// https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html
Checksum bool
// To be not used by external applications
Internal AdvancedGetOptions
}
// StatObjectOptions are used to specify additional headers or options
// during GET info/stat requests.
type StatObjectOptions = GetObjectOptions
// Header returns the http.Header representation of the GET options.
func (o GetObjectOptions) Header() http.Header {
headers := make(http.Header, len(o.headers))
for k, v := range o.headers {
headers.Set(k, v)
}
if o.ServerSideEncryption != nil && o.ServerSideEncryption.Type() == encrypt.SSEC {
o.ServerSideEncryption.Marshal(headers)
}
// this header is set for active-active replication scenario where GET/HEAD
// to site A is proxy'd to site B if object/version missing on site A.
if o.Internal.ReplicationProxyRequest != "" {
headers.Set(minIOBucketReplicationProxyRequest, o.Internal.ReplicationProxyRequest)
}
if o.Checksum {
headers.Set("x-amz-checksum-mode", "ENABLED")
}
return headers
}
// Set adds a key value pair to the options. The
// key-value pair will be part of the HTTP GET request
// headers.
func (o *GetObjectOptions) Set(key, value string) {
if o.headers == nil {
o.headers = make(map[string]string)
}
o.headers[http.CanonicalHeaderKey(key)] = value
}
// SetReqParam - set request query string parameter
// supported key: see supportedQueryValues and allowedCustomQueryPrefix.
// If an unsupported key is passed in, it will be ignored and nothing will be done.
func (o *GetObjectOptions) SetReqParam(key, value string) {
if !isCustomQueryValue(key) && !isStandardQueryValue(key) {
// do nothing
return
}
if o.reqParams == nil {
o.reqParams = make(url.Values)
}
o.reqParams.Set(key, value)
}
// AddReqParam - add request query string parameter
// supported key: see supportedQueryValues and allowedCustomQueryPrefix.
// If an unsupported key is passed in, it will be ignored and nothing will be done.
func (o *GetObjectOptions) AddReqParam(key, value string) {
if !isCustomQueryValue(key) && !isStandardQueryValue(key) {
// do nothing
return
}
if o.reqParams == nil {
o.reqParams = make(url.Values)
}
o.reqParams.Add(key, value)
}
// SetMatchETag - set match etag.
func (o *GetObjectOptions) SetMatchETag(etag string) error {
if etag == "" {
return errInvalidArgument("ETag cannot be empty.")
}
o.Set("If-Match", "\""+etag+"\"")
return nil
}
// SetMatchETagExcept - set match etag except.
func (o *GetObjectOptions) SetMatchETagExcept(etag string) error {
if etag == "" {
return errInvalidArgument("ETag cannot be empty.")
}
o.Set("If-None-Match", "\""+etag+"\"")
return nil
}
// SetUnmodified - set unmodified time since.
func (o *GetObjectOptions) SetUnmodified(modTime time.Time) error {
if modTime.IsZero() {
return errInvalidArgument("Modified since cannot be empty.")
}
o.Set("If-Unmodified-Since", modTime.Format(http.TimeFormat))
return nil
}
// SetModified - set modified time since.
func (o *GetObjectOptions) SetModified(modTime time.Time) error {
if modTime.IsZero() {
return errInvalidArgument("Modified since cannot be empty.")
}
o.Set("If-Modified-Since", modTime.Format(http.TimeFormat))
return nil
}
// SetRange - set the start and end offset of the object to be read.
// See https://tools.ietf.org/html/rfc7233#section-3.1 for reference.
func (o *GetObjectOptions) SetRange(start, end int64) error {
switch {
case start == 0 && end < 0:
// Read last '-end' bytes. `bytes=-N`.
o.Set("Range", fmt.Sprintf("bytes=%d", end))
case 0 < start && end == 0:
// Read everything starting from offset
// 'start'. `bytes=N-`.
o.Set("Range", fmt.Sprintf("bytes=%d-", start))
case 0 <= start && start <= end:
// Read everything starting at 'start' till the
// 'end'. `bytes=N-M`
o.Set("Range", fmt.Sprintf("bytes=%d-%d", start, end))
default:
// All other cases such as
// bytes=-3-
// bytes=5-3
// bytes=-2-4
// bytes=-3-0
// bytes=-3--2
// are invalid.
return errInvalidArgument(
fmt.Sprintf(
"Invalid range specified: start=%d end=%d",
start, end))
}
return nil
}
// toQueryValues - Convert the versionId, partNumber, and reqParams in Options to query string parameters.
func (o *GetObjectOptions) toQueryValues() url.Values {
urlValues := make(url.Values)
if o.VersionID != "" {
urlValues.Set("versionId", o.VersionID)
}
if o.PartNumber > 0 {
urlValues.Set("partNumber", strconv.Itoa(o.PartNumber))
}
if o.reqParams != nil {
for key, values := range o.reqParams {
for _, value := range values {
urlValues.Add(key, value)
}
}
}
return urlValues
}
Server/vendor/github.com/minio/minio-go/v7/api-inventory-ext.go
Generated Vendored
+332
View File
@@ -0,0 +1,332 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2015-2025 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"context"
"encoding/json"
"io"
"iter"
"net/http"
"net/url"
"strings"
"time"
"github.com/minio/minio-go/v7/pkg/s3utils"
)
// This file contains the inventory API extension for MinIO server. It is not
// compatible with AWS S3.
func makeInventoryReqMetadata(bucket string, urlParams ...string) requestMetadata {
urlValues := make(url.Values)
urlValues.Set("minio-inventory", "")
// If an odd number of parameters is given, we skip the last pair to avoid
// an out of bounds access.
for i := 0; i+1 < len(urlParams); i += 2 {
urlValues.Set(urlParams[i], urlParams[i+1])
}
return requestMetadata{
bucketName: bucket,
queryValues: urlValues,
}
}
// GenerateInventoryConfigYAML generates a YAML template for an inventory configuration.
// This is a MinIO-specific API and is not compatible with AWS S3.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucket: Name of the bucket
// - id: Unique identifier for the inventory configuration
//
// Returns a YAML template string that can be customized and used with PutBucketInventoryConfiguration.
func (c *Client) GenerateInventoryConfigYAML(ctx context.Context, bucket, id string) (string, error) {
if err := s3utils.CheckValidBucketName(bucket); err != nil {
return "", err
}
if id == "" {
return "", errInvalidArgument("inventory ID cannot be empty")
}
reqMeta := makeInventoryReqMetadata(bucket, "generate", "", "id", id)
resp, err := c.executeMethod(ctx, http.MethodGet, reqMeta)
defer closeResponse(resp)
if err != nil {
return "", err
}
if resp.StatusCode != http.StatusOK {
return "", httpRespToErrorResponse(resp, bucket, "")
}
buf := new(strings.Builder)
_, err = io.Copy(buf, resp.Body)
return buf.String(), err
}
// inventoryPutConfigOpts is a placeholder for future options that may be added.
type inventoryPutConfigOpts struct{}
// InventoryPutConfigOption is to allow for functional options for
// PutBucketInventoryConfiguration. It may be used in the future to customize
// the PutBucketInventoryConfiguration request, but currently does not do
// anything.
type InventoryPutConfigOption func(*inventoryPutConfigOpts)
// PutBucketInventoryConfiguration creates or updates an inventory configuration for a bucket.
// This is a MinIO-specific API and is not compatible with AWS S3.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucket: Name of the bucket
// - id: Unique identifier for the inventory configuration
// - yamlDef: YAML definition of the inventory configuration
//
// Returns an error if the operation fails, or if bucket name, id, or yamlDef is empty.
func (c *Client) PutBucketInventoryConfiguration(ctx context.Context, bucket string, id string, yamlDef string, _ ...InventoryPutConfigOption) error {
if err := s3utils.CheckValidBucketName(bucket); err != nil {
return err
}
if id == "" {
return errInvalidArgument("inventory ID cannot be empty")
}
if yamlDef == "" {
return errInvalidArgument("YAML definition cannot be empty")
}
reqMeta := makeInventoryReqMetadata(bucket, "id", id)
reqMeta.contentBody = strings.NewReader(yamlDef)
reqMeta.contentLength = int64(len(yamlDef))
reqMeta.contentMD5Base64 = sumMD5Base64([]byte(yamlDef))
resp, err := c.executeMethod(ctx, http.MethodPut, reqMeta)
defer closeResponse(resp)
if err != nil {
return err
}
if resp.StatusCode != http.StatusOK {
return httpRespToErrorResponse(resp, bucket, "")
}
return nil
}
// GetBucketInventoryConfiguration retrieves the inventory configuration for a bucket.
// This is a MinIO-specific API and is not compatible with AWS S3.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucket: Name of the bucket
// - id: Unique identifier for the inventory configuration
//
// Returns the inventory configuration or an error if the operation fails or if the configuration doesn't exist.
func (c *Client) GetBucketInventoryConfiguration(ctx context.Context, bucket, id string) (*InventoryConfiguration, error) {
if err := s3utils.CheckValidBucketName(bucket); err != nil {
return nil, err
}
if id == "" {
return nil, errInvalidArgument("inventory ID cannot be empty")
}
reqMeta := makeInventoryReqMetadata(bucket, "id", id)
resp, err := c.executeMethod(ctx, http.MethodGet, reqMeta)
defer closeResponse(resp)
if err != nil {
return nil, err
}
if resp.StatusCode != http.StatusOK {
return nil, httpRespToErrorResponse(resp, bucket, "")
}
decoder := json.NewDecoder(resp.Body)
var ic InventoryConfiguration
err = decoder.Decode(&ic)
if err != nil {
return nil, err
}
return &ic, nil
}
// DeleteBucketInventoryConfiguration deletes an inventory configuration from a bucket.
// This is a MinIO-specific API and is not compatible with AWS S3.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucket: Name of the bucket
// - id: Unique identifier for the inventory configuration to delete
//
// Returns an error if the operation fails or if the configuration doesn't exist.
func (c *Client) DeleteBucketInventoryConfiguration(ctx context.Context, bucket, id string) error {
if err := s3utils.CheckValidBucketName(bucket); err != nil {
return err
}
if id == "" {
return errInvalidArgument("inventory ID cannot be empty")
}
reqMeta := makeInventoryReqMetadata(bucket, "id", id)
resp, err := c.executeMethod(ctx, http.MethodDelete, reqMeta)
defer closeResponse(resp)
if err != nil {
return err
}
if resp.StatusCode != http.StatusOK {
return httpRespToErrorResponse(resp, bucket, "")
}
return nil
}
// InventoryConfiguration represents the inventory configuration
type InventoryConfiguration struct {
Bucket string `json:"bucket"`
ID string `json:"id"`
User string `json:"user"`
YamlDef string `json:"yamlDef,omitempty"`
}
// InventoryListResult represents the result of listing inventory
// configurations.
type InventoryListResult struct {
Items []InventoryConfiguration `json:"items"`
NextContinuationToken string `json:"nextContinuationToken,omitempty"`
}
// ListBucketInventoryConfigurations lists up to 100 inventory configurations for a bucket.
// This is a MinIO-specific API and is not compatible with AWS S3.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucket: Name of the bucket
// - continuationToken: Token for pagination (empty string for first request)
//
// Returns a list result with configurations and a continuation token for the next page, or an error.
func (c *Client) ListBucketInventoryConfigurations(ctx context.Context, bucket, continuationToken string) (lr *InventoryListResult, err error) {
if err := s3utils.CheckValidBucketName(bucket); err != nil {
return nil, err
}
reqMeta := makeInventoryReqMetadata(bucket, "continuation-token", continuationToken)
resp, err := c.executeMethod(ctx, http.MethodGet, reqMeta)
defer closeResponse(resp)
if err != nil {
return nil, err
}
if resp.StatusCode != http.StatusOK {
return nil, httpRespToErrorResponse(resp, bucket, "")
}
decoder := json.NewDecoder(resp.Body)
err = decoder.Decode(&lr)
if err != nil {
return nil, err
}
return lr, nil
}
// ListBucketInventoryConfigurationsIterator returns an iterator that lists all inventory configurations
// for a bucket. This is a MinIO-specific API and is not compatible with AWS S3.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucket: Name of the bucket
//
// Returns an iterator that yields InventoryConfiguration values and errors. The iterator automatically
// handles pagination and fetches all configurations.
func (c *Client) ListBucketInventoryConfigurationsIterator(ctx context.Context, bucket string) iter.Seq2[InventoryConfiguration, error] {
return func(yield func(InventoryConfiguration, error) bool) {
if err := s3utils.CheckValidBucketName(bucket); err != nil {
yield(InventoryConfiguration{}, err)
return
}
var continuationToken string
for {
listResult, err := c.ListBucketInventoryConfigurations(ctx, bucket, continuationToken)
if err != nil {
yield(InventoryConfiguration{}, err)
return
}
for _, item := range listResult.Items {
if !yield(item, nil) {
return
}
}
if listResult.NextContinuationToken == "" {
return
}
continuationToken = listResult.NextContinuationToken
}
}
}
// InventoryJobStatus represents the status of an inventory job.
type InventoryJobStatus struct {
Bucket string `json:"bucket"`
ID string `json:"id"`
User string `json:"user"`
AccessKey string `json:"accessKey"`
Schedule string `json:"schedule"`
State string `json:"state"`
NextScheduledTime time.Time `json:"nextScheduledTime,omitempty"`
StartTime time.Time `json:"startTime,omitempty"`
EndTime time.Time `json:"endTime,omitempty"`
LastUpdate time.Time `json:"lastUpdate,omitempty"`
Scanned string `json:"scanned,omitempty"`
Matched string `json:"matched,omitempty"`
ScannedCount uint64 `json:"scannedCount,omitempty"`
MatchedCount uint64 `json:"matchedCount,omitempty"`
RecordsWritten uint64 `json:"recordsWritten,omitempty"`
OutputFilesCount uint64 `json:"outputFilesCount,omitempty"`
ExecutionTime string `json:"executionTime,omitempty"`
NumStarts uint64 `json:"numStarts,omitempty"`
NumErrors uint64 `json:"numErrors,omitempty"`
NumLockLosses uint64 `json:"numLockLosses,omitempty"`
ManifestPath string `json:"manifestPath,omitempty"`
RetryAttempts uint64 `json:"retryAttempts,omitempty"`
LastFailTime time.Time `json:"lastFailTime,omitempty"`
LastFailErrors []string `json:"lastFailErrors,omitempty"`
}
// GetBucketInventoryJobStatus retrieves the status of an inventory job for a bucket.
// This is a MinIO-specific API and is not compatible with AWS S3.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucket: Name of the bucket
// - id: Unique identifier for the inventory job
//
// Returns the inventory job status including execution state, progress, and error information, or an error if the operation fails.
func (c *Client) GetBucketInventoryJobStatus(ctx context.Context, bucket, id string) (*InventoryJobStatus, error) {
if err := s3utils.CheckValidBucketName(bucket); err != nil {
return nil, err
}
if id == "" {
return nil, errInvalidArgument("inventory ID cannot be empty")
}
reqMeta := makeInventoryReqMetadata(bucket, "id", id, "status", "")
resp, err := c.executeMethod(ctx, http.MethodGet, reqMeta)
defer closeResponse(resp)
if err != nil {
return nil, err
}
if resp.StatusCode != http.StatusOK {
return nil, httpRespToErrorResponse(resp, bucket, "")
}
decoder := json.NewDecoder(resp.Body)
var jStatus InventoryJobStatus
err = decoder.Decode(&jStatus)
if err != nil {
return nil, err
}
return &jStatus, nil
}
Server/vendor/github.com/minio/minio-go/v7/api-list.go
Generated Vendored
+1142
View File
File diff suppressed because it is too large Load Diff
Server/vendor/github.com/minio/minio-go/v7/api-object-legal-hold.go
Generated Vendored
+193
View File
@@ -0,0 +1,193 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2020 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"bytes"
"context"
"encoding/xml"
"fmt"
"net/http"
"net/url"
"github.com/minio/minio-go/v7/pkg/s3utils"
)
// objectLegalHold - object legal hold specified in
// https://docs.aws.amazon.com/AmazonS3/latest/API/archive-RESTObjectPUTLegalHold.html
type objectLegalHold struct {
XMLNS string `xml:"xmlns,attr,omitempty"`
XMLName xml.Name `xml:"LegalHold"`
Status LegalHoldStatus `xml:"Status,omitempty"`
}
// PutObjectLegalHoldOptions represents options specified by user for PutObjectLegalHold call
type PutObjectLegalHoldOptions struct {
VersionID string
Status *LegalHoldStatus
}
// GetObjectLegalHoldOptions represents options specified by user for GetObjectLegalHold call
type GetObjectLegalHoldOptions struct {
VersionID string
}
// LegalHoldStatus - object legal hold status.
type LegalHoldStatus string
const (
// LegalHoldEnabled indicates legal hold is enabled
LegalHoldEnabled LegalHoldStatus = "ON"
// LegalHoldDisabled indicates legal hold is disabled
LegalHoldDisabled LegalHoldStatus = "OFF"
)
func (r LegalHoldStatus) String() string {
return string(r)
}
// IsValid - check whether this legal hold status is valid or not.
func (r LegalHoldStatus) IsValid() bool {
return r == LegalHoldEnabled || r == LegalHoldDisabled
}
func newObjectLegalHold(status *LegalHoldStatus) (*objectLegalHold, error) {
if status == nil {
return nil, fmt.Errorf("Status not set")
}
if !status.IsValid() {
return nil, fmt.Errorf("invalid legal hold status `%v`", status)
}
legalHold := &objectLegalHold{
Status: *status,
}
return legalHold, nil
}
// PutObjectLegalHold sets the legal hold status for an object and specific version.
// Legal hold prevents an object version from being overwritten or deleted, regardless of retention settings.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucketName: Name of the bucket
// - objectName: Name of the object
// - opts: Options including Status (LegalHoldEnabled or LegalHoldDisabled) and optional VersionID
//
// Returns an error if the operation fails or if the status is invalid.
func (c *Client) PutObjectLegalHold(ctx context.Context, bucketName, objectName string, opts PutObjectLegalHoldOptions) error {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return err
}
if err := s3utils.CheckValidObjectName(objectName); err != nil {
return err
}
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("legal-hold", "")
if opts.VersionID != "" {
urlValues.Set("versionId", opts.VersionID)
}
lh, err := newObjectLegalHold(opts.Status)
if err != nil {
return err
}
lhData, err := xml.Marshal(lh)
if err != nil {
return err
}
reqMetadata := requestMetadata{
bucketName: bucketName,
objectName: objectName,
queryValues: urlValues,
contentBody: bytes.NewReader(lhData),
contentLength: int64(len(lhData)),
contentMD5Base64: sumMD5Base64(lhData),
contentSHA256Hex: sum256Hex(lhData),
}
// Execute PUT Object Legal Hold.
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
defer closeResponse(resp)
if err != nil {
return err
}
if resp != nil {
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusNoContent {
return httpRespToErrorResponse(resp, bucketName, objectName)
}
}
return nil
}
// GetObjectLegalHold retrieves the legal hold status for an object and specific version.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucketName: Name of the bucket
// - objectName: Name of the object
// - opts: Options including optional VersionID to target a specific version
//
// Returns the legal hold status (LegalHoldEnabled or LegalHoldDisabled) or an error if the operation fails.
func (c *Client) GetObjectLegalHold(ctx context.Context, bucketName, objectName string, opts GetObjectLegalHoldOptions) (status *LegalHoldStatus, err error) {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return nil, err
}
if err := s3utils.CheckValidObjectName(objectName); err != nil {
return nil, err
}
urlValues := make(url.Values)
urlValues.Set("legal-hold", "")
if opts.VersionID != "" {
urlValues.Set("versionId", opts.VersionID)
}
// Execute GET on bucket to list objects.
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
bucketName: bucketName,
objectName: objectName,
queryValues: urlValues,
contentSHA256Hex: emptySHA256Hex,
})
defer closeResponse(resp)
if err != nil {
return nil, err
}
if resp != nil {
if resp.StatusCode != http.StatusOK {
return nil, httpRespToErrorResponse(resp, bucketName, objectName)
}
}
lh := &objectLegalHold{}
if err = xml.NewDecoder(resp.Body).Decode(lh); err != nil {
return nil, err
}
return &lh.Status, nil
}
Server/vendor/github.com/minio/minio-go/v7/api-object-lock.go
Generated Vendored
+241
View File
@@ -0,0 +1,241 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2019 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"bytes"
"context"
"encoding/xml"
"fmt"
"net/http"
"net/url"
"time"
"github.com/minio/minio-go/v7/pkg/s3utils"
)
// RetentionMode - object retention mode.
type RetentionMode string
const (
// Governance - governance mode.
Governance RetentionMode = "GOVERNANCE"
// Compliance - compliance mode.
Compliance RetentionMode = "COMPLIANCE"
)
func (r RetentionMode) String() string {
return string(r)
}
// IsValid - check whether this retention mode is valid or not.
func (r RetentionMode) IsValid() bool {
return r == Governance || r == Compliance
}
// ValidityUnit - retention validity unit.
type ValidityUnit string
const (
// Days - denotes no. of days.
Days ValidityUnit = "DAYS"
// Years - denotes no. of years.
Years ValidityUnit = "YEARS"
)
func (unit ValidityUnit) String() string {
return string(unit)
}
// IsValid - check whether this validity unit is valid or not.
func (unit ValidityUnit) isValid() bool {
return unit == Days || unit == Years
}
// Retention - bucket level retention configuration.
type Retention struct {
Mode RetentionMode
Validity time.Duration
}
func (r Retention) String() string {
return fmt.Sprintf("{Mode:%v, Validity:%v}", r.Mode, r.Validity)
}
// IsEmpty - returns whether retention is empty or not.
func (r Retention) IsEmpty() bool {
return r.Mode == "" || r.Validity == 0
}
// objectLockConfig - object lock configuration specified in
// https://docs.aws.amazon.com/AmazonS3/latest/API/Type_API_ObjectLockConfiguration.html
type objectLockConfig struct {
XMLNS string `xml:"xmlns,attr,omitempty"`
XMLName xml.Name `xml:"ObjectLockConfiguration"`
ObjectLockEnabled string `xml:"ObjectLockEnabled"`
Rule *struct {
DefaultRetention struct {
Mode RetentionMode `xml:"Mode"`
Days *uint `xml:"Days"`
Years *uint `xml:"Years"`
} `xml:"DefaultRetention"`
} `xml:"Rule,omitempty"`
}
func newObjectLockConfig(mode *RetentionMode, validity *uint, unit *ValidityUnit) (*objectLockConfig, error) {
config := &objectLockConfig{
ObjectLockEnabled: "Enabled",
}
if mode != nil && validity != nil && unit != nil {
if !mode.IsValid() {
return nil, fmt.Errorf("invalid retention mode `%v`", mode)
}
if !unit.isValid() {
return nil, fmt.Errorf("invalid validity unit `%v`", unit)
}
config.Rule = &struct {
DefaultRetention struct {
Mode RetentionMode `xml:"Mode"`
Days *uint `xml:"Days"`
Years *uint `xml:"Years"`
} `xml:"DefaultRetention"`
}{}
config.Rule.DefaultRetention.Mode = *mode
if *unit == Days {
config.Rule.DefaultRetention.Days = validity
} else {
config.Rule.DefaultRetention.Years = validity
}
return config, nil
}
if mode == nil && validity == nil && unit == nil {
return config, nil
}
return nil, fmt.Errorf("all of retention mode, validity and validity unit must be passed")
}
// SetBucketObjectLockConfig sets object lock configuration in given bucket. mode, validity and unit are either all set or all nil.
func (c *Client) SetBucketObjectLockConfig(ctx context.Context, bucketName string, mode *RetentionMode, validity *uint, unit *ValidityUnit) error {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return err
}
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("object-lock", "")
config, err := newObjectLockConfig(mode, validity, unit)
if err != nil {
return err
}
configData, err := xml.Marshal(config)
if err != nil {
return err
}
reqMetadata := requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
contentBody: bytes.NewReader(configData),
contentLength: int64(len(configData)),
contentMD5Base64: sumMD5Base64(configData),
contentSHA256Hex: sum256Hex(configData),
}
// Execute PUT bucket object lock configuration.
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
defer closeResponse(resp)
if err != nil {
return err
}
if resp != nil {
if resp.StatusCode != http.StatusOK {
return httpRespToErrorResponse(resp, bucketName, "")
}
}
return nil
}
// GetObjectLockConfig gets object lock configuration of given bucket.
func (c *Client) GetObjectLockConfig(ctx context.Context, bucketName string) (objectLock string, mode *RetentionMode, validity *uint, unit *ValidityUnit, err error) {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return "", nil, nil, nil, err
}
urlValues := make(url.Values)
urlValues.Set("object-lock", "")
// Execute GET on bucket to list objects.
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
contentSHA256Hex: emptySHA256Hex,
})
defer closeResponse(resp)
if err != nil {
return "", nil, nil, nil, err
}
if resp != nil {
if resp.StatusCode != http.StatusOK {
return "", nil, nil, nil, httpRespToErrorResponse(resp, bucketName, "")
}
}
config := &objectLockConfig{}
if err = xml.NewDecoder(resp.Body).Decode(config); err != nil {
return "", nil, nil, nil, err
}
if config.Rule != nil {
mode = &config.Rule.DefaultRetention.Mode
if config.Rule.DefaultRetention.Days != nil {
validity = config.Rule.DefaultRetention.Days
days := Days
unit = &days
} else {
validity = config.Rule.DefaultRetention.Years
years := Years
unit = &years
}
return config.ObjectLockEnabled, mode, validity, unit, nil
}
return config.ObjectLockEnabled, nil, nil, nil, nil
}
// GetBucketObjectLockConfig gets object lock configuration of given bucket.
func (c *Client) GetBucketObjectLockConfig(ctx context.Context, bucketName string) (mode *RetentionMode, validity *uint, unit *ValidityUnit, err error) {
_, mode, validity, unit, err = c.GetObjectLockConfig(ctx, bucketName)
return mode, validity, unit, err
}
// SetObjectLockConfig sets object lock configuration in given bucket. mode, validity and unit are either all set or all nil.
func (c *Client) SetObjectLockConfig(ctx context.Context, bucketName string, mode *RetentionMode, validity *uint, unit *ValidityUnit) error {
return c.SetBucketObjectLockConfig(ctx, bucketName, mode, validity, unit)
}
Server/vendor/github.com/minio/minio-go/v7/api-object-retention.go
Generated Vendored
+182
View File
@@ -0,0 +1,182 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2019-2020 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"bytes"
"context"
"encoding/xml"
"fmt"
"net/http"
"net/url"
"time"
"github.com/minio/minio-go/v7/pkg/s3utils"
)
// objectRetention - object retention specified in
// https://docs.aws.amazon.com/AmazonS3/latest/API/Type_API_ObjectLockConfiguration.html
type objectRetention struct {
XMLNS string `xml:"xmlns,attr,omitempty"`
XMLName xml.Name `xml:"Retention"`
Mode RetentionMode `xml:"Mode,omitempty"`
RetainUntilDate *time.Time `type:"timestamp" timestampFormat:"iso8601" xml:"RetainUntilDate,omitempty"`
}
func newObjectRetention(mode *RetentionMode, date *time.Time) (*objectRetention, error) {
objectRetention := &objectRetention{}
if date != nil && !date.IsZero() {
objectRetention.RetainUntilDate = date
}
if mode != nil {
if !mode.IsValid() {
return nil, fmt.Errorf("invalid retention mode `%v`", mode)
}
objectRetention.Mode = *mode
}
return objectRetention, nil
}
// PutObjectRetentionOptions represents options specified by user for PutObject call
type PutObjectRetentionOptions struct {
GovernanceBypass bool
Mode *RetentionMode
RetainUntilDate *time.Time
VersionID string
}
// PutObjectRetention sets the retention configuration for an object and specific version.
// Object retention prevents an object version from being deleted or overwritten for a specified period.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucketName: Name of the bucket
// - objectName: Name of the object
// - opts: Options including Mode (GOVERNANCE or COMPLIANCE), RetainUntilDate, optional VersionID, and GovernanceBypass
//
// Returns an error if the operation fails or if the retention settings are invalid.
func (c *Client) PutObjectRetention(ctx context.Context, bucketName, objectName string, opts PutObjectRetentionOptions) error {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return err
}
if err := s3utils.CheckValidObjectName(objectName); err != nil {
return err
}
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("retention", "")
if opts.VersionID != "" {
urlValues.Set("versionId", opts.VersionID)
}
retention, err := newObjectRetention(opts.Mode, opts.RetainUntilDate)
if err != nil {
return err
}
retentionData, err := xml.Marshal(retention)
if err != nil {
return err
}
// Build headers.
headers := make(http.Header)
if opts.GovernanceBypass {
// Set the bypass goverenance retention header
headers.Set(amzBypassGovernance, "true")
}
reqMetadata := requestMetadata{
bucketName: bucketName,
objectName: objectName,
queryValues: urlValues,
contentBody: bytes.NewReader(retentionData),
contentLength: int64(len(retentionData)),
contentMD5Base64: sumMD5Base64(retentionData),
contentSHA256Hex: sum256Hex(retentionData),
customHeader: headers,
}
// Execute PUT Object Retention.
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
defer closeResponse(resp)
if err != nil {
return err
}
if resp != nil {
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusNoContent {
return httpRespToErrorResponse(resp, bucketName, objectName)
}
}
return nil
}
// GetObjectRetention retrieves the retention configuration for an object and specific version.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucketName: Name of the bucket
// - objectName: Name of the object
// - versionID: Optional version ID to target a specific version (empty string for current version)
//
// Returns the retention mode (GOVERNANCE or COMPLIANCE), retain-until date, and any error.
func (c *Client) GetObjectRetention(ctx context.Context, bucketName, objectName, versionID string) (mode *RetentionMode, retainUntilDate *time.Time, err error) {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return nil, nil, err
}
if err := s3utils.CheckValidObjectName(objectName); err != nil {
return nil, nil, err
}
urlValues := make(url.Values)
urlValues.Set("retention", "")
if versionID != "" {
urlValues.Set("versionId", versionID)
}
// Execute GET on bucket to list objects.
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
bucketName: bucketName,
objectName: objectName,
queryValues: urlValues,
contentSHA256Hex: emptySHA256Hex,
})
defer closeResponse(resp)
if err != nil {
return nil, nil, err
}
if resp != nil {
if resp.StatusCode != http.StatusOK {
return nil, nil, httpRespToErrorResponse(resp, bucketName, objectName)
}
}
retention := &objectRetention{}
if err = xml.NewDecoder(resp.Body).Decode(retention); err != nil {
return nil, nil, err
}
return &retention.Mode, retention.RetainUntilDate, nil
}
Server/vendor/github.com/minio/minio-go/v7/api-object-tagging.go
Generated Vendored
+202
View File
@@ -0,0 +1,202 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2020 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"bytes"
"context"
"encoding/xml"
"net/http"
"net/url"
"github.com/minio/minio-go/v7/pkg/s3utils"
"github.com/minio/minio-go/v7/pkg/tags"
)
// PutObjectTaggingOptions holds an object version id
// to update tag(s) of a specific object version
type PutObjectTaggingOptions struct {
VersionID string
Internal AdvancedObjectTaggingOptions
}
// AdvancedObjectTaggingOptions for internal use by MinIO server - not intended for client use.
type AdvancedObjectTaggingOptions struct {
ReplicationProxyRequest string
}
// PutObjectTagging replaces or creates object tag(s) and can target a specific object version
// in a versioned bucket.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucketName: Name of the bucket
// - objectName: Name of the object
// - otags: Tags to apply to the object
// - opts: Options including VersionID to target a specific version
//
// Returns an error if the operation fails.
func (c *Client) PutObjectTagging(ctx context.Context, bucketName, objectName string, otags *tags.Tags, opts PutObjectTaggingOptions) error {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return err
}
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("tagging", "")
if opts.VersionID != "" {
urlValues.Set("versionId", opts.VersionID)
}
headers := make(http.Header, 0)
if opts.Internal.ReplicationProxyRequest != "" {
headers.Set(minIOBucketReplicationProxyRequest, opts.Internal.ReplicationProxyRequest)
}
reqBytes, err := xml.Marshal(otags)
if err != nil {
return err
}
reqMetadata := requestMetadata{
bucketName: bucketName,
objectName: objectName,
queryValues: urlValues,
contentBody: bytes.NewReader(reqBytes),
contentLength: int64(len(reqBytes)),
contentMD5Base64: sumMD5Base64(reqBytes),
customHeader: headers,
}
// Execute PUT to set a object tagging.
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
defer closeResponse(resp)
if err != nil {
return err
}
if resp != nil {
if resp.StatusCode != http.StatusOK {
return httpRespToErrorResponse(resp, bucketName, objectName)
}
}
return nil
}
// GetObjectTaggingOptions holds the object version ID
// to fetch the tagging key/value pairs
type GetObjectTaggingOptions struct {
VersionID string
Internal AdvancedObjectTaggingOptions
}
// GetObjectTagging retrieves object tag(s) with options to target a specific object version
// in a versioned bucket.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucketName: Name of the bucket
// - objectName: Name of the object
// - opts: Options including VersionID to target a specific version
//
// Returns the object's tags or an error if the operation fails.
func (c *Client) GetObjectTagging(ctx context.Context, bucketName, objectName string, opts GetObjectTaggingOptions) (*tags.Tags, error) {
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("tagging", "")
if opts.VersionID != "" {
urlValues.Set("versionId", opts.VersionID)
}
headers := make(http.Header, 0)
if opts.Internal.ReplicationProxyRequest != "" {
headers.Set(minIOBucketReplicationProxyRequest, opts.Internal.ReplicationProxyRequest)
}
// Execute GET on object to get object tag(s)
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
bucketName: bucketName,
objectName: objectName,
queryValues: urlValues,
customHeader: headers,
})
defer closeResponse(resp)
if err != nil {
return nil, err
}
if resp != nil {
if resp.StatusCode != http.StatusOK {
return nil, httpRespToErrorResponse(resp, bucketName, objectName)
}
}
return tags.ParseObjectXML(resp.Body)
}
// RemoveObjectTaggingOptions holds the version id of the object to remove
type RemoveObjectTaggingOptions struct {
VersionID string
Internal AdvancedObjectTaggingOptions
}
// RemoveObjectTagging removes object tag(s) with options to target a specific object version
// in a versioned bucket.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucketName: Name of the bucket
// - objectName: Name of the object
// - opts: Options including VersionID to target a specific version
//
// Returns an error if the operation fails.
func (c *Client) RemoveObjectTagging(ctx context.Context, bucketName, objectName string, opts RemoveObjectTaggingOptions) error {
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("tagging", "")
if opts.VersionID != "" {
urlValues.Set("versionId", opts.VersionID)
}
headers := make(http.Header, 0)
if opts.Internal.ReplicationProxyRequest != "" {
headers.Set(minIOBucketReplicationProxyRequest, opts.Internal.ReplicationProxyRequest)
}
// Execute DELETE on object to remove object tag(s)
resp, err := c.executeMethod(ctx, http.MethodDelete, requestMetadata{
bucketName: bucketName,
objectName: objectName,
queryValues: urlValues,
customHeader: headers,
})
defer closeResponse(resp)
if err != nil {
return err
}
if resp != nil {
// S3 returns "204 No content" after Object tag deletion.
if resp.StatusCode != http.StatusNoContent {
return httpRespToErrorResponse(resp, bucketName, objectName)
}
}
return err
}
Server/vendor/github.com/minio/minio-go/v7/api-presigned.go
Generated Vendored
+228
View File
@@ -0,0 +1,228 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2015-2017 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"context"
"errors"
"net/http"
"net/url"
"time"
"github.com/minio/minio-go/v7/pkg/s3utils"
"github.com/minio/minio-go/v7/pkg/signer"
)
// presignURL - Returns a presigned URL for an input 'method'.
// Expires maximum is 7days - ie. 604800 and minimum is 1.
func (c *Client) presignURL(ctx context.Context, method, bucketName, objectName string, expires time.Duration, reqParams url.Values, extraHeaders http.Header) (u *url.URL, err error) {
// Input validation.
if method == "" {
return nil, errInvalidArgument("method cannot be empty.")
}
if err = s3utils.CheckValidBucketName(bucketName); err != nil {
return nil, err
}
if err = isValidExpiry(expires); err != nil {
return nil, err
}
// Convert expires into seconds.
expireSeconds := int64(expires / time.Second)
reqMetadata := requestMetadata{
presignURL: true,
bucketName: bucketName,
objectName: objectName,
expires: expireSeconds,
queryValues: reqParams,
extraPresignHeader: extraHeaders,
}
// Instantiate a new request.
// Since expires is set newRequest will presign the request.
var req *http.Request
if req, err = c.newRequest(ctx, method, reqMetadata); err != nil {
return nil, err
}
return req.URL, nil
}
// PresignedGetObject - Returns a presigned URL to access an object
// data without credentials. URL can have a maximum expiry of
// upto 7days or a minimum of 1sec. Additionally you can override
// a set of response headers using the query parameters.
func (c *Client) PresignedGetObject(ctx context.Context, bucketName, objectName string, expires time.Duration, reqParams url.Values) (u *url.URL, err error) {
if err = s3utils.CheckValidObjectName(objectName); err != nil {
return nil, err
}
return c.presignURL(ctx, http.MethodGet, bucketName, objectName, expires, reqParams, nil)
}
// PresignedHeadObject - Returns a presigned URL to access
// object metadata without credentials. URL can have a maximum expiry
// of upto 7days or a minimum of 1sec. Additionally you can override
// a set of response headers using the query parameters.
func (c *Client) PresignedHeadObject(ctx context.Context, bucketName, objectName string, expires time.Duration, reqParams url.Values) (u *url.URL, err error) {
if err = s3utils.CheckValidObjectName(objectName); err != nil {
return nil, err
}
return c.presignURL(ctx, http.MethodHead, bucketName, objectName, expires, reqParams, nil)
}
// PresignedPutObject - Returns a presigned URL to upload an object
// without credentials. URL can have a maximum expiry of upto 7days
// or a minimum of 1sec.
func (c *Client) PresignedPutObject(ctx context.Context, bucketName, objectName string, expires time.Duration) (u *url.URL, err error) {
if err = s3utils.CheckValidObjectName(objectName); err != nil {
return nil, err
}
return c.presignURL(ctx, http.MethodPut, bucketName, objectName, expires, nil, nil)
}
// PresignHeader - similar to Presign() but allows including HTTP headers that
// will be used to build the signature. The request using the resulting URL will
// need to have the exact same headers to be added for signature validation to
// pass.
//
// FIXME: The extra header parameter should be included in Presign() in the next
// major version bump, and this function should then be deprecated.
func (c *Client) PresignHeader(ctx context.Context, method, bucketName, objectName string, expires time.Duration, reqParams url.Values, extraHeaders http.Header) (u *url.URL, err error) {
return c.presignURL(ctx, method, bucketName, objectName, expires, reqParams, extraHeaders)
}
// Presign - returns a presigned URL for any http method of your choice along
// with custom request params and extra signed headers. URL can have a maximum
// expiry of upto 7days or a minimum of 1sec.
func (c *Client) Presign(ctx context.Context, method, bucketName, objectName string, expires time.Duration, reqParams url.Values) (u *url.URL, err error) {
return c.presignURL(ctx, method, bucketName, objectName, expires, reqParams, nil)
}
// PresignedPostPolicy - Returns POST urlString, form data to upload an object.
func (c *Client) PresignedPostPolicy(ctx context.Context, p *PostPolicy) (u *url.URL, formData map[string]string, err error) {
// Validate input arguments.
if p.expiration.IsZero() {
return nil, nil, errors.New("Expiration time must be specified")
}
if _, ok := p.formData["key"]; !ok {
return nil, nil, errors.New("object key must be specified")
}
if _, ok := p.formData["bucket"]; !ok {
return nil, nil, errors.New("bucket name must be specified")
}
bucketName := p.formData["bucket"]
// Fetch the bucket location.
location, err := c.getBucketLocation(ctx, bucketName)
if err != nil {
return nil, nil, err
}
isVirtualHost := c.isVirtualHostStyleRequest(*c.endpointURL, bucketName)
u, err = c.makeTargetURL(bucketName, "", location, isVirtualHost, nil)
if err != nil {
return nil, nil, err
}
// Get credentials from the configured credentials provider.
credValues, err := c.credsProvider.GetWithContext(c.CredContext())
if err != nil {
return nil, nil, err
}
var (
signerType = credValues.SignerType
sessionToken = credValues.SessionToken
accessKeyID = credValues.AccessKeyID
secretAccessKey = credValues.SecretAccessKey
)
if signerType.IsAnonymous() {
return nil, nil, errInvalidArgument("Presigned operations are not supported for anonymous credentials")
}
// Keep time.
t := time.Now().UTC()
// For signature version '2' handle here.
if signerType.IsV2() {
policyBase64 := p.base64()
p.formData["policy"] = policyBase64
// For Google endpoint set this value to be 'GoogleAccessId'.
if s3utils.IsGoogleEndpoint(*c.endpointURL) {
p.formData["GoogleAccessId"] = accessKeyID
} else {
// For all other endpoints set this value to be 'AWSAccessKeyId'.
p.formData["AWSAccessKeyId"] = accessKeyID
}
// Sign the policy.
p.formData["signature"] = signer.PostPresignSignatureV2(policyBase64, secretAccessKey)
return u, p.formData, nil
}
// Add date policy.
if err = p.addNewPolicy(policyCondition{
matchType: "eq",
condition: "$x-amz-date",
value: t.Format(iso8601DateFormat),
}); err != nil {
return nil, nil, err
}
// Add algorithm policy.
if err = p.addNewPolicy(policyCondition{
matchType: "eq",
condition: "$x-amz-algorithm",
value: signV4Algorithm,
}); err != nil {
return nil, nil, err
}
// Add a credential policy.
credential := signer.GetCredential(accessKeyID, location, t, signer.ServiceTypeS3)
if err = p.addNewPolicy(policyCondition{
matchType: "eq",
condition: "$x-amz-credential",
value: credential,
}); err != nil {
return nil, nil, err
}
if sessionToken != "" {
if err = p.addNewPolicy(policyCondition{
matchType: "eq",
condition: "$x-amz-security-token",
value: sessionToken,
}); err != nil {
return nil, nil, err
}
}
// Get base64 encoded policy.
policyBase64 := p.base64()
// Fill in the form data.
p.formData["policy"] = policyBase64
p.formData["x-amz-algorithm"] = signV4Algorithm
p.formData["x-amz-credential"] = credential
p.formData["x-amz-date"] = t.Format(iso8601DateFormat)
if sessionToken != "" {
p.formData["x-amz-security-token"] = sessionToken
}
p.formData["x-amz-signature"] = signer.PostPresignSignatureV4(policyBase64, t, secretAccessKey, location)
return u, p.formData, nil
}
Server/vendor/github.com/minio/minio-go/v7/api-prompt-object.go
Generated Vendored
+78
View File
@@ -0,0 +1,78 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2015-2024 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"bytes"
"context"
"encoding/json"
"io"
"net/http"
"github.com/minio/minio-go/v7/pkg/s3utils"
)
// PromptObject performs language model inference with the prompt and referenced object as context.
// Inference is performed using a Lambda handler that can process the prompt and object.
// Currently, this functionality is limited to certain MinIO servers.
func (c *Client) PromptObject(ctx context.Context, bucketName, objectName, prompt string, opts PromptObjectOptions) (io.ReadCloser, error) {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return nil, ErrorResponse{
StatusCode: http.StatusBadRequest,
Code: InvalidBucketName,
Message: err.Error(),
}
}
if err := s3utils.CheckValidObjectName(objectName); err != nil {
return nil, ErrorResponse{
StatusCode: http.StatusBadRequest,
Code: XMinioInvalidObjectName,
Message: err.Error(),
}
}
opts.AddLambdaArnToReqParams(opts.LambdaArn)
opts.SetHeader("Content-Type", "application/json")
opts.AddPromptArg("prompt", prompt)
promptReqBytes, err := json.Marshal(opts.PromptArgs)
if err != nil {
return nil, err
}
// Execute POST on bucket/object.
resp, err := c.executeMethod(ctx, http.MethodPost, requestMetadata{
bucketName: bucketName,
objectName: objectName,
queryValues: opts.toQueryValues(),
customHeader: opts.Header(),
contentSHA256Hex: sum256Hex(promptReqBytes),
contentBody: bytes.NewReader(promptReqBytes),
contentLength: int64(len(promptReqBytes)),
})
if err != nil {
return nil, err
}
if resp.StatusCode != http.StatusOK {
defer closeResponse(resp)
return nil, httpRespToErrorResponse(resp, bucketName, objectName)
}
return resp.Body, nil
}
Server/vendor/github.com/minio/minio-go/v7/api-prompt-options.go
Generated Vendored
+84
View File
@@ -0,0 +1,84 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2015-2024 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"net/http"
"net/url"
)
// PromptObjectOptions provides options to PromptObject call.
// LambdaArn is the ARN of the Prompt Lambda to be invoked.
// PromptArgs is a map of key-value pairs to be passed to the inference action on the Prompt Lambda.
// "prompt" is a reserved key and should not be used as a key in PromptArgs.
type PromptObjectOptions struct {
LambdaArn string
PromptArgs map[string]any
headers map[string]string
reqParams url.Values
}
// Header returns the http.Header representation of the POST options.
func (o PromptObjectOptions) Header() http.Header {
headers := make(http.Header, len(o.headers))
for k, v := range o.headers {
headers.Set(k, v)
}
return headers
}
// AddPromptArg Add a key value pair to the prompt arguments where the key is a string and
// the value is a JSON serializable.
func (o *PromptObjectOptions) AddPromptArg(key string, value any) {
if o.PromptArgs == nil {
o.PromptArgs = make(map[string]any)
}
o.PromptArgs[key] = value
}
// AddLambdaArnToReqParams adds the lambdaArn to the request query string parameters.
func (o *PromptObjectOptions) AddLambdaArnToReqParams(lambdaArn string) {
if o.reqParams == nil {
o.reqParams = make(url.Values)
}
o.reqParams.Add("lambdaArn", lambdaArn)
}
// SetHeader adds a key value pair to the options. The
// key-value pair will be part of the HTTP POST request
// headers.
func (o *PromptObjectOptions) SetHeader(key, value string) {
if o.headers == nil {
o.headers = make(map[string]string)
}
o.headers[http.CanonicalHeaderKey(key)] = value
}
// toQueryValues - Convert the reqParams in Options to query string parameters.
func (o *PromptObjectOptions) toQueryValues() url.Values {
urlValues := make(url.Values)
if o.reqParams != nil {
for key, values := range o.reqParams {
for _, value := range values {
urlValues.Add(key, value)
}
}
}
return urlValues
}
Server/vendor/github.com/minio/minio-go/v7/api-put-bucket.go
Generated Vendored
+130
View File
@@ -0,0 +1,130 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2015-2020 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"bytes"
"context"
"encoding/xml"
"net/http"
"github.com/minio/minio-go/v7/pkg/s3utils"
)
// Bucket operations
func (c *Client) makeBucket(ctx context.Context, bucketName string, opts MakeBucketOptions) (err error) {
// Validate the input arguments.
if err := s3utils.CheckValidBucketNameStrict(bucketName); err != nil {
return err
}
err = c.doMakeBucket(ctx, bucketName, opts)
if err != nil && (opts.Region == "" || opts.Region == "us-east-1") {
if resp, ok := err.(ErrorResponse); ok && resp.Code == AuthorizationHeaderMalformed && resp.Region != "" {
opts.Region = resp.Region
err = c.doMakeBucket(ctx, bucketName, opts)
}
}
return err
}
func (c *Client) doMakeBucket(ctx context.Context, bucketName string, opts MakeBucketOptions) (err error) {
defer func() {
// Save the location into cache on a successful makeBucket response.
if err == nil {
c.bucketLocCache.Set(bucketName, opts.Region)
}
}()
// If location is empty, treat is a default region 'us-east-1'.
if opts.Region == "" {
opts.Region = "us-east-1"
// For custom region clients, default
// to custom region instead not 'us-east-1'.
if c.region != "" {
opts.Region = c.region
}
}
// PUT bucket request metadata.
reqMetadata := requestMetadata{
bucketName: bucketName,
bucketLocation: opts.Region,
}
headers := make(http.Header)
if opts.ObjectLocking {
headers.Add("x-amz-bucket-object-lock-enabled", "true")
}
if opts.ForceCreate {
headers.Add("x-minio-force-create", "true")
}
reqMetadata.customHeader = headers
// If location is not 'us-east-1' create bucket location config.
if opts.Region != "us-east-1" && opts.Region != "" {
createBucketConfig := createBucketConfiguration{}
createBucketConfig.Location = opts.Region
var createBucketConfigBytes []byte
createBucketConfigBytes, err = xml.Marshal(createBucketConfig)
if err != nil {
return err
}
reqMetadata.contentMD5Base64 = sumMD5Base64(createBucketConfigBytes)
reqMetadata.contentSHA256Hex = sum256Hex(createBucketConfigBytes)
reqMetadata.contentBody = bytes.NewReader(createBucketConfigBytes)
reqMetadata.contentLength = int64(len(createBucketConfigBytes))
}
// Execute PUT to create a new bucket.
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
defer closeResponse(resp)
if err != nil {
return err
}
if resp != nil {
if resp.StatusCode != http.StatusOK {
return httpRespToErrorResponse(resp, bucketName, "")
}
}
// Success.
return nil
}
// MakeBucketOptions holds all options to tweak bucket creation
type MakeBucketOptions struct {
// Bucket location
Region string
// Enable object locking
ObjectLocking bool
// ForceCreate - this is a MinIO specific extension.
ForceCreate bool
}
// MakeBucket creates a new bucket with bucketName with a context to control cancellations and timeouts.
//
// Location is an optional argument, by default all buckets are
// created in US Standard Region.
//
// For Amazon S3 for more supported regions - http://docs.aws.amazon.com/general/latest/gr/rande.html
// For Google Cloud Storage for more supported regions - https://cloud.google.com/storage/docs/bucket-locations
func (c *Client) MakeBucket(ctx context.Context, bucketName string, opts MakeBucketOptions) (err error) {
return c.makeBucket(ctx, bucketName, opts)
}
Server/vendor/github.com/minio/minio-go/v7/api-put-object-common.go
Generated Vendored
+151
View File
@@ -0,0 +1,151 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2015-2017 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"context"
"io"
"math"
"os"
"github.com/minio/minio-go/v7/pkg/s3utils"
)
const nullVersionID = "null"
// Verify if reader is *minio.Object
func isObject(reader io.Reader) (ok bool) {
_, ok = reader.(*Object)
return ok
}
// Verify if reader is a generic ReaderAt
func isReadAt(reader io.Reader) (ok bool) {
var v *os.File
v, ok = reader.(*os.File)
if ok {
// Stdin, Stdout and Stderr all have *os.File type
// which happen to also be io.ReaderAt compatible
// we need to add special conditions for them to
// be ignored by this function.
for _, f := range []string{
"/dev/stdin",
"/dev/stdout",
"/dev/stderr",
} {
if f == v.Name() {
ok = false
break
}
}
} else {
_, ok = reader.(io.ReaderAt)
}
return ok
}
// OptimalPartInfo - calculate the optimal part info for a given
// object size.
//
// NOTE: Assumption here is that for any object to be uploaded to any S3 compatible
// object storage it will have the following parameters as constants.
//
// maxPartsCount - 10000
// minPartSize - 16MiB
// maxObjectSize - ~48.83TiB (maxPartSize * maxPartsCount)
func OptimalPartInfo(objectSize int64, configuredPartSize uint64) (totalPartsCount int, partSize, lastPartSize int64, err error) {
// When object size is unknown (-1), default to 5TiB to limit memory usage.
// This results in ~537MiB part sizes. For larger objects (up to ~48.83TiB),
// callers should set configuredPartSize explicitly to control memory usage.
var unknownSize bool
if objectSize == -1 {
unknownSize = true
objectSize = maxMultipartPutObjectSize
}
// object size is larger than supported maximum.
if objectSize > maxObjectSize {
err = errEntityTooLarge(objectSize, maxObjectSize, "", "")
return totalPartsCount, partSize, lastPartSize, err
}
var partSizeFlt float64
if configuredPartSize > 0 {
if int64(configuredPartSize) > objectSize {
err = errEntityTooLarge(int64(configuredPartSize), objectSize, "", "")
return totalPartsCount, partSize, lastPartSize, err
}
if !unknownSize {
if objectSize > (int64(configuredPartSize) * maxPartsCount) {
err = errInvalidArgument("Part size * max_parts(10000) is lesser than input objectSize.")
return totalPartsCount, partSize, lastPartSize, err
}
}
if configuredPartSize < absMinPartSize {
err = errInvalidArgument("Input part size is smaller than allowed minimum of 5MiB.")
return totalPartsCount, partSize, lastPartSize, err
}
if configuredPartSize > maxPartSize {
err = errInvalidArgument("Input part size is bigger than allowed maximum of 5GiB.")
return totalPartsCount, partSize, lastPartSize, err
}
partSizeFlt = float64(configuredPartSize)
if unknownSize {
// If input has unknown size and part size is configured
// keep it to maximum allowed as per 10000 parts.
objectSize = int64(configuredPartSize) * maxPartsCount
}
} else {
configuredPartSize = minPartSize
// Use floats for part size for all calculations to avoid
// overflows during float64 to int64 conversions.
partSizeFlt = float64(objectSize / maxPartsCount)
partSizeFlt = math.Ceil(partSizeFlt/float64(configuredPartSize)) * float64(configuredPartSize)
}
// Total parts count.
totalPartsCount = int(math.Ceil(float64(objectSize) / partSizeFlt))
// Part size.
partSize = int64(partSizeFlt)
// Last part size.
lastPartSize = objectSize - int64(totalPartsCount-1)*partSize
return totalPartsCount, partSize, lastPartSize, nil
}
// getUploadID - fetch upload id if already present for an object name
// or initiate a new request to fetch a new upload id.
func (c *Client) newUploadID(ctx context.Context, bucketName, objectName string, opts PutObjectOptions) (uploadID string, err error) {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return "", err
}
if err := s3utils.CheckValidObjectName(objectName); err != nil {
return "", err
}
// Initiate multipart upload for an object.
initMultipartUploadResult, err := c.initiateMultipartUpload(ctx, bucketName, objectName, opts)
if err != nil {
return "", err
}
return initMultipartUploadResult.UploadID, nil
}
Server/vendor/github.com/minio/minio-go/v7/api-put-object-fan-out.go
Generated Vendored
+167
View File
@@ -0,0 +1,167 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2023 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"context"
"encoding/json"
"errors"
"io"
"mime/multipart"
"net/http"
"strconv"
"strings"
"time"
"github.com/minio/minio-go/v7/pkg/encrypt"
)
// PutObjectFanOutEntry is per object entry fan-out metadata
type PutObjectFanOutEntry struct {
Key string `json:"key"`
UserMetadata map[string]string `json:"metadata,omitempty"`
UserTags map[string]string `json:"tags,omitempty"`
ContentType string `json:"contentType,omitempty"`
ContentEncoding string `json:"contentEncoding,omitempty"`
ContentDisposition string `json:"contentDisposition,omitempty"`
ContentLanguage string `json:"contentLanguage,omitempty"`
CacheControl string `json:"cacheControl,omitempty"`
Retention RetentionMode `json:"retention,omitempty"`
RetainUntilDate *time.Time `json:"retainUntil,omitempty"`
}
// PutObjectFanOutRequest this is the request structure sent
// to the server to fan-out the stream to multiple objects.
type PutObjectFanOutRequest struct {
Entries []PutObjectFanOutEntry
Checksum Checksum
SSE encrypt.ServerSide
}
// PutObjectFanOutResponse this is the response structure sent
// by the server upon success or failure for each object
// fan-out keys. Additionally, this response carries ETag,
// VersionID and LastModified for each object fan-out.
type PutObjectFanOutResponse struct {
Key string `json:"key"`
ETag string `json:"etag,omitempty"`
VersionID string `json:"versionId,omitempty"`
LastModified *time.Time `json:"lastModified,omitempty"`
Error string `json:"error,omitempty"`
}
// PutObjectFanOut - is a variant of PutObject instead of writing a single object from a single
// stream multiple objects are written, defined via a list of PutObjectFanOutRequests. Each entry
// in PutObjectFanOutRequest carries an object keyname and its relevant metadata if any. `Key` is
// mandatory, rest of the other options in PutObjectFanOutRequest are optional.
func (c *Client) PutObjectFanOut(ctx context.Context, bucket string, fanOutData io.Reader, fanOutReq PutObjectFanOutRequest) ([]PutObjectFanOutResponse, error) {
if len(fanOutReq.Entries) == 0 {
return nil, errInvalidArgument("fan out requests cannot be empty")
}
policy := NewPostPolicy()
policy.SetBucket(bucket)
policy.SetKey(strconv.FormatInt(time.Now().UnixNano(), 16))
// Expires in 15 minutes.
policy.SetExpires(time.Now().UTC().Add(15 * time.Minute))
// Set encryption headers if any.
policy.SetEncryption(fanOutReq.SSE)
// Set checksum headers if any.
err := policy.SetChecksum(fanOutReq.Checksum)
if err != nil {
return nil, err
}
url, formData, err := c.PresignedPostPolicy(ctx, policy)
if err != nil {
return nil, err
}
r, w := io.Pipe()
req, err := http.NewRequest(http.MethodPost, url.String(), r)
if err != nil {
w.Close()
return nil, err
}
var b strings.Builder
enc := json.NewEncoder(&b)
for _, req := range fanOutReq.Entries {
if req.Key == "" {
w.Close()
return nil, errors.New("PutObjectFanOutRequest.Key is mandatory and cannot be empty")
}
if err = enc.Encode(&req); err != nil {
w.Close()
return nil, err
}
}
mwriter := multipart.NewWriter(w)
req.Header.Add("Content-Type", mwriter.FormDataContentType())
go func() {
defer w.Close()
defer mwriter.Close()
for k, v := range formData {
if err := mwriter.WriteField(k, v); err != nil {
return
}
}
if err := mwriter.WriteField("x-minio-fanout-list", b.String()); err != nil {
return
}
mw, err := mwriter.CreateFormFile("file", "fanout-content")
if err != nil {
return
}
if _, err = io.Copy(mw, fanOutData); err != nil {
return
}
}()
resp, err := c.do(req)
if err != nil {
return nil, err
}
defer closeResponse(resp)
if resp.StatusCode != http.StatusOK {
return nil, httpRespToErrorResponse(resp, bucket, "fanout-content")
}
dec := json.NewDecoder(resp.Body)
fanOutResp := make([]PutObjectFanOutResponse, 0, len(fanOutReq.Entries))
for dec.More() {
var m PutObjectFanOutResponse
if err = dec.Decode(&m); err != nil {
return nil, err
}
fanOutResp = append(fanOutResp, m)
}
return fanOutResp, nil
}
Server/vendor/github.com/minio/minio-go/v7/api-put-object-file-context.go
Generated Vendored
+64
View File
@@ -0,0 +1,64 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2017 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"context"
"mime"
"os"
"path/filepath"
"github.com/minio/minio-go/v7/pkg/s3utils"
)
// FPutObject - Create an object in a bucket, with contents from file at filePath. Allows request cancellation.
func (c *Client) FPutObject(ctx context.Context, bucketName, objectName, filePath string, opts PutObjectOptions) (info UploadInfo, err error) {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return UploadInfo{}, err
}
if err := s3utils.CheckValidObjectName(objectName); err != nil {
return UploadInfo{}, err
}
// Open the referenced file.
fileReader, err := os.Open(filePath)
// If any error fail quickly here.
if err != nil {
return UploadInfo{}, err
}
defer fileReader.Close()
// Save the file stat.
fileStat, err := fileReader.Stat()
if err != nil {
return UploadInfo{}, err
}
// Save the file size.
fileSize := fileStat.Size()
// Set contentType based on filepath extension if not given or default
// value of "application/octet-stream" if the extension has no associated type.
if opts.ContentType == "" {
if opts.ContentType = mime.TypeByExtension(filepath.Ext(filePath)); opts.ContentType == "" {
opts.ContentType = "application/octet-stream"
}
}
return c.PutObject(ctx, bucketName, objectName, fileReader, fileSize, opts)
}
Server/vendor/github.com/minio/minio-go/v7/api-put-object-multipart.go
Generated Vendored
+478
View File
@@ -0,0 +1,478 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2015-2017 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"bytes"
"context"
"encoding/base64"
"encoding/hex"
"encoding/xml"
"fmt"
"io"
"net/http"
"net/url"
"sort"
"strconv"
"strings"
"github.com/google/uuid"
"github.com/minio/minio-go/v7/pkg/encrypt"
"github.com/minio/minio-go/v7/pkg/s3utils"
)
func (c *Client) putObjectMultipart(ctx context.Context, bucketName, objectName string, reader io.Reader, size int64,
opts PutObjectOptions,
) (info UploadInfo, err error) {
info, err = c.putObjectMultipartNoStream(ctx, bucketName, objectName, reader, opts)
if err != nil {
errResp := ToErrorResponse(err)
// Verify if multipart functionality is not available, if not
// fall back to single PutObject operation.
if errResp.Code == AccessDenied && strings.Contains(errResp.Message, "Access Denied") {
// Verify if size of reader is greater than '5GiB'.
if size > maxSinglePutObjectSize {
return UploadInfo{}, errEntityTooLarge(size, maxSinglePutObjectSize, bucketName, objectName)
}
// Fall back to uploading as single PutObject operation.
return c.putObject(ctx, bucketName, objectName, reader, size, opts)
}
}
return info, err
}
func (c *Client) putObjectMultipartNoStream(ctx context.Context, bucketName, objectName string, reader io.Reader, opts PutObjectOptions) (info UploadInfo, err error) {
// Input validation.
if err = s3utils.CheckValidBucketName(bucketName); err != nil {
return UploadInfo{}, err
}
if err = s3utils.CheckValidObjectName(objectName); err != nil {
return UploadInfo{}, err
}
// Total data read and written to server. should be equal to
// 'size' at the end of the call.
var totalUploadedSize int64
// Complete multipart upload.
var complMultipartUpload completeMultipartUpload
// Calculate the optimal parts info for a given size.
totalPartsCount, partSize, _, err := OptimalPartInfo(-1, opts.PartSize)
if err != nil {
return UploadInfo{}, err
}
// Choose hash algorithms to be calculated by hashCopyN,
// avoid sha256 with non-v4 signature request or
// HTTPS connection.
hashAlgos, hashSums := c.hashMaterials(opts.SendContentMd5, !opts.DisableContentSha256)
// Initiate a new multipart upload.
uploadID, err := c.newUploadID(ctx, bucketName, objectName, opts)
if err != nil {
return UploadInfo{}, err
}
defer func() {
if err != nil {
c.abortMultipartUpload(ctx, bucketName, objectName, uploadID)
}
}()
// Part number always starts with '1'.
partNumber := 1
// Initialize parts uploaded map.
partsInfo := make(map[int]ObjectPart)
// Create a buffer.
buf := make([]byte, partSize)
// Create checksums
// CRC32C is ~50% faster on AMD64 @ 30GB/s
customHeader := make(http.Header)
crc := opts.AutoChecksum.Hasher()
for partNumber <= totalPartsCount {
length, rErr := readFull(reader, buf)
if rErr == io.EOF && partNumber > 1 {
break
}
if rErr != nil && rErr != io.ErrUnexpectedEOF && rErr != io.EOF {
return UploadInfo{}, rErr
}
// Calculates hash sums while copying partSize bytes into cw.
for k, v := range hashAlgos {
v.Write(buf[:length])
hashSums[k] = v.Sum(nil)
v.Close()
}
// Update progress reader appropriately to the latest offset
// as we read from the source.
rd := newHook(bytes.NewReader(buf[:length]), opts.Progress)
// Checksums..
var (
md5Base64 string
sha256Hex string
)
if hashSums["md5"] != nil {
md5Base64 = base64.StdEncoding.EncodeToString(hashSums["md5"])
}
if hashSums["sha256"] != nil {
sha256Hex = hex.EncodeToString(hashSums["sha256"])
}
if opts.AutoChecksum.IsSet() {
crc.Reset()
crc.Write(buf[:length])
cSum := crc.Sum(nil)
customHeader.Set(opts.AutoChecksum.Key(), base64.StdEncoding.EncodeToString(cSum))
customHeader.Set(amzChecksumAlgo, opts.AutoChecksum.String())
if opts.AutoChecksum.FullObjectRequested() {
customHeader.Set(amzChecksumMode, ChecksumFullObjectMode.String())
}
}
p := uploadPartParams{bucketName: bucketName, objectName: objectName, uploadID: uploadID, reader: rd, partNumber: partNumber, md5Base64: md5Base64, sha256Hex: sha256Hex, size: int64(length), sse: opts.ServerSideEncryption, streamSha256: !opts.DisableContentSha256, customHeader: customHeader}
// Proceed to upload the part.
objPart, uerr := c.uploadPart(ctx, p)
if uerr != nil {
return UploadInfo{}, uerr
}
// Save successfully uploaded part metadata.
partsInfo[partNumber] = objPart
// Save successfully uploaded size.
totalUploadedSize += int64(length)
// Increment part number.
partNumber++
// For unknown size, Read EOF we break away.
// We do not have to upload till totalPartsCount.
if rErr == io.EOF {
break
}
}
// Loop over total uploaded parts to save them in
// Parts array before completing the multipart request.
allParts := make([]ObjectPart, 0, len(partsInfo))
for i := 1; i < partNumber; i++ {
part, ok := partsInfo[i]
if !ok {
return UploadInfo{}, errInvalidArgument(fmt.Sprintf("Missing part number %d", i))
}
allParts = append(allParts, part)
complMultipartUpload.Parts = append(complMultipartUpload.Parts, CompletePart{
ETag: part.ETag,
PartNumber: part.PartNumber,
ChecksumCRC32: part.ChecksumCRC32,
ChecksumCRC32C: part.ChecksumCRC32C,
ChecksumSHA1: part.ChecksumSHA1,
ChecksumSHA256: part.ChecksumSHA256,
ChecksumCRC64NVME: part.ChecksumCRC64NVME,
ChecksumMD5: part.ChecksumMD5,
ChecksumSHA512: part.ChecksumSHA512,
ChecksumXXHash64: part.ChecksumXXHash64,
ChecksumXXHash3: part.ChecksumXXHash3,
ChecksumXXHash128: part.ChecksumXXHash128,
})
}
// Sort all completed parts.
sort.Sort(completedParts(complMultipartUpload.Parts))
opts = PutObjectOptions{
ServerSideEncryption: opts.ServerSideEncryption,
AutoChecksum: opts.AutoChecksum,
}
applyAutoChecksum(&opts, allParts)
uploadInfo, err := c.completeMultipartUpload(ctx, bucketName, objectName, uploadID, complMultipartUpload, opts)
if err != nil {
return UploadInfo{}, err
}
uploadInfo.Size = totalUploadedSize
return uploadInfo, nil
}
// initiateMultipartUpload - Initiates a multipart upload and returns an upload ID.
func (c *Client) initiateMultipartUpload(ctx context.Context, bucketName, objectName string, opts PutObjectOptions) (initiateMultipartUploadResult, error) {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return initiateMultipartUploadResult{}, err
}
if err := s3utils.CheckValidObjectName(objectName); err != nil {
return initiateMultipartUploadResult{}, err
}
// Initialize url queries.
urlValues := make(url.Values)
urlValues.Set("uploads", "")
if opts.Internal.SourceVersionID != "" {
if opts.Internal.SourceVersionID != nullVersionID {
if _, err := uuid.Parse(opts.Internal.SourceVersionID); err != nil {
return initiateMultipartUploadResult{}, errInvalidArgument(err.Error())
}
}
urlValues.Set("versionId", opts.Internal.SourceVersionID)
}
// Set ContentType header.
customHeader := opts.Header()
reqMetadata := requestMetadata{
bucketName: bucketName,
objectName: objectName,
queryValues: urlValues,
customHeader: customHeader,
}
// Execute POST on an objectName to initiate multipart upload.
resp, err := c.executeMethod(ctx, http.MethodPost, reqMetadata)
defer closeResponse(resp)
if err != nil {
return initiateMultipartUploadResult{}, err
}
if resp != nil {
if resp.StatusCode != http.StatusOK {
return initiateMultipartUploadResult{}, httpRespToErrorResponse(resp, bucketName, objectName)
}
}
// Decode xml for new multipart upload.
initiateMultipartUploadResult := initiateMultipartUploadResult{}
err = xmlDecoder(resp.Body, &initiateMultipartUploadResult)
if err != nil {
return initiateMultipartUploadResult, err
}
return initiateMultipartUploadResult, nil
}
type uploadPartParams struct {
bucketName string
objectName string
uploadID string
reader io.Reader
partNumber int
md5Base64 string
sha256Hex string
size int64
sse encrypt.ServerSide
streamSha256 bool
customHeader http.Header
trailer http.Header
}
// uploadPart - Uploads a part in a multipart upload.
func (c *Client) uploadPart(ctx context.Context, p uploadPartParams) (ObjectPart, error) {
// Input validation.
if err := s3utils.CheckValidBucketName(p.bucketName); err != nil {
return ObjectPart{}, err
}
if err := s3utils.CheckValidObjectName(p.objectName); err != nil {
return ObjectPart{}, err
}
if p.size > maxPartSize {
return ObjectPart{}, errEntityTooLarge(p.size, maxPartSize, p.bucketName, p.objectName)
}
if p.size <= -1 {
return ObjectPart{}, errEntityTooSmall(p.size, p.bucketName, p.objectName)
}
if p.partNumber <= 0 {
return ObjectPart{}, errInvalidArgument("Part number cannot be negative or equal to zero.")
}
if p.uploadID == "" {
return ObjectPart{}, errInvalidArgument("UploadID cannot be empty.")
}
// Get resources properly escaped and lined up before using them in http request.
urlValues := make(url.Values)
// Set part number.
urlValues.Set("partNumber", strconv.Itoa(p.partNumber))
// Set upload id.
urlValues.Set("uploadId", p.uploadID)
// Set encryption headers, if any.
if p.customHeader == nil {
p.customHeader = make(http.Header)
}
// https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html
// Server-side encryption is supported by the S3 Multipart Upload actions.
// Unless you are using a customer-provided encryption key, you don't need
// to specify the encryption parameters in each UploadPart request.
if p.sse != nil && p.sse.Type() == encrypt.SSEC {
p.sse.Marshal(p.customHeader)
}
reqMetadata := requestMetadata{
bucketName: p.bucketName,
objectName: p.objectName,
queryValues: urlValues,
customHeader: p.customHeader,
contentBody: p.reader,
contentLength: p.size,
contentMD5Base64: p.md5Base64,
contentSHA256Hex: p.sha256Hex,
streamSha256: p.streamSha256,
trailer: p.trailer,
}
// Execute PUT on each part.
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
defer closeResponse(resp)
if err != nil {
return ObjectPart{}, err
}
if resp != nil {
if resp.StatusCode != http.StatusOK {
return ObjectPart{}, httpRespToErrorResponse(resp, p.bucketName, p.objectName)
}
}
// Once successfully uploaded, return completed part.
h := resp.Header
objPart := ObjectPart{
ChecksumCRC32: h.Get(ChecksumCRC32.Key()),
ChecksumCRC32C: h.Get(ChecksumCRC32C.Key()),
ChecksumSHA1: h.Get(ChecksumSHA1.Key()),
ChecksumSHA256: h.Get(ChecksumSHA256.Key()),
ChecksumCRC64NVME: h.Get(ChecksumCRC64NVME.Key()),
ChecksumMD5: h.Get(ChecksumMD5.Key()),
ChecksumSHA512: h.Get(ChecksumSHA512.Key()),
ChecksumXXHash64: h.Get(ChecksumXXHash64.Key()),
ChecksumXXHash3: h.Get(ChecksumXXHash3.Key()),
ChecksumXXHash128: h.Get(ChecksumXXHash128.Key()),
}
objPart.Size = p.size
objPart.PartNumber = p.partNumber
// Trim off the odd double quotes from ETag in the beginning and end.
objPart.ETag = trimEtag(h.Get("ETag"))
return objPart, nil
}
// completeMultipartUpload - Completes a multipart upload by assembling previously uploaded parts.
func (c *Client) completeMultipartUpload(ctx context.Context, bucketName, objectName, uploadID string,
complete completeMultipartUpload, opts PutObjectOptions,
) (UploadInfo, error) {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return UploadInfo{}, err
}
if err := s3utils.CheckValidObjectName(objectName); err != nil {
return UploadInfo{}, err
}
// Initialize url queries.
urlValues := make(url.Values)
urlValues.Set("uploadId", uploadID)
// Marshal complete multipart body.
completeMultipartUploadBytes, err := xml.Marshal(complete)
if err != nil {
return UploadInfo{}, err
}
headers := opts.Header()
if s3utils.IsAmazonEndpoint(*c.endpointURL) {
headers.Del(encrypt.SseKmsKeyID) // Remove X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id not supported in CompleteMultipartUpload
headers.Del(encrypt.SseGenericHeader) // Remove X-Amz-Server-Side-Encryption not supported in CompleteMultipartUpload
headers.Del(encrypt.SseEncryptionContext) // Remove X-Amz-Server-Side-Encryption-Context not supported in CompleteMultipartUpload
}
// Instantiate all the complete multipart buffer.
completeMultipartUploadBuffer := bytes.NewReader(completeMultipartUploadBytes)
reqMetadata := requestMetadata{
bucketName: bucketName,
objectName: objectName,
queryValues: urlValues,
contentBody: completeMultipartUploadBuffer,
contentLength: int64(len(completeMultipartUploadBytes)),
contentSHA256Hex: sum256Hex(completeMultipartUploadBytes),
customHeader: headers,
expect200OKWithError: true,
}
// Execute POST to complete multipart upload for an objectName.
resp, err := c.executeMethod(ctx, http.MethodPost, reqMetadata)
defer closeResponse(resp)
if err != nil {
return UploadInfo{}, err
}
if resp != nil {
if resp.StatusCode != http.StatusOK {
return UploadInfo{}, httpRespToErrorResponse(resp, bucketName, objectName)
}
}
// Read resp.Body into a []bytes to parse for Error response inside the body
var b []byte
b, err = io.ReadAll(resp.Body)
if err != nil {
return UploadInfo{}, err
}
// Decode completed multipart upload response on success.
completeMultipartUploadResult := completeMultipartUploadResult{}
err = xmlDecoder(bytes.NewReader(b), &completeMultipartUploadResult)
if err != nil {
// xml parsing failure due to presence an ill-formed xml fragment
return UploadInfo{}, err
} else if completeMultipartUploadResult.Bucket == "" {
// xml's Decode method ignores well-formed xml that don't apply to the type of value supplied.
// In this case, it would leave completeMultipartUploadResult with the corresponding zero-values
// of the members.
// Decode completed multipart upload response on failure
completeMultipartUploadErr := ErrorResponse{}
err = xmlDecoder(bytes.NewReader(b), &completeMultipartUploadErr)
if err != nil {
// xml parsing failure due to presence an ill-formed xml fragment
return UploadInfo{}, err
}
return UploadInfo{}, completeMultipartUploadErr
}
// extract lifecycle expiry date and rule ID
expTime, ruleID := amzExpirationToExpiryDateRuleID(resp.Header.Get(amzExpiration))
return UploadInfo{
Bucket: completeMultipartUploadResult.Bucket,
Key: completeMultipartUploadResult.Key,
ETag: trimEtag(completeMultipartUploadResult.ETag),
VersionID: resp.Header.Get(amzVersionID),
Location: completeMultipartUploadResult.Location,
Expiration: expTime,
ExpirationRuleID: ruleID,
ChecksumSHA256: completeMultipartUploadResult.ChecksumSHA256,
ChecksumSHA1: completeMultipartUploadResult.ChecksumSHA1,
ChecksumCRC32: completeMultipartUploadResult.ChecksumCRC32,
ChecksumCRC32C: completeMultipartUploadResult.ChecksumCRC32C,
ChecksumCRC64NVME: completeMultipartUploadResult.ChecksumCRC64NVME,
ChecksumMD5: completeMultipartUploadResult.ChecksumMD5,
ChecksumSHA512: completeMultipartUploadResult.ChecksumSHA512,
ChecksumXXHash64: completeMultipartUploadResult.ChecksumXXHash64,
ChecksumXXHash3: completeMultipartUploadResult.ChecksumXXHash3,
ChecksumXXHash128: completeMultipartUploadResult.ChecksumXXHash128,
ChecksumMode: completeMultipartUploadResult.ChecksumType,
}, nil
}
Server/vendor/github.com/minio/minio-go/v7/api-put-object-streaming.go
Generated Vendored
+821
View File
@@ -0,0 +1,821 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2017 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"bytes"
"context"
"encoding/base64"
"fmt"
"io"
"net/http"
"net/url"
"sort"
"strings"
"sync"
"github.com/google/uuid"
"github.com/minio/minio-go/v7/pkg/s3utils"
)
// putObjectMultipartStream - upload a large object using
// multipart upload and streaming signature for signing payload.
// Comprehensive put object operation involving multipart uploads.
//
// Following code handles these types of readers.
//
// - *minio.Object
// - Any reader which has a method 'ReadAt()'
func (c *Client) putObjectMultipartStream(ctx context.Context, bucketName, objectName string,
reader io.Reader, size int64, opts PutObjectOptions,
) (info UploadInfo, err error) {
if opts.ConcurrentStreamParts && opts.NumThreads > 1 {
info, err = c.putObjectMultipartStreamParallel(ctx, bucketName, objectName, reader, opts)
} else if !isObject(reader) && isReadAt(reader) && !opts.SendContentMd5 {
// Verify if the reader implements ReadAt and it is not a *minio.Object then we will use parallel uploader.
info, err = c.putObjectMultipartStreamFromReadAt(ctx, bucketName, objectName, reader.(io.ReaderAt), size, opts)
} else {
info, err = c.putObjectMultipartStreamOptionalChecksum(ctx, bucketName, objectName, reader, size, opts)
}
if err != nil && s3utils.IsGoogleEndpoint(*c.endpointURL) {
errResp := ToErrorResponse(err)
// Verify if multipart functionality is not available, if not
// fall back to single PutObject operation.
if errResp.Code == AccessDenied && strings.Contains(errResp.Message, "Access Denied") {
// Verify if size of reader is greater than '5GiB'.
if size > maxSinglePutObjectSize {
return UploadInfo{}, errEntityTooLarge(size, maxSinglePutObjectSize, bucketName, objectName)
}
// Fall back to uploading as single PutObject operation.
return c.putObject(ctx, bucketName, objectName, reader, size, opts)
}
}
return info, err
}
// uploadedPartRes - the response received from a part upload.
type uploadedPartRes struct {
Error error // Any error encountered while uploading the part.
PartNum int // Number of the part uploaded.
Size int64 // Size of the part uploaded.
Part ObjectPart
}
type uploadPartReq struct {
PartNum int // Number of the part uploaded.
Part ObjectPart // Size of the part uploaded.
}
// putObjectMultipartFromReadAt - Uploads files bigger than 128MiB.
// Supports all readers which implements io.ReaderAt interface
// (ReadAt method).
//
// NOTE: This function is meant to be used for all readers which
// implement io.ReaderAt which allows us for resuming multipart
// uploads but reading at an offset, which would avoid re-read the
// data which was already uploaded. Internally this function uses
// temporary files for staging all the data, these temporary files are
// cleaned automatically when the caller i.e http client closes the
// stream after uploading all the contents successfully.
func (c *Client) putObjectMultipartStreamFromReadAt(ctx context.Context, bucketName, objectName string,
reader io.ReaderAt, size int64, opts PutObjectOptions,
) (info UploadInfo, err error) {
// Input validation.
if err = s3utils.CheckValidBucketName(bucketName); err != nil {
return UploadInfo{}, err
}
if err = s3utils.CheckValidObjectName(objectName); err != nil {
return UploadInfo{}, err
}
// Calculate the optimal parts info for a given size.
totalPartsCount, partSize, lastPartSize, err := OptimalPartInfo(size, opts.PartSize)
if err != nil {
return UploadInfo{}, err
}
// Initiate a new multipart upload.
uploadID, err := c.newUploadID(ctx, bucketName, objectName, opts)
if err != nil {
return UploadInfo{}, err
}
withChecksum := c.trailingHeaderSupport
// Aborts the multipart upload in progress, if the
// function returns any error, since we do not resume
// we should purge the parts which have been uploaded
// to relinquish storage space.
defer func() {
if err != nil {
c.abortMultipartUpload(ctx, bucketName, objectName, uploadID)
}
}()
// Total data read and written to server. should be equal to 'size' at the end of the call.
var totalUploadedSize int64
// Complete multipart upload.
var complMultipartUpload completeMultipartUpload
// Declare a channel that sends the next part number to be uploaded.
uploadPartsCh := make(chan uploadPartReq)
// Declare a channel that sends back the response of a part upload.
uploadedPartsCh := make(chan uploadedPartRes)
// Used for readability, lastPartNumber is always totalPartsCount.
lastPartNumber := totalPartsCount
partitionCtx, partitionCancel := context.WithCancel(ctx)
defer partitionCancel()
// Send each part number to the channel to be processed.
go func() {
defer close(uploadPartsCh)
for p := 1; p <= totalPartsCount; p++ {
select {
case <-partitionCtx.Done():
return
case uploadPartsCh <- uploadPartReq{PartNum: p}:
}
}
}()
// Receive each part number from the channel allowing three parallel uploads.
for w := 1; w <= opts.getNumThreads(); w++ {
go func(partSize int64) {
for {
var uploadReq uploadPartReq
var ok bool
select {
case <-ctx.Done():
return
case uploadReq, ok = <-uploadPartsCh:
if !ok {
return
}
// Each worker will draw from the part channel and upload in parallel.
}
// If partNumber was not uploaded we calculate the missing
// part offset and size. For all other part numbers we
// calculate offset based on multiples of partSize.
readOffset := int64(uploadReq.PartNum-1) * partSize
// As a special case if partNumber is lastPartNumber, we
// calculate the offset based on the last part size.
if uploadReq.PartNum == lastPartNumber {
readOffset = size - lastPartSize
partSize = lastPartSize
}
sectionReader := newHook(io.NewSectionReader(reader, readOffset, partSize), opts.Progress)
trailer := make(http.Header, 1)
if withChecksum {
crc := opts.AutoChecksum.Hasher()
trailer.Set(opts.AutoChecksum.Key(), base64.StdEncoding.EncodeToString(crc.Sum(nil)))
sectionReader = newHashReaderWrapper(sectionReader, crc, func(hash []byte) {
trailer.Set(opts.AutoChecksum.Key(), base64.StdEncoding.EncodeToString(hash))
})
}
// Proceed to upload the part.
p := uploadPartParams{
bucketName: bucketName,
objectName: objectName,
uploadID: uploadID,
reader: sectionReader,
partNumber: uploadReq.PartNum,
size: partSize,
sse: opts.ServerSideEncryption,
streamSha256: !opts.DisableContentSha256,
sha256Hex: "",
trailer: trailer,
}
objPart, err := c.uploadPart(ctx, p)
if err != nil {
select {
case <-ctx.Done():
case uploadedPartsCh <- uploadedPartRes{
Error: err,
}:
}
// Exit the goroutine.
return
}
// Save successfully uploaded part metadata.
uploadReq.Part = objPart
// Send successful part info through the channel.
select {
case <-ctx.Done():
case uploadedPartsCh <- uploadedPartRes{
Size: objPart.Size,
PartNum: uploadReq.PartNum,
Part: uploadReq.Part,
}:
}
}
}(partSize)
}
// Gather the responses as they occur and update any
// progress bar.
allParts := make([]ObjectPart, 0, totalPartsCount)
for u := 1; u <= totalPartsCount; u++ {
select {
case <-ctx.Done():
return UploadInfo{}, ctx.Err()
case uploadRes := <-uploadedPartsCh:
if uploadRes.Error != nil {
return UploadInfo{}, uploadRes.Error
}
allParts = append(allParts, uploadRes.Part)
// Update the totalUploadedSize.
totalUploadedSize += uploadRes.Size
complMultipartUpload.Parts = append(complMultipartUpload.Parts, CompletePart{
ETag: uploadRes.Part.ETag,
PartNumber: uploadRes.Part.PartNumber,
ChecksumCRC32: uploadRes.Part.ChecksumCRC32,
ChecksumCRC32C: uploadRes.Part.ChecksumCRC32C,
ChecksumSHA1: uploadRes.Part.ChecksumSHA1,
ChecksumSHA256: uploadRes.Part.ChecksumSHA256,
ChecksumCRC64NVME: uploadRes.Part.ChecksumCRC64NVME,
ChecksumMD5: uploadRes.Part.ChecksumMD5,
ChecksumSHA512: uploadRes.Part.ChecksumSHA512,
ChecksumXXHash64: uploadRes.Part.ChecksumXXHash64,
ChecksumXXHash3: uploadRes.Part.ChecksumXXHash3,
ChecksumXXHash128: uploadRes.Part.ChecksumXXHash128,
})
}
}
// Verify if we uploaded all the data.
if totalUploadedSize != size {
return UploadInfo{}, errUnexpectedEOF(totalUploadedSize, size, bucketName, objectName)
}
// Sort all completed parts.
sort.Sort(completedParts(complMultipartUpload.Parts))
opts = PutObjectOptions{
ServerSideEncryption: opts.ServerSideEncryption,
AutoChecksum: opts.AutoChecksum,
}
if withChecksum {
applyAutoChecksum(&opts, allParts)
}
uploadInfo, err := c.completeMultipartUpload(ctx, bucketName, objectName, uploadID, complMultipartUpload, opts)
if err != nil {
return UploadInfo{}, err
}
uploadInfo.Size = totalUploadedSize
return uploadInfo, nil
}
func (c *Client) putObjectMultipartStreamOptionalChecksum(ctx context.Context, bucketName, objectName string,
reader io.Reader, size int64, opts PutObjectOptions,
) (info UploadInfo, err error) {
// Input validation.
if err = s3utils.CheckValidBucketName(bucketName); err != nil {
return UploadInfo{}, err
}
if err = s3utils.CheckValidObjectName(objectName); err != nil {
return UploadInfo{}, err
}
// Calculate the optimal parts info for a given size.
totalPartsCount, partSize, lastPartSize, err := OptimalPartInfo(size, opts.PartSize)
if err != nil {
return UploadInfo{}, err
}
// Initiates a new multipart request
uploadID, err := c.newUploadID(ctx, bucketName, objectName, opts)
if err != nil {
return UploadInfo{}, err
}
// Aborts the multipart upload if the function returns
// any error, since we do not resume we should purge
// the parts which have been uploaded to relinquish
// storage space.
defer func() {
if err != nil {
c.abortMultipartUpload(ctx, bucketName, objectName, uploadID)
}
}()
// Create checksums
// CRC32C is ~50% faster on AMD64 @ 30GB/s
customHeader := make(http.Header)
crc := opts.AutoChecksum.Hasher()
md5Hash := c.md5Hasher()
defer md5Hash.Close()
// Total data read and written to server. should be equal to 'size' at the end of the call.
var totalUploadedSize int64
// Initialize parts uploaded map.
partsInfo := make(map[int]ObjectPart)
// Create a buffer.
buf := make([]byte, partSize)
// Avoid declaring variables in the for loop
var md5Base64 string
// Part number always starts with '1'.
var partNumber int
for partNumber = 1; partNumber <= totalPartsCount; partNumber++ {
// Proceed to upload the part.
if partNumber == totalPartsCount {
partSize = lastPartSize
}
length, rerr := readFull(reader, buf)
if rerr == io.EOF && partNumber > 1 {
break
}
if rerr != nil && rerr != io.ErrUnexpectedEOF && err != io.EOF {
return UploadInfo{}, rerr
}
// Calculate md5sum.
if opts.SendContentMd5 {
md5Hash.Reset()
md5Hash.Write(buf[:length])
md5Base64 = base64.StdEncoding.EncodeToString(md5Hash.Sum(nil))
}
if opts.AutoChecksum.IsSet() {
// Add CRC32C instead.
crc.Reset()
crc.Write(buf[:length])
cSum := crc.Sum(nil)
customHeader.Set(opts.AutoChecksum.Key(), base64.StdEncoding.EncodeToString(cSum))
customHeader.Set(amzChecksumAlgo, opts.AutoChecksum.String())
if opts.AutoChecksum.FullObjectRequested() {
customHeader.Set(amzChecksumMode, ChecksumFullObjectMode.String())
}
}
// Update progress reader appropriately to the latest offset
// as we read from the source.
hooked := newHook(bytes.NewReader(buf[:length]), opts.Progress)
p := uploadPartParams{bucketName: bucketName, objectName: objectName, uploadID: uploadID, reader: hooked, partNumber: partNumber, md5Base64: md5Base64, size: partSize, sse: opts.ServerSideEncryption, streamSha256: !opts.DisableContentSha256, customHeader: customHeader}
objPart, uerr := c.uploadPart(ctx, p)
if uerr != nil {
return UploadInfo{}, uerr
}
// Save successfully uploaded part metadata.
partsInfo[partNumber] = objPart
// Save successfully uploaded size.
totalUploadedSize += partSize
}
// Verify if we uploaded all the data.
if size > 0 {
if totalUploadedSize != size {
return UploadInfo{}, errUnexpectedEOF(totalUploadedSize, size, bucketName, objectName)
}
}
// Complete multipart upload.
var complMultipartUpload completeMultipartUpload
// Loop over total uploaded parts to save them in
// Parts array before completing the multipart request.
allParts := make([]ObjectPart, 0, len(partsInfo))
for i := 1; i < partNumber; i++ {
part, ok := partsInfo[i]
if !ok {
return UploadInfo{}, errInvalidArgument(fmt.Sprintf("Missing part number %d", i))
}
allParts = append(allParts, part)
complMultipartUpload.Parts = append(complMultipartUpload.Parts, CompletePart{
ETag: part.ETag,
PartNumber: part.PartNumber,
ChecksumCRC32: part.ChecksumCRC32,
ChecksumCRC32C: part.ChecksumCRC32C,
ChecksumSHA1: part.ChecksumSHA1,
ChecksumSHA256: part.ChecksumSHA256,
ChecksumCRC64NVME: part.ChecksumCRC64NVME,
ChecksumMD5: part.ChecksumMD5,
ChecksumSHA512: part.ChecksumSHA512,
ChecksumXXHash64: part.ChecksumXXHash64,
ChecksumXXHash3: part.ChecksumXXHash3,
ChecksumXXHash128: part.ChecksumXXHash128,
})
}
// Sort all completed parts.
sort.Sort(completedParts(complMultipartUpload.Parts))
opts = PutObjectOptions{
ServerSideEncryption: opts.ServerSideEncryption,
AutoChecksum: opts.AutoChecksum,
}
applyAutoChecksum(&opts, allParts)
uploadInfo, err := c.completeMultipartUpload(ctx, bucketName, objectName, uploadID, complMultipartUpload, opts)
if err != nil {
return UploadInfo{}, err
}
uploadInfo.Size = totalUploadedSize
return uploadInfo, nil
}
// putObjectMultipartStreamParallel uploads opts.NumThreads parts in parallel.
// This is expected to take opts.PartSize * opts.NumThreads * (GOGC / 100) bytes of buffer.
func (c *Client) putObjectMultipartStreamParallel(ctx context.Context, bucketName, objectName string,
reader io.Reader, opts PutObjectOptions,
) (info UploadInfo, err error) {
// Input validation.
if err = s3utils.CheckValidBucketName(bucketName); err != nil {
return UploadInfo{}, err
}
if err = s3utils.CheckValidObjectName(objectName); err != nil {
return UploadInfo{}, err
}
// Cancel all when an error occurs.
ctx, cancel := context.WithCancel(ctx)
defer cancel()
// Calculate the optimal parts info for a given size.
totalPartsCount, partSize, _, err := OptimalPartInfo(-1, opts.PartSize)
if err != nil {
return UploadInfo{}, err
}
// Initiates a new multipart request
uploadID, err := c.newUploadID(ctx, bucketName, objectName, opts)
if err != nil {
return UploadInfo{}, err
}
// Aborts the multipart upload if the function returns
// any error, since we do not resume we should purge
// the parts which have been uploaded to relinquish
// storage space.
defer func() {
if err != nil {
c.abortMultipartUpload(ctx, bucketName, objectName, uploadID)
}
}()
// Create checksums
// CRC32C is ~50% faster on AMD64 @ 30GB/s
crc := opts.AutoChecksum.Hasher()
// Total data read and written to server. should be equal to 'size' at the end of the call.
var totalUploadedSize int64
// Initialize parts uploaded map.
partsInfo := make(map[int]ObjectPart)
// Create a buffer.
nBuffers := int64(opts.NumThreads)
bufs := make(chan []byte, nBuffers)
all := make([]byte, nBuffers*partSize)
for i := int64(0); i < nBuffers; i++ {
bufs <- all[i*partSize : i*partSize+partSize]
}
var wg sync.WaitGroup
var mu sync.Mutex
errCh := make(chan error, opts.NumThreads)
reader = newHook(reader, opts.Progress)
// Part number always starts with '1'.
var partNumber int
for partNumber = 1; partNumber <= totalPartsCount; partNumber++ {
// Proceed to upload the part.
var buf []byte
select {
case buf = <-bufs:
case err = <-errCh:
cancel()
wg.Wait()
return UploadInfo{}, err
}
if int64(len(buf)) != partSize {
return UploadInfo{}, fmt.Errorf("read buffer < %d than expected partSize: %d", len(buf), partSize)
}
length, rerr := readFull(reader, buf)
if rerr == io.EOF && partNumber > 1 {
// Done
break
}
if rerr != nil && rerr != io.ErrUnexpectedEOF && rerr != io.EOF {
cancel()
wg.Wait()
return UploadInfo{}, rerr
}
wg.Add(1)
go func(partNumber int) {
// Calculate md5sum.
customHeader := make(http.Header)
if opts.AutoChecksum.IsSet() {
// Add Checksum instead.
crc.Reset()
crc.Write(buf[:length])
cSum := crc.Sum(nil)
customHeader.Set(opts.AutoChecksum.Key(), base64.StdEncoding.EncodeToString(cSum))
customHeader.Set(amzChecksumAlgo, opts.AutoChecksum.String())
if opts.AutoChecksum.FullObjectRequested() {
customHeader.Set(amzChecksumMode, ChecksumFullObjectMode.String())
}
}
// Avoid declaring variables in the for loop
var md5Base64 string
if opts.SendContentMd5 {
md5Hash := c.md5Hasher()
md5Hash.Write(buf[:length])
md5Base64 = base64.StdEncoding.EncodeToString(md5Hash.Sum(nil))
md5Hash.Close()
}
defer wg.Done()
p := uploadPartParams{
bucketName: bucketName,
objectName: objectName,
uploadID: uploadID,
reader: bytes.NewReader(buf[:length]),
partNumber: partNumber,
md5Base64: md5Base64,
size: int64(length),
sse: opts.ServerSideEncryption,
streamSha256: !opts.DisableContentSha256,
customHeader: customHeader,
}
objPart, uerr := c.uploadPart(ctx, p)
if uerr != nil {
errCh <- uerr
return
}
// Save successfully uploaded part metadata.
mu.Lock()
partsInfo[partNumber] = objPart
mu.Unlock()
// Send buffer back so it can be reused.
bufs <- buf
}(partNumber)
// Save successfully uploaded size.
totalUploadedSize += int64(length)
}
wg.Wait()
// Collect any error
select {
case err = <-errCh:
return UploadInfo{}, err
default:
}
// Complete multipart upload.
var complMultipartUpload completeMultipartUpload
// Loop over total uploaded parts to save them in
// Parts array before completing the multipart request.
allParts := make([]ObjectPart, 0, len(partsInfo))
for i := 1; i < partNumber; i++ {
part, ok := partsInfo[i]
if !ok {
return UploadInfo{}, errInvalidArgument(fmt.Sprintf("Missing part number %d", i))
}
allParts = append(allParts, part)
complMultipartUpload.Parts = append(complMultipartUpload.Parts, CompletePart{
ETag: part.ETag,
PartNumber: part.PartNumber,
ChecksumCRC32: part.ChecksumCRC32,
ChecksumCRC32C: part.ChecksumCRC32C,
ChecksumSHA1: part.ChecksumSHA1,
ChecksumSHA256: part.ChecksumSHA256,
ChecksumCRC64NVME: part.ChecksumCRC64NVME,
ChecksumMD5: part.ChecksumMD5,
ChecksumSHA512: part.ChecksumSHA512,
ChecksumXXHash64: part.ChecksumXXHash64,
ChecksumXXHash3: part.ChecksumXXHash3,
ChecksumXXHash128: part.ChecksumXXHash128,
})
}
// Sort all completed parts.
sort.Sort(completedParts(complMultipartUpload.Parts))
opts = PutObjectOptions{
ServerSideEncryption: opts.ServerSideEncryption,
AutoChecksum: opts.AutoChecksum,
}
applyAutoChecksum(&opts, allParts)
uploadInfo, err := c.completeMultipartUpload(ctx, bucketName, objectName, uploadID, complMultipartUpload, opts)
if err != nil {
return UploadInfo{}, err
}
uploadInfo.Size = totalUploadedSize
return uploadInfo, nil
}
// putObject special function used Google Cloud Storage. This special function
// is used for Google Cloud Storage since Google's multipart API is not S3 compatible.
func (c *Client) putObject(ctx context.Context, bucketName, objectName string, reader io.Reader, size int64, opts PutObjectOptions) (info UploadInfo, err error) {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return UploadInfo{}, err
}
if err := s3utils.CheckValidObjectName(objectName); err != nil {
return UploadInfo{}, err
}
// Size -1 is only supported on Google Cloud Storage, we error
// out in all other situations.
if size < 0 && !s3utils.IsGoogleEndpoint(*c.endpointURL) {
return UploadInfo{}, errEntityTooSmall(size, bucketName, objectName)
}
if opts.SendContentMd5 && s3utils.IsGoogleEndpoint(*c.endpointURL) && size < 0 {
return UploadInfo{}, errInvalidArgument("MD5Sum cannot be calculated with size '-1'")
}
var readSeeker io.Seeker
if size > 0 {
if isReadAt(reader) && !isObject(reader) {
seeker, ok := reader.(io.Seeker)
if ok {
offset, err := seeker.Seek(0, io.SeekCurrent)
if err != nil {
return UploadInfo{}, errInvalidArgument(err.Error())
}
reader = io.NewSectionReader(reader.(io.ReaderAt), offset, size)
readSeeker = reader.(io.Seeker)
}
}
}
var md5Base64 string
if opts.SendContentMd5 {
// Calculate md5sum.
hash := c.md5Hasher()
if readSeeker != nil {
if _, err := io.Copy(hash, reader); err != nil {
return UploadInfo{}, err
}
// Seek back to beginning of io.NewSectionReader's offset.
_, err = readSeeker.Seek(0, io.SeekStart)
if err != nil {
return UploadInfo{}, errInvalidArgument(err.Error())
}
} else {
// Create a buffer.
buf := make([]byte, size)
length, err := readFull(reader, buf)
if err != nil && err != io.ErrUnexpectedEOF && err != io.EOF {
return UploadInfo{}, err
}
hash.Write(buf[:length])
reader = bytes.NewReader(buf[:length])
}
md5Base64 = base64.StdEncoding.EncodeToString(hash.Sum(nil))
hash.Close()
}
// Update progress reader appropriately to the latest offset as we
// read from the source.
progressReader := newHook(reader, opts.Progress)
// This function does not calculate sha256 and md5sum for payload.
// Execute put object.
return c.putObjectDo(ctx, bucketName, objectName, progressReader, md5Base64, "", size, opts)
}
// putObjectDo - executes the put object http operation.
// NOTE: You must have WRITE permissions on a bucket to add an object to it.
func (c *Client) putObjectDo(ctx context.Context, bucketName, objectName string, reader io.Reader, md5Base64, sha256Hex string, size int64, opts PutObjectOptions) (UploadInfo, error) {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return UploadInfo{}, err
}
if err := s3utils.CheckValidObjectName(objectName); err != nil {
return UploadInfo{}, err
}
// Set headers.
customHeader := opts.Header()
// Populate request metadata.
reqMetadata := requestMetadata{
bucketName: bucketName,
objectName: objectName,
customHeader: customHeader,
contentBody: reader,
contentLength: size,
contentMD5Base64: md5Base64,
contentSHA256Hex: sha256Hex,
streamSha256: !opts.DisableContentSha256,
}
// Add CRC when client supports it, MD5 is not set, not Google and we don't add SHA256 to chunks.
addCrc := c.trailingHeaderSupport && md5Base64 == "" && !s3utils.IsGoogleEndpoint(*c.endpointURL) && (opts.DisableContentSha256 || c.secure)
if opts.Checksum.IsSet() {
reqMetadata.addCrc = &opts.Checksum
} else if addCrc {
// If user has added checksums, don't add them ourselves.
for k := range opts.UserMetadata {
if strings.HasPrefix(strings.ToLower(k), "x-amz-checksum-") {
addCrc = false
}
}
if addCrc {
opts.AutoChecksum.SetDefault(ChecksumFullObjectCRC32C)
reqMetadata.addCrc = &opts.AutoChecksum
}
}
if opts.Internal.SourceVersionID != "" {
if opts.Internal.SourceVersionID != nullVersionID {
if _, err := uuid.Parse(opts.Internal.SourceVersionID); err != nil {
return UploadInfo{}, errInvalidArgument(err.Error())
}
}
urlValues := make(url.Values)
urlValues.Set("versionId", opts.Internal.SourceVersionID)
reqMetadata.queryValues = urlValues
}
// Execute PUT an objectName.
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
defer closeResponse(resp)
if err != nil {
return UploadInfo{}, err
}
if resp != nil {
if resp.StatusCode != http.StatusOK {
return UploadInfo{}, httpRespToErrorResponse(resp, bucketName, objectName)
}
}
// extract lifecycle expiry date and rule ID
expTime, ruleID := amzExpirationToExpiryDateRuleID(resp.Header.Get(amzExpiration))
h := resp.Header
return UploadInfo{
Bucket: bucketName,
Key: objectName,
ETag: trimEtag(h.Get("ETag")),
VersionID: h.Get(amzVersionID),
Size: size,
Expiration: expTime,
ExpirationRuleID: ruleID,
// Checksum values
ChecksumCRC32: h.Get(ChecksumCRC32.Key()),
ChecksumCRC32C: h.Get(ChecksumCRC32C.Key()),
ChecksumSHA1: h.Get(ChecksumSHA1.Key()),
ChecksumSHA256: h.Get(ChecksumSHA256.Key()),
ChecksumCRC64NVME: h.Get(ChecksumCRC64NVME.Key()),
ChecksumMD5: h.Get(ChecksumMD5.Key()),
ChecksumSHA512: h.Get(ChecksumSHA512.Key()),
ChecksumXXHash64: h.Get(ChecksumXXHash64.Key()),
ChecksumXXHash3: h.Get(ChecksumXXHash3.Key()),
ChecksumXXHash128: h.Get(ChecksumXXHash128.Key()),
ChecksumMode: h.Get(ChecksumFullObjectMode.Key()),
}, nil
}
Server/vendor/github.com/minio/minio-go/v7/api-put-object.go
Generated Vendored
+529
View File
@@ -0,0 +1,529 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2015-2017 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"bytes"
"context"
"encoding/base64"
"errors"
"fmt"
"io"
"net/http"
"sort"
"strings"
"time"
"github.com/minio/minio-go/v7/pkg/encrypt"
"github.com/minio/minio-go/v7/pkg/s3utils"
"github.com/minio/minio-go/v7/pkg/tags"
"golang.org/x/net/http/httpguts"
)
// ReplicationStatus represents replication status of object
type ReplicationStatus string
const (
// ReplicationStatusPending indicates replication is pending
ReplicationStatusPending ReplicationStatus = "PENDING"
// ReplicationStatusComplete indicates replication completed ok
ReplicationStatusComplete ReplicationStatus = "COMPLETED"
// ReplicationStatusFailed indicates replication failed
ReplicationStatusFailed ReplicationStatus = "FAILED"
// ReplicationStatusReplica indicates object is a replica of a source
ReplicationStatusReplica ReplicationStatus = "REPLICA"
// ReplicationStatusReplicaEdge indicates object is a replica of a edge source
ReplicationStatusReplicaEdge ReplicationStatus = "REPLICA-EDGE"
)
// Empty returns true if no replication status set.
func (r ReplicationStatus) Empty() bool {
return r == ""
}
// AdvancedPutOptions for internal use - to be utilized by replication, ILM transition
// implementation on MinIO server
type AdvancedPutOptions struct {
SourceVersionID string
SourceETag string
ReplicationStatus ReplicationStatus
SourceMTime time.Time
ReplicationRequest bool
RetentionTimestamp time.Time
TaggingTimestamp time.Time
LegalholdTimestamp time.Time
ReplicationValidityCheck bool
}
// PutObjectOptions represents options specified by user for PutObject call
type PutObjectOptions struct {
UserMetadata map[string]string
UserTags map[string]string
Progress io.Reader
ContentType string
ContentEncoding string
ContentDisposition string
ContentLanguage string
CacheControl string
Expires time.Time
Mode RetentionMode
RetainUntilDate time.Time
ServerSideEncryption encrypt.ServerSide
NumThreads uint
StorageClass string
WebsiteRedirectLocation string
PartSize uint64
LegalHold LegalHoldStatus
SendContentMd5 bool
DisableContentSha256 bool
DisableMultipart bool
// AutoChecksum is the type of checksum that will be added if no other checksum is added,
// like MD5 or SHA256 streaming checksum, and it is feasible for the upload type.
// If none is specified CRC32C is used, since it is generally the fastest.
AutoChecksum ChecksumType
// Checksum will force a checksum of the specific type.
// This requires that the client was created with "TrailingHeaders:true" option,
// and that the destination server supports it.
// Unavailable with V2 signatures & Google endpoints.
// This will disable content MD5 checksums if set.
Checksum ChecksumType
// ConcurrentStreamParts will create NumThreads buffers of PartSize bytes,
// fill them serially and upload them in parallel.
// This can be used for faster uploads on non-seekable or slow-to-seek input.
ConcurrentStreamParts bool
Internal AdvancedPutOptions
customHeaders http.Header
}
// SetMatchETag if etag matches while PUT MinIO returns an error
// this is a MinIO specific extension to support optimistic locking
// semantics.
func (opts *PutObjectOptions) SetMatchETag(etag string) {
if opts.customHeaders == nil {
opts.customHeaders = http.Header{}
}
if etag == "*" {
opts.customHeaders.Set("If-Match", "*")
} else {
opts.customHeaders.Set("If-Match", "\""+etag+"\"")
}
}
// SetMatchETagExcept if etag does not match while PUT MinIO returns an
// error this is a MinIO specific extension to support optimistic locking
// semantics.
func (opts *PutObjectOptions) SetMatchETagExcept(etag string) {
if opts.customHeaders == nil {
opts.customHeaders = http.Header{}
}
if etag == "*" {
opts.customHeaders.Set("If-None-Match", "*")
} else {
opts.customHeaders.Set("If-None-Match", "\""+etag+"\"")
}
}
// getNumThreads - gets the number of threads to be used in the multipart
// put object operation
func (opts PutObjectOptions) getNumThreads() (numThreads int) {
if opts.NumThreads > 0 {
numThreads = int(opts.NumThreads)
} else {
numThreads = totalWorkers
}
return numThreads
}
// Header - constructs the headers from metadata entered by user in
// PutObjectOptions struct
func (opts PutObjectOptions) Header() (header http.Header) {
header = make(http.Header)
contentType := opts.ContentType
if contentType == "" {
contentType = "application/octet-stream"
}
header.Set("Content-Type", contentType)
if opts.ContentEncoding != "" {
header.Set("Content-Encoding", opts.ContentEncoding)
}
if opts.ContentDisposition != "" {
header.Set("Content-Disposition", opts.ContentDisposition)
}
if opts.ContentLanguage != "" {
header.Set("Content-Language", opts.ContentLanguage)
}
if opts.CacheControl != "" {
header.Set("Cache-Control", opts.CacheControl)
}
if !opts.Expires.IsZero() {
header.Set("Expires", opts.Expires.UTC().Format(http.TimeFormat))
}
if opts.Mode != "" {
header.Set(amzLockMode, opts.Mode.String())
}
if !opts.RetainUntilDate.IsZero() {
header.Set("X-Amz-Object-Lock-Retain-Until-Date", opts.RetainUntilDate.Format(time.RFC3339))
}
if opts.LegalHold != "" {
header.Set(amzLegalHoldHeader, opts.LegalHold.String())
}
if opts.ServerSideEncryption != nil {
opts.ServerSideEncryption.Marshal(header)
}
if opts.StorageClass != "" {
header.Set(amzStorageClass, opts.StorageClass)
}
if opts.WebsiteRedirectLocation != "" {
header.Set(amzWebsiteRedirectLocation, opts.WebsiteRedirectLocation)
}
if !opts.Internal.ReplicationStatus.Empty() {
header.Set(amzBucketReplicationStatus, string(opts.Internal.ReplicationStatus))
}
if !opts.Internal.SourceMTime.IsZero() {
header.Set(minIOBucketSourceMTime, opts.Internal.SourceMTime.Format(time.RFC3339Nano))
}
if opts.Internal.SourceETag != "" {
header.Set(minIOBucketSourceETag, opts.Internal.SourceETag)
}
if opts.Internal.ReplicationRequest {
header.Set(minIOBucketReplicationRequest, "true")
}
if opts.Internal.ReplicationValidityCheck {
header.Set(minIOBucketReplicationCheck, "true")
}
if !opts.Internal.LegalholdTimestamp.IsZero() {
header.Set(minIOBucketReplicationObjectLegalHoldTimestamp, opts.Internal.LegalholdTimestamp.Format(time.RFC3339Nano))
}
if !opts.Internal.RetentionTimestamp.IsZero() {
header.Set(minIOBucketReplicationObjectRetentionTimestamp, opts.Internal.RetentionTimestamp.Format(time.RFC3339Nano))
}
if !opts.Internal.TaggingTimestamp.IsZero() {
header.Set(minIOBucketReplicationTaggingTimestamp, opts.Internal.TaggingTimestamp.Format(time.RFC3339Nano))
}
if len(opts.UserTags) != 0 {
if tags, _ := tags.NewTags(opts.UserTags, true); tags != nil {
header.Set(amzTaggingHeader, tags.String())
}
}
for k, v := range opts.UserMetadata {
if isAmzHeader(k) || isStandardHeader(k) || isStorageClassHeader(k) || isMinioHeader(k) {
header.Set(k, v)
} else {
header.Set("x-amz-meta-"+k, v)
}
}
// set any other additional custom headers.
for k, v := range opts.customHeaders {
header[k] = v
}
return header
}
// validate() checks if the UserMetadata map has standard headers or and raises an error if so.
func (opts PutObjectOptions) validate(c *Client) (err error) {
for k, v := range opts.UserMetadata {
if !httpguts.ValidHeaderFieldName(k) || isStandardHeader(k) || isSSEHeader(k) || isStorageClassHeader(k) || isMinioHeader(k) {
return errInvalidArgument(k + " unsupported user defined metadata name")
}
if !httpguts.ValidHeaderFieldValue(v) {
return errInvalidArgument(v + " unsupported user defined metadata value")
}
}
if opts.Mode != "" && !opts.Mode.IsValid() {
return errInvalidArgument(opts.Mode.String() + " unsupported retention mode")
}
if opts.LegalHold != "" && !opts.LegalHold.IsValid() {
return errInvalidArgument(opts.LegalHold.String() + " unsupported legal-hold status")
}
checkCrc := false
for k := range opts.UserMetadata {
if strings.HasPrefix(k, "x-amz-checksum-") {
checkCrc = true
break
}
}
if opts.Checksum.IsSet() || checkCrc {
switch {
case !c.trailingHeaderSupport:
return errInvalidArgument("Checksum requires Client with TrailingHeaders enabled")
case c.overrideSignerType.IsV2():
return errInvalidArgument("Checksum cannot be used with v2 signatures")
case s3utils.IsGoogleEndpoint(*c.endpointURL):
return errInvalidArgument("Checksum cannot be used with GCS endpoints")
}
}
return nil
}
// completedParts is a collection of parts sortable by their part numbers.
// used for sorting the uploaded parts before completing the multipart request.
type completedParts []CompletePart
func (a completedParts) Len() int { return len(a) }
func (a completedParts) Swap(i, j int) { a[i], a[j] = a[j], a[i] }
func (a completedParts) Less(i, j int) bool { return a[i].PartNumber < a[j].PartNumber }
// PutObject creates an object in a bucket.
//
// You must have WRITE permissions on a bucket to create an object.
//
// - For size smaller than 16MiB PutObject automatically does a
// single atomic PUT operation.
//
// - For size larger than 16MiB PutObject automatically does a
// multipart upload operation.
//
// - For size input as -1 PutObject does a multipart Put operation
// until input stream reaches EOF. Maximum object size that can
// be uploaded through this operation will be 5TiB by default.
// For larger objects (up to ~48.83TiB), set PutObjectOptions.PartSize
// to control memory usage and enable uploads beyond 5TiB.
//
// WARNING: Passing down '-1' will use memory and these cannot
// be reused for best outcomes for PutObject(), pass the size always.
//
// NOTE: Upon errors during upload multipart operation is entirely aborted.
func (c *Client) PutObject(ctx context.Context, bucketName, objectName string, reader io.Reader, size int64,
opts PutObjectOptions,
) (info UploadInfo, err error) {
if size < 0 && opts.DisableMultipart {
return UploadInfo{}, errors.New("object size must be provided with disable multipart upload")
}
err = opts.validate(c)
if err != nil {
return UploadInfo{}, err
}
// Check for largest object size allowed.
if size > int64(maxObjectSize) {
return UploadInfo{}, errEntityTooLarge(size, maxObjectSize, bucketName, objectName)
}
if opts.Checksum.IsSet() {
opts.AutoChecksum = opts.Checksum
opts.SendContentMd5 = false
}
if c.trailingHeaderSupport {
opts.AutoChecksum.SetDefault(ChecksumCRC32C)
addAutoChecksumHeaders(&opts)
}
// NOTE: Streaming signature is not supported by GCS.
if s3utils.IsGoogleEndpoint(*c.endpointURL) {
return c.putObject(ctx, bucketName, objectName, reader, size, opts)
}
partSize := opts.PartSize
if opts.PartSize == 0 {
partSize = minPartSize
}
if c.overrideSignerType.IsV2() {
if size >= 0 && size < int64(partSize) || opts.DisableMultipart {
return c.putObject(ctx, bucketName, objectName, reader, size, opts)
}
return c.putObjectMultipart(ctx, bucketName, objectName, reader, size, opts)
}
if size < 0 {
if opts.DisableMultipart {
return UploadInfo{}, errors.New("no length provided and multipart disabled")
}
if opts.ConcurrentStreamParts && opts.NumThreads > 1 {
return c.putObjectMultipartStreamParallel(ctx, bucketName, objectName, reader, opts)
}
return c.putObjectMultipartStreamNoLength(ctx, bucketName, objectName, reader, opts)
}
if size <= int64(partSize) || opts.DisableMultipart {
return c.putObject(ctx, bucketName, objectName, reader, size, opts)
}
return c.putObjectMultipartStream(ctx, bucketName, objectName, reader, size, opts)
}
func (c *Client) putObjectMultipartStreamNoLength(ctx context.Context, bucketName, objectName string, reader io.Reader, opts PutObjectOptions) (info UploadInfo, err error) {
// Input validation.
if err = s3utils.CheckValidBucketName(bucketName); err != nil {
return UploadInfo{}, err
}
if err = s3utils.CheckValidObjectName(objectName); err != nil {
return UploadInfo{}, err
}
// Total data read and written to server. should be equal to
// 'size' at the end of the call.
var totalUploadedSize int64
// Complete multipart upload.
var complMultipartUpload completeMultipartUpload
// Calculate the optimal parts info for a given size.
totalPartsCount, partSize, _, err := OptimalPartInfo(-1, opts.PartSize)
if err != nil {
return UploadInfo{}, err
}
// Initiate a new multipart upload.
uploadID, err := c.newUploadID(ctx, bucketName, objectName, opts)
if err != nil {
return UploadInfo{}, err
}
defer func() {
if err != nil {
c.abortMultipartUpload(ctx, bucketName, objectName, uploadID)
}
}()
// Part number always starts with '1'.
partNumber := 1
// Initialize parts uploaded map.
partsInfo := make(map[int]ObjectPart)
// Create a buffer.
buf := make([]byte, partSize)
// Create checksums
// CRC32C is ~50% faster on AMD64 @ 30GB/s
customHeader := make(http.Header)
crc := opts.AutoChecksum.Hasher()
for partNumber <= totalPartsCount {
length, rerr := readFull(reader, buf)
if rerr == io.EOF && partNumber > 1 {
break
}
if rerr != nil && rerr != io.ErrUnexpectedEOF && rerr != io.EOF {
return UploadInfo{}, rerr
}
var md5Base64 string
if opts.SendContentMd5 {
// Calculate md5sum.
hash := c.md5Hasher()
hash.Write(buf[:length])
md5Base64 = base64.StdEncoding.EncodeToString(hash.Sum(nil))
hash.Close()
}
if opts.AutoChecksum.IsSet() {
crc.Reset()
crc.Write(buf[:length])
cSum := crc.Sum(nil)
customHeader.Set(opts.AutoChecksum.Key(), base64.StdEncoding.EncodeToString(cSum))
customHeader.Set(amzChecksumAlgo, opts.AutoChecksum.String())
if opts.AutoChecksum.FullObjectRequested() {
customHeader.Set(amzChecksumMode, ChecksumFullObjectMode.String())
}
}
// Update progress reader appropriately to the latest offset
// as we read from the source.
rd := newHook(bytes.NewReader(buf[:length]), opts.Progress)
// Proceed to upload the part.
p := uploadPartParams{bucketName: bucketName, objectName: objectName, uploadID: uploadID, reader: rd, partNumber: partNumber, md5Base64: md5Base64, size: int64(length), sse: opts.ServerSideEncryption, streamSha256: !opts.DisableContentSha256, customHeader: customHeader}
objPart, uerr := c.uploadPart(ctx, p)
if uerr != nil {
return UploadInfo{}, uerr
}
// Save successfully uploaded part metadata.
partsInfo[partNumber] = objPart
// Save successfully uploaded size.
totalUploadedSize += int64(length)
// Increment part number.
partNumber++
// For unknown size, Read EOF we break away.
// We do not have to upload till totalPartsCount.
if rerr == io.EOF {
break
}
}
// Loop over total uploaded parts to save them in
// Parts array before completing the multipart request.
allParts := make([]ObjectPart, 0, len(partsInfo))
for i := 1; i < partNumber; i++ {
part, ok := partsInfo[i]
if !ok {
return UploadInfo{}, errInvalidArgument(fmt.Sprintf("Missing part number %d", i))
}
allParts = append(allParts, part)
complMultipartUpload.Parts = append(complMultipartUpload.Parts, CompletePart{
ETag: part.ETag,
PartNumber: part.PartNumber,
ChecksumCRC32: part.ChecksumCRC32,
ChecksumCRC32C: part.ChecksumCRC32C,
ChecksumSHA1: part.ChecksumSHA1,
ChecksumSHA256: part.ChecksumSHA256,
ChecksumCRC64NVME: part.ChecksumCRC64NVME,
ChecksumMD5: part.ChecksumMD5,
ChecksumSHA512: part.ChecksumSHA512,
ChecksumXXHash64: part.ChecksumXXHash64,
ChecksumXXHash3: part.ChecksumXXHash3,
ChecksumXXHash128: part.ChecksumXXHash128,
})
}
// Sort all completed parts.
sort.Sort(completedParts(complMultipartUpload.Parts))
opts = PutObjectOptions{
ServerSideEncryption: opts.ServerSideEncryption,
AutoChecksum: opts.AutoChecksum,
}
applyAutoChecksum(&opts, allParts)
uploadInfo, err := c.completeMultipartUpload(ctx, bucketName, objectName, uploadID, complMultipartUpload, opts)
if err != nil {
return UploadInfo{}, err
}
uploadInfo.Size = totalUploadedSize
return uploadInfo, nil
}
Server/vendor/github.com/minio/minio-go/v7/api-putobject-snowball.go
Generated Vendored
+246
View File
@@ -0,0 +1,246 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2021 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"archive/tar"
"bufio"
"bytes"
"context"
"fmt"
"io"
"net/http"
"os"
"strings"
"sync"
"time"
"github.com/klauspost/compress/s2"
)
// SnowballOptions contains options for PutObjectsSnowball calls.
type SnowballOptions struct {
// Opts is options applied to all objects.
Opts PutObjectOptions
// Processing options:
// InMemory specifies that all objects should be collected in memory
// before they are uploaded.
// If false a temporary file will be created.
InMemory bool
// Compress enabled content compression before upload.
// Compression will typically reduce memory and network usage,
// Compression can safely be enabled with MinIO hosts.
Compress bool
// SkipErrs if enabled will skip any errors while reading the
// object content while creating the snowball archive
SkipErrs bool
}
// SnowballObject contains information about a single object to be added to the snowball.
type SnowballObject struct {
// Key is the destination key, including prefix.
Key string
// Size is the content size of this object.
Size int64
// Modtime to apply to the object.
// If Modtime is the zero value current time will be used.
ModTime time.Time
// Content of the object.
// Exactly 'Size' number of bytes must be provided.
Content io.Reader
// VersionID of the object; if empty, a new versionID will be generated
VersionID string
// Headers contains more options for this object upload, the same as you
// would include in a regular PutObject operation, such as user metadata
// and content-disposition, expires, ..
Headers http.Header
// Close will be called when an object has finished processing.
// Note that if PutObjectsSnowball returns because of an error,
// objects not consumed from the input will NOT have been closed.
// Leave as nil for no callback.
Close func()
}
type nopReadSeekCloser struct {
io.ReadSeeker
}
func (n nopReadSeekCloser) Close() error {
return nil
}
// This is available as io.ReadSeekCloser from go1.16
type readSeekCloser interface {
io.Reader
io.Closer
io.Seeker
}
// PutObjectsSnowball will put multiple objects with a single put call.
// A (compressed) TAR file will be created which will contain multiple objects.
// The key for each object will be used for the destination in the specified bucket.
// Total size should be < 5TB.
// This function blocks until 'objs' is closed and the content has been uploaded.
func (c *Client) PutObjectsSnowball(ctx context.Context, bucketName string, opts SnowballOptions, objs <-chan SnowballObject) (err error) {
err = opts.Opts.validate(c)
if err != nil {
return err
}
var tmpWriter io.Writer
var getTmpReader func() (rc readSeekCloser, sz int64, err error)
if opts.InMemory {
b := bytes.NewBuffer(nil)
tmpWriter = b
getTmpReader = func() (readSeekCloser, int64, error) {
return nopReadSeekCloser{bytes.NewReader(b.Bytes())}, int64(b.Len()), nil
}
} else {
f, err := os.CreateTemp("", "s3-putsnowballobjects-*")
if err != nil {
return err
}
name := f.Name()
tmpWriter = f
var once sync.Once
defer once.Do(func() {
f.Close()
})
defer os.Remove(name)
getTmpReader = func() (readSeekCloser, int64, error) {
once.Do(func() {
f.Close()
})
f, err := os.Open(name)
if err != nil {
return nil, 0, err
}
st, err := f.Stat()
if err != nil {
return nil, 0, err
}
return f, st.Size(), nil
}
}
flush := func() error { return nil }
if !opts.Compress {
if !opts.InMemory {
// Insert buffer for writes.
buf := bufio.NewWriterSize(tmpWriter, 1<<20)
flush = buf.Flush
tmpWriter = buf
}
} else {
s2c := s2.NewWriter(tmpWriter, s2.WriterBetterCompression())
flush = s2c.Close
defer s2c.Close()
tmpWriter = s2c
}
t := tar.NewWriter(tmpWriter)
objectLoop:
for {
select {
case <-ctx.Done():
return ctx.Err()
case obj, ok := <-objs:
if !ok {
break objectLoop
}
closeObj := func() {}
if obj.Close != nil {
closeObj = obj.Close
}
// Trim accidental slash prefix.
obj.Key = strings.TrimPrefix(obj.Key, "/")
header := tar.Header{
Typeflag: tar.TypeReg,
Name: obj.Key,
Size: obj.Size,
ModTime: obj.ModTime,
Format: tar.FormatPAX,
}
if header.ModTime.IsZero() {
header.ModTime = time.Now().UTC()
}
header.PAXRecords = make(map[string]string)
if obj.VersionID != "" {
header.PAXRecords["minio.versionId"] = obj.VersionID
}
for k, vals := range obj.Headers {
header.PAXRecords["minio.metadata."+k] = strings.Join(vals, ",")
}
if err := t.WriteHeader(&header); err != nil {
closeObj()
return err
}
n, err := io.Copy(t, obj.Content)
if err != nil {
closeObj()
if opts.SkipErrs {
continue
}
return err
}
if n != obj.Size {
closeObj()
if opts.SkipErrs {
continue
}
return io.ErrUnexpectedEOF
}
closeObj()
}
}
// Flush tar
err = t.Flush()
if err != nil {
return err
}
// Flush compression
err = flush()
if err != nil {
return err
}
if opts.Opts.UserMetadata == nil {
opts.Opts.UserMetadata = map[string]string{}
}
opts.Opts.UserMetadata["X-Amz-Meta-Snowball-Auto-Extract"] = "true"
opts.Opts.DisableMultipart = true
rc, sz, err := getTmpReader()
if err != nil {
return err
}
defer rc.Close()
rand := c.random.Uint64()
_, err = c.PutObject(ctx, bucketName, fmt.Sprintf("snowball-upload-%x.tar", rand), rc, sz, opts.Opts)
return err
}
Server/vendor/github.com/minio/minio-go/v7/api-remove.go
Generated Vendored
+796
View File
@@ -0,0 +1,796 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2015-2020 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"bytes"
"context"
"encoding/xml"
"io"
"iter"
"net/http"
"net/url"
"time"
"github.com/minio/minio-go/v7/pkg/s3utils"
)
// useMultiDeleteForBulkDelete returns true if the client should use
// multi-object delete API for bulk delete operations. Returns false
// for endpoints that do not support multi-object delete (e.g., GCS).
func (c *Client) useMultiDeleteForBulkDelete() bool {
// NOTE: GCS does not support multi-object delete API.
return !s3utils.IsGoogleEndpoint(*c.endpointURL)
}
//revive:disable
// Deprecated: BucketOptions will be renamed to RemoveBucketOptions in future versions.
type BucketOptions = RemoveBucketOptions
//revive:enable
// RemoveBucketOptions special headers to purge buckets, only
// useful when endpoint is MinIO
type RemoveBucketOptions struct {
ForceDelete bool
}
// RemoveBucketWithOptions deletes the bucket name.
//
// All objects (including all object versions and delete markers)
// in the bucket will be deleted forcibly if bucket options set
// ForceDelete to 'true'.
func (c *Client) RemoveBucketWithOptions(ctx context.Context, bucketName string, opts RemoveBucketOptions) error {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return err
}
// Build headers.
headers := make(http.Header)
if opts.ForceDelete {
headers.Set(minIOForceDelete, "true")
}
// Execute DELETE on bucket.
resp, err := c.executeMethod(ctx, http.MethodDelete, requestMetadata{
bucketName: bucketName,
contentSHA256Hex: emptySHA256Hex,
customHeader: headers,
})
defer closeResponse(resp)
if err != nil {
return err
}
if resp != nil {
if resp.StatusCode != http.StatusNoContent {
return httpRespToErrorResponse(resp, bucketName, "")
}
}
// Remove the location from cache on a successful delete.
c.bucketLocCache.Delete(bucketName)
return nil
}
// RemoveBucket deletes the bucket name.
//
// All objects (including all object versions and delete markers).
// in the bucket must be deleted before successfully attempting this request.
func (c *Client) RemoveBucket(ctx context.Context, bucketName string) error {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return err
}
// Execute DELETE on bucket.
resp, err := c.executeMethod(ctx, http.MethodDelete, requestMetadata{
bucketName: bucketName,
contentSHA256Hex: emptySHA256Hex,
})
defer closeResponse(resp)
if err != nil {
return err
}
if resp != nil {
if resp.StatusCode != http.StatusNoContent {
return httpRespToErrorResponse(resp, bucketName, "")
}
}
// Remove the location from cache on a successful delete.
c.bucketLocCache.Delete(bucketName)
return nil
}
// AdvancedRemoveOptions intended for internal use by replication
type AdvancedRemoveOptions struct {
ReplicationDeleteMarker bool
ReplicationStatus ReplicationStatus
ReplicationMTime time.Time
ReplicationRequest bool
ReplicationValidityCheck bool // check permissions
}
// RemoveObjectOptions represents options specified by user for RemoveObject call
type RemoveObjectOptions struct {
ForceDelete bool
GovernanceBypass bool
VersionID string
Internal AdvancedRemoveOptions
}
// RemoveObject removes an object from a bucket.
func (c *Client) RemoveObject(ctx context.Context, bucketName, objectName string, opts RemoveObjectOptions) error {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return err
}
if err := s3utils.CheckValidObjectName(objectName); err != nil {
return err
}
res := c.removeObject(ctx, bucketName, objectName, opts)
return res.Err
}
func (c *Client) removeObject(ctx context.Context, bucketName, objectName string, opts RemoveObjectOptions) RemoveObjectResult {
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
if opts.VersionID != "" {
urlValues.Set("versionId", opts.VersionID)
}
// Build headers.
headers := make(http.Header)
if opts.GovernanceBypass {
// Set the bypass goverenance retention header
headers.Set(amzBypassGovernance, "true")
}
if opts.Internal.ReplicationDeleteMarker {
headers.Set(minIOBucketReplicationDeleteMarker, "true")
}
if !opts.Internal.ReplicationMTime.IsZero() {
headers.Set(minIOBucketSourceMTime, opts.Internal.ReplicationMTime.Format(time.RFC3339Nano))
}
if !opts.Internal.ReplicationStatus.Empty() {
headers.Set(amzBucketReplicationStatus, string(opts.Internal.ReplicationStatus))
}
if opts.Internal.ReplicationRequest {
headers.Set(minIOBucketReplicationRequest, "true")
}
if opts.Internal.ReplicationValidityCheck {
headers.Set(minIOBucketReplicationCheck, "true")
}
if opts.ForceDelete {
headers.Set(minIOForceDelete, "true")
}
// Execute DELETE on objectName.
resp, err := c.executeMethod(ctx, http.MethodDelete, requestMetadata{
bucketName: bucketName,
objectName: objectName,
contentSHA256Hex: emptySHA256Hex,
queryValues: urlValues,
customHeader: headers,
})
defer closeResponse(resp)
if err != nil {
return RemoveObjectResult{Err: err}
}
if resp != nil {
// if some unexpected error happened and max retry is reached, we want to let client know
if resp.StatusCode != http.StatusNoContent {
err := httpRespToErrorResponse(resp, bucketName, objectName)
return RemoveObjectResult{Err: err}
}
}
// DeleteObject always responds with http '204' even for
// objects which do not exist. So no need to handle them
// specifically.
return RemoveObjectResult{
ObjectName: objectName,
ObjectVersionID: opts.VersionID,
DeleteMarker: resp.Header.Get("x-amz-delete-marker") == "true",
DeleteMarkerVersionID: resp.Header.Get("x-amz-version-id"),
}
}
// RemoveObjectError - container of Multi Delete S3 API error
type RemoveObjectError struct {
ObjectName string
VersionID string
Err error
}
func (err *RemoveObjectError) Error() string {
// This should never happen as we will have a non-nil error with no underlying error.
if err.Err == nil {
return "unexpected remove object error result"
}
return err.Err.Error()
}
// RemoveObjectResult - container of Multi Delete S3 API result
type RemoveObjectResult struct {
ObjectName string
ObjectVersionID string
DeleteMarker bool
DeleteMarkerVersionID string
Err error
}
// generateRemoveMultiObjects - generate the XML request for remove multi objects request
func generateRemoveMultiObjectsRequest(objects []ObjectInfo) []byte {
delObjects := []deleteObject{}
for _, obj := range objects {
delObjects = append(delObjects, deleteObject{
Key: obj.Key,
VersionID: obj.VersionID,
})
}
xmlBytes, _ := xml.Marshal(deleteMultiObjects{Objects: delObjects, Quiet: false})
return xmlBytes
}
// processRemoveMultiObjectsResponse - parse the remove multi objects web service
// and return the success/failure result status for each object
func processRemoveMultiObjectsResponse(body io.Reader, resultCh chan<- RemoveObjectResult) {
// Parse multi delete XML response
rmResult := &deleteMultiObjectsResult{}
err := xmlDecoder(body, rmResult)
if err != nil {
resultCh <- RemoveObjectResult{ObjectName: "", Err: err}
return
}
// Fill deletion that returned success
for _, obj := range rmResult.DeletedObjects {
resultCh <- RemoveObjectResult{
ObjectName: obj.Key,
// Only filled with versioned buckets
ObjectVersionID: obj.VersionID,
DeleteMarker: obj.DeleteMarker,
DeleteMarkerVersionID: obj.DeleteMarkerVersionID,
}
}
// Fill deletion that returned an error.
for _, obj := range rmResult.UnDeletedObjects {
// Version does not exist is not an error ignore and continue.
switch obj.Code {
case InvalidArgument, NoSuchVersion:
continue
}
resultCh <- RemoveObjectResult{
ObjectName: obj.Key,
ObjectVersionID: obj.VersionID,
Err: ErrorResponse{
Code: obj.Code,
Message: obj.Message,
},
}
}
}
// RemoveObjectsOptions represents options specified by user for RemoveObjects call
type RemoveObjectsOptions struct {
GovernanceBypass bool
}
// RemoveObjects removes multiple objects from a bucket while
// it is possible to specify objects versions which are received from
// objectsCh. Remove failures are sent back via error channel.
func (c *Client) RemoveObjects(ctx context.Context, bucketName string, objectsCh <-chan ObjectInfo, opts RemoveObjectsOptions) <-chan RemoveObjectError {
errorCh := make(chan RemoveObjectError, 1)
// Validate if bucket name is valid.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
defer close(errorCh)
errorCh <- RemoveObjectError{
Err: err,
}
return errorCh
}
// Validate objects channel to be properly allocated.
if objectsCh == nil {
defer close(errorCh)
errorCh <- RemoveObjectError{
Err: errInvalidArgument("Objects channel cannot be nil"),
}
return errorCh
}
resultCh := make(chan RemoveObjectResult, 1)
go c.removeObjects(ctx, bucketName, objectsCh, resultCh, opts)
go func() {
defer close(errorCh)
for res := range resultCh {
// Send only errors to the error channel
if res.Err == nil {
continue
}
errorCh <- RemoveObjectError{
ObjectName: res.ObjectName,
VersionID: res.ObjectVersionID,
Err: res.Err,
}
}
}()
return errorCh
}
// RemoveObjectsWithIter bulk deletes multiple objects from a bucket.
// Objects (with optional versions) to be removed must be provided with
// an iterator. Objects are removed asynchronously and results must be
// consumed. If the returned result iterator is stopped, the context is
// canceled, or a remote call failed, the provided iterator will no
// longer accept more objects.
func (c *Client) RemoveObjectsWithIter(ctx context.Context, bucketName string, objectsIter iter.Seq[ObjectInfo], opts RemoveObjectsOptions) (iter.Seq[RemoveObjectResult], error) {
// Validate if bucket name is valid.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return nil, err
}
// Validate objects channel to be properly allocated.
if objectsIter == nil {
return nil, errInvalidArgument("Objects iter can never by nil")
}
return func(yield func(RemoveObjectResult) bool) {
select {
case <-ctx.Done():
return
default:
}
c.removeObjectsIter(ctx, bucketName, objectsIter, yield, opts)
}, nil
}
// RemoveObjectsWithResult removes multiple objects from a bucket while
// it is possible to specify objects versions which are received from
// objectsCh. Remove results, successes and failures are sent back via
// RemoveObjectResult channel
func (c *Client) RemoveObjectsWithResult(ctx context.Context, bucketName string, objectsCh <-chan ObjectInfo, opts RemoveObjectsOptions) <-chan RemoveObjectResult {
resultCh := make(chan RemoveObjectResult, 1)
// Validate if bucket name is valid.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
defer close(resultCh)
resultCh <- RemoveObjectResult{
Err: err,
}
return resultCh
}
// Validate objects channel to be properly allocated.
if objectsCh == nil {
defer close(resultCh)
resultCh <- RemoveObjectResult{
Err: errInvalidArgument("Objects channel cannot be nil"),
}
return resultCh
}
go c.removeObjects(ctx, bucketName, objectsCh, resultCh, opts)
return resultCh
}
// Return true if the character is within the allowed characters in an XML 1.0 document
// The list of allowed characters can be found here: https://www.w3.org/TR/xml/#charsets
func validXMLChar(r rune) (ok bool) {
return r == 0x09 ||
r == 0x0A ||
r == 0x0D ||
r >= 0x20 && r <= 0xD7FF ||
r >= 0xE000 && r <= 0xFFFD ||
r >= 0x10000 && r <= 0x10FFFF
}
func hasInvalidXMLChar(str string) bool {
for _, s := range str {
if !validXMLChar(s) {
return true
}
}
return false
}
// Generate and call MultiDelete S3 requests based on entries received from the iterator.
func (c *Client) removeObjectsIter(ctx context.Context, bucketName string, objectsIter iter.Seq[ObjectInfo], yield func(RemoveObjectResult) bool, opts RemoveObjectsOptions) {
// NOTE: GCS does not support multi-object delete, use single DELETE requests.
if !c.useMultiDeleteForBulkDelete() {
c.removeObjectsSingleIter(ctx, bucketName, objectsIter, yield, opts)
return
}
maxEntries := 1000
urlValues := make(url.Values)
urlValues.Set("delete", "")
// Build headers.
headers := make(http.Header)
if opts.GovernanceBypass {
// Set the bypass goverenance retention header
headers.Set(amzBypassGovernance, "true")
}
processRemoveMultiObjectsResponseIter := func(batch []ObjectInfo, yield func(RemoveObjectResult) bool) bool {
if len(batch) == 0 {
return false
}
// Generate remove multi objects XML request
removeBytes := generateRemoveMultiObjectsRequest(batch)
// Execute POST on bucket to remove objects.
resp, err := c.executeMethod(ctx, http.MethodPost, requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
contentBody: bytes.NewReader(removeBytes),
contentLength: int64(len(removeBytes)),
contentMD5Base64: sumMD5Base64(removeBytes),
contentSHA256Hex: sum256Hex(removeBytes),
customHeader: headers,
})
if resp != nil {
defer closeResponse(resp)
if resp.StatusCode != http.StatusOK {
err = httpRespToErrorResponse(resp, bucketName, "")
}
}
if err != nil {
for _, b := range batch {
if !yield(RemoveObjectResult{
ObjectName: b.Key,
ObjectVersionID: b.VersionID,
Err: err,
}) {
return false
}
}
return false
}
// Parse multi delete XML response
rmResult := &deleteMultiObjectsResult{}
if err := xmlDecoder(resp.Body, rmResult); err != nil {
yield(RemoveObjectResult{ObjectName: "", Err: err})
return false
}
// Fill deletion that returned an error.
for _, obj := range rmResult.UnDeletedObjects {
// Version does not exist is not an error ignore and continue.
switch obj.Code {
case "InvalidArgument", "NoSuchVersion":
continue
}
if !yield(RemoveObjectResult{
ObjectName: obj.Key,
ObjectVersionID: obj.VersionID,
Err: ErrorResponse{
Code: obj.Code,
Message: obj.Message,
},
}) {
return false
}
}
// Fill deletion that returned success
for _, obj := range rmResult.DeletedObjects {
if !yield(RemoveObjectResult{
ObjectName: obj.Key,
// Only filled with versioned buckets
ObjectVersionID: obj.VersionID,
DeleteMarker: obj.DeleteMarker,
DeleteMarkerVersionID: obj.DeleteMarkerVersionID,
}) {
return false
}
}
return true
}
var batch []ObjectInfo
next, stop := iter.Pull(objectsIter)
defer stop()
for {
// Loop over entries by 1000 and call MultiDelete requests
object, ok := next()
if !ok {
// delete the remaining batch.
processRemoveMultiObjectsResponseIter(batch, yield)
return
}
if hasInvalidXMLChar(object.Key) {
// Use single DELETE so the object name will be in the request URL instead of the multi-delete XML document.
removeResult := c.removeObject(ctx, bucketName, object.Key, RemoveObjectOptions{
VersionID: object.VersionID,
GovernanceBypass: opts.GovernanceBypass,
})
if err := removeResult.Err; err != nil {
// Version does not exist is not an error ignore and continue.
switch ToErrorResponse(err).Code {
case "InvalidArgument", "NoSuchVersion":
continue
}
}
if !yield(removeResult) {
return
}
continue
}
batch = append(batch, object)
if len(batch) < maxEntries {
continue
}
if !processRemoveMultiObjectsResponseIter(batch, yield) {
return
}
batch = batch[:0]
}
}
// Generate and call MultiDelete S3 requests based on entries received from objectsCh
func (c *Client) removeObjects(ctx context.Context, bucketName string, objectsCh <-chan ObjectInfo, resultCh chan<- RemoveObjectResult, opts RemoveObjectsOptions) {
// Close result channel when delete finishes.
defer close(resultCh)
// NOTE: GCS does not support multi-object delete, use single DELETE requests.
if !c.useMultiDeleteForBulkDelete() {
c.removeObjectsSingle(ctx, bucketName, objectsCh, resultCh, opts)
return
}
maxEntries := 1000
finish := false
urlValues := make(url.Values)
urlValues.Set("delete", "")
// Loop over entries by 1000 and call MultiDelete requests
for !finish {
count := 0
var batch []ObjectInfo
// Try to gather 1000 entries
for object := range objectsCh {
if hasInvalidXMLChar(object.Key) {
// Use single DELETE so the object name will be in the request URL instead of the multi-delete XML document.
removeResult := c.removeObject(ctx, bucketName, object.Key, RemoveObjectOptions{
VersionID: object.VersionID,
GovernanceBypass: opts.GovernanceBypass,
})
if err := removeResult.Err; err != nil {
// Version does not exist is not an error ignore and continue.
switch ToErrorResponse(err).Code {
case InvalidArgument, NoSuchVersion:
continue
}
resultCh <- removeResult
}
resultCh <- removeResult
continue
}
batch = append(batch, object)
if count++; count >= maxEntries {
break
}
}
if count == 0 {
// Multi Objects Delete API doesn't accept empty object list, quit immediately
break
}
if count < maxEntries {
// We didn't have 1000 entries, so this is the last batch
finish = true
}
// Build headers.
headers := make(http.Header)
if opts.GovernanceBypass {
// Set the bypass goverenance retention header
headers.Set(amzBypassGovernance, "true")
}
// Generate remove multi objects XML request
removeBytes := generateRemoveMultiObjectsRequest(batch)
// Execute POST on bucket to remove objects.
resp, err := c.executeMethod(ctx, http.MethodPost, requestMetadata{
bucketName: bucketName,
queryValues: urlValues,
contentBody: bytes.NewReader(removeBytes),
contentLength: int64(len(removeBytes)),
contentMD5Base64: sumMD5Base64(removeBytes),
contentSHA256Hex: sum256Hex(removeBytes),
customHeader: headers,
expect200OKWithError: true,
})
if resp != nil && resp.StatusCode != http.StatusOK {
err = httpRespToErrorResponse(resp, bucketName, "")
}
if err != nil {
for _, b := range batch {
resultCh <- RemoveObjectResult{
ObjectName: b.Key,
ObjectVersionID: b.VersionID,
Err: err,
}
}
continue
}
// Process multiobjects remove xml response
processRemoveMultiObjectsResponse(resp.Body, resultCh)
closeResponse(resp)
}
}
// removeObjectsSingle deletes objects one by one using single DELETE requests.
// This is used for endpoints that do not support multi-object delete (e.g., GCS).
func (c *Client) removeObjectsSingle(ctx context.Context, bucketName string, objectsCh <-chan ObjectInfo, resultCh chan<- RemoveObjectResult, opts RemoveObjectsOptions) {
for {
select {
case <-ctx.Done():
return
case object, ok := <-objectsCh:
if !ok {
return
}
removeResult := c.removeObject(ctx, bucketName, object.Key, RemoveObjectOptions{
VersionID: object.VersionID,
GovernanceBypass: opts.GovernanceBypass,
})
if err := removeResult.Err; err != nil {
// Version/object does not exist is not an error, ignore and continue.
switch ToErrorResponse(err).Code {
case NoSuchVersion, NoSuchKey:
continue
}
}
select {
case <-ctx.Done():
return
case resultCh <- removeResult:
}
}
}
}
// removeObjectsSingleIter deletes objects one by one using single DELETE requests.
// This is used for endpoints that do not support multi-object delete (e.g., GCS).
func (c *Client) removeObjectsSingleIter(ctx context.Context, bucketName string, objectsIter iter.Seq[ObjectInfo], yield func(RemoveObjectResult) bool, opts RemoveObjectsOptions) {
for object := range objectsIter {
select {
case <-ctx.Done():
return
default:
}
removeResult := c.removeObject(ctx, bucketName, object.Key, RemoveObjectOptions{
VersionID: object.VersionID,
GovernanceBypass: opts.GovernanceBypass,
})
if err := removeResult.Err; err != nil {
// Version/object does not exist is not an error, ignore and continue.
switch ToErrorResponse(err).Code {
case NoSuchVersion, NoSuchKey:
continue
}
}
if !yield(removeResult) {
return
}
}
}
// RemoveIncompleteUpload aborts an partially uploaded object.
func (c *Client) RemoveIncompleteUpload(ctx context.Context, bucketName, objectName string) error {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return err
}
if err := s3utils.CheckValidObjectName(objectName); err != nil {
return err
}
// Find multipart upload ids of the object to be aborted.
uploadIDs, err := c.findUploadIDs(ctx, bucketName, objectName)
if err != nil {
return err
}
for _, uploadID := range uploadIDs {
// abort incomplete multipart upload, based on the upload id passed.
err := c.abortMultipartUpload(ctx, bucketName, objectName, uploadID)
if err != nil {
return err
}
}
return nil
}
// abortMultipartUpload aborts a multipart upload for the given
// uploadID, all previously uploaded parts are deleted.
func (c *Client) abortMultipartUpload(ctx context.Context, bucketName, objectName, uploadID string) error {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return err
}
if err := s3utils.CheckValidObjectName(objectName); err != nil {
return err
}
// Initialize url queries.
urlValues := make(url.Values)
urlValues.Set("uploadId", uploadID)
// Execute DELETE on multipart upload.
resp, err := c.executeMethod(ctx, http.MethodDelete, requestMetadata{
bucketName: bucketName,
objectName: objectName,
queryValues: urlValues,
contentSHA256Hex: emptySHA256Hex,
})
defer closeResponse(resp)
if err != nil {
return err
}
if resp != nil {
if resp.StatusCode != http.StatusNoContent {
// Abort has no response body, handle it for any errors.
var errorResponse ErrorResponse
switch resp.StatusCode {
case http.StatusNotFound:
// This is needed specifically for abort and it cannot
// be converged into default case.
errorResponse = ErrorResponse{
Code: NoSuchUpload,
Message: "The specified multipart upload does not exist.",
BucketName: bucketName,
Key: objectName,
RequestID: resp.Header.Get("x-amz-request-id"),
HostID: resp.Header.Get("x-amz-id-2"),
Region: resp.Header.Get("x-amz-bucket-region"),
}
default:
return httpRespToErrorResponse(resp, bucketName, objectName)
}
return errorResponse
}
}
return nil
}
Server/vendor/github.com/minio/minio-go/v7/api-restore.go
Generated Vendored
+182
View File
@@ -0,0 +1,182 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* (C) 2018-2021 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"bytes"
"context"
"encoding/xml"
"net/http"
"net/url"
"github.com/minio/minio-go/v7/pkg/s3utils"
"github.com/minio/minio-go/v7/pkg/tags"
)
// RestoreType represents the restore request type
type RestoreType string
const (
// RestoreSelect represents the restore SELECT operation
RestoreSelect = RestoreType("SELECT")
)
// TierType represents a retrieval tier
type TierType string
const (
// TierStandard is the standard retrieval tier
TierStandard = TierType("Standard")
// TierBulk is the bulk retrieval tier
TierBulk = TierType("Bulk")
// TierExpedited is the expedited retrieval tier
TierExpedited = TierType("Expedited")
)
// GlacierJobParameters represents the retrieval tier parameter
type GlacierJobParameters struct {
Tier TierType
}
// Encryption contains the type of server-side encryption used during object retrieval
type Encryption struct {
EncryptionType string
KMSContext string
KMSKeyID string `xml:"KMSKeyId"`
}
// MetadataEntry represents a metadata information of the restored object.
type MetadataEntry struct {
Name string
Value string
}
// S3 holds properties of the copy of the archived object
type S3 struct {
AccessControlList *AccessControlList `xml:"AccessControlList,omitempty"`
BucketName string
Prefix string
CannedACL *string `xml:"CannedACL,omitempty"`
Encryption *Encryption `xml:"Encryption,omitempty"`
StorageClass *string `xml:"StorageClass,omitempty"`
Tagging *tags.Tags `xml:"Tagging,omitempty"`
UserMetadata *MetadataEntry `xml:"UserMetadata,omitempty"`
}
// SelectParameters holds the select request parameters
type SelectParameters struct {
XMLName xml.Name `xml:"SelectParameters"`
ExpressionType QueryExpressionType
Expression string
InputSerialization SelectObjectInputSerialization
OutputSerialization SelectObjectOutputSerialization
}
// OutputLocation holds properties of the copy of the archived object
type OutputLocation struct {
XMLName xml.Name `xml:"OutputLocation"`
S3 S3 `xml:"S3"`
}
// RestoreRequest holds properties of the restore object request
type RestoreRequest struct {
XMLName xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ RestoreRequest"`
Type *RestoreType `xml:"Type,omitempty"`
Tier *TierType `xml:"Tier,omitempty"`
Days *int `xml:"Days,omitempty"`
GlacierJobParameters *GlacierJobParameters `xml:"GlacierJobParameters,omitempty"`
Description *string `xml:"Description,omitempty"`
SelectParameters *SelectParameters `xml:"SelectParameters,omitempty"`
OutputLocation *OutputLocation `xml:"OutputLocation,omitempty"`
}
// SetDays sets the days parameter of the restore request
func (r *RestoreRequest) SetDays(v int) {
r.Days = &v
}
// SetGlacierJobParameters sets the GlacierJobParameters of the restore request
func (r *RestoreRequest) SetGlacierJobParameters(v GlacierJobParameters) {
r.GlacierJobParameters = &v
}
// SetType sets the type of the restore request
func (r *RestoreRequest) SetType(v RestoreType) {
r.Type = &v
}
// SetTier sets the retrieval tier of the restore request
func (r *RestoreRequest) SetTier(v TierType) {
r.Tier = &v
}
// SetDescription sets the description of the restore request
func (r *RestoreRequest) SetDescription(v string) {
r.Description = &v
}
// SetSelectParameters sets SelectParameters of the restore select request
func (r *RestoreRequest) SetSelectParameters(v SelectParameters) {
r.SelectParameters = &v
}
// SetOutputLocation sets the properties of the copy of the archived object
func (r *RestoreRequest) SetOutputLocation(v OutputLocation) {
r.OutputLocation = &v
}
// RestoreObject is a implementation of https://docs.aws.amazon.com/AmazonS3/latest/API/API_RestoreObject.html AWS S3 API
func (c *Client) RestoreObject(ctx context.Context, bucketName, objectName, versionID string, req RestoreRequest) error {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return err
}
if err := s3utils.CheckValidObjectName(objectName); err != nil {
return err
}
restoreRequestBytes, err := xml.Marshal(req)
if err != nil {
return err
}
urlValues := make(url.Values)
urlValues.Set("restore", "")
if versionID != "" {
urlValues.Set("versionId", versionID)
}
// Execute POST on bucket/object.
resp, err := c.executeMethod(ctx, http.MethodPost, requestMetadata{
bucketName: bucketName,
objectName: objectName,
queryValues: urlValues,
contentMD5Base64: sumMD5Base64(restoreRequestBytes),
contentSHA256Hex: sum256Hex(restoreRequestBytes),
contentBody: bytes.NewReader(restoreRequestBytes),
contentLength: int64(len(restoreRequestBytes)),
})
defer closeResponse(resp)
if err != nil {
return err
}
if resp.StatusCode != http.StatusAccepted && resp.StatusCode != http.StatusOK {
return httpRespToErrorResponse(resp, bucketName, "")
}
return nil
}
Server/vendor/github.com/minio/minio-go/v7/api-s3-datatypes.go
Generated Vendored
+509
View File
@@ -0,0 +1,509 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2015-2020 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"encoding/base64"
"encoding/xml"
"errors"
"io"
"reflect"
"time"
)
// listAllMyBucketsResult container for listBuckets response.
type listAllMyBucketsResult struct {
// Container for one or more buckets.
Buckets struct {
Bucket []BucketInfo
}
Owner owner
}
// listAllMyDirectoryBucketsResult container for listDirectoryBuckets response.
type listAllMyDirectoryBucketsResult struct {
Buckets struct {
Bucket []BucketInfo
}
ContinuationToken string
}
// owner container for bucket owner information.
type owner struct {
DisplayName string
ID string
}
// CommonPrefix container for prefix response.
type CommonPrefix struct {
Prefix string
}
// ListBucketV2Result container for listObjects response version 2.
type ListBucketV2Result struct {
// A response can contain CommonPrefixes only if you have
// specified a delimiter.
CommonPrefixes []CommonPrefix
// Metadata about each object returned.
Contents []ObjectInfo
Delimiter string
// Encoding type used to encode object keys in the response.
EncodingType string
// A flag that indicates whether or not ListObjects returned all of the results
// that satisfied the search criteria.
IsTruncated bool
MaxKeys int64
Name string
// Hold the token that will be sent in the next request to fetch the next group of keys
NextContinuationToken string
ContinuationToken string
Prefix string
// FetchOwner and StartAfter are currently not used
FetchOwner string
StartAfter string
}
// Version is an element in the list object versions response
type Version struct {
ETag string
IsLatest bool
Key string
LastModified time.Time
Owner Owner
Size int64
StorageClass string
VersionID string `xml:"VersionId"`
// x-amz-meta-* headers stripped "x-amz-meta-" prefix containing the first value.
// Only returned by MinIO servers.
UserMetadata StringMap `json:"userMetadata,omitempty"`
// x-amz-tagging values in their k/v values.
// Only returned by MinIO servers.
UserTags URLMap `json:"userTags,omitempty" xml:"UserTags"`
Internal *struct {
K int // Data blocks
M int // Parity blocks
} `xml:"Internal"`
// Checksum values. Only returned by AiStor servers.
ChecksumCRC32 string `xml:",omitempty"`
ChecksumCRC32C string `xml:",omitempty"`
ChecksumSHA1 string `xml:",omitempty"`
ChecksumSHA256 string `xml:",omitempty"`
ChecksumCRC64NVME string `xml:",omitempty"`
ChecksumMD5 string `xml:",omitempty"`
ChecksumSHA512 string `xml:",omitempty"`
ChecksumXXHash64 string `xml:"ChecksumXXHASH64,omitempty"`
ChecksumXXHash3 string `xml:"ChecksumXXHASH3,omitempty"`
ChecksumXXHash128 string `xml:"ChecksumXXHASH128,omitempty"`
ChecksumAlgorithm string `xml:",omitempty"`
ChecksumType string `xml:",omitempty"`
isDeleteMarker bool
}
// ListVersionsResult is an element in the list object versions response
// and has a special Unmarshaler because we need to preserver the order
// of <Version> and <DeleteMarker> in ListVersionsResult.Versions slice
type ListVersionsResult struct {
Versions []Version
CommonPrefixes []CommonPrefix
Name string
Prefix string
Delimiter string
MaxKeys int64
EncodingType string
IsTruncated bool
KeyMarker string
VersionIDMarker string
NextKeyMarker string
NextVersionIDMarker string
}
// UnmarshalXML is a custom unmarshal code for the response of ListObjectVersions, the custom
// code will unmarshal <Version> and <DeleteMarker> tags and save them in Versions field to
// preserve the lexical order of the listing.
func (l *ListVersionsResult) UnmarshalXML(d *xml.Decoder, _ xml.StartElement) (err error) {
for {
// Read tokens from the XML document in a stream.
t, err := d.Token()
if err != nil {
if err == io.EOF {
break
}
return err
}
se, ok := t.(xml.StartElement)
if ok {
tagName := se.Name.Local
switch tagName {
case "Name", "Prefix",
"Delimiter", "EncodingType",
"KeyMarker", "NextKeyMarker":
var s string
if err = d.DecodeElement(&s, &se); err != nil {
return err
}
v := reflect.ValueOf(l).Elem().FieldByName(tagName)
if v.IsValid() {
v.SetString(s)
}
case "VersionIdMarker":
// VersionIdMarker is a special case because of 'Id' instead of 'ID' in field name
var s string
if err = d.DecodeElement(&s, &se); err != nil {
return err
}
l.VersionIDMarker = s
case "NextVersionIdMarker":
// NextVersionIdMarker is a special case because of 'Id' instead of 'ID' in field name
var s string
if err = d.DecodeElement(&s, &se); err != nil {
return err
}
l.NextVersionIDMarker = s
case "IsTruncated": // bool
var b bool
if err = d.DecodeElement(&b, &se); err != nil {
return err
}
l.IsTruncated = b
case "MaxKeys": // int64
var i int64
if err = d.DecodeElement(&i, &se); err != nil {
return err
}
l.MaxKeys = i
case "CommonPrefixes":
var cp CommonPrefix
if err = d.DecodeElement(&cp, &se); err != nil {
return err
}
l.CommonPrefixes = append(l.CommonPrefixes, cp)
case "DeleteMarker", "Version":
var v Version
if err = d.DecodeElement(&v, &se); err != nil {
return err
}
if tagName == "DeleteMarker" {
v.isDeleteMarker = true
}
l.Versions = append(l.Versions, v)
default:
return errors.New("unrecognized option:" + tagName)
}
}
}
return nil
}
// ListBucketResult container for listObjects response.
type ListBucketResult struct {
// A response can contain CommonPrefixes only if you have
// specified a delimiter.
CommonPrefixes []CommonPrefix
// Metadata about each object returned.
Contents []ObjectInfo
Delimiter string
// Encoding type used to encode object keys in the response.
EncodingType string
// A flag that indicates whether or not ListObjects returned all of the results
// that satisfied the search criteria.
IsTruncated bool
Marker string
MaxKeys int64
Name string
// When response is truncated (the IsTruncated element value in
// the response is true), you can use the key name in this field
// as marker in the subsequent request to get next set of objects.
// Object storage lists objects in alphabetical order Note: This
// element is returned only if you have delimiter request
// parameter specified. If response does not include the NextMaker
// and it is truncated, you can use the value of the last Key in
// the response as the marker in the subsequent request to get the
// next set of object keys.
NextMarker string
Prefix string
}
// ListMultipartUploadsResult container for ListMultipartUploads response
type ListMultipartUploadsResult struct {
Bucket string
KeyMarker string
UploadIDMarker string `xml:"UploadIdMarker"`
NextKeyMarker string
NextUploadIDMarker string `xml:"NextUploadIdMarker"`
EncodingType string
MaxUploads int64
IsTruncated bool
Uploads []ObjectMultipartInfo `xml:"Upload"`
Prefix string
Delimiter string
// A response can contain CommonPrefixes only if you specify a delimiter.
CommonPrefixes []CommonPrefix
}
// initiator container for who initiated multipart upload.
type initiator struct {
ID string
DisplayName string
}
// copyObjectResult container for copy object response.
type copyObjectResult struct {
ETag string
LastModified time.Time // time string format "2006-01-02T15:04:05.000Z"
}
// ObjectPart container for particular part of an object.
type ObjectPart struct {
// Part number identifies the part.
PartNumber int
// Date and time the part was uploaded.
LastModified time.Time
// Entity tag returned when the part was uploaded, usually md5sum
// of the part.
ETag string
// Size of the uploaded part data.
Size int64
// Checksum values of each part.
ChecksumCRC32 string
ChecksumCRC32C string
ChecksumSHA1 string
ChecksumSHA256 string
ChecksumCRC64NVME string
ChecksumMD5 string
ChecksumSHA512 string
ChecksumXXHash64 string `xml:"ChecksumXXHASH64,omitempty"`
ChecksumXXHash3 string `xml:"ChecksumXXHASH3,omitempty"`
ChecksumXXHash128 string `xml:"ChecksumXXHASH128,omitempty"`
}
// Checksum will return the checksum for the given type.
// Will return the empty string if not set.
func (c ObjectPart) Checksum(t ChecksumType) string {
switch {
case t.Is(ChecksumCRC32C):
return c.ChecksumCRC32C
case t.Is(ChecksumCRC32):
return c.ChecksumCRC32
case t.Is(ChecksumSHA1):
return c.ChecksumSHA1
case t.Is(ChecksumSHA256):
return c.ChecksumSHA256
case t.Is(ChecksumCRC64NVME):
return c.ChecksumCRC64NVME
case t.Is(ChecksumMD5):
return c.ChecksumMD5
case t.Is(ChecksumSHA512):
return c.ChecksumSHA512
case t.Is(ChecksumXXHash64):
return c.ChecksumXXHash64
case t.Is(ChecksumXXHash3):
return c.ChecksumXXHash3
case t.Is(ChecksumXXHash128):
return c.ChecksumXXHash128
}
return ""
}
// ChecksumRaw returns the decoded checksum from the part.
func (c ObjectPart) ChecksumRaw(t ChecksumType) ([]byte, error) {
b64 := c.Checksum(t)
if b64 == "" {
return nil, errors.New("no checksum set")
}
decoded, err := base64.StdEncoding.DecodeString(b64)
if err != nil {
return nil, err
}
if len(decoded) != t.RawByteLen() {
return nil, errors.New("checksum length mismatch")
}
return decoded, nil
}
// ListObjectPartsResult container for ListObjectParts response.
type ListObjectPartsResult struct {
Bucket string
Key string
UploadID string `xml:"UploadId"`
Initiator initiator
Owner owner
StorageClass string
PartNumberMarker int
NextPartNumberMarker int
MaxParts int
// ChecksumAlgorithm will be CRC32, CRC32C, etc.
ChecksumAlgorithm string
// ChecksumType is FULL_OBJECT or COMPOSITE (assume COMPOSITE when unset)
ChecksumType string
// Indicates whether the returned list of parts is truncated.
IsTruncated bool
ObjectParts []ObjectPart `xml:"Part"`
EncodingType string
}
// initiateMultipartUploadResult container for InitiateMultiPartUpload
// response.
type initiateMultipartUploadResult struct {
Bucket string
Key string
UploadID string `xml:"UploadId"`
}
// completeMultipartUploadResult container for completed multipart
// upload response.
type completeMultipartUploadResult struct {
Location string
Bucket string
Key string
ETag string
// Checksum values, hash of hashes of parts.
ChecksumCRC32 string
ChecksumCRC32C string
ChecksumSHA1 string
ChecksumSHA256 string
ChecksumCRC64NVME string
ChecksumMD5 string
ChecksumSHA512 string
ChecksumXXHash64 string `xml:"ChecksumXXHASH64"`
ChecksumXXHash3 string `xml:"ChecksumXXHASH3"`
ChecksumXXHash128 string `xml:"ChecksumXXHASH128"`
ChecksumType string
}
// CompletePart sub container lists individual part numbers and their
// md5sum, part of completeMultipartUpload.
type CompletePart struct {
// Part number identifies the part.
PartNumber int
ETag string
// Checksum values
ChecksumCRC32 string `xml:"ChecksumCRC32,omitempty"`
ChecksumCRC32C string `xml:"ChecksumCRC32C,omitempty"`
ChecksumSHA1 string `xml:"ChecksumSHA1,omitempty"`
ChecksumSHA256 string `xml:"ChecksumSHA256,omitempty"`
ChecksumCRC64NVME string `xml:",omitempty"`
ChecksumMD5 string `xml:",omitempty"`
ChecksumSHA512 string `xml:",omitempty"`
ChecksumXXHash64 string `xml:"ChecksumXXHASH64,omitempty"`
ChecksumXXHash3 string `xml:"ChecksumXXHASH3,omitempty"`
ChecksumXXHash128 string `xml:"ChecksumXXHASH128,omitempty"`
}
// Checksum will return the checksum for the given type.
// Will return the empty string if not set.
func (c CompletePart) Checksum(t ChecksumType) string {
switch {
case t.Is(ChecksumCRC32C):
return c.ChecksumCRC32C
case t.Is(ChecksumCRC32):
return c.ChecksumCRC32
case t.Is(ChecksumSHA1):
return c.ChecksumSHA1
case t.Is(ChecksumSHA256):
return c.ChecksumSHA256
case t.Is(ChecksumCRC64NVME):
return c.ChecksumCRC64NVME
case t.Is(ChecksumMD5):
return c.ChecksumMD5
case t.Is(ChecksumSHA512):
return c.ChecksumSHA512
case t.Is(ChecksumXXHash64):
return c.ChecksumXXHash64
case t.Is(ChecksumXXHash3):
return c.ChecksumXXHash3
case t.Is(ChecksumXXHash128):
return c.ChecksumXXHash128
}
return ""
}
// completeMultipartUpload container for completing multipart upload.
type completeMultipartUpload struct {
XMLName xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ CompleteMultipartUpload" json:"-"`
Parts []CompletePart `xml:"Part"`
}
// createBucketConfiguration container for bucket configuration.
type createBucketConfiguration struct {
XMLName xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ CreateBucketConfiguration" json:"-"`
Location string `xml:"LocationConstraint"`
}
// deleteObject container for Delete element in MultiObjects Delete XML request
type deleteObject struct {
Key string
VersionID string `xml:"VersionId,omitempty"`
}
// deletedObject container for Deleted element in MultiObjects Delete XML response
type deletedObject struct {
Key string
VersionID string `xml:"VersionId,omitempty"`
// These fields are ignored.
DeleteMarker bool
DeleteMarkerVersionID string `xml:"DeleteMarkerVersionId,omitempty"`
}
// nonDeletedObject container for Error element (failed deletion) in MultiObjects Delete XML response
type nonDeletedObject struct {
Key string
Code string
Message string
VersionID string `xml:"VersionId"`
}
// deletedMultiObjects container for MultiObjects Delete XML request
type deleteMultiObjects struct {
XMLName xml.Name `xml:"Delete"`
Quiet bool
Objects []deleteObject `xml:"Object"`
}
// deletedMultiObjectsResult container for MultiObjects Delete XML response
type deleteMultiObjectsResult struct {
XMLName xml.Name `xml:"DeleteResult"`
DeletedObjects []deletedObject `xml:"Deleted"`
UnDeletedObjects []nonDeletedObject `xml:"Error"`
}
Server/vendor/github.com/minio/minio-go/v7/api-select.go
Generated Vendored
+755
View File
@@ -0,0 +1,755 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* (C) 2018-2020 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"bytes"
"context"
"encoding/binary"
"encoding/xml"
"errors"
"fmt"
"hash"
"hash/crc32"
"io"
"net/http"
"net/url"
"strings"
"github.com/minio/minio-go/v7/pkg/encrypt"
"github.com/minio/minio-go/v7/pkg/s3utils"
)
// CSVFileHeaderInfo - is the parameter for whether to utilize headers.
type CSVFileHeaderInfo string
// Constants for file header info.
const (
CSVFileHeaderInfoNone CSVFileHeaderInfo = "NONE"
CSVFileHeaderInfoIgnore CSVFileHeaderInfo = "IGNORE"
CSVFileHeaderInfoUse CSVFileHeaderInfo = "USE"
)
// SelectCompressionType - is the parameter for what type of compression is
// present
type SelectCompressionType string
// Constants for compression types under select API.
const (
SelectCompressionNONE SelectCompressionType = "NONE"
SelectCompressionGZIP SelectCompressionType = "GZIP"
SelectCompressionBZIP SelectCompressionType = "BZIP2"
// Non-standard compression schemes, supported by MinIO hosts:
SelectCompressionZSTD SelectCompressionType = "ZSTD" // Zstandard compression.
SelectCompressionLZ4 SelectCompressionType = "LZ4" // LZ4 Stream
SelectCompressionS2 SelectCompressionType = "S2" // S2 Stream
SelectCompressionSNAPPY SelectCompressionType = "SNAPPY" // Snappy stream
)
// CSVQuoteFields - is the parameter for how CSV fields are quoted.
type CSVQuoteFields string
// Constants for csv quote styles.
const (
CSVQuoteFieldsAlways CSVQuoteFields = "Always"
CSVQuoteFieldsAsNeeded CSVQuoteFields = "AsNeeded"
)
// QueryExpressionType - is of what syntax the expression is, this should only
// be SQL
type QueryExpressionType string
// Constants for expression type.
const (
QueryExpressionTypeSQL QueryExpressionType = "SQL"
)
// JSONType determines json input serialization type.
type JSONType string
// Constants for JSONTypes.
const (
JSONDocumentType JSONType = "DOCUMENT"
JSONLinesType JSONType = "LINES"
)
// ParquetInputOptions parquet input specific options
type ParquetInputOptions struct{}
// CSVInputOptions csv input specific options
type CSVInputOptions struct {
FileHeaderInfo CSVFileHeaderInfo
fileHeaderInfoSet bool
RecordDelimiter string
recordDelimiterSet bool
FieldDelimiter string
fieldDelimiterSet bool
QuoteCharacter string
quoteCharacterSet bool
QuoteEscapeCharacter string
quoteEscapeCharacterSet bool
Comments string
commentsSet bool
}
// SetFileHeaderInfo sets the file header info in the CSV input options
func (c *CSVInputOptions) SetFileHeaderInfo(val CSVFileHeaderInfo) {
c.FileHeaderInfo = val
c.fileHeaderInfoSet = true
}
// SetRecordDelimiter sets the record delimiter in the CSV input options
func (c *CSVInputOptions) SetRecordDelimiter(val string) {
c.RecordDelimiter = val
c.recordDelimiterSet = true
}
// SetFieldDelimiter sets the field delimiter in the CSV input options
func (c *CSVInputOptions) SetFieldDelimiter(val string) {
c.FieldDelimiter = val
c.fieldDelimiterSet = true
}
// SetQuoteCharacter sets the quote character in the CSV input options
func (c *CSVInputOptions) SetQuoteCharacter(val string) {
c.QuoteCharacter = val
c.quoteCharacterSet = true
}
// SetQuoteEscapeCharacter sets the quote escape character in the CSV input options
func (c *CSVInputOptions) SetQuoteEscapeCharacter(val string) {
c.QuoteEscapeCharacter = val
c.quoteEscapeCharacterSet = true
}
// SetComments sets the comments character in the CSV input options
func (c *CSVInputOptions) SetComments(val string) {
c.Comments = val
c.commentsSet = true
}
// MarshalXML - produces the xml representation of the CSV input options struct
func (c CSVInputOptions) MarshalXML(e *xml.Encoder, start xml.StartElement) error {
if err := e.EncodeToken(start); err != nil {
return err
}
if c.FileHeaderInfo != "" || c.fileHeaderInfoSet {
if err := e.EncodeElement(c.FileHeaderInfo, xml.StartElement{Name: xml.Name{Local: "FileHeaderInfo"}}); err != nil {
return err
}
}
if c.RecordDelimiter != "" || c.recordDelimiterSet {
if err := e.EncodeElement(c.RecordDelimiter, xml.StartElement{Name: xml.Name{Local: "RecordDelimiter"}}); err != nil {
return err
}
}
if c.FieldDelimiter != "" || c.fieldDelimiterSet {
if err := e.EncodeElement(c.FieldDelimiter, xml.StartElement{Name: xml.Name{Local: "FieldDelimiter"}}); err != nil {
return err
}
}
if c.QuoteCharacter != "" || c.quoteCharacterSet {
if err := e.EncodeElement(c.QuoteCharacter, xml.StartElement{Name: xml.Name{Local: "QuoteCharacter"}}); err != nil {
return err
}
}
if c.QuoteEscapeCharacter != "" || c.quoteEscapeCharacterSet {
if err := e.EncodeElement(c.QuoteEscapeCharacter, xml.StartElement{Name: xml.Name{Local: "QuoteEscapeCharacter"}}); err != nil {
return err
}
}
if c.Comments != "" || c.commentsSet {
if err := e.EncodeElement(c.Comments, xml.StartElement{Name: xml.Name{Local: "Comments"}}); err != nil {
return err
}
}
return e.EncodeToken(xml.EndElement{Name: start.Name})
}
// CSVOutputOptions csv output specific options
type CSVOutputOptions struct {
QuoteFields CSVQuoteFields
quoteFieldsSet bool
RecordDelimiter string
recordDelimiterSet bool
FieldDelimiter string
fieldDelimiterSet bool
QuoteCharacter string
quoteCharacterSet bool
QuoteEscapeCharacter string
quoteEscapeCharacterSet bool
}
// SetQuoteFields sets the quote field parameter in the CSV output options
func (c *CSVOutputOptions) SetQuoteFields(val CSVQuoteFields) {
c.QuoteFields = val
c.quoteFieldsSet = true
}
// SetRecordDelimiter sets the record delimiter character in the CSV output options
func (c *CSVOutputOptions) SetRecordDelimiter(val string) {
c.RecordDelimiter = val
c.recordDelimiterSet = true
}
// SetFieldDelimiter sets the field delimiter character in the CSV output options
func (c *CSVOutputOptions) SetFieldDelimiter(val string) {
c.FieldDelimiter = val
c.fieldDelimiterSet = true
}
// SetQuoteCharacter sets the quote character in the CSV output options
func (c *CSVOutputOptions) SetQuoteCharacter(val string) {
c.QuoteCharacter = val
c.quoteCharacterSet = true
}
// SetQuoteEscapeCharacter sets the quote escape character in the CSV output options
func (c *CSVOutputOptions) SetQuoteEscapeCharacter(val string) {
c.QuoteEscapeCharacter = val
c.quoteEscapeCharacterSet = true
}
// MarshalXML - produces the xml representation of the CSVOutputOptions struct
func (c CSVOutputOptions) MarshalXML(e *xml.Encoder, start xml.StartElement) error {
if err := e.EncodeToken(start); err != nil {
return err
}
if c.QuoteFields != "" || c.quoteFieldsSet {
if err := e.EncodeElement(c.QuoteFields, xml.StartElement{Name: xml.Name{Local: "QuoteFields"}}); err != nil {
return err
}
}
if c.RecordDelimiter != "" || c.recordDelimiterSet {
if err := e.EncodeElement(c.RecordDelimiter, xml.StartElement{Name: xml.Name{Local: "RecordDelimiter"}}); err != nil {
return err
}
}
if c.FieldDelimiter != "" || c.fieldDelimiterSet {
if err := e.EncodeElement(c.FieldDelimiter, xml.StartElement{Name: xml.Name{Local: "FieldDelimiter"}}); err != nil {
return err
}
}
if c.QuoteCharacter != "" || c.quoteCharacterSet {
if err := e.EncodeElement(c.QuoteCharacter, xml.StartElement{Name: xml.Name{Local: "QuoteCharacter"}}); err != nil {
return err
}
}
if c.QuoteEscapeCharacter != "" || c.quoteEscapeCharacterSet {
if err := e.EncodeElement(c.QuoteEscapeCharacter, xml.StartElement{Name: xml.Name{Local: "QuoteEscapeCharacter"}}); err != nil {
return err
}
}
return e.EncodeToken(xml.EndElement{Name: start.Name})
}
// JSONInputOptions json input specific options
type JSONInputOptions struct {
Type JSONType
typeSet bool
}
// SetType sets the JSON type in the JSON input options
func (j *JSONInputOptions) SetType(typ JSONType) {
j.Type = typ
j.typeSet = true
}
// MarshalXML - produces the xml representation of the JSONInputOptions struct
func (j JSONInputOptions) MarshalXML(e *xml.Encoder, start xml.StartElement) error {
if err := e.EncodeToken(start); err != nil {
return err
}
if j.Type != "" || j.typeSet {
if err := e.EncodeElement(j.Type, xml.StartElement{Name: xml.Name{Local: "Type"}}); err != nil {
return err
}
}
return e.EncodeToken(xml.EndElement{Name: start.Name})
}
// JSONOutputOptions - json output specific options
type JSONOutputOptions struct {
RecordDelimiter string
recordDelimiterSet bool
}
// SetRecordDelimiter sets the record delimiter in the JSON output options
func (j *JSONOutputOptions) SetRecordDelimiter(val string) {
j.RecordDelimiter = val
j.recordDelimiterSet = true
}
// MarshalXML - produces the xml representation of the JSONOutputOptions struct
func (j JSONOutputOptions) MarshalXML(e *xml.Encoder, start xml.StartElement) error {
if err := e.EncodeToken(start); err != nil {
return err
}
if j.RecordDelimiter != "" || j.recordDelimiterSet {
if err := e.EncodeElement(j.RecordDelimiter, xml.StartElement{Name: xml.Name{Local: "RecordDelimiter"}}); err != nil {
return err
}
}
return e.EncodeToken(xml.EndElement{Name: start.Name})
}
// SelectObjectInputSerialization - input serialization parameters
type SelectObjectInputSerialization struct {
CompressionType SelectCompressionType `xml:"CompressionType,omitempty"`
Parquet *ParquetInputOptions `xml:"Parquet,omitempty"`
CSV *CSVInputOptions `xml:"CSV,omitempty"`
JSON *JSONInputOptions `xml:"JSON,omitempty"`
}
// SelectObjectOutputSerialization - output serialization parameters.
type SelectObjectOutputSerialization struct {
CSV *CSVOutputOptions `xml:"CSV,omitempty"`
JSON *JSONOutputOptions `xml:"JSON,omitempty"`
}
// SelectObjectOptions - represents the input select body
type SelectObjectOptions struct {
XMLName xml.Name `xml:"SelectObjectContentRequest" json:"-"`
ServerSideEncryption encrypt.ServerSide `xml:"-"`
Expression string
ExpressionType QueryExpressionType
InputSerialization SelectObjectInputSerialization
OutputSerialization SelectObjectOutputSerialization
RequestProgress struct {
Enabled bool
}
}
// Header returns the http.Header representation of the SelectObject options.
func (o SelectObjectOptions) Header() http.Header {
headers := make(http.Header)
if o.ServerSideEncryption != nil && o.ServerSideEncryption.Type() == encrypt.SSEC {
o.ServerSideEncryption.Marshal(headers)
}
return headers
}
// SelectObjectType - is the parameter which defines what type of object the
// operation is being performed on.
type SelectObjectType string
// Constants for input data types.
const (
SelectObjectTypeCSV SelectObjectType = "CSV"
SelectObjectTypeJSON SelectObjectType = "JSON"
SelectObjectTypeParquet SelectObjectType = "Parquet"
)
// preludeInfo is used for keeping track of necessary information from the
// prelude.
type preludeInfo struct {
totalLen uint32
headerLen uint32
}
// SelectResults is used for the streaming responses from the server.
type SelectResults struct {
pipeReader *io.PipeReader
resp *http.Response
stats *StatsMessage
progress *ProgressMessage
}
// ProgressMessage is a struct for progress xml message.
type ProgressMessage struct {
XMLName xml.Name `xml:"Progress" json:"-"`
StatsMessage
}
// StatsMessage is a struct for stat xml message.
type StatsMessage struct {
XMLName xml.Name `xml:"Stats" json:"-"`
BytesScanned int64
BytesProcessed int64
BytesReturned int64
}
// messageType represents the type of message.
type messageType string
const (
errorMsg messageType = "error"
commonMsg messageType = "event"
)
// eventType represents the type of event.
type eventType string
// list of event-types returned by Select API.
const (
endEvent eventType = "End"
recordsEvent eventType = "Records"
progressEvent eventType = "Progress"
statsEvent eventType = "Stats"
)
// contentType represents content type of event.
type contentType string
const (
xmlContent contentType = "text/xml"
)
// SelectObjectContent is a implementation of http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectSELECTContent.html AWS S3 API.
func (c *Client) SelectObjectContent(ctx context.Context, bucketName, objectName string, opts SelectObjectOptions) (*SelectResults, error) {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return nil, err
}
if err := s3utils.CheckValidObjectName(objectName); err != nil {
return nil, err
}
selectReqBytes, err := xml.Marshal(opts)
if err != nil {
return nil, err
}
urlValues := make(url.Values)
urlValues.Set("select", "")
urlValues.Set("select-type", "2")
// Execute POST on bucket/object.
resp, err := c.executeMethod(ctx, http.MethodPost, requestMetadata{
bucketName: bucketName,
objectName: objectName,
queryValues: urlValues,
customHeader: opts.Header(),
contentMD5Base64: sumMD5Base64(selectReqBytes),
contentSHA256Hex: sum256Hex(selectReqBytes),
contentBody: bytes.NewReader(selectReqBytes),
contentLength: int64(len(selectReqBytes)),
})
if err != nil {
return nil, err
}
return NewSelectResults(resp, bucketName)
}
// NewSelectResults creates a Select Result parser that parses the response
// and returns a Reader that will return parsed and assembled select output.
func NewSelectResults(resp *http.Response, bucketName string) (*SelectResults, error) {
if resp.StatusCode != http.StatusOK {
return nil, httpRespToErrorResponse(resp, bucketName, "")
}
pipeReader, pipeWriter := io.Pipe()
streamer := &SelectResults{
resp: resp,
stats: &StatsMessage{},
progress: &ProgressMessage{},
pipeReader: pipeReader,
}
streamer.start(pipeWriter)
return streamer, nil
}
// Close - closes the underlying response body and the stream reader.
func (s *SelectResults) Close() error {
defer closeResponse(s.resp)
return s.pipeReader.Close()
}
// Read - is a reader compatible implementation for SelectObjectContent records.
func (s *SelectResults) Read(b []byte) (n int, err error) {
return s.pipeReader.Read(b)
}
// Stats - information about a request's stats when processing is complete.
func (s *SelectResults) Stats() *StatsMessage {
return s.stats
}
// Progress - information about the progress of a request.
func (s *SelectResults) Progress() *ProgressMessage {
return s.progress
}
// start is the main function that decodes the large byte array into
// several events that are sent through the eventstream.
func (s *SelectResults) start(pipeWriter *io.PipeWriter) {
go func() {
for {
var prelude preludeInfo
headers := make(http.Header)
var err error
// Create CRC code
crc := crc32.New(crc32.IEEETable)
crcReader := io.TeeReader(s.resp.Body, crc)
// Extract the prelude(12 bytes) into a struct to extract relevant information.
prelude, err = processPrelude(crcReader, crc)
if err != nil {
pipeWriter.CloseWithError(err)
closeResponse(s.resp)
return
}
// Extract the headers(variable bytes) into a struct to extract relevant information
if prelude.headerLen > 0 {
if err = extractHeader(io.LimitReader(crcReader, int64(prelude.headerLen)), headers); err != nil {
pipeWriter.CloseWithError(err)
closeResponse(s.resp)
return
}
}
// Get the actual payload length so that the appropriate amount of
// bytes can be read or parsed.
payloadLen := prelude.PayloadLen()
m := messageType(headers.Get("message-type"))
switch m {
case errorMsg:
pipeWriter.CloseWithError(errors.New(headers.Get("error-code") + ":\"" + headers.Get("error-message") + "\""))
closeResponse(s.resp)
return
case commonMsg:
// Get content-type of the payload.
c := contentType(headers.Get("content-type"))
// Get event type of the payload.
e := eventType(headers.Get("event-type"))
// Handle all supported events.
switch e {
case endEvent:
pipeWriter.Close()
closeResponse(s.resp)
return
case recordsEvent:
if _, err = io.Copy(pipeWriter, io.LimitReader(crcReader, payloadLen)); err != nil {
pipeWriter.CloseWithError(err)
closeResponse(s.resp)
return
}
case progressEvent:
switch c {
case xmlContent:
if err = xmlDecoder(io.LimitReader(crcReader, payloadLen), s.progress); err != nil {
pipeWriter.CloseWithError(err)
closeResponse(s.resp)
return
}
default:
pipeWriter.CloseWithError(fmt.Errorf("Unexpected content-type %s sent for event-type %s", c, progressEvent))
closeResponse(s.resp)
return
}
case statsEvent:
switch c {
case xmlContent:
if err = xmlDecoder(io.LimitReader(crcReader, payloadLen), s.stats); err != nil {
pipeWriter.CloseWithError(err)
closeResponse(s.resp)
return
}
default:
pipeWriter.CloseWithError(fmt.Errorf("Unexpected content-type %s sent for event-type %s", c, statsEvent))
closeResponse(s.resp)
return
}
}
}
// Ensures that the full message's CRC is correct and
// that the message is not corrupted
if err := checkCRC(s.resp.Body, crc.Sum32()); err != nil {
pipeWriter.CloseWithError(err)
closeResponse(s.resp)
return
}
}
}()
}
// PayloadLen is a function that calculates the length of the payload.
func (p preludeInfo) PayloadLen() int64 {
return int64(p.totalLen - p.headerLen - 16)
}
// processPrelude is the function that reads the 12 bytes of the prelude and
// ensures the CRC is correct while also extracting relevant information into
// the struct,
func processPrelude(prelude io.Reader, crc hash.Hash32) (preludeInfo, error) {
var err error
pInfo := preludeInfo{}
// reads total length of the message (first 4 bytes)
pInfo.totalLen, err = extractUint32(prelude)
if err != nil {
return pInfo, err
}
// reads total header length of the message (2nd 4 bytes)
pInfo.headerLen, err = extractUint32(prelude)
if err != nil {
return pInfo, err
}
// checks that the CRC is correct (3rd 4 bytes)
preCRC := crc.Sum32()
if err := checkCRC(prelude, preCRC); err != nil {
return pInfo, err
}
return pInfo, nil
}
// extracts the relevant information from the Headers.
func extractHeader(body io.Reader, myHeaders http.Header) error {
for {
// extracts the first part of the header,
headerTypeName, err := extractHeaderType(body)
if err != nil {
// Since end of file, we have read all of our headers
if err == io.EOF {
break
}
return err
}
// reads the 7 present in the header and ignores it.
extractUint8(body)
headerValueName, err := extractHeaderValue(body)
if err != nil {
return err
}
myHeaders.Set(headerTypeName, headerValueName)
}
return nil
}
// extractHeaderType extracts the first half of the header message, the header type.
func extractHeaderType(body io.Reader) (string, error) {
// extracts 2 bit integer
headerNameLen, err := extractUint8(body)
if err != nil {
return "", err
}
// extracts the string with the appropriate number of bytes
headerName, err := extractString(body, int(headerNameLen))
if err != nil {
return "", err
}
return strings.TrimPrefix(headerName, ":"), nil
}
// extractsHeaderValue extracts the second half of the header message, the
// header value
func extractHeaderValue(body io.Reader) (string, error) {
bodyLen, err := extractUint16(body)
if err != nil {
return "", err
}
bodyName, err := extractString(body, int(bodyLen))
if err != nil {
return "", err
}
return bodyName, nil
}
// extracts a string from byte array of a particular number of bytes.
func extractString(source io.Reader, lenBytes int) (string, error) {
myVal := make([]byte, lenBytes)
_, err := source.Read(myVal)
if err != nil {
return "", err
}
return string(myVal), nil
}
// extractUint32 extracts a 4 byte integer from the byte array.
func extractUint32(r io.Reader) (uint32, error) {
buf := make([]byte, 4)
_, err := readFull(r, buf)
if err != nil {
return 0, err
}
return binary.BigEndian.Uint32(buf), nil
}
// extractUint16 extracts a 2 byte integer from the byte array.
func extractUint16(r io.Reader) (uint16, error) {
buf := make([]byte, 2)
_, err := readFull(r, buf)
if err != nil {
return 0, err
}
return binary.BigEndian.Uint16(buf), nil
}
// extractUint8 extracts a 1 byte integer from the byte array.
func extractUint8(r io.Reader) (uint8, error) {
buf := make([]byte, 1)
_, err := readFull(r, buf)
if err != nil {
return 0, err
}
return buf[0], nil
}
// checkCRC ensures that the CRC matches with the one from the reader.
func checkCRC(r io.Reader, expect uint32) error {
msgCRC, err := extractUint32(r)
if err != nil {
return err
}
if msgCRC != expect {
return fmt.Errorf("Checksum Mismatch, MessageCRC of 0x%X does not equal expected CRC of 0x%X", msgCRC, expect)
}
return nil
}
Server/vendor/github.com/minio/minio-go/v7/api-stat.go
Generated Vendored
+124
View File
@@ -0,0 +1,124 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2015-2020 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"context"
"net/http"
"github.com/minio/minio-go/v7/pkg/s3utils"
)
// BucketExists verifies if bucket exists and you have permission to access it. Allows for a Context to
// control cancellations and timeouts.
func (c *Client) BucketExists(ctx context.Context, bucketName string) (bool, error) {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return false, err
}
// Execute HEAD on bucketName.
resp, err := c.executeMethod(ctx, http.MethodHead, requestMetadata{
bucketName: bucketName,
contentSHA256Hex: emptySHA256Hex,
})
defer closeResponse(resp)
if err != nil {
if ToErrorResponse(err).Code == NoSuchBucket {
return false, nil
}
return false, err
}
if resp != nil {
resperr := httpRespToErrorResponse(resp, bucketName, "")
if ToErrorResponse(resperr).Code == NoSuchBucket {
return false, nil
}
if resp.StatusCode != http.StatusOK {
return false, httpRespToErrorResponse(resp, bucketName, "")
}
}
return true, nil
}
// StatObject verifies if object exists, you have permission to access it
// and returns information about the object.
func (c *Client) StatObject(ctx context.Context, bucketName, objectName string, opts StatObjectOptions) (ObjectInfo, error) {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return ObjectInfo{}, ErrorResponse{
StatusCode: http.StatusBadRequest,
Code: InvalidBucketName,
Message: err.Error(),
}
}
if err := s3utils.CheckValidObjectName(objectName); err != nil {
return ObjectInfo{}, ErrorResponse{
StatusCode: http.StatusBadRequest,
Code: XMinioInvalidObjectName,
Message: err.Error(),
}
}
headers := opts.Header()
if opts.Internal.ReplicationDeleteMarker {
headers.Set(minIOBucketReplicationDeleteMarker, "true")
}
if opts.Internal.IsReplicationReadyForDeleteMarker {
headers.Set(isMinioTgtReplicationReady, "true")
}
// Execute HEAD on objectName.
resp, err := c.executeMethod(ctx, http.MethodHead, requestMetadata{
bucketName: bucketName,
objectName: objectName,
queryValues: opts.toQueryValues(),
contentSHA256Hex: emptySHA256Hex,
customHeader: headers,
})
defer closeResponse(resp)
if err != nil {
return ObjectInfo{}, err
}
if resp != nil {
deleteMarker := resp.Header.Get(amzDeleteMarker) == "true"
replicationReady := resp.Header.Get(minioTgtReplicationReady) == "true"
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusPartialContent {
if resp.StatusCode == http.StatusMethodNotAllowed && opts.VersionID != "" && deleteMarker {
errResp := ErrorResponse{
StatusCode: resp.StatusCode,
Code: MethodNotAllowed,
Message: s3ErrorResponseMap[MethodNotAllowed],
BucketName: bucketName,
Key: objectName,
}
return ObjectInfo{
VersionID: resp.Header.Get(amzVersionID),
IsDeleteMarker: deleteMarker,
}, errResp
}
return ObjectInfo{
VersionID: resp.Header.Get(amzVersionID),
IsDeleteMarker: deleteMarker,
ReplicationReady: replicationReady, // whether delete marker can be replicated
}, httpRespToErrorResponse(resp, bucketName, objectName)
}
}
return ToObjectInfo(bucketName, objectName, resp.Header)
}
Server/vendor/github.com/minio/minio-go/v7/api-update-object-encryption.go
Generated Vendored
+130
View File
@@ -0,0 +1,130 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2025-2026 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"bytes"
"context"
"encoding/xml"
"net/http"
"net/url"
"github.com/minio/minio-go/v7/pkg/s3utils"
)
// updateObjectEncryptionSSEKMS represents the SSE-KMS element in the request body.
type updateObjectEncryptionSSEKMS struct {
BucketKeyEnabled bool `xml:"BucketKeyEnabled,omitempty"`
KMSKeyArn string `xml:"KMSKeyArn"`
}
// updateObjectEncryptionRequest represents the XML request body for UpdateObjectEncryption.
type updateObjectEncryptionRequest struct {
XMLName xml.Name `xml:"ObjectEncryption"`
XMLNS string `xml:"xmlns,attr"`
SSEKMS *updateObjectEncryptionSSEKMS `xml:"SSE-KMS"`
}
// UpdateObjectEncryptionOptions holds options for the UpdateObjectEncryption call.
type UpdateObjectEncryptionOptions struct {
// KMSKeyArn is the KMS key name or ARN to encrypt the object with.
KMSKeyArn string
// BucketKeyEnabled enables S3 Bucket Key for KMS encryption.
BucketKeyEnabled bool
// VersionID targets a specific object version.
VersionID string
}
// UpdateObjectEncryptionResult holds the result of an UpdateObjectEncryption call.
type UpdateObjectEncryptionResult struct {
// VersionID is the version ID of the object that was updated, if versioning is enabled.
VersionID string
}
// UpdateObjectEncryption changes the encryption configuration of an existing object in-place.
// The object must already be encrypted with SSE-S3 or SSE-KMS. SSE-C objects are not supported.
// This operation rotates the data encryption key envelope without re-reading/re-writing object data.
//
// Parameters:
// - ctx: Context for request cancellation and timeout
// - bucketName: Name of the bucket
// - objectName: Name of the object
// - opts: Options including KMSKeyArn (required), optional BucketKeyEnabled, and optional VersionID
//
// Returns the version ID of the updated object (if versioning is enabled) and an error if the operation fails.
func (c *Client) UpdateObjectEncryption(ctx context.Context, bucketName, objectName string, opts UpdateObjectEncryptionOptions) (UpdateObjectEncryptionResult, error) {
// Input validation.
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return UpdateObjectEncryptionResult{}, err
}
if err := s3utils.CheckValidObjectName(objectName); err != nil {
return UpdateObjectEncryptionResult{}, err
}
if opts.KMSKeyArn == "" {
return UpdateObjectEncryptionResult{}, errInvalidArgument("KMSKeyArn is required for UpdateObjectEncryption.")
}
// Get resources properly escaped and lined up before
// using them in http request.
urlValues := make(url.Values)
urlValues.Set("encryption", "")
if opts.VersionID != "" {
urlValues.Set("versionId", opts.VersionID)
}
reqBody := updateObjectEncryptionRequest{
XMLNS: "http://s3.amazonaws.com/doc/2006-03-01/",
SSEKMS: &updateObjectEncryptionSSEKMS{
BucketKeyEnabled: opts.BucketKeyEnabled,
KMSKeyArn: opts.KMSKeyArn,
},
}
bodyData, err := xml.Marshal(reqBody)
if err != nil {
return UpdateObjectEncryptionResult{}, err
}
reqMetadata := requestMetadata{
bucketName: bucketName,
objectName: objectName,
queryValues: urlValues,
contentBody: bytes.NewReader(bodyData),
contentLength: int64(len(bodyData)),
contentMD5Base64: sumMD5Base64(bodyData),
contentSHA256Hex: sum256Hex(bodyData),
}
// Execute PUT Object Encryption.
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
defer closeResponse(resp)
if err != nil {
return UpdateObjectEncryptionResult{}, err
}
if resp.StatusCode != http.StatusOK {
return UpdateObjectEncryptionResult{}, httpRespToErrorResponse(resp, bucketName, objectName)
}
return UpdateObjectEncryptionResult{
VersionID: resp.Header.Get(amzVersionID),
}, nil
}
Server/vendor/github.com/minio/minio-go/v7/api.go
Generated Vendored
+1151
View File
File diff suppressed because it is too large Load Diff
Server/vendor/github.com/minio/minio-go/v7/bucket-cache.go
Generated Vendored
+219
View File
@@ -0,0 +1,219 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2015-2017 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"context"
"net"
"net/http"
"net/url"
"path"
"github.com/minio/minio-go/v7/pkg/credentials"
"github.com/minio/minio-go/v7/pkg/s3utils"
"github.com/minio/minio-go/v7/pkg/signer"
)
// GetBucketLocation - get location for the bucket name from location cache, if not
// fetch freshly by making a new request.
func (c *Client) GetBucketLocation(ctx context.Context, bucketName string) (string, error) {
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return "", err
}
return c.getBucketLocation(ctx, bucketName)
}
// getBucketLocation - Get location for the bucketName from location map cache, if not
// fetch freshly by making a new request.
func (c *Client) getBucketLocation(ctx context.Context, bucketName string) (string, error) {
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
return "", err
}
// Region set then no need to fetch bucket location.
if c.region != "" {
return c.region, nil
}
if location, ok := c.bucketLocCache.Get(bucketName); ok {
return location, nil
}
// Initialize a new request.
req, err := c.getBucketLocationRequest(ctx, bucketName)
if err != nil {
return "", err
}
// Initiate the request.
resp, err := c.do(req)
defer closeResponse(resp)
if err != nil {
return "", err
}
location, err := processBucketLocationResponse(resp, bucketName)
if err != nil {
return "", err
}
c.bucketLocCache.Set(bucketName, location)
return location, nil
}
// processes the getBucketLocation http response from the server.
func processBucketLocationResponse(resp *http.Response, bucketName string) (bucketLocation string, err error) {
if resp != nil {
if resp.StatusCode != http.StatusOK {
err = httpRespToErrorResponse(resp, bucketName, "")
errResp := ToErrorResponse(err)
// For access denied error, it could be an anonymous
// request. Move forward and let the top level callers
// succeed if possible based on their policy.
switch errResp.Code {
case NotImplemented:
switch errResp.Server {
case "AmazonSnowball":
return "snowball", nil
case "cloudflare":
return "us-east-1", nil
}
case AuthorizationHeaderMalformed:
fallthrough
case InvalidRegion:
fallthrough
case AccessDenied:
if errResp.Region == "" {
return "us-east-1", nil
}
return errResp.Region, nil
}
return "", err
}
}
// Extract location.
var locationConstraint string
err = xmlDecoder(resp.Body, &locationConstraint)
if err != nil {
return "", err
}
location := locationConstraint
// Location is empty will be 'us-east-1'.
if location == "" {
location = "us-east-1"
}
// Location can be 'EU' convert it to meaningful 'eu-west-1'.
if location == "EU" {
location = "eu-west-1"
}
// Save the location into cache.
// Return.
return location, nil
}
// getBucketLocationRequest - Wrapper creates a new getBucketLocation request.
func (c *Client) getBucketLocationRequest(ctx context.Context, bucketName string) (*http.Request, error) {
// Set location query.
urlValues := make(url.Values)
urlValues.Set("location", "")
// Set get bucket location always as path style.
targetURL := *c.endpointURL
// as it works in makeTargetURL method from api.go file
if h, p, err := net.SplitHostPort(targetURL.Host); err == nil {
if targetURL.Scheme == "http" && p == "80" || targetURL.Scheme == "https" && p == "443" {
targetURL.Host = h
if ip := net.ParseIP(h); ip != nil && ip.To4() == nil {
targetURL.Host = "[" + h + "]"
}
}
}
isVirtualStyle := c.isVirtualHostStyleRequest(targetURL, bucketName)
var urlStr string
if isVirtualStyle {
urlStr = c.endpointURL.Scheme + "://" + bucketName + "." + targetURL.Host + "/?location"
} else {
targetURL.Path = path.Join(bucketName, "") + "/"
targetURL.RawQuery = urlValues.Encode()
urlStr = targetURL.String()
}
// Get a new HTTP request for the method.
req, err := http.NewRequestWithContext(ctx, http.MethodGet, urlStr, nil)
if err != nil {
return nil, err
}
// Set UserAgent for the request.
c.setUserAgent(req)
// Get credentials from the configured credentials provider.
value, err := c.credsProvider.GetWithContext(c.CredContext())
if err != nil {
return nil, err
}
var (
signerType = value.SignerType
accessKeyID = value.AccessKeyID
secretAccessKey = value.SecretAccessKey
sessionToken = value.SessionToken
)
// Custom signer set then override the behavior.
if c.overrideSignerType != credentials.SignatureDefault {
signerType = c.overrideSignerType
}
// If signerType returned by credentials helper is anonymous,
// then do not sign regardless of signerType override.
if value.SignerType == credentials.SignatureAnonymous {
signerType = credentials.SignatureAnonymous
}
if signerType.IsAnonymous() {
return req, nil
}
if signerType.IsV2() {
req = signer.SignV2(*req, accessKeyID, secretAccessKey, isVirtualStyle)
return req, nil
}
// Set sha256 sum for signature calculation only with signature version '4'.
contentSha256 := emptySHA256Hex
if c.secure {
contentSha256 = unsignedPayload
}
req.Header.Set("X-Amz-Content-Sha256", contentSha256)
if s3utils.IsAmazonOutpostsEndpoint(*c.endpointURL) {
region := getDefaultLocation(*c.endpointURL, c.region)
req = signer.SignV4Outposts(*req, accessKeyID, secretAccessKey, sessionToken, region)
} else {
req = signer.SignV4(*req, accessKeyID, secretAccessKey, sessionToken, "us-east-1")
}
return req, nil
}
Server/vendor/github.com/minio/minio-go/v7/checksum.go
Generated Vendored
+537
View File
@@ -0,0 +1,537 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2015-2023 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"crypto/md5"
"crypto/sha1"
"crypto/sha256"
"crypto/sha512"
"encoding/base64"
"encoding/binary"
"errors"
"hash"
"io"
"math/bits"
"net/http"
"sort"
"strings"
"github.com/cespare/xxhash/v2"
"github.com/klauspost/crc32"
"github.com/minio/crc64nvme"
"github.com/zeebo/xxh3"
)
// ChecksumMode contains information about the checksum mode on the object
type ChecksumMode uint32
const (
// ChecksumFullObjectMode Full object checksum `csumCombine(csum1, csum2...)...), csumN...)`
ChecksumFullObjectMode ChecksumMode = 1 << iota
// ChecksumCompositeMode Composite checksum `csum([csum1 + csum2 ... + csumN])`
ChecksumCompositeMode
// Keep after all valid checksums
checksumLastMode
// checksumModeMask is a mask for valid checksum mode types.
checksumModeMask = checksumLastMode - 1
// ChecksumUnknownMode indicates no or unknown checksum mode.
ChecksumUnknownMode ChecksumMode = 0
)
// Is returns if c is all of t.
func (c ChecksumMode) Is(t ChecksumMode) bool {
return c&t == t
}
// Key returns the header key.
func (c ChecksumMode) Key() string {
return amzChecksumMode
}
func (c ChecksumMode) String() string {
switch c & checksumModeMask {
case ChecksumFullObjectMode:
return amzChecksumModeFullObject
case ChecksumCompositeMode:
return amzChecksumModeComposite
}
return ""
}
// ChecksumType contains information about the checksum type.
type ChecksumType uint32
const (
// ChecksumSHA256 indicates a SHA256 checksum.
ChecksumSHA256 ChecksumType = 1 << iota
// ChecksumSHA1 indicates a SHA-1 checksum.
ChecksumSHA1
// ChecksumCRC32 indicates a CRC32 checksum with IEEE table.
ChecksumCRC32
// ChecksumCRC32C indicates a CRC32 checksum with Castagnoli table.
ChecksumCRC32C
// ChecksumCRC64NVME indicates CRC64 with 0xad93d23594c93659 polynomial.
ChecksumCRC64NVME
// ChecksumMD5 indicates an MD5 checksum.
ChecksumMD5
// ChecksumSHA512 indicates a SHA-512 checksum.
ChecksumSHA512
// ChecksumXXHash64 indicates an XXHash64 checksum.
ChecksumXXHash64
// ChecksumXXHash3 indicates an XXH3-64 checksum.
ChecksumXXHash3
// ChecksumXXHash128 indicates an XXH3-128 checksum.
ChecksumXXHash128
// Keep after all valid checksums
checksumLast
// ChecksumFullObject is a modifier that can be used on CRC32 and CRC32C
// to indicate full object checksums.
ChecksumFullObject
// checksumMask is a mask for valid checksum types.
checksumMask = checksumLast - 1
// ChecksumNone indicates no checksum.
ChecksumNone ChecksumType = 0
// ChecksumFullObjectCRC32 indicates full object CRC32
ChecksumFullObjectCRC32 = ChecksumCRC32 | ChecksumFullObject
// ChecksumFullObjectCRC32C indicates full object CRC32C
ChecksumFullObjectCRC32C = ChecksumCRC32C | ChecksumFullObject
amzChecksumAlgo = "x-amz-checksum-algorithm"
amzChecksumCRC32 = "x-amz-checksum-crc32"
amzChecksumCRC32C = "x-amz-checksum-crc32c"
amzChecksumSHA1 = "x-amz-checksum-sha1"
amzChecksumSHA256 = "x-amz-checksum-sha256"
amzChecksumCRC64NVME = "x-amz-checksum-crc64nvme"
amzChecksumMD5 = "x-amz-checksum-md5"
amzChecksumSHA512 = "x-amz-checksum-sha512"
amzChecksumXXHash64 = "x-amz-checksum-xxhash64"
amzChecksumXXHash3 = "x-amz-checksum-xxhash3"
amzChecksumXXHash128 = "x-amz-checksum-xxhash128"
amzChecksumMode = "x-amz-checksum-type"
amzChecksumModeComposite = "COMPOSITE"
amzChecksumModeFullObject = "FULL_OBJECT"
)
// Base returns the base type, without modifiers.
func (c ChecksumType) Base() ChecksumType {
return c & checksumMask
}
// Is returns if c is all of t.
func (c ChecksumType) Is(t ChecksumType) bool {
return c&t == t
}
// Key returns the header key.
// returns empty string if invalid or none.
func (c ChecksumType) Key() string {
switch c & checksumMask {
case ChecksumCRC32:
return amzChecksumCRC32
case ChecksumCRC32C:
return amzChecksumCRC32C
case ChecksumSHA1:
return amzChecksumSHA1
case ChecksumSHA256:
return amzChecksumSHA256
case ChecksumCRC64NVME:
return amzChecksumCRC64NVME
case ChecksumMD5:
return amzChecksumMD5
case ChecksumSHA512:
return amzChecksumSHA512
case ChecksumXXHash64:
return amzChecksumXXHash64
case ChecksumXXHash3:
return amzChecksumXXHash3
case ChecksumXXHash128:
return amzChecksumXXHash128
}
return ""
}
// CanComposite will return if the checksum type can be used for composite multipart upload on AWS.
func (c ChecksumType) CanComposite() bool {
switch c & checksumMask {
case ChecksumSHA256, ChecksumSHA1, ChecksumCRC32, ChecksumCRC32C,
ChecksumMD5, ChecksumSHA512, ChecksumXXHash64, ChecksumXXHash3, ChecksumXXHash128:
return true
}
return false
}
// CanMergeCRC will return if the checksum type can be used for multipart upload on AWS.
func (c ChecksumType) CanMergeCRC() bool {
switch c & checksumMask {
case ChecksumCRC32, ChecksumCRC32C, ChecksumCRC64NVME:
return true
}
return false
}
// FullObjectRequested will return if the checksum type indicates full object checksum was requested.
func (c ChecksumType) FullObjectRequested() bool {
switch c & (ChecksumFullObject | checksumMask) {
case ChecksumFullObjectCRC32C, ChecksumFullObjectCRC32, ChecksumCRC64NVME:
return true
}
return false
}
// KeyCapitalized returns the capitalized key as used in HTTP headers.
func (c ChecksumType) KeyCapitalized() string {
return http.CanonicalHeaderKey(c.Key())
}
// RawByteLen returns the size of the un-encoded checksum.
func (c ChecksumType) RawByteLen() int {
switch c & checksumMask {
case ChecksumCRC32, ChecksumCRC32C:
return 4
case ChecksumSHA1:
return sha1.Size
case ChecksumSHA256:
return sha256.Size
case ChecksumCRC64NVME:
return crc64nvme.Size
case ChecksumXXHash64, ChecksumXXHash3:
return 8
case ChecksumMD5:
return md5.Size
case ChecksumSHA512:
return sha512.Size
case ChecksumXXHash128:
return 16
}
return 0
}
const crc64NVMEPolynomial = 0xad93d23594c93659
// Hasher returns a hasher corresponding to the checksum type.
// Returns nil if no checksum.
func (c ChecksumType) Hasher() hash.Hash {
switch c & checksumMask {
case ChecksumCRC32:
return crc32.NewIEEE()
case ChecksumCRC32C:
return crc32.New(crc32.MakeTable(crc32.Castagnoli))
case ChecksumSHA1:
return sha1.New()
case ChecksumSHA256:
return sha256.New()
case ChecksumCRC64NVME:
return crc64nvme.New()
case ChecksumMD5:
return md5.New()
case ChecksumSHA512:
return sha512.New()
case ChecksumXXHash64:
return xxhash.New()
case ChecksumXXHash3:
return xxh3.New()
case ChecksumXXHash128:
return xxh3.New128()
}
return nil
}
// IsSet returns whether the type is valid and known.
func (c ChecksumType) IsSet() bool {
return bits.OnesCount32(uint32(c&checksumMask)) == 1
}
// SetDefault will set the checksum if not already set.
func (c *ChecksumType) SetDefault(t ChecksumType) {
if !c.IsSet() {
*c = t
}
}
// EncodeToString the encoded hash value of the content provided in b.
func (c ChecksumType) EncodeToString(b []byte) string {
if !c.IsSet() {
return ""
}
h := c.Hasher()
h.Write(b)
return base64.StdEncoding.EncodeToString(h.Sum(nil))
}
// String returns the type as a string.
// Empty string for unset and "<invalid>" if not valid.
func (c ChecksumType) String() string {
switch c & checksumMask {
case ChecksumCRC32:
return "CRC32"
case ChecksumCRC32C:
return "CRC32C"
case ChecksumSHA1:
return "SHA1"
case ChecksumSHA256:
return "SHA256"
case ChecksumNone:
return ""
case ChecksumCRC64NVME:
return "CRC64NVME"
case ChecksumMD5:
return "MD5"
case ChecksumSHA512:
return "SHA512"
case ChecksumXXHash64:
return "XXHASH64"
case ChecksumXXHash3:
return "XXHASH3"
case ChecksumXXHash128:
return "XXHASH128"
}
return "<invalid>"
}
// ChecksumReader reads all of r and returns a checksum of type c.
// Returns any error that may have occurred while reading.
func (c ChecksumType) ChecksumReader(r io.Reader) (Checksum, error) {
h := c.Hasher()
if h == nil {
return Checksum{}, nil
}
_, err := io.Copy(h, r)
if err != nil {
return Checksum{}, err
}
return NewChecksum(c, h.Sum(nil)), nil
}
// ChecksumBytes returns a checksum of the content b with type c.
func (c ChecksumType) ChecksumBytes(b []byte) Checksum {
h := c.Hasher()
if h == nil {
return Checksum{}
}
n, err := h.Write(b)
if err != nil || n != len(b) {
// Shouldn't happen with these checksummers.
return Checksum{}
}
return NewChecksum(c, h.Sum(nil))
}
// Checksum is a type and encoded value.
type Checksum struct {
Type ChecksumType
r []byte
}
// NewChecksum sets the checksum to the value of b,
// which is the raw hash output.
// If the length of c does not match t.RawByteLen,
// a checksum with ChecksumNone is returned.
func NewChecksum(t ChecksumType, b []byte) Checksum {
if t.IsSet() && len(b) == t.RawByteLen() {
return Checksum{Type: t, r: b}
}
return Checksum{}
}
// NewChecksumString sets the checksum to the value of s,
// which is the base 64 encoded raw hash output.
// If the length of c does not match t.RawByteLen, it is not added.
func NewChecksumString(t ChecksumType, s string) Checksum {
b, _ := base64.StdEncoding.DecodeString(s)
if t.IsSet() && len(b) == t.RawByteLen() {
return Checksum{Type: t, r: b}
}
return Checksum{}
}
// IsSet returns whether the checksum is valid and known.
func (c Checksum) IsSet() bool {
return c.Type.IsSet() && len(c.r) == c.Type.RawByteLen()
}
// Encoded returns the encoded value.
// Returns the empty string if not set or valid.
func (c Checksum) Encoded() string {
if !c.IsSet() {
return ""
}
return base64.StdEncoding.EncodeToString(c.r)
}
// Raw returns the raw checksum value if set.
func (c Checksum) Raw() []byte {
if !c.IsSet() {
return nil
}
return c.r
}
// CompositeChecksum returns the composite checksum of all provided parts.
func (c ChecksumType) CompositeChecksum(p []ObjectPart) (*Checksum, error) {
if !c.CanComposite() {
return nil, errors.New("cannot do composite checksum")
}
sort.Slice(p, func(i, j int) bool {
return p[i].PartNumber < p[j].PartNumber
})
c = c.Base()
crcBytes := make([]byte, 0, len(p)*c.RawByteLen())
for _, part := range p {
pCrc, err := part.ChecksumRaw(c)
if err != nil {
return nil, err
}
crcBytes = append(crcBytes, pCrc...)
}
h := c.Hasher()
h.Write(crcBytes)
return &Checksum{Type: c, r: h.Sum(nil)}, nil
}
// FullObjectChecksum will return the full object checksum from provided parts.
func (c ChecksumType) FullObjectChecksum(p []ObjectPart) (*Checksum, error) {
if !c.CanMergeCRC() {
return nil, errors.New("cannot merge this checksum type")
}
c = c.Base()
sort.Slice(p, func(i, j int) bool {
return p[i].PartNumber < p[j].PartNumber
})
switch len(p) {
case 0:
return nil, errors.New("no parts given")
case 1:
check, err := p[0].ChecksumRaw(c)
if err != nil {
return nil, err
}
return &Checksum{
Type: c,
r: check,
}, nil
}
var merged uint32
var merged64 uint64
first, err := p[0].ChecksumRaw(c)
if err != nil {
return nil, err
}
sz := p[0].Size
switch c {
case ChecksumCRC32, ChecksumCRC32C:
merged = binary.BigEndian.Uint32(first)
case ChecksumCRC64NVME:
merged64 = binary.BigEndian.Uint64(first)
}
poly32 := uint32(crc32.IEEE)
if c.Is(ChecksumCRC32C) {
poly32 = crc32.Castagnoli
}
for _, part := range p[1:] {
if part.Size == 0 {
continue
}
sz += part.Size
pCrc, err := part.ChecksumRaw(c)
if err != nil {
return nil, err
}
switch c {
case ChecksumCRC32, ChecksumCRC32C:
merged = crc32Combine(poly32, merged, binary.BigEndian.Uint32(pCrc), part.Size)
case ChecksumCRC64NVME:
merged64 = crc64Combine(bits.Reverse64(crc64NVMEPolynomial), merged64, binary.BigEndian.Uint64(pCrc), part.Size)
}
}
var tmp [8]byte
switch c {
case ChecksumCRC32, ChecksumCRC32C:
binary.BigEndian.PutUint32(tmp[:], merged)
return &Checksum{
Type: c,
r: tmp[:4],
}, nil
case ChecksumCRC64NVME:
binary.BigEndian.PutUint64(tmp[:], merged64)
return &Checksum{
Type: c,
r: tmp[:8],
}, nil
default:
return nil, errors.New("unknown checksum type")
}
}
func addAutoChecksumHeaders(opts *PutObjectOptions) {
if opts.UserMetadata == nil {
opts.UserMetadata = make(map[string]string, 1)
}
addChecksum := true
for k := range opts.UserMetadata {
if strings.HasPrefix(strings.ToLower(k), "x-amz-checksum-") {
addChecksum = false
}
}
if addChecksum && opts.AutoChecksum.IsSet() {
opts.UserMetadata[amzChecksumAlgo] = opts.AutoChecksum.String()
if opts.AutoChecksum.FullObjectRequested() {
opts.UserMetadata[amzChecksumMode] = ChecksumFullObjectMode.String()
}
}
}
func applyAutoChecksum(opts *PutObjectOptions, allParts []ObjectPart) {
if !opts.AutoChecksum.IsSet() {
return
}
if opts.AutoChecksum.CanComposite() && !opts.AutoChecksum.Is(ChecksumFullObject) {
// Add composite hash of hashes.
crc, err := opts.AutoChecksum.CompositeChecksum(allParts)
if err == nil {
opts.UserMetadata = map[string]string{
opts.AutoChecksum.Key(): crc.Encoded(),
amzChecksumMode: ChecksumCompositeMode.String(),
}
}
} else if opts.AutoChecksum.CanMergeCRC() {
crc, err := opts.AutoChecksum.FullObjectChecksum(allParts)
if err == nil {
opts.UserMetadata = map[string]string{
opts.AutoChecksum.Key(): crc.Encoded(),
amzChecksumMode: ChecksumFullObjectMode.String(),
}
}
}
}
Server/vendor/github.com/minio/minio-go/v7/code_of_conduct.md
Generated Vendored
+52
View File
@@ -0,0 +1,52 @@
Contributor Covenant Code of Conduct
====================================
Our Pledge
----------
In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
Our Standards
-------------
Examples of behavior that contributes to creating a positive environment include:
- Using welcoming and inclusive language
- Being respectful of differing viewpoints and experiences
- Gracefully accepting constructive criticism
- Focusing on what is best for the community
- Showing empathy towards other community members
Examples of unacceptable behavior by participants include:
- The use of sexualized language or imagery and unwelcome sexual attention or advances
- Trolling, insulting/derogatory comments, and personal or political attacks
- Public or private harassment
- Publishing others' private information, such as a physical or electronic address, without explicit permission
- Other conduct which could reasonably be considered inappropriate in a professional setting
Our Responsibilities
--------------------
Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior, in compliance with the licensing terms applying to the Project developments.
Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful. However, these actions shall respect the licensing terms of the Project Developments that will always supersede such Code of Conduct.
Scope
-----
This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
Enforcement
-----------
Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at dev@min.io. The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.
Attribution
-----------
This Code of Conduct is adapted from the [Contributor Covenant](http://contributor-covenant.org), version 1.4, available at [http://contributor-covenant.org/version/1/4](http://contributor-covenant.org/version/1/4/)
This version includes a clarification to ensure that the code of conduct is in compliance with the free software licensing terms of the project.
Server/vendor/github.com/minio/minio-go/v7/constants.go
Generated Vendored
+134
View File
@@ -0,0 +1,134 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2015-2017 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
// Multipart upload defaults.
// absMinPartSize - absolute minimum part size (5 MiB) below which
// a part in a multipart upload may not be uploaded.
const absMinPartSize = 1024 * 1024 * 5
// minPartSize - minimum part size 16MiB per object after which
// putObject behaves internally as multipart.
const minPartSize = 1024 * 1024 * 16
// maxPartsCount - maximum number of parts for a single multipart session.
const maxPartsCount = 10000
// maxPartSize - maximum part size 5GiB for a single multipart upload
// operation.
const maxPartSize = 1024 * 1024 * 1024 * 5
// maxSinglePutObjectSize - maximum size 5GiB of object per PUT
// operation.
const maxSinglePutObjectSize = 1024 * 1024 * 1024 * 5
// maxMultipartPutObjectSize - maximum size 5TiB of object for
// Multipart operation.
const maxMultipartPutObjectSize = 1024 * 1024 * 1024 * 1024 * 5
// maxObjectSize - maximum size of an object calculated from
// maxPartSize * maxPartsCount = 5GiB * 10000 = ~48.83TiB
const maxObjectSize = maxPartSize * maxPartsCount
// unsignedPayload - value to be set to X-Amz-Content-Sha256 header when
// we don't want to sign the request payload
const unsignedPayload = "UNSIGNED-PAYLOAD"
// unsignedPayloadTrailer value to be set to X-Amz-Content-Sha256 header when
// we don't want to sign the request payload, but have a trailer.
const unsignedPayloadTrailer = "STREAMING-UNSIGNED-PAYLOAD-TRAILER"
// Total number of parallel workers used for multipart operation.
const totalWorkers = 4
// Signature related constants.
const (
signV4Algorithm = "AWS4-HMAC-SHA256"
iso8601DateFormat = "20060102T150405Z"
)
const (
// GetObjectAttributesTags are tags used to defined
// return values for the GetObjectAttributes API
GetObjectAttributesTags = "ETag,Checksum,StorageClass,ObjectSize,ObjectParts"
// GetObjectAttributesMaxParts defined the default maximum
// number of parts returned by GetObjectAttributes
GetObjectAttributesMaxParts = 1000
)
const (
// Response Headers
// ETag is a common response header
ETag = "ETag"
// Storage class header.
amzStorageClass = "X-Amz-Storage-Class"
// Website redirect location header
amzWebsiteRedirectLocation = "X-Amz-Website-Redirect-Location"
// GetObjectAttributes headers
amzPartNumberMarker = "X-Amz-Part-Number-Marker"
amzExpectedBucketOnwer = "X-Amz-Expected-Bucket-Owner"
amzMaxParts = "X-Amz-Max-Parts"
amzObjectAttributes = "X-Amz-Object-Attributes"
// Object Tagging headers
amzTaggingHeader = "X-Amz-Tagging"
amzTaggingHeaderDirective = "X-Amz-Tagging-Directive"
amzVersionID = "X-Amz-Version-Id"
amzTaggingCount = "X-Amz-Tagging-Count"
amzExpiration = "X-Amz-Expiration"
amzRestore = "X-Amz-Restore"
amzReplicationStatus = "X-Amz-Replication-Status"
amzDeleteMarker = "X-Amz-Delete-Marker"
// Object legal hold header
amzLegalHoldHeader = "X-Amz-Object-Lock-Legal-Hold"
// Object retention header
amzLockMode = "X-Amz-Object-Lock-Mode"
amzLockRetainUntil = "X-Amz-Object-Lock-Retain-Until-Date"
amzBypassGovernance = "X-Amz-Bypass-Governance-Retention"
// Replication status
amzBucketReplicationStatus = "X-Amz-Replication-Status"
// Minio specific Replication/lifecycle transition extension
minIOBucketSourceMTime = "X-Minio-Source-Mtime"
minIOBucketSourceETag = "X-Minio-Source-Etag"
minIOBucketReplicationDeleteMarker = "X-Minio-Source-DeleteMarker"
minIOBucketReplicationProxyRequest = "X-Minio-Source-Proxy-Request"
minIOBucketReplicationRequest = "X-Minio-Source-Replication-Request"
minIOBucketReplicationCheck = "X-Minio-Source-Replication-Check"
// Header indicates last tag update time on source
minIOBucketReplicationTaggingTimestamp = "X-Minio-Source-Replication-Tagging-Timestamp"
// Header indicates last retention update time on source
minIOBucketReplicationObjectRetentionTimestamp = "X-Minio-Source-Replication-Retention-Timestamp"
// Header indicates last legalhold update time on source
minIOBucketReplicationObjectLegalHoldTimestamp = "X-Minio-Source-Replication-LegalHold-Timestamp"
minIOForceDelete = "x-minio-force-delete"
// Header indicates delete marker replication request can be sent by source now.
minioTgtReplicationReady = "X-Minio-Replication-Ready"
// Header asks if delete marker replication request can be sent by source now.
isMinioTgtReplicationReady = "X-Minio-Check-Replication-Ready"
)
Server/vendor/github.com/minio/minio-go/v7/core.go
Generated Vendored
+151
View File
@@ -0,0 +1,151 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2015-2017 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"context"
"io"
"net/http"
"github.com/minio/minio-go/v7/pkg/encrypt"
)
// Core - Inherits Client and adds new methods to expose the low level S3 APIs.
type Core struct {
*Client
}
// NewCore - Returns new initialized a Core client, this CoreClient should be
// only used under special conditions such as need to access lower primitives
// and being able to use them to write your own wrappers.
func NewCore(endpoint string, opts *Options) (*Core, error) {
var s3Client Core
client, err := New(endpoint, opts)
if err != nil {
return nil, err
}
s3Client.Client = client
return &s3Client, nil
}
// ListObjects - List all the objects at a prefix, optionally with marker and delimiter
// you can further filter the results.
func (c Core) ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (result ListBucketResult, err error) {
return c.listObjectsQuery(context.Background(), bucket, prefix, marker, delimiter, maxKeys, nil)
}
// ListObjectsV2 - Lists all the objects at a prefix, similar to ListObjects() but uses
// continuationToken instead of marker to support iteration over the results.
func (c Core) ListObjectsV2(bucketName, objectPrefix, startAfter, continuationToken, delimiter string, maxkeys int) (ListBucketV2Result, error) {
return c.listObjectsV2Query(context.Background(), bucketName, objectPrefix, continuationToken, true, false, delimiter, startAfter, maxkeys, nil)
}
// CopyObject - copies an object from source object to destination object on server side.
func (c Core) CopyObject(ctx context.Context, sourceBucket, sourceObject, destBucket, destObject string, metadata map[string]string, srcOpts CopySrcOptions, dstOpts PutObjectOptions) (ObjectInfo, error) {
return c.copyObjectDo(ctx, sourceBucket, sourceObject, destBucket, destObject, metadata, srcOpts, dstOpts)
}
// CopyObjectPart - creates a part in a multipart upload by copying (a
// part of) an existing object.
func (c Core) CopyObjectPart(ctx context.Context, srcBucket, srcObject, destBucket, destObject, uploadID string,
partID int, startOffset, length int64, metadata map[string]string,
) (p CompletePart, err error) {
return c.copyObjectPartDo(ctx, srcBucket, srcObject, destBucket, destObject, uploadID,
partID, startOffset, length, metadata)
}
// PutObject - Upload object. Uploads using single PUT call.
func (c Core) PutObject(ctx context.Context, bucket, object string, data io.Reader, size int64, md5Base64, sha256Hex string, opts PutObjectOptions) (UploadInfo, error) {
hookReader := newHook(data, opts.Progress)
return c.putObjectDo(ctx, bucket, object, hookReader, md5Base64, sha256Hex, size, opts)
}
// NewMultipartUpload - Initiates new multipart upload and returns the new uploadID.
func (c Core) NewMultipartUpload(ctx context.Context, bucket, object string, opts PutObjectOptions) (uploadID string, err error) {
result, err := c.initiateMultipartUpload(ctx, bucket, object, opts)
return result.UploadID, err
}
// ListMultipartUploads - List incomplete uploads.
func (c Core) ListMultipartUploads(ctx context.Context, bucket, prefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (result ListMultipartUploadsResult, err error) {
return c.listMultipartUploadsQuery(ctx, bucket, keyMarker, uploadIDMarker, prefix, delimiter, maxUploads)
}
// PutObjectPartOptions contains options for PutObjectPart API
type PutObjectPartOptions struct {
Md5Base64, Sha256Hex string
SSE encrypt.ServerSide
CustomHeader, Trailer http.Header
DisableContentSha256 bool
}
// PutObjectPart - Upload an object part.
func (c Core) PutObjectPart(ctx context.Context, bucket, object, uploadID string, partID int,
data io.Reader, size int64, opts PutObjectPartOptions,
) (ObjectPart, error) {
p := uploadPartParams{
bucketName: bucket,
objectName: object,
uploadID: uploadID,
reader: data,
partNumber: partID,
md5Base64: opts.Md5Base64,
sha256Hex: opts.Sha256Hex,
size: size,
sse: opts.SSE,
streamSha256: !opts.DisableContentSha256,
customHeader: opts.CustomHeader,
trailer: opts.Trailer,
}
return c.uploadPart(ctx, p)
}
// ListObjectParts - List uploaded parts of an incomplete upload.x
func (c Core) ListObjectParts(ctx context.Context, bucket, object, uploadID string, partNumberMarker, maxParts int) (result ListObjectPartsResult, err error) {
return c.listObjectPartsQuery(ctx, bucket, object, uploadID, partNumberMarker, maxParts)
}
// CompleteMultipartUpload - Concatenate uploaded parts and commit to an object.
func (c Core) CompleteMultipartUpload(ctx context.Context, bucket, object, uploadID string, parts []CompletePart, opts PutObjectOptions) (UploadInfo, error) {
res, err := c.completeMultipartUpload(ctx, bucket, object, uploadID, completeMultipartUpload{
Parts: parts,
}, opts)
return res, err
}
// AbortMultipartUpload - Abort an incomplete upload.
func (c Core) AbortMultipartUpload(ctx context.Context, bucket, object, uploadID string) error {
return c.abortMultipartUpload(ctx, bucket, object, uploadID)
}
// GetBucketPolicy - fetches bucket access policy for a given bucket.
func (c Core) GetBucketPolicy(ctx context.Context, bucket string) (string, error) {
return c.getBucketPolicy(ctx, bucket)
}
// PutBucketPolicy - applies a new bucket access policy for a given bucket.
func (c Core) PutBucketPolicy(ctx context.Context, bucket, bucketPolicy string) error {
return c.putBucketPolicy(ctx, bucket, bucketPolicy)
}
// GetObject is a lower level API implemented to support reading
// partial objects and also downloading objects with special conditions
// matching etag, modtime etc.
func (c Core) GetObject(ctx context.Context, bucketName, objectName string, opts GetObjectOptions) (io.ReadCloser, ObjectInfo, http.Header, error) {
return c.getObject(ctx, bucketName, objectName, opts)
}
Server/vendor/github.com/minio/minio-go/v7/create-session.go
Generated Vendored
+182
View File
@@ -0,0 +1,182 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2015-2025 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"context"
"encoding/xml"
"errors"
"net"
"net/http"
"net/url"
"path"
"time"
"github.com/minio/minio-go/v7/pkg/credentials"
"github.com/minio/minio-go/v7/pkg/s3utils"
"github.com/minio/minio-go/v7/pkg/signer"
)
// SessionMode - session mode type there are only two types
type SessionMode string
// Session constants
const (
SessionReadWrite SessionMode = "ReadWrite"
SessionReadOnly SessionMode = "ReadOnly"
)
type createSessionResult struct {
XMLName xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ CreateSessionResult"`
Credentials struct {
AccessKey string `xml:"AccessKeyId" json:"accessKey,omitempty"`
SecretKey string `xml:"SecretAccessKey" json:"secretKey,omitempty"`
SessionToken string `xml:"SessionToken" json:"sessionToken,omitempty"`
Expiration time.Time `xml:"Expiration" json:"expiration,omitempty"`
} `xml:",omitempty"`
}
// CreateSession - https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateSession.html
// the returning credentials may be cached depending on the expiration of the original
// credential, credentials will get renewed 10 secs earlier than when its gonna expire
// allowing for some leeway in the renewal process.
func (c *Client) CreateSession(ctx context.Context, bucketName string, sessionMode SessionMode) (cred credentials.Value, err error) {
if err := s3utils.CheckValidBucketNameS3Express(bucketName); err != nil {
return credentials.Value{}, err
}
v, ok := c.bucketSessionCache.Get(bucketName)
if ok && v.Expiration.After(time.Now().Add(10*time.Second)) {
// Verify if the credentials will not expire
// in another 10 seconds, if not we renew it again.
return v, nil
}
req, err := c.createSessionRequest(ctx, bucketName, sessionMode)
if err != nil {
return credentials.Value{}, err
}
resp, err := c.do(req)
defer closeResponse(resp)
if err != nil {
return credentials.Value{}, err
}
if resp.StatusCode != http.StatusOK {
return credentials.Value{}, httpRespToErrorResponse(resp, bucketName, "")
}
credSession := &createSessionResult{}
dec := xml.NewDecoder(resp.Body)
if err = dec.Decode(credSession); err != nil {
return credentials.Value{}, err
}
defer c.bucketSessionCache.Set(bucketName, cred)
return credentials.Value{
AccessKeyID: credSession.Credentials.AccessKey,
SecretAccessKey: credSession.Credentials.SecretKey,
SessionToken: credSession.Credentials.SessionToken,
Expiration: credSession.Credentials.Expiration,
}, nil
}
// createSessionRequest - Wrapper creates a new CreateSession request.
func (c *Client) createSessionRequest(ctx context.Context, bucketName string, sessionMode SessionMode) (*http.Request, error) {
// Set location query.
urlValues := make(url.Values)
urlValues.Set("session", "")
// Set get bucket location always as path style.
targetURL := *c.endpointURL
// Fetch new host based on the bucket location.
host := getS3ExpressEndpoint(c.region, s3utils.IsS3ExpressBucket(bucketName))
// as it works in makeTargetURL method from api.go file
if h, p, err := net.SplitHostPort(host); err == nil {
if targetURL.Scheme == "http" && p == "80" || targetURL.Scheme == "https" && p == "443" {
host = h
if ip := net.ParseIP(h); ip != nil && ip.To4() == nil {
host = "[" + h + "]"
}
}
}
isVirtualStyle := c.isVirtualHostStyleRequest(targetURL, bucketName)
var urlStr string
if isVirtualStyle {
urlStr = c.endpointURL.Scheme + "://" + bucketName + "." + host + "/?session"
} else {
targetURL.Path = path.Join(bucketName, "") + "/"
targetURL.RawQuery = urlValues.Encode()
urlStr = targetURL.String()
}
// Get a new HTTP request for the method.
req, err := http.NewRequestWithContext(ctx, http.MethodGet, urlStr, nil)
if err != nil {
return nil, err
}
// Set UserAgent for the request.
c.setUserAgent(req)
// Get credentials from the configured credentials provider.
value, err := c.credsProvider.GetWithContext(c.CredContext())
if err != nil {
return nil, err
}
var (
signerType = value.SignerType
accessKeyID = value.AccessKeyID
secretAccessKey = value.SecretAccessKey
sessionToken = value.SessionToken
)
// Custom signer set then override the behavior.
if c.overrideSignerType != credentials.SignatureDefault {
signerType = c.overrideSignerType
}
// If signerType returned by credentials helper is anonymous,
// then do not sign regardless of signerType override.
if value.SignerType == credentials.SignatureAnonymous {
signerType = credentials.SignatureAnonymous
}
if signerType.IsAnonymous() || signerType.IsV2() {
return req, errors.New("Only signature v4 is supported for CreateSession() API")
}
// Set sha256 sum for signature calculation only with signature version '4'.
contentSha256 := emptySHA256Hex
if c.secure {
contentSha256 = unsignedPayload
}
req.Header.Set("X-Amz-Content-Sha256", contentSha256)
req.Header.Set("x-amz-create-session-mode", string(sessionMode))
req = signer.SignV4Express(*req, accessKeyID, secretAccessKey, sessionToken, c.region)
return req, nil
}
Server/vendor/github.com/minio/minio-go/v7/endpoints.go
Generated Vendored
+276
View File
@@ -0,0 +1,276 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2015-2024 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
type awsS3Endpoint struct {
endpoint string
dualstackEndpoint string
}
type awsS3ExpressEndpoint struct {
regionalEndpoint string
zonalEndpoints []string
}
var awsS3ExpressEndpointMap = map[string]awsS3ExpressEndpoint{
"us-east-1": {
"s3express-control.us-east-1.amazonaws.com",
[]string{
"s3express-use1-az4.us-east-1.amazonaws.com",
"s3express-use1-az5.us-east-1.amazonaws.com",
"3express-use1-az6.us-east-1.amazonaws.com",
},
},
"us-east-2": {
"s3express-control.us-east-2.amazonaws.com",
[]string{
"s3express-use2-az1.us-east-2.amazonaws.com",
"s3express-use2-az2.us-east-2.amazonaws.com",
},
},
"us-west-2": {
"s3express-control.us-west-2.amazonaws.com",
[]string{
"s3express-usw2-az1.us-west-2.amazonaws.com",
"s3express-usw2-az3.us-west-2.amazonaws.com",
"s3express-usw2-az4.us-west-2.amazonaws.com",
},
},
"ap-south-1": {
"s3express-control.ap-south-1.amazonaws.com",
[]string{
"s3express-aps1-az1.ap-south-1.amazonaws.com",
"s3express-aps1-az3.ap-south-1.amazonaws.com",
},
},
"ap-northeast-1": {
"s3express-control.ap-northeast-1.amazonaws.com",
[]string{
"s3express-apne1-az1.ap-northeast-1.amazonaws.com",
"s3express-apne1-az4.ap-northeast-1.amazonaws.com",
},
},
"eu-west-1": {
"s3express-control.eu-west-1.amazonaws.com",
[]string{
"s3express-euw1-az1.eu-west-1.amazonaws.com",
"s3express-euw1-az3.eu-west-1.amazonaws.com",
},
},
"eu-north-1": {
"s3express-control.eu-north-1.amazonaws.com",
[]string{
"s3express-eun1-az1.eu-north-1.amazonaws.com",
"s3express-eun1-az2.eu-north-1.amazonaws.com",
"s3express-eun1-az3.eu-north-1.amazonaws.com",
},
},
}
// awsS3EndpointMap Amazon S3 endpoint map.
var awsS3EndpointMap = map[string]awsS3Endpoint{
"us-east-1": {
"s3.us-east-1.amazonaws.com",
"s3.dualstack.us-east-1.amazonaws.com",
},
"us-east-2": {
"s3.us-east-2.amazonaws.com",
"s3.dualstack.us-east-2.amazonaws.com",
},
"us-iso-east-1": {
"s3.us-iso-east-1.c2s.ic.gov",
"", // dualstack endpoint doesn't exist
},
"us-isob-east-1": {
"s3.us-isob-east-1.sc2s.sgov.gov",
"", // dualstack endpoint doesn't exist
},
"us-iso-west-1": {
"s3.us-iso-west-1.c2s.ic.gov",
"", // dualstack endpoint doesn't exist
},
"us-west-2": {
"s3.us-west-2.amazonaws.com",
"s3.dualstack.us-west-2.amazonaws.com",
},
"us-west-1": {
"s3.us-west-1.amazonaws.com",
"s3.dualstack.us-west-1.amazonaws.com",
},
"ca-central-1": {
"s3.ca-central-1.amazonaws.com",
"s3.dualstack.ca-central-1.amazonaws.com",
},
"ca-west-1": {
"s3.ca-west-1.amazonaws.com",
"s3.dualstack.ca-west-1.amazonaws.com",
},
"eu-west-1": {
"s3.eu-west-1.amazonaws.com",
"s3.dualstack.eu-west-1.amazonaws.com",
},
"eu-west-2": {
"s3.eu-west-2.amazonaws.com",
"s3.dualstack.eu-west-2.amazonaws.com",
},
"eu-west-3": {
"s3.eu-west-3.amazonaws.com",
"s3.dualstack.eu-west-3.amazonaws.com",
},
"eu-central-1": {
"s3.eu-central-1.amazonaws.com",
"s3.dualstack.eu-central-1.amazonaws.com",
},
"eu-central-2": {
"s3.eu-central-2.amazonaws.com",
"s3.dualstack.eu-central-2.amazonaws.com",
},
"eu-north-1": {
"s3.eu-north-1.amazonaws.com",
"s3.dualstack.eu-north-1.amazonaws.com",
},
"eu-south-1": {
"s3.eu-south-1.amazonaws.com",
"s3.dualstack.eu-south-1.amazonaws.com",
},
"eu-south-2": {
"s3.eu-south-2.amazonaws.com",
"s3.dualstack.eu-south-2.amazonaws.com",
},
"ap-east-1": {
"s3.ap-east-1.amazonaws.com",
"s3.dualstack.ap-east-1.amazonaws.com",
},
"ap-south-1": {
"s3.ap-south-1.amazonaws.com",
"s3.dualstack.ap-south-1.amazonaws.com",
},
"ap-south-2": {
"s3.ap-south-2.amazonaws.com",
"s3.dualstack.ap-south-2.amazonaws.com",
},
"ap-southeast-1": {
"s3.ap-southeast-1.amazonaws.com",
"s3.dualstack.ap-southeast-1.amazonaws.com",
},
"ap-southeast-2": {
"s3.ap-southeast-2.amazonaws.com",
"s3.dualstack.ap-southeast-2.amazonaws.com",
},
"ap-southeast-3": {
"s3.ap-southeast-3.amazonaws.com",
"s3.dualstack.ap-southeast-3.amazonaws.com",
},
"ap-southeast-4": {
"s3.ap-southeast-4.amazonaws.com",
"s3.dualstack.ap-southeast-4.amazonaws.com",
},
"ap-northeast-1": {
"s3.ap-northeast-1.amazonaws.com",
"s3.dualstack.ap-northeast-1.amazonaws.com",
},
"ap-northeast-2": {
"s3.ap-northeast-2.amazonaws.com",
"s3.dualstack.ap-northeast-2.amazonaws.com",
},
"ap-northeast-3": {
"s3.ap-northeast-3.amazonaws.com",
"s3.dualstack.ap-northeast-3.amazonaws.com",
},
"af-south-1": {
"s3.af-south-1.amazonaws.com",
"s3.dualstack.af-south-1.amazonaws.com",
},
"me-central-1": {
"s3.me-central-1.amazonaws.com",
"s3.dualstack.me-central-1.amazonaws.com",
},
"me-south-1": {
"s3.me-south-1.amazonaws.com",
"s3.dualstack.me-south-1.amazonaws.com",
},
"sa-east-1": {
"s3.sa-east-1.amazonaws.com",
"s3.dualstack.sa-east-1.amazonaws.com",
},
"us-gov-west-1": {
"s3.us-gov-west-1.amazonaws.com",
"s3.dualstack.us-gov-west-1.amazonaws.com",
},
"us-gov-east-1": {
"s3.us-gov-east-1.amazonaws.com",
"s3.dualstack.us-gov-east-1.amazonaws.com",
},
"cn-north-1": {
"s3.cn-north-1.amazonaws.com.cn",
"s3.dualstack.cn-north-1.amazonaws.com.cn",
},
"cn-northwest-1": {
"s3.cn-northwest-1.amazonaws.com.cn",
"s3.dualstack.cn-northwest-1.amazonaws.com.cn",
},
"il-central-1": {
"s3.il-central-1.amazonaws.com",
"s3.dualstack.il-central-1.amazonaws.com",
},
"ap-southeast-5": {
"s3.ap-southeast-5.amazonaws.com",
"s3.dualstack.ap-southeast-5.amazonaws.com",
},
"ap-southeast-7": {
"s3.ap-southeast-7.amazonaws.com",
"s3.dualstack.ap-southeast-7.amazonaws.com",
},
"mx-central-1": {
"s3.mx-central-1.amazonaws.com",
"s3.dualstack.mx-central-1.amazonaws.com",
},
"ap-east-2": {
"s3.ap-east-2.amazonaws.com",
"s3.dualstack.ap-east-2.amazonaws.com",
},
}
// getS3ExpressEndpoint get Amazon S3 Express endpoing based on the region
// optionally if zonal is set returns first zonal endpoint.
func getS3ExpressEndpoint(region string, zonal bool) (endpoint string) {
s3ExpEndpoint, ok := awsS3ExpressEndpointMap[region]
if !ok {
return ""
}
if zonal {
return s3ExpEndpoint.zonalEndpoints[0]
}
return s3ExpEndpoint.regionalEndpoint
}
// getS3Endpoint get Amazon S3 endpoint based on the bucket location.
func getS3Endpoint(bucketLocation string, useDualstack bool) (endpoint string) {
s3Endpoint, ok := awsS3EndpointMap[bucketLocation]
if !ok {
// Default to 's3.us-east-1.amazonaws.com' endpoint.
if useDualstack {
return "s3.dualstack.us-east-1.amazonaws.com"
}
return "s3.us-east-1.amazonaws.com"
}
if useDualstack && s3Endpoint.dualstackEndpoint != "" {
return s3Endpoint.dualstackEndpoint
}
return s3Endpoint.endpoint
}
Server/vendor/github.com/minio/minio-go/v7/functional_tests.go
Generated Vendored
+15077
View File
File diff suppressed because it is too large Load Diff
Server/vendor/github.com/minio/minio-go/v7/hook-reader.go
Generated Vendored
+93
View File
@@ -0,0 +1,93 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2015-2017 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package minio
import (
"fmt"
"io"
)
// hookReader hooks additional reader in the source stream. It is
// useful for making progress bars. Second reader is appropriately
// notified about the exact number of bytes read from the primary
// source on each Read operation.
type hookReader struct {
source io.Reader
hook io.Reader
}
// Seek implements io.Seeker. Seeks source first, and if necessary
// seeks hook if Seek method is appropriately found.
func (hr *hookReader) Seek(offset int64, whence int) (n int64, err error) {
// Verify for source has embedded Seeker, use it.
sourceSeeker, ok := hr.source.(io.Seeker)
if ok {
n, err = sourceSeeker.Seek(offset, whence)
if err != nil {
return 0, err
}
}
if hr.hook != nil {
// Verify if hook has embedded Seeker, use it.
hookSeeker, ok := hr.hook.(io.Seeker)
if ok {
var m int64
m, err = hookSeeker.Seek(offset, whence)
if err != nil {
return 0, err
}
if n != m {
return 0, fmt.Errorf("hook seeker seeked %d bytes, expected source %d bytes", m, n)
}
}
}
return n, nil
}
// Read implements io.Reader. Always reads from the source, the return
// value 'n' number of bytes are reported through the hook. Returns
// error for all non io.EOF conditions.
func (hr *hookReader) Read(b []byte) (n int, err error) {
n, err = hr.source.Read(b)
if err != nil && err != io.EOF {
return n, err
}
if hr.hook != nil {
// Progress the hook with the total read bytes from the source.
if _, herr := hr.hook.Read(b[:n]); herr != nil {
if herr != io.EOF {
return n, herr
}
}
}
return n, err
}
// newHook returns a io.ReadSeeker which implements hookReader that
// reports the data read from the source to the hook.
func newHook(source, hook io.Reader) io.Reader {
if hook == nil {
return &hookReader{source: source}
}
return &hookReader{
source: source,
hook: hook,
}
}
Server/vendor/github.com/minio/minio-go/v7/pkg/cors/cors.go
Generated Vendored
+91
View File
@@ -0,0 +1,91 @@
/*
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
* Copyright 2015-2024 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cors
import (
"encoding/xml"
"fmt"
"io"
"strings"
"github.com/dustin/go-humanize"
)
const defaultXMLNS = "http://s3.amazonaws.com/doc/2006-03-01/"
// Config is the container for a CORS configuration for a bucket.
type Config struct {
XMLNS string `xml:"xmlns,attr,omitempty"`
XMLName xml.Name `xml:"CORSConfiguration"`
CORSRules []Rule `xml:"CORSRule"`
}
// Rule is a single rule in a CORS configuration.
type Rule struct {
AllowedHeader []string `xml:"AllowedHeader,omitempty"`
AllowedMethod []string `xml:"AllowedMethod,omitempty"`
AllowedOrigin []string `xml:"AllowedOrigin,omitempty"`
ExposeHeader []string `xml:"ExposeHeader,omitempty"`
ID string `xml:"ID,omitempty"`
MaxAgeSeconds int `xml:"MaxAgeSeconds,omitempty"`
}
// NewConfig creates a new CORS configuration with the given rules.
func NewConfig(rules []Rule) *Config {
return &Config{
XMLNS: defaultXMLNS,
XMLName: xml.Name{
Local: "CORSConfiguration",
Space: defaultXMLNS,
},
CORSRules: rules,
}
}
// ParseBucketCorsConfig parses a CORS configuration in XML from an io.Reader.
func ParseBucketCorsConfig(reader io.Reader) (*Config, error) {
var c Config
// Max size of cors document is 64KiB according to https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketCors.html
// This limiter is just for safety so has a max of 128KiB
err := xml.NewDecoder(io.LimitReader(reader, 128*humanize.KiByte)).Decode(&c)
if err != nil {
return nil, fmt.Errorf("decoding xml: %w", err)
}
if c.XMLNS == "" {
c.XMLNS = defaultXMLNS
}
for i, rule := range c.CORSRules {
for j, method := range rule.AllowedMethod {
c.CORSRules[i].AllowedMethod[j] = strings.ToUpper(method)
}
}
return &c, nil
}
// ToXML marshals the CORS configuration to XML.
func (c Config) ToXML() ([]byte, error) {
if c.XMLNS == "" {
c.XMLNS = defaultXMLNS
}
data, err := xml.Marshal(&c)
if err != nil {
return nil, fmt.Errorf("marshaling xml: %w", err)
}
return append([]byte(xml.Header), data...), nil
}

Some files were not shown because too many files have changed in this diff Show More