sevenhill/QSfera
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial QSfera import

Browse Source
This commit is contained in:
Курнат Андрей
2026-06-07 10:20:04 +03:00
commit 2315f25754
16485 changed files with 4826827 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Server/vendor/github.com/klauspost/crc32/.gitignore
Generated
+24
View File
@@ -0,0 +1,24 @@
# Compiled Object files, Static and Dynamic libs (Shared Objects)
*.o
*.a
*.so
# Folders
_obj
_test
# Architecture specific extensions/prefixes
*.[568vq]
[568vq].out
*.cgo1.go
*.cgo2.c
_cgo_defun.c
_cgo_gotypes.go
_cgo_export.*
_testmain.go
*.exe
*.test
*.prof
Server/vendor/github.com/klauspost/crc32/LICENSE
Generated Vendored
+27
View File
@@ -0,0 +1,27 @@
Copyright (c) 2012 The Go Authors. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Server/vendor/github.com/klauspost/crc32/README.md
Generated Vendored
+42
View File
@@ -0,0 +1,42 @@
# 2025 revival
For IEEE checksums AVX512 can be used to speed up CRC32 checksums by approximately 2x.
Castagnoli checksums (CRC32C) can also be computer with AVX512,
but the performance gain is not as significant enough for the downsides of using it at this point.
# crc32
This package is a drop-in replacement for the standard library `hash/crc32` package,
that features AVX 512 optimizations on x64 platforms, for a 2x speedup for IEEE CRC32 checksums.
# usage
Install using `go get github.com/klauspost/crc32`. This library is based on Go 1.24
Replace `import "hash/crc32"` with `import "github.com/klauspost/crc32"` and you are good to go.
# changes
* 2025: Revived and updated to Go 1.24, with AVX 512 optimizations.
# performance
AVX512 are enabled above 1KB input size. This rather high limit is due to AVX512 may be slower to ramp up than
the regular SSE4 implementation for smaller inputs. This is not reflected in the benchmarks below.
| Benchmark | Old MB/s | New MB/s | Speedup |
|-----------------------------------------------|----------|----------|---------|
| BenchmarkCRC32/poly=IEEE/size=512/align=0-32 | 17996.39 | 17969.94 | 1.00x |
| BenchmarkCRC32/poly=IEEE/size=512/align=1-32 | 18021.48 | 17945.55 | 1.00x |
| BenchmarkCRC32/poly=IEEE/size=1kB/align=0-32 | 19921.70 | 45613.77 | 2.29x |
| BenchmarkCRC32/poly=IEEE/size=1kB/align=1-32 | 19946.60 | 46819.09 | 2.35x |
| BenchmarkCRC32/poly=IEEE/size=4kB/align=0-32 | 21538.65 | 48600.93 | 2.26x |
| BenchmarkCRC32/poly=IEEE/size=4kB/align=1-32 | 21449.20 | 48477.84 | 2.26x |
| BenchmarkCRC32/poly=IEEE/size=32kB/align=0-32 | 21785.49 | 46013.10 | 2.11x |
| BenchmarkCRC32/poly=IEEE/size=32kB/align=1-32 | 21946.47 | 45954.10 | 2.09x |
cpu: AMD Ryzen 9 9950X 16-Core Processor
# license
Standard Go license. See [LICENSE](LICENSE) for details.
Server/vendor/github.com/klauspost/crc32/crc32.go
Generated Vendored
+253
View File
@@ -0,0 +1,253 @@
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package crc32 implements the 32-bit cyclic redundancy check, or CRC-32,
// checksum. See https://en.wikipedia.org/wiki/Cyclic_redundancy_check for
// information.
//
// Polynomials are represented in LSB-first form also known as reversed representation.
//
// See https://en.wikipedia.org/wiki/Mathematics_of_cyclic_redundancy_checks#Reversed_representations_and_reciprocal_polynomials
// for information.
package crc32
import (
"encoding/binary"
"errors"
"hash"
"sync"
"sync/atomic"
)
// The size of a CRC-32 checksum in bytes.
const Size = 4
// Predefined polynomials.
const (
// IEEE is by far and away the most common CRC-32 polynomial.
// Used by ethernet (IEEE 802.3), v.42, fddi, gzip, zip, png, ...
IEEE = 0xedb88320
// Castagnoli's polynomial, used in iSCSI.
// Has better error detection characteristics than IEEE.
// https://dx.doi.org/10.1109/26.231911
Castagnoli = 0x82f63b78
// Koopman's polynomial.
// Also has better error detection characteristics than IEEE.
// https://dx.doi.org/10.1109/DSN.2002.1028931
Koopman = 0xeb31d82e
)
// Table is a 256-word table representing the polynomial for efficient processing.
type Table [256]uint32
// This file makes use of functions implemented in architecture-specific files.
// The interface that they implement is as follows:
//
// // archAvailableIEEE reports whether an architecture-specific CRC32-IEEE
// // algorithm is available.
// archAvailableIEEE() bool
//
// // archInitIEEE initializes the architecture-specific CRC3-IEEE algorithm.
// // It can only be called if archAvailableIEEE() returns true.
// archInitIEEE()
//
// // archUpdateIEEE updates the given CRC32-IEEE. It can only be called if
// // archInitIEEE() was previously called.
// archUpdateIEEE(crc uint32, p []byte) uint32
//
// // archAvailableCastagnoli reports whether an architecture-specific
// // CRC32-C algorithm is available.
// archAvailableCastagnoli() bool
//
// // archInitCastagnoli initializes the architecture-specific CRC32-C
// // algorithm. It can only be called if archAvailableCastagnoli() returns
// // true.
// archInitCastagnoli()
//
// // archUpdateCastagnoli updates the given CRC32-C. It can only be called
// // if archInitCastagnoli() was previously called.
// archUpdateCastagnoli(crc uint32, p []byte) uint32
// castagnoliTable points to a lazily initialized Table for the Castagnoli
// polynomial. MakeTable will always return this value when asked to make a
// Castagnoli table so we can compare against it to find when the caller is
// using this polynomial.
var castagnoliTable *Table
var castagnoliTable8 *slicing8Table
var updateCastagnoli func(crc uint32, p []byte) uint32
var haveCastagnoli atomic.Bool
var castagnoliInitOnce = sync.OnceFunc(func() {
castagnoliTable = simpleMakeTable(Castagnoli)
if archAvailableCastagnoli() {
archInitCastagnoli()
updateCastagnoli = archUpdateCastagnoli
} else {
// Initialize the slicing-by-8 table.
castagnoliTable8 = slicingMakeTable(Castagnoli)
updateCastagnoli = func(crc uint32, p []byte) uint32 {
return slicingUpdate(crc, castagnoliTable8, p)
}
}
haveCastagnoli.Store(true)
})
// IEEETable is the table for the [IEEE] polynomial.
var IEEETable = simpleMakeTable(IEEE)
// ieeeTable8 is the slicing8Table for IEEE
var ieeeTable8 *slicing8Table
var updateIEEE func(crc uint32, p []byte) uint32
var ieeeInitOnce = sync.OnceFunc(func() {
if archAvailableIEEE() {
archInitIEEE()
updateIEEE = archUpdateIEEE
} else {
// Initialize the slicing-by-8 table.
ieeeTable8 = slicingMakeTable(IEEE)
updateIEEE = func(crc uint32, p []byte) uint32 {
return slicingUpdate(crc, ieeeTable8, p)
}
}
})
// MakeTable returns a [Table] constructed from the specified polynomial.
// The contents of this [Table] must not be modified.
func MakeTable(poly uint32) *Table {
switch poly {
case IEEE:
ieeeInitOnce()
return IEEETable
case Castagnoli:
castagnoliInitOnce()
return castagnoliTable
default:
return simpleMakeTable(poly)
}
}
// digest represents the partial evaluation of a checksum.
type digest struct {
crc uint32
tab *Table
}
// New creates a new [hash.Hash32] computing the CRC-32 checksum using the
// polynomial represented by the [Table]. Its Sum method will lay the
// value out in big-endian byte order. The returned Hash32 also
// implements [encoding.BinaryMarshaler] and [encoding.BinaryUnmarshaler] to
// marshal and unmarshal the internal state of the hash.
func New(tab *Table) hash.Hash32 {
if tab == IEEETable {
ieeeInitOnce()
}
return &digest{0, tab}
}
// NewIEEE creates a new [hash.Hash32] computing the CRC-32 checksum using
// the [IEEE] polynomial. Its Sum method will lay the value out in
// big-endian byte order. The returned Hash32 also implements
// [encoding.BinaryMarshaler] and [encoding.BinaryUnmarshaler] to marshal
// and unmarshal the internal state of the hash.
func NewIEEE() hash.Hash32 { return New(IEEETable) }
func (d *digest) Size() int { return Size }
func (d *digest) BlockSize() int { return 1 }
func (d *digest) Reset() { d.crc = 0 }
const (
magic = "crc\x01"
marshaledSize = len(magic) + 4 + 4
)
func (d *digest) AppendBinary(b []byte) ([]byte, error) {
b = append(b, magic...)
b = binary.BigEndian.AppendUint32(b, tableSum(d.tab))
b = binary.BigEndian.AppendUint32(b, d.crc)
return b, nil
}
func (d *digest) MarshalBinary() ([]byte, error) {
return d.AppendBinary(make([]byte, 0, marshaledSize))
}
func (d *digest) UnmarshalBinary(b []byte) error {
if len(b) < len(magic) || string(b[:len(magic)]) != magic {
return errors.New("hash/crc32: invalid hash state identifier")
}
if len(b) != marshaledSize {
return errors.New("hash/crc32: invalid hash state size")
}
if tableSum(d.tab) != binary.BigEndian.Uint32(b[4:]) {
return errors.New("hash/crc32: tables do not match")
}
d.crc = binary.BigEndian.Uint32(b[8:])
return nil
}
func update(crc uint32, tab *Table, p []byte, checkInitIEEE bool) uint32 {
switch {
case haveCastagnoli.Load() && tab == castagnoliTable:
return updateCastagnoli(crc, p)
case tab == IEEETable:
if checkInitIEEE {
ieeeInitOnce()
}
return updateIEEE(crc, p)
default:
return simpleUpdate(crc, tab, p)
}
}
// Update returns the result of adding the bytes in p to the crc.
func Update(crc uint32, tab *Table, p []byte) uint32 {
// Unfortunately, because IEEETable is exported, IEEE may be used without a
// call to MakeTable. We have to make sure it gets initialized in that case.
return update(crc, tab, p, true)
}
func (d *digest) Write(p []byte) (n int, err error) {
// We only create digest objects through New() which takes care of
// initialization in this case.
d.crc = update(d.crc, d.tab, p, false)
return len(p), nil
}
func (d *digest) Sum32() uint32 { return d.crc }
func (d *digest) Sum(in []byte) []byte {
s := d.Sum32()
return append(in, byte(s>>24), byte(s>>16), byte(s>>8), byte(s))
}
// Checksum returns the CRC-32 checksum of data
// using the polynomial represented by the [Table].
func Checksum(data []byte, tab *Table) uint32 { return Update(0, tab, data) }
// ChecksumIEEE returns the CRC-32 checksum of data
// using the [IEEE] polynomial.
func ChecksumIEEE(data []byte) uint32 {
ieeeInitOnce()
return updateIEEE(0, data)
}
// tableSum returns the IEEE checksum of table t.
func tableSum(t *Table) uint32 {
var a [1024]byte
b := a[:0]
if t != nil {
for _, x := range t {
b = binary.BigEndian.AppendUint32(b, x)
}
}
return ChecksumIEEE(b)
}
Server/vendor/github.com/klauspost/crc32/crc32_amd64.go
Generated Vendored
+253
View File
@@ -0,0 +1,253 @@
// Copyright 2011 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// AMD64-specific hardware-assisted CRC32 algorithms. See crc32.go for a
// description of the interface that each architecture-specific file
// implements.
package crc32
import (
"unsafe"
"golang.org/x/sys/cpu"
)
// This file contains the code to call the SSE 4.2 version of the Castagnoli
// and IEEE CRC.
// castagnoliSSE42 is defined in crc32_amd64.s and uses the SSE 4.2 CRC32
// instruction.
//
//go:noescape
func castagnoliSSE42(crc uint32, p []byte) uint32
// castagnoliSSE42Triple is defined in crc32_amd64.s and uses the SSE 4.2 CRC32
// instruction.
//
//go:noescape
func castagnoliSSE42Triple(
crcA, crcB, crcC uint32,
a, b, c []byte,
rounds uint32,
) (retA uint32, retB uint32, retC uint32)
// ieeeCLMUL is defined in crc_amd64.s and uses the PCLMULQDQ
// instruction as well as SSE 4.1.
//
//go:noescape
func ieeeCLMUL(crc uint32, p []byte) uint32
// castagnoliCLMULAvx512 is defined in crc_amd64.s and uses the PCLMULQDQ
// instruction as well as SSE 4.1.
//
//go:noescape
func castagnoliCLMULAvx512(crc uint32, p []byte) uint32
// ieeeCLMUL is defined in crc_amd64.s and uses the PCLMULQDQ
// instruction as well as SSE 4.1.
//
//go:noescape
func ieeeCLMULAvx512(crc uint32, p []byte) uint32
const castagnoliK1 = 168
const castagnoliK2 = 1344
type sse42Table [4]Table
var castagnoliSSE42TableK1 *sse42Table
var castagnoliSSE42TableK2 *sse42Table
func archAvailableCastagnoli() bool {
return cpu.X86.HasSSE42
}
func archInitCastagnoli() {
if !cpu.X86.HasSSE42 {
panic("arch-specific Castagnoli not available")
}
castagnoliSSE42TableK1 = new(sse42Table)
castagnoliSSE42TableK2 = new(sse42Table)
// See description in updateCastagnoli.
// t[0][i] = CRC(i000, O)
// t[1][i] = CRC(0i00, O)
// t[2][i] = CRC(00i0, O)
// t[3][i] = CRC(000i, O)
// where O is a sequence of K zeros.
var tmp [castagnoliK2]byte
for b := 0; b < 4; b++ {
for i := 0; i < 256; i++ {
val := uint32(i) << uint32(b*8)
castagnoliSSE42TableK1[b][i] = castagnoliSSE42(val, tmp[:castagnoliK1])
castagnoliSSE42TableK2[b][i] = castagnoliSSE42(val, tmp[:])
}
}
}
// castagnoliShift computes the CRC32-C of K1 or K2 zeroes (depending on the
// table given) with the given initial crc value. This corresponds to
// CRC(crc, O) in the description in updateCastagnoli.
func castagnoliShift(table *sse42Table, crc uint32) uint32 {
return table[3][crc>>24] ^
table[2][(crc>>16)&0xFF] ^
table[1][(crc>>8)&0xFF] ^
table[0][crc&0xFF]
}
func archUpdateCastagnoli(crc uint32, p []byte) uint32 {
if !cpu.X86.HasSSE42 {
panic("not available")
}
// This method is inspired from the algorithm in Intel's white paper:
// "Fast CRC Computation for iSCSI Polynomial Using CRC32 Instruction"
// The same strategy of splitting the buffer in three is used but the
// combining calculation is different; the complete derivation is explained
// below.
//
// -- The basic idea --
//
// The CRC32 instruction (available in SSE4.2) can process 8 bytes at a
// time. In recent Intel architectures the instruction takes 3 cycles;
// however the processor can pipeline up to three instructions if they
// don't depend on each other.
//
// Roughly this means that we can process three buffers in about the same
// time we can process one buffer.
//
// The idea is then to split the buffer in three, CRC the three pieces
// separately and then combine the results.
//
// Combining the results requires precomputed tables, so we must choose a
// fixed buffer length to optimize. The longer the length, the faster; but
// only buffers longer than this length will use the optimization. We choose
// two cutoffs and compute tables for both:
// - one around 512: 168*3=504
// - one around 4KB: 1344*3=4032
//
// -- The nitty gritty --
//
// Let CRC(I, X) be the non-inverted CRC32-C of the sequence X (with
// initial non-inverted CRC I). This function has the following properties:
// (a) CRC(I, AB) = CRC(CRC(I, A), B)
// (b) CRC(I, A xor B) = CRC(I, A) xor CRC(0, B)
//
// Say we want to compute CRC(I, ABC) where A, B, C are three sequences of
// K bytes each, where K is a fixed constant. Let O be the sequence of K zero
// bytes.
//
// CRC(I, ABC) = CRC(I, ABO xor C)
// = CRC(I, ABO) xor CRC(0, C)
// = CRC(CRC(I, AB), O) xor CRC(0, C)
// = CRC(CRC(I, AO xor B), O) xor CRC(0, C)
// = CRC(CRC(I, AO) xor CRC(0, B), O) xor CRC(0, C)
// = CRC(CRC(CRC(I, A), O) xor CRC(0, B), O) xor CRC(0, C)
//
// The castagnoliSSE42Triple function can compute CRC(I, A), CRC(0, B),
// and CRC(0, C) efficiently. We just need to find a way to quickly compute
// CRC(uvwx, O) given a 4-byte initial value uvwx. We can precompute these
// values; since we can't have a 32-bit table, we break it up into four
// 8-bit tables:
//
// CRC(uvwx, O) = CRC(u000, O) xor
// CRC(0v00, O) xor
// CRC(00w0, O) xor
// CRC(000x, O)
//
// We can compute tables corresponding to the four terms for all 8-bit
// values.
crc = ^crc
// Disabled, since it is not significantly faster than the SSE 4.2 version, even on Zen 5.
if false && len(p) >= 2048 && cpu.X86.HasAVX512F && cpu.X86.HasAVX512VL && cpu.X86.HasAVX512VPCLMULQDQ && cpu.X86.HasPCLMULQDQ {
left := len(p) & 15
do := len(p) - left
crc = castagnoliCLMULAvx512(crc, p[:do])
return ^castagnoliSSE42(crc, p[do:])
}
// If a buffer is long enough to use the optimization, process the first few
// bytes to align the buffer to an 8 byte boundary (if necessary).
if len(p) >= castagnoliK1*3 {
delta := int(uintptr(unsafe.Pointer(&p[0])) & 7)
if delta != 0 {
delta = 8 - delta
crc = castagnoliSSE42(crc, p[:delta])
p = p[delta:]
}
}
// Process 3*K2 at a time.
for len(p) >= castagnoliK2*3 {
// Compute CRC(I, A), CRC(0, B), and CRC(0, C).
crcA, crcB, crcC := castagnoliSSE42Triple(
crc, 0, 0,
p, p[castagnoliK2:], p[castagnoliK2*2:],
castagnoliK2/24)
// CRC(I, AB) = CRC(CRC(I, A), O) xor CRC(0, B)
crcAB := castagnoliShift(castagnoliSSE42TableK2, crcA) ^ crcB
// CRC(I, ABC) = CRC(CRC(I, AB), O) xor CRC(0, C)
crc = castagnoliShift(castagnoliSSE42TableK2, crcAB) ^ crcC
p = p[castagnoliK2*3:]
}
// Process 3*K1 at a time.
for len(p) >= castagnoliK1*3 {
// Compute CRC(I, A), CRC(0, B), and CRC(0, C).
crcA, crcB, crcC := castagnoliSSE42Triple(
crc, 0, 0,
p, p[castagnoliK1:], p[castagnoliK1*2:],
castagnoliK1/24)
// CRC(I, AB) = CRC(CRC(I, A), O) xor CRC(0, B)
crcAB := castagnoliShift(castagnoliSSE42TableK1, crcA) ^ crcB
// CRC(I, ABC) = CRC(CRC(I, AB), O) xor CRC(0, C)
crc = castagnoliShift(castagnoliSSE42TableK1, crcAB) ^ crcC
p = p[castagnoliK1*3:]
}
// Use the simple implementation for what's left.
crc = castagnoliSSE42(crc, p)
return ^crc
}
func archAvailableIEEE() bool {
return cpu.X86.HasPCLMULQDQ && cpu.X86.HasSSE41
}
var archIeeeTable8 *slicing8Table
func archInitIEEE() {
if !cpu.X86.HasPCLMULQDQ || !cpu.X86.HasSSE41 {
panic("not available")
}
// We still use slicing-by-8 for small buffers.
archIeeeTable8 = slicingMakeTable(IEEE)
}
func archUpdateIEEE(crc uint32, p []byte) uint32 {
if !cpu.X86.HasPCLMULQDQ || !cpu.X86.HasSSE41 {
panic("not available")
}
if len(p) >= 64 {
if len(p) >= 1024 && cpu.X86.HasAVX512F && cpu.X86.HasAVX512VL && cpu.X86.HasAVX512VPCLMULQDQ && cpu.X86.HasPCLMULQDQ {
left := len(p) & 15
do := len(p) - left
crc = ^ieeeCLMULAvx512(^crc, p[:do])
p = p[do:]
} else {
left := len(p) & 15
do := len(p) - left
crc = ^ieeeCLMUL(^crc, p[:do])
p = p[do:]
}
}
if len(p) == 0 {
return crc
}
return slicingUpdate(crc, archIeeeTable8, p)
}
Server/vendor/github.com/klauspost/crc32/crc32_amd64.s
Generated Vendored
+527
View File
@@ -0,0 +1,527 @@
// Copyright 2011 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
#include "textflag.h"
// castagnoliSSE42 updates the (non-inverted) crc with the given buffer.
//
// func castagnoliSSE42(crc uint32, p []byte) uint32
TEXT ·castagnoliSSE42(SB), NOSPLIT, $0
MOVL crc+0(FP), AX // CRC value
MOVQ p+8(FP), SI // data pointer
MOVQ p_len+16(FP), CX // len(p)
// If there are fewer than 8 bytes to process, skip alignment.
CMPQ CX, $8
JL less_than_8
MOVQ SI, BX
ANDQ $7, BX
JZ aligned
// Process the first few bytes to 8-byte align the input.
// BX = 8 - BX. We need to process this many bytes to align.
SUBQ $1, BX
XORQ $7, BX
BTQ $0, BX
JNC align_2
CRC32B (SI), AX
DECQ CX
INCQ SI
align_2:
BTQ $1, BX
JNC align_4
CRC32W (SI), AX
SUBQ $2, CX
ADDQ $2, SI
align_4:
BTQ $2, BX
JNC aligned
CRC32L (SI), AX
SUBQ $4, CX
ADDQ $4, SI
aligned:
// The input is now 8-byte aligned and we can process 8-byte chunks.
CMPQ CX, $8
JL less_than_8
CRC32Q (SI), AX
ADDQ $8, SI
SUBQ $8, CX
JMP aligned
less_than_8:
// We may have some bytes left over; process 4 bytes, then 2, then 1.
BTQ $2, CX
JNC less_than_4
CRC32L (SI), AX
ADDQ $4, SI
less_than_4:
BTQ $1, CX
JNC less_than_2
CRC32W (SI), AX
ADDQ $2, SI
less_than_2:
BTQ $0, CX
JNC done
CRC32B (SI), AX
done:
MOVL AX, ret+32(FP)
RET
// castagnoliSSE42Triple updates three (non-inverted) crcs with (24*rounds)
// bytes from each buffer.
//
// func castagnoliSSE42Triple(
// crc1, crc2, crc3 uint32,
// a, b, c []byte,
// rounds uint32,
// ) (retA uint32, retB uint32, retC uint32)
TEXT ·castagnoliSSE42Triple(SB), NOSPLIT, $0
MOVL crcA+0(FP), AX
MOVL crcB+4(FP), CX
MOVL crcC+8(FP), DX
MOVQ a+16(FP), R8 // data pointer
MOVQ b+40(FP), R9 // data pointer
MOVQ c+64(FP), R10 // data pointer
MOVL rounds+88(FP), R11
loop:
CRC32Q (R8), AX
CRC32Q (R9), CX
CRC32Q (R10), DX
CRC32Q 8(R8), AX
CRC32Q 8(R9), CX
CRC32Q 8(R10), DX
CRC32Q 16(R8), AX
CRC32Q 16(R9), CX
CRC32Q 16(R10), DX
ADDQ $24, R8
ADDQ $24, R9
ADDQ $24, R10
DECQ R11
JNZ loop
MOVL AX, retA+96(FP)
MOVL CX, retB+100(FP)
MOVL DX, retC+104(FP)
RET
// CRC32 polynomial data
//
// These constants are lifted from the
// Linux kernel, since they avoid the costly
// PSHUFB 16 byte reversal proposed in the
// original Intel paper.
DATA r2r1<>+0(SB)/8, $0x154442bd4
DATA r2r1<>+8(SB)/8, $0x1c6e41596
DATA r4r3<>+0(SB)/8, $0x1751997d0
DATA r4r3<>+8(SB)/8, $0x0ccaa009e
DATA rupoly<>+0(SB)/8, $0x1db710641
DATA rupoly<>+8(SB)/8, $0x1f7011641
DATA r5<>+0(SB)/8, $0x163cd6124
GLOBL r2r1<>(SB), RODATA, $16
GLOBL r4r3<>(SB), RODATA, $16
GLOBL rupoly<>(SB), RODATA, $16
GLOBL r5<>(SB), RODATA, $8
// Based on https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/fast-crc-computation-generic-polynomials-pclmulqdq-paper.pdf
// len(p) must be at least 64, and must be a multiple of 16.
// func ieeeCLMUL(crc uint32, p []byte) uint32
TEXT ·ieeeCLMUL(SB), NOSPLIT, $0
MOVL crc+0(FP), X0 // Initial CRC value
MOVQ p+8(FP), SI // data pointer
MOVQ p_len+16(FP), CX // len(p)
MOVOU (SI), X1
MOVOU 16(SI), X2
MOVOU 32(SI), X3
MOVOU 48(SI), X4
PXOR X0, X1
ADDQ $64, SI // buf+=64
SUBQ $64, CX // len-=64
CMPQ CX, $64 // Less than 64 bytes left
JB remain64
MOVOA r2r1<>+0(SB), X0
loopback64:
MOVOA X1, X5
MOVOA X2, X6
MOVOA X3, X7
MOVOA X4, X8
PCLMULQDQ $0, X0, X1
PCLMULQDQ $0, X0, X2
PCLMULQDQ $0, X0, X3
PCLMULQDQ $0, X0, X4
// Load next early
MOVOU (SI), X11
MOVOU 16(SI), X12
MOVOU 32(SI), X13
MOVOU 48(SI), X14
PCLMULQDQ $0x11, X0, X5
PCLMULQDQ $0x11, X0, X6
PCLMULQDQ $0x11, X0, X7
PCLMULQDQ $0x11, X0, X8
PXOR X5, X1
PXOR X6, X2
PXOR X7, X3
PXOR X8, X4
PXOR X11, X1
PXOR X12, X2
PXOR X13, X3
PXOR X14, X4
ADDQ $0x40, DI
ADDQ $64, SI // buf+=64
SUBQ $64, CX // len-=64
CMPQ CX, $64 // Less than 64 bytes left?
JGE loopback64
// Fold result into a single register (X1)
remain64:
MOVOA r4r3<>+0(SB), X0
MOVOA X1, X5
PCLMULQDQ $0, X0, X1
PCLMULQDQ $0x11, X0, X5
PXOR X5, X1
PXOR X2, X1
MOVOA X1, X5
PCLMULQDQ $0, X0, X1
PCLMULQDQ $0x11, X0, X5
PXOR X5, X1
PXOR X3, X1
MOVOA X1, X5
PCLMULQDQ $0, X0, X1
PCLMULQDQ $0x11, X0, X5
PXOR X5, X1
PXOR X4, X1
// If there is less than 16 bytes left we are done
CMPQ CX, $16
JB finish
// Encode 16 bytes
remain16:
MOVOU (SI), X10
MOVOA X1, X5
PCLMULQDQ $0, X0, X1
PCLMULQDQ $0x11, X0, X5
PXOR X5, X1
PXOR X10, X1
SUBQ $16, CX
ADDQ $16, SI
CMPQ CX, $16
JGE remain16
finish:
// Fold final result into 32 bits and return it
PCMPEQB X3, X3
PCLMULQDQ $1, X1, X0
PSRLDQ $8, X1
PXOR X0, X1
MOVOA X1, X2
MOVQ r5<>+0(SB), X0
// Creates 32 bit mask. Note that we don't care about upper half.
PSRLQ $32, X3
PSRLDQ $4, X2
PAND X3, X1
PCLMULQDQ $0, X0, X1
PXOR X2, X1
MOVOA rupoly<>+0(SB), X0
MOVOA X1, X2
PAND X3, X1
PCLMULQDQ $0x10, X0, X1
PAND X3, X1
PCLMULQDQ $0, X0, X1
PXOR X2, X1
PEXTRD $1, X1, AX
MOVL AX, ret+32(FP)
RET
DATA r2r1X<>+0(SB)/8, $0x154442bd4
DATA r2r1X<>+8(SB)/8, $0x1c6e41596
DATA r2r1X<>+16(SB)/8, $0x154442bd4
DATA r2r1X<>+24(SB)/8, $0x1c6e41596
DATA r2r1X<>+32(SB)/8, $0x154442bd4
DATA r2r1X<>+40(SB)/8, $0x1c6e41596
DATA r2r1X<>+48(SB)/8, $0x154442bd4
DATA r2r1X<>+56(SB)/8, $0x1c6e41596
GLOBL r2r1X<>(SB), RODATA, $64
// Based on https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/fast-crc-computation-generic-polynomials-pclmulqdq-paper.pdf
// len(p) must be at least 128, and must be a multiple of 16.
// func ieeeCLMULAvx512(crc uint32, p []byte) uint32
TEXT ·ieeeCLMULAvx512(SB), NOSPLIT, $0
MOVL crc+0(FP), AX // Initial CRC value
MOVQ p+8(FP), SI // data pointer
MOVQ p_len+16(FP), CX // len(p)
VPXORQ Z0, Z0, Z0
VMOVDQU64 (SI), Z1
VMOVQ AX, X0
VPXORQ Z0, Z1, Z1 // Merge initial CRC value into Z1
ADDQ $64, SI // buf+=64
SUBQ $64, CX // len-=64
VMOVDQU64 r2r1X<>+0(SB), Z0
loopback64:
// Load next early
VMOVDQU64 (SI), Z11
VPCLMULQDQ $0x11, Z0, Z1, Z5
VPCLMULQDQ $0, Z0, Z1, Z1
VPTERNLOGD $0x96, Z11, Z5, Z1 // Combine results with xor into Z1
ADDQ $0x40, DI
ADDQ $64, SI // buf+=64
SUBQ $64, CX // len-=64
CMPQ CX, $64 // Less than 64 bytes left?
JGE loopback64
// Fold result into a single register (X1)
remain64:
VEXTRACTF32X4 $1, Z1, X2 // X2: Second 128-bit lane
VEXTRACTF32X4 $2, Z1, X3 // X3: Third 128-bit lane
VEXTRACTF32X4 $3, Z1, X4 // X4: Fourth 128-bit lane
MOVOA r4r3<>+0(SB), X0
MOVOA X1, X5
PCLMULQDQ $0, X0, X1
PCLMULQDQ $0x11, X0, X5
PXOR X5, X1
PXOR X2, X1
MOVOA X1, X5
PCLMULQDQ $0, X0, X1
PCLMULQDQ $0x11, X0, X5
PXOR X5, X1
PXOR X3, X1
MOVOA X1, X5
PCLMULQDQ $0, X0, X1
PCLMULQDQ $0x11, X0, X5
PXOR X5, X1
PXOR X4, X1
// If there is less than 16 bytes left we are done
CMPQ CX, $16
JB finish
// Encode 16 bytes
remain16:
MOVOU (SI), X10
MOVOA X1, X5
PCLMULQDQ $0, X0, X1
PCLMULQDQ $0x11, X0, X5
PXOR X5, X1
PXOR X10, X1
SUBQ $16, CX
ADDQ $16, SI
CMPQ CX, $16
JGE remain16
finish:
// Fold final result into 32 bits and return it
PCMPEQB X3, X3
PCLMULQDQ $1, X1, X0
PSRLDQ $8, X1
PXOR X0, X1
MOVOA X1, X2
MOVQ r5<>+0(SB), X0
// Creates 32 bit mask. Note that we don't care about upper half.
PSRLQ $32, X3
PSRLDQ $4, X2
PAND X3, X1
PCLMULQDQ $0, X0, X1
PXOR X2, X1
MOVOA rupoly<>+0(SB), X0
MOVOA X1, X2
PAND X3, X1
PCLMULQDQ $0x10, X0, X1
PAND X3, X1
PCLMULQDQ $0, X0, X1
PXOR X2, X1
PEXTRD $1, X1, AX
MOVL AX, ret+32(FP)
VZEROUPPER
RET
// Castagonli Polynomial constants
DATA r2r1C<>+0(SB)/8, $0x0740eef02
DATA r2r1C<>+8(SB)/8, $0x09e4addf8
DATA r2r1C<>+16(SB)/8, $0x0740eef02
DATA r2r1C<>+24(SB)/8, $0x09e4addf8
DATA r2r1C<>+32(SB)/8, $0x0740eef02
DATA r2r1C<>+40(SB)/8, $0x09e4addf8
DATA r2r1C<>+48(SB)/8, $0x0740eef02
DATA r2r1C<>+56(SB)/8, $0x09e4addf8
GLOBL r2r1C<>(SB), RODATA, $64
DATA r4r3C<>+0(SB)/8, $0xf20c0dfe
DATA r4r3C<>+8(SB)/8, $0x14cd00bd6
DATA rupolyC<>+0(SB)/8, $0x105ec76f0
DATA rupolyC<>+8(SB)/8, $0xdea713f1
DATA r5C<>+0(SB)/8, $0xdd45aab8
GLOBL r4r3C<>(SB), RODATA, $16
GLOBL rupolyC<>(SB), RODATA, $16
GLOBL r5C<>(SB), RODATA, $8
// Based on https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/fast-crc-computation-generic-polynomials-pclmulqdq-paper.pdf
// len(p) must be at least 128, and must be a multiple of 16.
// func castagnoliCLMULAvx512(crc uint32, p []byte) uint32
TEXT ·castagnoliCLMULAvx512(SB), NOSPLIT, $0
MOVL crc+0(FP), AX // Initial CRC value
MOVQ p+8(FP), SI // data pointer
MOVQ p_len+16(FP), CX // len(p)
VPXORQ Z0, Z0, Z0
VMOVDQU64 (SI), Z1
VMOVQ AX, X0
VPXORQ Z0, Z1, Z1 // Merge initial CRC value into Z1
ADDQ $64, SI // buf+=64
SUBQ $64, CX // len-=64
VMOVDQU64 r2r1C<>+0(SB), Z0
loopback64:
// Load next early
VMOVDQU64 (SI), Z11
VPCLMULQDQ $0x11, Z0, Z1, Z5
VPCLMULQDQ $0, Z0, Z1, Z1
VPTERNLOGD $0x96, Z11, Z5, Z1 // Combine results with xor into Z1
ADDQ $0x40, DI
ADDQ $64, SI // buf+=64
SUBQ $64, CX // len-=64
CMPQ CX, $64 // Less than 64 bytes left?
JGE loopback64
// Fold result into a single register (X1)
remain64:
VEXTRACTF32X4 $1, Z1, X2 // X2: Second 128-bit lane
VEXTRACTF32X4 $2, Z1, X3 // X3: Third 128-bit lane
VEXTRACTF32X4 $3, Z1, X4 // X4: Fourth 128-bit lane
MOVOA r4r3C<>+0(SB), X0
MOVOA X1, X5
PCLMULQDQ $0, X0, X1
PCLMULQDQ $0x11, X0, X5
PXOR X5, X1
PXOR X2, X1
MOVOA X1, X5
PCLMULQDQ $0, X0, X1
PCLMULQDQ $0x11, X0, X5
PXOR X5, X1
PXOR X3, X1
MOVOA X1, X5
PCLMULQDQ $0, X0, X1
PCLMULQDQ $0x11, X0, X5
PXOR X5, X1
PXOR X4, X1
// If there is less than 16 bytes left we are done
CMPQ CX, $16
JB finish
// Encode 16 bytes
remain16:
MOVOU (SI), X10
MOVOA X1, X5
PCLMULQDQ $0, X0, X1
PCLMULQDQ $0x11, X0, X5
PXOR X5, X1
PXOR X10, X1
SUBQ $16, CX
ADDQ $16, SI
CMPQ CX, $16
JGE remain16
finish:
// Fold final result into 32 bits and return it
PCMPEQB X3, X3
PCLMULQDQ $1, X1, X0
PSRLDQ $8, X1
PXOR X0, X1
MOVOA X1, X2
MOVQ r5C<>+0(SB), X0
// Creates 32 bit mask. Note that we don't care about upper half.
PSRLQ $32, X3
PSRLDQ $4, X2
PAND X3, X1
PCLMULQDQ $0, X0, X1
PXOR X2, X1
MOVOA rupolyC<>+0(SB), X0
MOVOA X1, X2
PAND X3, X1
PCLMULQDQ $0x10, X0, X1
PAND X3, X1
PCLMULQDQ $0, X0, X1
PXOR X2, X1
PEXTRD $1, X1, AX
MOVL AX, ret+32(FP)
VZEROUPPER
RET
Server/vendor/github.com/klauspost/crc32/crc32_arm64.go
Generated Vendored
+50
View File
@@ -0,0 +1,50 @@
// Copyright 2017 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// ARM64-specific hardware-assisted CRC32 algorithms. See crc32.go for a
// description of the interface that each architecture-specific file
// implements.
package crc32
import "golang.org/x/sys/cpu"
func castagnoliUpdate(crc uint32, p []byte) uint32
func ieeeUpdate(crc uint32, p []byte) uint32
func archAvailableCastagnoli() bool {
return cpu.ARM64.HasCRC32
}
func archInitCastagnoli() {
if !cpu.ARM64.HasCRC32 {
panic("arch-specific crc32 instruction for Castagnoli not available")
}
}
func archUpdateCastagnoli(crc uint32, p []byte) uint32 {
if !cpu.ARM64.HasCRC32 {
panic("arch-specific crc32 instruction for Castagnoli not available")
}
return ^castagnoliUpdate(^crc, p)
}
func archAvailableIEEE() bool {
return cpu.ARM64.HasCRC32
}
func archInitIEEE() {
if !cpu.ARM64.HasCRC32 {
panic("arch-specific crc32 instruction for IEEE not available")
}
}
func archUpdateIEEE(crc uint32, p []byte) uint32 {
if !cpu.ARM64.HasCRC32 {
panic("arch-specific crc32 instruction for IEEE not available")
}
return ^ieeeUpdate(^crc, p)
}
Server/vendor/github.com/klauspost/crc32/crc32_arm64.s
Generated Vendored
+97
View File
@@ -0,0 +1,97 @@
// Copyright 2017 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
#include "textflag.h"
// castagnoliUpdate updates the non-inverted crc with the given data.
// func castagnoliUpdate(crc uint32, p []byte) uint32
TEXT ·castagnoliUpdate(SB), NOSPLIT, $0-36
MOVWU crc+0(FP), R9 // CRC value
MOVD p+8(FP), R13 // data pointer
MOVD p_len+16(FP), R11 // len(p)
update:
CMP $16, R11
BLT less_than_16
LDP.P 16(R13), (R8, R10)
CRC32CX R8, R9
CRC32CX R10, R9
SUB $16, R11
JMP update
less_than_16:
TBZ $3, R11, less_than_8
MOVD.P 8(R13), R10
CRC32CX R10, R9
less_than_8:
TBZ $2, R11, less_than_4
MOVWU.P 4(R13), R10
CRC32CW R10, R9
less_than_4:
TBZ $1, R11, less_than_2
MOVHU.P 2(R13), R10
CRC32CH R10, R9
less_than_2:
TBZ $0, R11, done
MOVBU (R13), R10
CRC32CB R10, R9
done:
MOVWU R9, ret+32(FP)
RET
// ieeeUpdate updates the non-inverted crc with the given data.
// func ieeeUpdate(crc uint32, p []byte) uint32
TEXT ·ieeeUpdate(SB), NOSPLIT, $0-36
MOVWU crc+0(FP), R9 // CRC value
MOVD p+8(FP), R13 // data pointer
MOVD p_len+16(FP), R11 // len(p)
update:
CMP $16, R11
BLT less_than_16
LDP.P 16(R13), (R8, R10)
CRC32X R8, R9
CRC32X R10, R9
SUB $16, R11
JMP update
less_than_16:
TBZ $3, R11, less_than_8
MOVD.P 8(R13), R10
CRC32X R10, R9
less_than_8:
TBZ $2, R11, less_than_4
MOVWU.P 4(R13), R10
CRC32W R10, R9
less_than_4:
TBZ $1, R11, less_than_2
MOVHU.P 2(R13), R10
CRC32H R10, R9
less_than_2:
TBZ $0, R11, done
MOVBU (R13), R10
CRC32B R10, R9
done:
MOVWU R9, ret+32(FP)
RET
Server/vendor/github.com/klauspost/crc32/crc32_generic.go
Generated Vendored
+91
View File
@@ -0,0 +1,91 @@
// Copyright 2011 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// This file contains CRC32 algorithms that are not specific to any architecture
// and don't use hardware acceleration.
//
// The simple (and slow) CRC32 implementation only uses a 256*4 bytes table.
//
// The slicing-by-8 algorithm is a faster implementation that uses a bigger
// table (8*256*4 bytes).
package crc32
import "encoding/binary"
// simpleMakeTable allocates and constructs a Table for the specified
// polynomial. The table is suitable for use with the simple algorithm
// (simpleUpdate).
func simpleMakeTable(poly uint32) *Table {
t := new(Table)
simplePopulateTable(poly, t)
return t
}
// simplePopulateTable constructs a Table for the specified polynomial, suitable
// for use with simpleUpdate.
func simplePopulateTable(poly uint32, t *Table) {
for i := 0; i < 256; i++ {
crc := uint32(i)
for j := 0; j < 8; j++ {
if crc&1 == 1 {
crc = (crc >> 1) ^ poly
} else {
crc >>= 1
}
}
t[i] = crc
}
}
// simpleUpdate uses the simple algorithm to update the CRC, given a table that
// was previously computed using simpleMakeTable.
func simpleUpdate(crc uint32, tab *Table, p []byte) uint32 {
crc = ^crc
for _, v := range p {
crc = tab[byte(crc)^v] ^ (crc >> 8)
}
return ^crc
}
// Use slicing-by-8 when payload >= this value.
const slicing8Cutoff = 16
// slicing8Table is array of 8 Tables, used by the slicing-by-8 algorithm.
type slicing8Table [8]Table
// slicingMakeTable constructs a slicing8Table for the specified polynomial. The
// table is suitable for use with the slicing-by-8 algorithm (slicingUpdate).
func slicingMakeTable(poly uint32) *slicing8Table {
t := new(slicing8Table)
simplePopulateTable(poly, &t[0])
for i := 0; i < 256; i++ {
crc := t[0][i]
for j := 1; j < 8; j++ {
crc = t[0][crc&0xFF] ^ (crc >> 8)
t[j][i] = crc
}
}
return t
}
// slicingUpdate uses the slicing-by-8 algorithm to update the CRC, given a
// table that was previously computed using slicingMakeTable.
func slicingUpdate(crc uint32, tab *slicing8Table, p []byte) uint32 {
if len(p) >= slicing8Cutoff {
crc = ^crc
for len(p) > 8 {
crc ^= binary.LittleEndian.Uint32(p)
crc = tab[0][p[7]] ^ tab[1][p[6]] ^ tab[2][p[5]] ^ tab[3][p[4]] ^
tab[4][crc>>24] ^ tab[5][(crc>>16)&0xFF] ^
tab[6][(crc>>8)&0xFF] ^ tab[7][crc&0xFF]
p = p[8:]
}
crc = ^crc
}
if len(p) == 0 {
return crc
}
return simpleUpdate(crc, &tab[0], p)
}
Server/vendor/github.com/klauspost/crc32/crc32_loong64.go
Generated Vendored
+50
View File
@@ -0,0 +1,50 @@
// Copyright 2024 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// LoongArch64-specific hardware-assisted CRC32 algorithms. See crc32.go for a
// description of the interface that each architecture-specific file
// implements.
package crc32
import "golang.org/x/sys/cpu"
func castagnoliUpdate(crc uint32, p []byte) uint32
func ieeeUpdate(crc uint32, p []byte) uint32
func archAvailableCastagnoli() bool {
return cpu.Loong64.HasCRC32
}
func archInitCastagnoli() {
if !cpu.Loong64.HasCRC32 {
panic("arch-specific crc32 instruction for Castagnoli not available")
}
}
func archUpdateCastagnoli(crc uint32, p []byte) uint32 {
if !cpu.Loong64.HasCRC32 {
panic("arch-specific crc32 instruction for Castagnoli not available")
}
return ^castagnoliUpdate(^crc, p)
}
func archAvailableIEEE() bool {
return cpu.Loong64.HasCRC32
}
func archInitIEEE() {
if !cpu.Loong64.HasCRC32 {
panic("arch-specific crc32 instruction for IEEE not available")
}
}
func archUpdateIEEE(crc uint32, p []byte) uint32 {
if !cpu.Loong64.HasCRC32 {
panic("arch-specific crc32 instruction for IEEE not available")
}
return ^ieeeUpdate(^crc, p)
}
Server/vendor/github.com/klauspost/crc32/crc32_loong64.s
Generated Vendored
+160
View File
@@ -0,0 +1,160 @@
// Copyright 2024 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
#include "textflag.h"
// castagnoliUpdate updates the non-inverted crc with the given data.
// func castagnoliUpdate(crc uint32, p []byte) uint32
TEXT ·castagnoliUpdate(SB), NOSPLIT, $0-36
MOVWU crc+0(FP), R4 // a0 = CRC value
MOVV p+8(FP), R5 // a1 = data pointer
MOVV p_len+16(FP), R6 // a2 = len(p)
SGT $8, R6, R12
BNE R12, less_than_8
AND $7, R5, R12
BEQ R12, aligned
// Process the first few bytes to 8-byte align the input.
// t0 = 8 - t0. We need to process this many bytes to align.
SUB $1, R12
XOR $7, R12
AND $1, R12, R13
BEQ R13, align_2
MOVB (R5), R13
CRCCWBW R4, R13, R4
ADDV $1, R5
ADDV $-1, R6
align_2:
AND $2, R12, R13
BEQ R13, align_4
MOVH (R5), R13
CRCCWHW R4, R13, R4
ADDV $2, R5
ADDV $-2, R6
align_4:
AND $4, R12, R13
BEQ R13, aligned
MOVW (R5), R13
CRCCWWW R4, R13, R4
ADDV $4, R5
ADDV $-4, R6
aligned:
// The input is now 8-byte aligned and we can process 8-byte chunks.
SGT $8, R6, R12
BNE R12, less_than_8
MOVV (R5), R13
CRCCWVW R4, R13, R4
ADDV $8, R5
ADDV $-8, R6
JMP aligned
less_than_8:
// We may have some bytes left over; process 4 bytes, then 2, then 1.
AND $4, R6, R12
BEQ R12, less_than_4
MOVW (R5), R13
CRCCWWW R4, R13, R4
ADDV $4, R5
ADDV $-4, R6
less_than_4:
AND $2, R6, R12
BEQ R12, less_than_2
MOVH (R5), R13
CRCCWHW R4, R13, R4
ADDV $2, R5
ADDV $-2, R6
less_than_2:
BEQ R6, done
MOVB (R5), R13
CRCCWBW R4, R13, R4
done:
MOVW R4, ret+32(FP)
RET
// ieeeUpdate updates the non-inverted crc with the given data.
// func ieeeUpdate(crc uint32, p []byte) uint32
TEXT ·ieeeUpdate(SB), NOSPLIT, $0-36
MOVWU crc+0(FP), R4 // a0 = CRC value
MOVV p+8(FP), R5 // a1 = data pointer
MOVV p_len+16(FP), R6 // a2 = len(p)
SGT $8, R6, R12
BNE R12, less_than_8
AND $7, R5, R12
BEQ R12, aligned
// Process the first few bytes to 8-byte align the input.
// t0 = 8 - t0. We need to process this many bytes to align.
SUB $1, R12
XOR $7, R12
AND $1, R12, R13
BEQ R13, align_2
MOVB (R5), R13
CRCWBW R4, R13, R4
ADDV $1, R5
ADDV $-1, R6
align_2:
AND $2, R12, R13
BEQ R13, align_4
MOVH (R5), R13
CRCWHW R4, R13, R4
ADDV $2, R5
ADDV $-2, R6
align_4:
AND $4, R12, R13
BEQ R13, aligned
MOVW (R5), R13
CRCWWW R4, R13, R4
ADDV $4, R5
ADDV $-4, R6
aligned:
// The input is now 8-byte aligned and we can process 8-byte chunks.
SGT $8, R6, R12
BNE R12, less_than_8
MOVV (R5), R13
CRCWVW R4, R13, R4
ADDV $8, R5
ADDV $-8, R6
JMP aligned
less_than_8:
// We may have some bytes left over; process 4 bytes, then 2, then 1.
AND $4, R6, R12
BEQ R12, less_than_4
MOVW (R5), R13
CRCWWW R4, R13, R4
ADDV $4, R5
ADDV $-4, R6
less_than_4:
AND $2, R6, R12
BEQ R12, less_than_2
MOVH (R5), R13
CRCWHW R4, R13, R4
ADDV $2, R5
ADDV $-2, R6
less_than_2:
BEQ R6, done
MOVB (R5), R13
CRCWBW R4, R13, R4
done:
MOVW R4, ret+32(FP)
RET
Server/vendor/github.com/klauspost/crc32/crc32_otherarch.go
Generated Vendored
+15
View File
@@ -0,0 +1,15 @@
// Copyright 2011 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
//go:build !amd64 && !s390x && !ppc64le && !arm64 && !loong64
package crc32
func archAvailableIEEE() bool { return false }
func archInitIEEE() { panic("not available") }
func archUpdateIEEE(crc uint32, p []byte) uint32 { panic("not available") }
func archAvailableCastagnoli() bool { return false }
func archInitCastagnoli() { panic("not available") }
func archUpdateCastagnoli(crc uint32, p []byte) uint32 { panic("not available") }
Server/vendor/github.com/klauspost/crc32/crc32_ppc64le.go
Generated Vendored
+88
View File
@@ -0,0 +1,88 @@
// Copyright 2017 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package crc32
import (
"unsafe"
)
const (
vecMinLen = 16
vecAlignMask = 15 // align to 16 bytes
crcIEEE = 1
crcCast = 2
)
//go:noescape
func ppc64SlicingUpdateBy8(crc uint32, table8 *slicing8Table, p []byte) uint32
// this function requires the buffer to be 16 byte aligned and > 16 bytes long.
//
//go:noescape
func vectorCrc32(crc uint32, poly uint32, p []byte) uint32
var archCastagnoliTable8 *slicing8Table
func archInitCastagnoli() {
archCastagnoliTable8 = slicingMakeTable(Castagnoli)
}
func archUpdateCastagnoli(crc uint32, p []byte) uint32 {
if len(p) >= 4*vecMinLen {
// If not aligned then process the initial unaligned bytes
if uint64(uintptr(unsafe.Pointer(&p[0])))&uint64(vecAlignMask) != 0 {
align := uint64(uintptr(unsafe.Pointer(&p[0]))) & uint64(vecAlignMask)
newlen := vecMinLen - align
crc = ppc64SlicingUpdateBy8(crc, archCastagnoliTable8, p[:newlen])
p = p[newlen:]
}
// p should be aligned now
aligned := len(p) & ^vecAlignMask
crc = vectorCrc32(crc, crcCast, p[:aligned])
p = p[aligned:]
}
if len(p) == 0 {
return crc
}
return ppc64SlicingUpdateBy8(crc, archCastagnoliTable8, p)
}
func archAvailableIEEE() bool {
return true
}
func archAvailableCastagnoli() bool {
return true
}
var archIeeeTable8 *slicing8Table
func archInitIEEE() {
// We still use slicing-by-8 for small buffers.
archIeeeTable8 = slicingMakeTable(IEEE)
}
// archUpdateIEEE calculates the checksum of p using vectorizedIEEE.
func archUpdateIEEE(crc uint32, p []byte) uint32 {
// Check if vector code should be used. If not aligned, then handle those
// first up to the aligned bytes.
if len(p) >= 4*vecMinLen {
if uint64(uintptr(unsafe.Pointer(&p[0])))&uint64(vecAlignMask) != 0 {
align := uint64(uintptr(unsafe.Pointer(&p[0]))) & uint64(vecAlignMask)
newlen := vecMinLen - align
crc = ppc64SlicingUpdateBy8(crc, archIeeeTable8, p[:newlen])
p = p[newlen:]
}
aligned := len(p) & ^vecAlignMask
crc = vectorCrc32(crc, crcIEEE, p[:aligned])
p = p[aligned:]
}
if len(p) == 0 {
return crc
}
return ppc64SlicingUpdateBy8(crc, archIeeeTable8, p)
}
Server/vendor/github.com/klauspost/crc32/crc32_ppc64le.s
Generated Vendored
+736
View File
@@ -0,0 +1,736 @@
// Copyright 2017 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// The vectorized implementation found below is a derived work
// from code written by Anton Blanchard <anton@au.ibm.com> found
// at https://github.com/antonblanchard/crc32-vpmsum. The original
// is dual licensed under GPL and Apache 2. As the copyright holder
// for the work, IBM has contributed this new work under
// the golang license.
// Changes include porting to Go assembler with modifications for
// the Go ABI for ppc64le.
#include "textflag.h"
#define POWER8_OFFSET 132
#define off16 R16
#define off32 R17
#define off48 R18
#define off64 R19
#define off80 R20
#define off96 R21
#define off112 R22
#define const1 V24
#define const2 V25
#define byteswap V26
#define mask_32bit V27
#define mask_64bit V28
#define zeroes V29
#define MAX_SIZE 32*1024
#define REFLECT
TEXT ·ppc64SlicingUpdateBy8(SB), NOSPLIT|NOFRAME, $0-44
MOVWZ crc+0(FP), R3 // incoming crc
MOVD table8+8(FP), R4 // *Table
MOVD p+16(FP), R5
MOVD p_len+24(FP), R6 // p len
CMP $0, R6 // len == 0?
BNE start
MOVW R3, ret+40(FP) // return crc
RET
start:
NOR R3, R3, R7 // ^crc
MOVWZ R7, R7 // 32 bits
CMP R6, $16
MOVD R6, CTR
BLT short
SRAD $3, R6, R8 // 8 byte chunks
MOVD R8, CTR
loop:
MOVWZ 0(R5), R8 // 0-3 bytes of p ?Endian?
MOVWZ 4(R5), R9 // 4-7 bytes of p
MOVD R4, R10 // &tab[0]
XOR R7, R8, R7 // crc ^= byte[0:3]
RLDICL $40, R9, $56, R17 // p[7]
SLD $2, R17, R17 // p[7]*4
RLDICL $40, R7, $56, R8 // crc>>24
SLD $2, R8, R8 // crc>>24*4
RLDICL $48, R9, $56, R18 // p[6]
SLD $2, R18, R18 // p[6]*4
MOVWZ (R10)(R17), R21 // tab[0][p[7]]
ADD $1024, R10, R10 // tab[1]
RLDICL $56, R9, $56, R19 // p[5]
SLD $2, R19, R19 // p[5]*4:1
MOVWZ (R10)(R18), R22 // tab[1][p[6]]
ADD $1024, R10, R10 // tab[2]
XOR R21, R22, R21 // xor done R22
CLRLSLDI $56, R9, $2, R20
MOVWZ (R10)(R19), R23 // tab[2][p[5]]
ADD $1024, R10, R10 // &tab[3]
XOR R21, R23, R21 // xor done R23
MOVWZ (R10)(R20), R24 // tab[3][p[4]]
ADD $1024, R10, R10 // &tab[4]
XOR R21, R24, R21 // xor done R24
MOVWZ (R10)(R8), R25 // tab[4][crc>>24]
RLDICL $48, R7, $56, R24 // crc>>16&0xFF
XOR R21, R25, R21 // xor done R25
ADD $1024, R10, R10 // &tab[5]
SLD $2, R24, R24 // crc>>16&0xFF*4
MOVWZ (R10)(R24), R26 // tab[5][crc>>16&0xFF]
XOR R21, R26, R21 // xor done R26
RLDICL $56, R7, $56, R25 // crc>>8
ADD $1024, R10, R10 // &tab[6]
SLD $2, R25, R25 // crc>>8&FF*2
MOVBZ R7, R26 // crc&0xFF
MOVWZ (R10)(R25), R27 // tab[6][crc>>8&0xFF]
ADD $1024, R10, R10 // &tab[7]
SLD $2, R26, R26 // crc&0xFF*2
XOR R21, R27, R21 // xor done R27
ADD $8, R5 // p = p[8:]
MOVWZ (R10)(R26), R28 // tab[7][crc&0xFF]
XOR R21, R28, R21 // xor done R28
MOVWZ R21, R7 // crc for next round
BDNZ loop
ANDCC $7, R6, R8 // any leftover bytes
BEQ done // none --> done
MOVD R8, CTR // byte count
PCALIGN $16 // align short loop
short:
MOVBZ 0(R5), R8 // get v
XOR R8, R7, R8 // byte(crc)^v -> R8
RLDIC $2, R8, $54, R8 // rldicl r8,r8,2,22
SRD $8, R7, R14 // crc>>8
MOVWZ (R4)(R8), R10
ADD $1, R5
XOR R10, R14, R7 // loop crc in R7
BDNZ short
done:
NOR R7, R7, R7 // ^crc
MOVW R7, ret+40(FP) // return crc
RET
#ifdef BYTESWAP_DATA
DATA ·byteswapcons+0(SB)/8, $0x0706050403020100
DATA ·byteswapcons+8(SB)/8, $0x0f0e0d0c0b0a0908
GLOBL ·byteswapcons+0(SB), RODATA, $16
#endif
TEXT ·vectorCrc32(SB), NOSPLIT|NOFRAME, $0-36
MOVWZ crc+0(FP), R3 // incoming crc
MOVWZ ctab+4(FP), R14 // crc poly id
MOVD p+8(FP), R4
MOVD p_len+16(FP), R5 // p len
// R3 = incoming crc
// R14 = constant table identifier
// R5 = address of bytes
// R6 = length of bytes
// defines for index loads
MOVD $16, off16
MOVD $32, off32
MOVD $48, off48
MOVD $64, off64
MOVD $80, off80
MOVD $96, off96
MOVD $112, off112
MOVD $0, R15
MOVD R3, R10 // save initial crc
NOR R3, R3, R3 // ^crc
MOVWZ R3, R3 // 32 bits
VXOR zeroes, zeroes, zeroes // clear the V reg
VSPLTISW $-1, V0
VSLDOI $4, V29, V0, mask_32bit
VSLDOI $8, V29, V0, mask_64bit
VXOR V8, V8, V8
MTVSRD R3, VS40 // crc initial value VS40 = V8
#ifdef REFLECT
VSLDOI $8, zeroes, V8, V8 // or: VSLDOI V29,V8,V27,4 for top 32 bits?
#else
VSLDOI $4, V8, zeroes, V8
#endif
#ifdef BYTESWAP_DATA
MOVD $·byteswapcons(SB), R3
LVX (R3), byteswap
#endif
CMPU R5, $256 // length of bytes
BLT short
RLDICR $0, R5, $56, R6 // chunk to process
// First step for larger sizes
l1:
MOVD $32768, R7
MOVD R7, R9
CMP R6, R7 // compare R6, R7 (MAX SIZE)
BGT top // less than MAX, just do remainder
MOVD R6, R7
top:
SUB R7, R6, R6
// mainloop does 128 bytes at a time
SRD $7, R7
// determine the offset into the constants table to start with.
// Each constant is 128 bytes, used against 16 bytes of data.
SLD $4, R7, R8
SRD $3, R9, R9
SUB R8, R9, R8
// The last iteration is reduced in a separate step
ADD $-1, R7
MOVD R7, CTR
// Determine which constant table (depends on poly)
CMP R14, $1
BNE castTable
MOVD $·IEEEConst(SB), R3
BR startConst
castTable:
MOVD $·CastConst(SB), R3
startConst:
ADD R3, R8, R3 // starting point in constants table
VXOR V0, V0, V0 // clear the V regs
VXOR V1, V1, V1
VXOR V2, V2, V2
VXOR V3, V3, V3
VXOR V4, V4, V4
VXOR V5, V5, V5
VXOR V6, V6, V6
VXOR V7, V7, V7
LVX (R3), const1 // loading constant values
CMP R15, $1 // Identify warm up pass
BEQ next
// First warm up pass: load the bytes to process
LVX (R4), V16
LVX (R4+off16), V17
LVX (R4+off32), V18
LVX (R4+off48), V19
LVX (R4+off64), V20
LVX (R4+off80), V21
LVX (R4+off96), V22
LVX (R4+off112), V23
ADD $128, R4 // bump up to next 128 bytes in buffer
VXOR V16, V8, V16 // xor in initial CRC in V8
next:
BC 18, 0, first_warm_up_done
ADD $16, R3 // bump up to next constants
LVX (R3), const2 // table values
VPMSUMD V16, const1, V8 // second warm up pass
LVX (R4), V16 // load from buffer
OR $0, R2, R2
VPMSUMD V17, const1, V9 // vpmsumd with constants
LVX (R4+off16), V17 // load next from buffer
OR $0, R2, R2
VPMSUMD V18, const1, V10 // vpmsumd with constants
LVX (R4+off32), V18 // load next from buffer
OR $0, R2, R2
VPMSUMD V19, const1, V11 // vpmsumd with constants
LVX (R4+off48), V19 // load next from buffer
OR $0, R2, R2
VPMSUMD V20, const1, V12 // vpmsumd with constants
LVX (R4+off64), V20 // load next from buffer
OR $0, R2, R2
VPMSUMD V21, const1, V13 // vpmsumd with constants
LVX (R4+off80), V21 // load next from buffer
OR $0, R2, R2
VPMSUMD V22, const1, V14 // vpmsumd with constants
LVX (R4+off96), V22 // load next from buffer
OR $0, R2, R2
VPMSUMD V23, const1, V15 // vpmsumd with constants
LVX (R4+off112), V23 // load next from buffer
ADD $128, R4 // bump up to next 128 bytes in buffer
BC 18, 0, first_cool_down
cool_top:
LVX (R3), const1 // constants
ADD $16, R3 // inc to next constants
OR $0, R2, R2
VXOR V0, V8, V0 // xor in previous vpmsumd
VPMSUMD V16, const2, V8 // vpmsumd with constants
LVX (R4), V16 // buffer
OR $0, R2, R2
VXOR V1, V9, V1 // xor in previous
VPMSUMD V17, const2, V9 // vpmsumd with constants
LVX (R4+off16), V17 // next in buffer
OR $0, R2, R2
VXOR V2, V10, V2 // xor in previous
VPMSUMD V18, const2, V10 // vpmsumd with constants
LVX (R4+off32), V18 // next in buffer
OR $0, R2, R2
VXOR V3, V11, V3 // xor in previous
VPMSUMD V19, const2, V11 // vpmsumd with constants
LVX (R4+off48), V19 // next in buffer
LVX (R3), const2 // get next constant
OR $0, R2, R2
VXOR V4, V12, V4 // xor in previous
VPMSUMD V20, const1, V12 // vpmsumd with constants
LVX (R4+off64), V20 // next in buffer
OR $0, R2, R2
VXOR V5, V13, V5 // xor in previous
VPMSUMD V21, const1, V13 // vpmsumd with constants
LVX (R4+off80), V21 // next in buffer
OR $0, R2, R2
VXOR V6, V14, V6 // xor in previous
VPMSUMD V22, const1, V14 // vpmsumd with constants
LVX (R4+off96), V22 // next in buffer
OR $0, R2, R2
VXOR V7, V15, V7 // xor in previous
VPMSUMD V23, const1, V15 // vpmsumd with constants
LVX (R4+off112), V23 // next in buffer
ADD $128, R4 // bump up buffer pointer
BDNZ cool_top // are we done?
first_cool_down:
// load the constants
// xor in the previous value
// vpmsumd the result with constants
LVX (R3), const1
ADD $16, R3
VXOR V0, V8, V0
VPMSUMD V16, const1, V8
OR $0, R2, R2
VXOR V1, V9, V1
VPMSUMD V17, const1, V9
OR $0, R2, R2
VXOR V2, V10, V2
VPMSUMD V18, const1, V10
OR $0, R2, R2
VXOR V3, V11, V3
VPMSUMD V19, const1, V11
OR $0, R2, R2
VXOR V4, V12, V4
VPMSUMD V20, const1, V12
OR $0, R2, R2
VXOR V5, V13, V5
VPMSUMD V21, const1, V13
OR $0, R2, R2
VXOR V6, V14, V6
VPMSUMD V22, const1, V14
OR $0, R2, R2
VXOR V7, V15, V7
VPMSUMD V23, const1, V15
OR $0, R2, R2
second_cool_down:
VXOR V0, V8, V0
VXOR V1, V9, V1
VXOR V2, V10, V2
VXOR V3, V11, V3
VXOR V4, V12, V4
VXOR V5, V13, V5
VXOR V6, V14, V6
VXOR V7, V15, V7
#ifdef REFLECT
VSLDOI $4, V0, zeroes, V0
VSLDOI $4, V1, zeroes, V1
VSLDOI $4, V2, zeroes, V2
VSLDOI $4, V3, zeroes, V3
VSLDOI $4, V4, zeroes, V4
VSLDOI $4, V5, zeroes, V5
VSLDOI $4, V6, zeroes, V6
VSLDOI $4, V7, zeroes, V7
#endif
LVX (R4), V8
LVX (R4+off16), V9
LVX (R4+off32), V10
LVX (R4+off48), V11
LVX (R4+off64), V12
LVX (R4+off80), V13
LVX (R4+off96), V14
LVX (R4+off112), V15
ADD $128, R4
VXOR V0, V8, V16
VXOR V1, V9, V17
VXOR V2, V10, V18
VXOR V3, V11, V19
VXOR V4, V12, V20
VXOR V5, V13, V21
VXOR V6, V14, V22
VXOR V7, V15, V23
MOVD $1, R15
CMP $0, R6
ADD $128, R6
BNE l1
ANDCC $127, R5
SUBC R5, $128, R6
ADD R3, R6, R3
SRD $4, R5, R7
MOVD R7, CTR
LVX (R3), V0
LVX (R3+off16), V1
LVX (R3+off32), V2
LVX (R3+off48), V3
LVX (R3+off64), V4
LVX (R3+off80), V5
LVX (R3+off96), V6
LVX (R3+off112), V7
ADD $128, R3
VPMSUMW V16, V0, V0
VPMSUMW V17, V1, V1
VPMSUMW V18, V2, V2
VPMSUMW V19, V3, V3
VPMSUMW V20, V4, V4
VPMSUMW V21, V5, V5
VPMSUMW V22, V6, V6
VPMSUMW V23, V7, V7
// now reduce the tail
CMP $0, R7
BEQ next1
LVX (R4), V16
LVX (R3), V17
VPMSUMW V16, V17, V16
VXOR V0, V16, V0
BC 18, 0, next1
LVX (R4+off16), V16
LVX (R3+off16), V17
VPMSUMW V16, V17, V16
VXOR V0, V16, V0
BC 18, 0, next1
LVX (R4+off32), V16
LVX (R3+off32), V17
VPMSUMW V16, V17, V16
VXOR V0, V16, V0
BC 18, 0, next1
LVX (R4+off48), V16
LVX (R3+off48), V17
VPMSUMW V16, V17, V16
VXOR V0, V16, V0
BC 18, 0, next1
LVX (R4+off64), V16
LVX (R3+off64), V17
VPMSUMW V16, V17, V16
VXOR V0, V16, V0
BC 18, 0, next1
LVX (R4+off80), V16
LVX (R3+off80), V17
VPMSUMW V16, V17, V16
VXOR V0, V16, V0
BC 18, 0, next1
LVX (R4+off96), V16
LVX (R3+off96), V17
VPMSUMW V16, V17, V16
VXOR V0, V16, V0
next1:
VXOR V0, V1, V0
VXOR V2, V3, V2
VXOR V4, V5, V4
VXOR V6, V7, V6
VXOR V0, V2, V0
VXOR V4, V6, V4
VXOR V0, V4, V0
barrett_reduction:
CMP R14, $1
BNE barcstTable
MOVD $·IEEEBarConst(SB), R3
BR startbarConst
barcstTable:
MOVD $·CastBarConst(SB), R3
startbarConst:
LVX (R3), const1
LVX (R3+off16), const2
VSLDOI $8, V0, V0, V1
VXOR V0, V1, V0
#ifdef REFLECT
VSPLTISB $1, V1
VSL V0, V1, V0
#endif
VAND V0, mask_64bit, V0
#ifndef REFLECT
VPMSUMD V0, const1, V1
VSLDOI $8, zeroes, V1, V1
VPMSUMD V1, const2, V1
VXOR V0, V1, V0
VSLDOI $8, V0, zeroes, V0
#else
VAND V0, mask_32bit, V1
VPMSUMD V1, const1, V1
VAND V1, mask_32bit, V1
VPMSUMD V1, const2, V1
VXOR V0, V1, V0
VSLDOI $4, V0, zeroes, V0
#endif
MFVSRD VS32, R3 // VS32 = V0
NOR R3, R3, R3 // return ^crc
MOVW R3, ret+32(FP)
RET
first_warm_up_done:
LVX (R3), const1
ADD $16, R3
VPMSUMD V16, const1, V8
VPMSUMD V17, const1, V9
VPMSUMD V18, const1, V10
VPMSUMD V19, const1, V11
VPMSUMD V20, const1, V12
VPMSUMD V21, const1, V13
VPMSUMD V22, const1, V14
VPMSUMD V23, const1, V15
BR second_cool_down
short:
CMP $0, R5
BEQ zero
// compute short constants
CMP R14, $1
BNE castshTable
MOVD $·IEEEConst(SB), R3
ADD $4080, R3
BR startshConst
castshTable:
MOVD $·CastConst(SB), R3
ADD $4080, R3
startshConst:
SUBC R5, $256, R6 // sub from 256
ADD R3, R6, R3
// calculate where to start
SRD $4, R5, R7
MOVD R7, CTR
VXOR V19, V19, V19
VXOR V20, V20, V20
LVX (R4), V0
LVX (R3), V16
VXOR V0, V8, V0
VPMSUMW V0, V16, V0
BC 18, 0, v0
LVX (R4+off16), V1
LVX (R3+off16), V17
VPMSUMW V1, V17, V1
BC 18, 0, v1
LVX (R4+off32), V2
LVX (R3+off32), V16
VPMSUMW V2, V16, V2
BC 18, 0, v2
LVX (R4+off48), V3
LVX (R3+off48), V17
VPMSUMW V3, V17, V3
BC 18, 0, v3
LVX (R4+off64), V4
LVX (R3+off64), V16
VPMSUMW V4, V16, V4
BC 18, 0, v4
LVX (R4+off80), V5
LVX (R3+off80), V17
VPMSUMW V5, V17, V5
BC 18, 0, v5
LVX (R4+off96), V6
LVX (R3+off96), V16
VPMSUMW V6, V16, V6
BC 18, 0, v6
LVX (R4+off112), V7
LVX (R3+off112), V17
VPMSUMW V7, V17, V7
BC 18, 0, v7
ADD $128, R3
ADD $128, R4
LVX (R4), V8
LVX (R3), V16
VPMSUMW V8, V16, V8
BC 18, 0, v8
LVX (R4+off16), V9
LVX (R3+off16), V17
VPMSUMW V9, V17, V9
BC 18, 0, v9
LVX (R4+off32), V10
LVX (R3+off32), V16
VPMSUMW V10, V16, V10
BC 18, 0, v10
LVX (R4+off48), V11
LVX (R3+off48), V17
VPMSUMW V11, V17, V11
BC 18, 0, v11
LVX (R4+off64), V12
LVX (R3+off64), V16
VPMSUMW V12, V16, V12
BC 18, 0, v12
LVX (R4+off80), V13
LVX (R3+off80), V17
VPMSUMW V13, V17, V13
BC 18, 0, v13
LVX (R4+off96), V14
LVX (R3+off96), V16
VPMSUMW V14, V16, V14
BC 18, 0, v14
LVX (R4+off112), V15
LVX (R3+off112), V17
VPMSUMW V15, V17, V15
VXOR V19, V15, V19
v14:
VXOR V20, V14, V20
v13:
VXOR V19, V13, V19
v12:
VXOR V20, V12, V20
v11:
VXOR V19, V11, V19
v10:
VXOR V20, V10, V20
v9:
VXOR V19, V9, V19
v8:
VXOR V20, V8, V20
v7:
VXOR V19, V7, V19
v6:
VXOR V20, V6, V20
v5:
VXOR V19, V5, V19
v4:
VXOR V20, V4, V20
v3:
VXOR V19, V3, V19
v2:
VXOR V20, V2, V20
v1:
VXOR V19, V1, V19
v0:
VXOR V20, V0, V20
VXOR V19, V20, V0
BR barrett_reduction
zero:
// This case is the original crc, so just return it
MOVW R10, ret+32(FP)
RET
Server/vendor/github.com/klauspost/crc32/crc32_s390x.go
Generated Vendored
+91
View File
@@ -0,0 +1,91 @@
// Copyright 2016 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package crc32
import "golang.org/x/sys/cpu"
const (
vxMinLen = 64
vxAlignMask = 15 // align to 16 bytes
)
// hasVX reports whether the machine has the z/Architecture
// vector facility installed and enabled.
var hasVX = cpu.S390X.HasVX
// vectorizedCastagnoli implements CRC32 using vector instructions.
// It is defined in crc32_s390x.s.
//
//go:noescape
func vectorizedCastagnoli(crc uint32, p []byte) uint32
// vectorizedIEEE implements CRC32 using vector instructions.
// It is defined in crc32_s390x.s.
//
//go:noescape
func vectorizedIEEE(crc uint32, p []byte) uint32
func archAvailableCastagnoli() bool {
return hasVX
}
var archCastagnoliTable8 *slicing8Table
func archInitCastagnoli() {
if !hasVX {
panic("not available")
}
// We still use slicing-by-8 for small buffers.
archCastagnoliTable8 = slicingMakeTable(Castagnoli)
}
// archUpdateCastagnoli calculates the checksum of p using
// vectorizedCastagnoli.
func archUpdateCastagnoli(crc uint32, p []byte) uint32 {
if !hasVX {
panic("not available")
}
// Use vectorized function if data length is above threshold.
if len(p) >= vxMinLen {
aligned := len(p) & ^vxAlignMask
crc = vectorizedCastagnoli(crc, p[:aligned])
p = p[aligned:]
}
if len(p) == 0 {
return crc
}
return slicingUpdate(crc, archCastagnoliTable8, p)
}
func archAvailableIEEE() bool {
return hasVX
}
var archIeeeTable8 *slicing8Table
func archInitIEEE() {
if !hasVX {
panic("not available")
}
// We still use slicing-by-8 for small buffers.
archIeeeTable8 = slicingMakeTable(IEEE)
}
// archUpdateIEEE calculates the checksum of p using vectorizedIEEE.
func archUpdateIEEE(crc uint32, p []byte) uint32 {
if !hasVX {
panic("not available")
}
// Use vectorized function if data length is above threshold.
if len(p) >= vxMinLen {
aligned := len(p) & ^vxAlignMask
crc = vectorizedIEEE(crc, p[:aligned])
p = p[aligned:]
}
if len(p) == 0 {
return crc
}
return slicingUpdate(crc, archIeeeTable8, p)
}
Server/vendor/github.com/klauspost/crc32/crc32_s390x.s
Generated Vendored
+225
View File
@@ -0,0 +1,225 @@
// Copyright 2016 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
#include "textflag.h"
// Vector register range containing CRC-32 constants
#define CONST_PERM_LE2BE V9
#define CONST_R2R1 V10
#define CONST_R4R3 V11
#define CONST_R5 V12
#define CONST_RU_POLY V13
#define CONST_CRC_POLY V14
// The CRC-32 constant block contains reduction constants to fold and
// process particular chunks of the input data stream in parallel.
//
// Note that the constant definitions below are extended in order to compute
// intermediate results with a single VECTOR GALOIS FIELD MULTIPLY instruction.
// The rightmost doubleword can be 0 to prevent contribution to the result or
// can be multiplied by 1 to perform an XOR without the need for a separate
// VECTOR EXCLUSIVE OR instruction.
//
// The polynomials used are bit-reflected:
//
// IEEE: P'(x) = 0x0edb88320
// Castagnoli: P'(x) = 0x082f63b78
// IEEE polynomial constants
DATA ·crclecons+0(SB)/8, $0x0F0E0D0C0B0A0908 // LE-to-BE mask
DATA ·crclecons+8(SB)/8, $0x0706050403020100
DATA ·crclecons+16(SB)/8, $0x00000001c6e41596 // R2
DATA ·crclecons+24(SB)/8, $0x0000000154442bd4 // R1
DATA ·crclecons+32(SB)/8, $0x00000000ccaa009e // R4
DATA ·crclecons+40(SB)/8, $0x00000001751997d0 // R3
DATA ·crclecons+48(SB)/8, $0x0000000000000000
DATA ·crclecons+56(SB)/8, $0x0000000163cd6124 // R5
DATA ·crclecons+64(SB)/8, $0x0000000000000000
DATA ·crclecons+72(SB)/8, $0x00000001F7011641 // u'
DATA ·crclecons+80(SB)/8, $0x0000000000000000
DATA ·crclecons+88(SB)/8, $0x00000001DB710641 // P'(x) << 1
GLOBL ·crclecons(SB), RODATA, $144
// Castagonli Polynomial constants
DATA ·crcclecons+0(SB)/8, $0x0F0E0D0C0B0A0908 // LE-to-BE mask
DATA ·crcclecons+8(SB)/8, $0x0706050403020100
DATA ·crcclecons+16(SB)/8, $0x000000009e4addf8 // R2
DATA ·crcclecons+24(SB)/8, $0x00000000740eef02 // R1
DATA ·crcclecons+32(SB)/8, $0x000000014cd00bd6 // R4
DATA ·crcclecons+40(SB)/8, $0x00000000f20c0dfe // R3
DATA ·crcclecons+48(SB)/8, $0x0000000000000000
DATA ·crcclecons+56(SB)/8, $0x00000000dd45aab8 // R5
DATA ·crcclecons+64(SB)/8, $0x0000000000000000
DATA ·crcclecons+72(SB)/8, $0x00000000dea713f1 // u'
DATA ·crcclecons+80(SB)/8, $0x0000000000000000
DATA ·crcclecons+88(SB)/8, $0x0000000105ec76f0 // P'(x) << 1
GLOBL ·crcclecons(SB), RODATA, $144
// The CRC-32 function(s) use these calling conventions:
//
// Parameters:
//
// R2: Initial CRC value, typically ~0; and final CRC (return) value.
// R3: Input buffer pointer, performance might be improved if the
// buffer is on a doubleword boundary.
// R4: Length of the buffer, must be 64 bytes or greater.
//
// Register usage:
//
// R5: CRC-32 constant pool base pointer.
// V0: Initial CRC value and intermediate constants and results.
// V1..V4: Data for CRC computation.
// V5..V8: Next data chunks that are fetched from the input buffer.
//
// V9..V14: CRC-32 constants.
// func vectorizedIEEE(crc uint32, p []byte) uint32
TEXT ·vectorizedIEEE(SB), NOSPLIT, $0
MOVWZ crc+0(FP), R2 // R2 stores the CRC value
MOVD p+8(FP), R3 // data pointer
MOVD p_len+16(FP), R4 // len(p)
MOVD $·crclecons(SB), R5
BR vectorizedBody<>(SB)
// func vectorizedCastagnoli(crc uint32, p []byte) uint32
TEXT ·vectorizedCastagnoli(SB), NOSPLIT, $0
MOVWZ crc+0(FP), R2 // R2 stores the CRC value
MOVD p+8(FP), R3 // data pointer
MOVD p_len+16(FP), R4 // len(p)
// R5: crc-32 constant pool base pointer, constant is used to reduce crc
MOVD $·crcclecons(SB), R5
BR vectorizedBody<>(SB)
TEXT vectorizedBody<>(SB), NOSPLIT, $0
XOR $0xffffffff, R2 // NOTW R2
VLM 0(R5), CONST_PERM_LE2BE, CONST_CRC_POLY
// Load the initial CRC value into the rightmost word of V0
VZERO V0
VLVGF $3, R2, V0
// Crash if the input size is less than 64-bytes.
CMP R4, $64
BLT crash
// Load a 64-byte data chunk and XOR with CRC
VLM 0(R3), V1, V4 // 64-bytes into V1..V4
// Reflect the data if the CRC operation is in the bit-reflected domain
VPERM V1, V1, CONST_PERM_LE2BE, V1
VPERM V2, V2, CONST_PERM_LE2BE, V2
VPERM V3, V3, CONST_PERM_LE2BE, V3
VPERM V4, V4, CONST_PERM_LE2BE, V4
VX V0, V1, V1 // V1 ^= CRC
ADD $64, R3 // BUF = BUF + 64
ADD $(-64), R4
// Check remaining buffer size and jump to proper folding method
CMP R4, $64
BLT less_than_64bytes
fold_64bytes_loop:
// Load the next 64-byte data chunk into V5 to V8
VLM 0(R3), V5, V8
VPERM V5, V5, CONST_PERM_LE2BE, V5
VPERM V6, V6, CONST_PERM_LE2BE, V6
VPERM V7, V7, CONST_PERM_LE2BE, V7
VPERM V8, V8, CONST_PERM_LE2BE, V8
// Perform a GF(2) multiplication of the doublewords in V1 with
// the reduction constants in V0. The intermediate result is
// then folded (accumulated) with the next data chunk in V5 and
// stored in V1. Repeat this step for the register contents
// in V2, V3, and V4 respectively.
VGFMAG CONST_R2R1, V1, V5, V1
VGFMAG CONST_R2R1, V2, V6, V2
VGFMAG CONST_R2R1, V3, V7, V3
VGFMAG CONST_R2R1, V4, V8, V4
// Adjust buffer pointer and length for next loop
ADD $64, R3 // BUF = BUF + 64
ADD $(-64), R4 // LEN = LEN - 64
CMP R4, $64
BGE fold_64bytes_loop
less_than_64bytes:
// Fold V1 to V4 into a single 128-bit value in V1
VGFMAG CONST_R4R3, V1, V2, V1
VGFMAG CONST_R4R3, V1, V3, V1
VGFMAG CONST_R4R3, V1, V4, V1
// Check whether to continue with 64-bit folding
CMP R4, $16
BLT final_fold
fold_16bytes_loop:
VL 0(R3), V2 // Load next data chunk
VPERM V2, V2, CONST_PERM_LE2BE, V2
VGFMAG CONST_R4R3, V1, V2, V1 // Fold next data chunk
// Adjust buffer pointer and size for folding next data chunk
ADD $16, R3
ADD $-16, R4
// Process remaining data chunks
CMP R4, $16
BGE fold_16bytes_loop
final_fold:
VLEIB $7, $0x40, V9
VSRLB V9, CONST_R4R3, V0
VLEIG $0, $1, V0
VGFMG V0, V1, V1
VLEIB $7, $0x20, V9 // Shift by words
VSRLB V9, V1, V2 // Store remaining bits in V2
VUPLLF V1, V1 // Split rightmost doubleword
VGFMAG CONST_R5, V1, V2, V1 // V1 = (V1 * R5) XOR V2
// The input values to the Barret reduction are the degree-63 polynomial
// in V1 (R(x)), degree-32 generator polynomial, and the reduction
// constant u. The Barret reduction result is the CRC value of R(x) mod
// P(x).
//
// The Barret reduction algorithm is defined as:
//
// 1. T1(x) = floor( R(x) / x^32 ) GF2MUL u
// 2. T2(x) = floor( T1(x) / x^32 ) GF2MUL P(x)
// 3. C(x) = R(x) XOR T2(x) mod x^32
//
// Note: To compensate the division by x^32, use the vector unpack
// instruction to move the leftmost word into the leftmost doubleword
// of the vector register. The rightmost doubleword is multiplied
// with zero to not contribute to the intermediate results.
// T1(x) = floor( R(x) / x^32 ) GF2MUL u
VUPLLF V1, V2
VGFMG CONST_RU_POLY, V2, V2
// Compute the GF(2) product of the CRC polynomial in VO with T1(x) in
// V2 and XOR the intermediate result, T2(x), with the value in V1.
// The final result is in the rightmost word of V2.
VUPLLF V2, V2
VGFMAG CONST_CRC_POLY, V2, V1, V2
done:
VLGVF $2, V2, R2
XOR $0xffffffff, R2 // NOTW R2
MOVWZ R2, ret + 32(FP)
RET
crash:
MOVD $0, (R0) // input size is less than 64-bytes
Server/vendor/github.com/klauspost/crc32/crc32_table_ppc64le.s
Generated Vendored
+3285
View File
File diff suppressed because it is too large Load Diff
Server/vendor/github.com/klauspost/crc32/gen.go
Generated Vendored
+7
View File
@@ -0,0 +1,7 @@
// Copyright 2023 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
//go:generate go run gen_const_ppc64le.go
package crc32