build(deps): bump github.com/olekukonko/tablewriter from 1.1.3 to 1.1.4
Bumps [github.com/olekukonko/tablewriter](https://github.com/olekukonko/tablewriter) from 1.1.3 to 1.1.4. - [Release notes](https://github.com/olekukonko/tablewriter/releases) - [Commits](https://github.com/olekukonko/tablewriter/compare/v1.1.3...v1.1.4) --- updated-dependencies: - dependency-name: github.com/olekukonko/tablewriter dependency-version: 1.1.4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
This commit is contained in:
dependabot[bot]
committed by
Ralf Haferkamp
Ralf Haferkamp
parent
8d30364a82
commit
57fdbb2d4c
@@ -1 +1,3 @@
|
||||
.DS_Store
|
||||
*.out
|
||||
*.test
|
||||
|
||||
+7
-5
@@ -18,13 +18,15 @@ by running `go generate` from the top package directory.
|
||||
|
||||
## Pull Requests and branches
|
||||
|
||||
For PRs (pull requests), you can use the gh CLI tool to retrieve details,
|
||||
or post comments. Then, compare the current branch with main. Reviewing a PR
|
||||
and reviewing a branch are about the same, but the PR may add context.
|
||||
For PRs (pull requests), you can use the gh CLI tool. Compare the current branch with main. Reviewing a PR and reviewing a branch are about the same, but the PR may add context.
|
||||
|
||||
Look for bugs. Think like GitHub Copilot or Cursor BugBot.
|
||||
Understand the goals of the PR. Note any API changes, especially breaking changes.
|
||||
|
||||
Offer to post a brief summary of the review to the PR, via the gh CLI tool.
|
||||
Look for thoroughness of tests, as well as GoDoc comments.
|
||||
|
||||
Retrieve and consider the comments on the PR, which may have come from GitHub Copilot or Cursor BugBot. Think like GitHub Copilot or Cursor BugBot.
|
||||
|
||||
Offer to optionally post a brief summary of the review to the PR, via the gh CLI tool.
|
||||
|
||||
## Comparisons to go-runewidth
|
||||
|
||||
|
||||
+46
-1
@@ -1,5 +1,50 @@
|
||||
# Changelog
|
||||
|
||||
## [0.10.0]
|
||||
|
||||
[Compare](https://github.com/clipperhouse/displaywidth/compare/v0.9.0...v0.10.0)
|
||||
|
||||
### Added
|
||||
- New `ControlSequences` option to treat ECMA-48/ANSI escape sequences as zero-width. (#20)
|
||||
- `TruncateString` and `TruncateBytes` now preserve trailing ANSI escape sequences (such as SGR resets) when `ControlSequences` is true, preventing color bleed in terminal output.
|
||||
|
||||
### Changed
|
||||
- Removed `stringish` dependency; generic type constraints are now inline `~string | []byte`.
|
||||
- Upgraded uax29 dependency to v2.6.0 for ANSI escape sequence support in the grapheme iterator.
|
||||
|
||||
## [0.9.0]
|
||||
|
||||
[Compare](https://github.com/clipperhouse/displaywidth/compare/v0.8.0...v0.9.0)
|
||||
|
||||
### Changed
|
||||
- Unicode 17 support: East Asian Width and emoji data updated to Unicode 17.0.0. (#18)
|
||||
- Upgraded uax29 dependency to v2.5.0 (Unicode 17 grapheme segmentation).
|
||||
|
||||
## [0.8.0]
|
||||
|
||||
[Compare](https://github.com/clipperhouse/displaywidth/compare/v0.7.0...v0.8.0)
|
||||
|
||||
### Changed
|
||||
- Performance: ASCII fast path that applies to any run of printable
|
||||
ASCII. 2x-10x faster for ASCII text vs v0.7.0. (#16)
|
||||
- Upgraded uax29 dependency to v2.4.0 for Unicode 16 support. Text that includes
|
||||
Indic_Conjunct_Break may segment differently (and more correctly). (#15)
|
||||
|
||||
## [0.7.0]
|
||||
|
||||
[Compare](https://github.com/clipperhouse/displaywidth/compare/v0.6.2...v0.7.0)
|
||||
|
||||
### Added
|
||||
- New `TruncateString` and `TruncateBytes` methods to truncate strings to a
|
||||
maximum display width, with optional tail (like an ellipsis). (#13)
|
||||
|
||||
## [0.6.2]
|
||||
|
||||
[Compare](https://github.com/clipperhouse/displaywidth/compare/v0.6.1...v0.6.2)
|
||||
|
||||
### Changed
|
||||
- Internal: reduced property categories for simpler trie.
|
||||
|
||||
## [0.6.1]
|
||||
|
||||
[Compare](https://github.com/clipperhouse/displaywidth/compare/v0.6.0...v0.6.1)
|
||||
@@ -19,7 +64,7 @@
|
||||
widths of grapheme clusters.
|
||||
|
||||
### Changed
|
||||
- Added ASCII fast paths
|
||||
- Fast ASCII lookups
|
||||
|
||||
## [0.5.0]
|
||||
|
||||
|
||||
+53
-48
@@ -61,8 +61,28 @@ func main() {
|
||||
|
||||
### Options
|
||||
|
||||
There is one option, `displaywidth.Options.EastAsianWidth`, which defines
|
||||
how [East Asian Ambiguous characters](https://www.unicode.org/reports/tr11/#Ambiguous)
|
||||
Create the options you need, and then use methods on the options struct.
|
||||
|
||||
```go
|
||||
var myOptions = displaywidth.Options{
|
||||
EastAsianWidth: true,
|
||||
ControlSequences: true,
|
||||
}
|
||||
|
||||
width := myOptions.String("Hello, 世界!")
|
||||
```
|
||||
|
||||
#### ControlSequences
|
||||
|
||||
`ControlSequences` specifies whether to ignore ECMA-48 escape sequences
|
||||
when calculating the display width. When `false` (default), ANSI escape
|
||||
sequences are treated as just a series of characters. When `true`, they are
|
||||
treated as a single zero-width unit.
|
||||
|
||||
#### EastAsianWidth
|
||||
|
||||
`EastAsianWidth` defines how
|
||||
[East Asian Ambiguous characters](https://www.unicode.org/reports/tr11/#Ambiguous)
|
||||
are treated.
|
||||
|
||||
When `false` (default), East Asian Ambiguous characters are treated as width 1.
|
||||
@@ -70,26 +90,8 @@ When `true`, they are treated as width 2.
|
||||
|
||||
You may wish to configure this based on environment variables or locale.
|
||||
`go-runewidth`, for example, does so
|
||||
[during package initialization](https://github.com/mattn/go-runewidth/blob/master/runewidth.go#L26C1-L45C2).
|
||||
[during package initialization](https://github.com/mattn/go-runewidth/blob/master/runewidth.go#L26C1-L45C2). `displaywidth` does not do this automatically, we prefer to leave it to you.
|
||||
|
||||
`displaywidth` does not do this automatically, we prefer to leave it to you.
|
||||
You might do something like:
|
||||
|
||||
```go
|
||||
var width displaywidth.Options // zero value is default
|
||||
|
||||
func init() {
|
||||
if os.Getenv("EAST_ASIAN_WIDTH") == "true" {
|
||||
width = displaywidth.Options{EastAsianWidth: true}
|
||||
}
|
||||
// or check locale, or any other logic you want
|
||||
}
|
||||
|
||||
// use it in your logic
|
||||
func myApp() {
|
||||
fmt.Println(width.String("Hello, 世界!"))
|
||||
}
|
||||
```
|
||||
|
||||
## Technical standards and compatibility
|
||||
|
||||
@@ -97,19 +99,16 @@ This package implements the Unicode East Asian Width standard
|
||||
([UAX #11](https://www.unicode.org/reports/tr11/tr11-43.html)), and handles
|
||||
[version selectors](https://en.wikipedia.org/wiki/Variation_Selectors_(Unicode_block)),
|
||||
and [regional indicator pairs](https://en.wikipedia.org/wiki/Regional_indicator_symbol)
|
||||
(flags). We implement [Unicode TR51](https://www.unicode.org/reports/tr51/tr51-27.html). We are keeping
|
||||
an eye on [emerging standards](https://www.jeffquast.com/post/state-of-terminal-emulation-2025/).
|
||||
(flags). We implement [Unicode TR51](https://www.unicode.org/reports/tr51/tr51-27.html)
|
||||
for emojis. We are keeping an eye on
|
||||
[emerging standards](https://www.jeffquast.com/post/state-of-terminal-emulation-2025/).
|
||||
|
||||
For control sequences, we implement the [ECMA-48](https://ecma-international.org/publications-and-standards/standards/ecma-48/) standard for 7-bit ASCII control sequences.
|
||||
|
||||
`clipperhouse/displaywidth`, `mattn/go-runewidth`, and `rivo/uniseg` will
|
||||
give the same outputs for most real-world text. Extensive details are in the
|
||||
[compatibility analysis](comparison/COMPATIBILITY_ANALYSIS.md).
|
||||
|
||||
If you wish to investigate the core logic, see the `lookupProperties` and `width`
|
||||
functions in [width.go](width.go#L139). The essential trie generation logic is in
|
||||
`buildPropertyBitmap` in [unicode.go](internal/gen/unicode.go#L316).
|
||||
|
||||
|
||||
## Prior Art
|
||||
|
||||
[mattn/go-runewidth](https://github.com/mattn/go-runewidth)
|
||||
@@ -133,33 +132,39 @@ goarch: arm64
|
||||
pkg: github.com/clipperhouse/displaywidth/comparison
|
||||
cpu: Apple M2
|
||||
|
||||
BenchmarkString_Mixed/clipperhouse/displaywidth-8 10326 ns/op 163.37 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkString_Mixed/mattn/go-runewidth-8 14415 ns/op 117.03 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkString_Mixed/rivo/uniseg-8 19343 ns/op 87.21 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkString_Mixed/clipperhouse/displaywidth-8 5784 ns/op 291.69 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkString_Mixed/mattn/go-runewidth-8 14751 ns/op 114.36 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkString_Mixed/rivo/uniseg-8 19360 ns/op 87.14 MB/s 0 B/op 0 allocs/op
|
||||
|
||||
BenchmarkString_EastAsian/clipperhouse/displaywidth-8 10561 ns/op 159.74 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkString_EastAsian/mattn/go-runewidth-8 23790 ns/op 70.91 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkString_EastAsian/rivo/uniseg-8 19322 ns/op 87.31 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkString_ASCII/clipperhouse/displaywidth-8 54.60 ns/op 2344.32 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkString_ASCII/mattn/go-runewidth-8 1195 ns/op 107.08 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkString_ASCII/rivo/uniseg-8 1578 ns/op 81.13 MB/s 0 B/op 0 allocs/op
|
||||
|
||||
BenchmarkString_ASCII/clipperhouse/displaywidth-8 1033 ns/op 123.88 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkString_ASCII/mattn/go-runewidth-8 1168 ns/op 109.59 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkString_ASCII/rivo/uniseg-8 1585 ns/op 80.74 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkString_EastAsian/clipperhouse/displaywidth-8 5837 ns/op 289.01 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkString_EastAsian/mattn/go-runewidth-8 24418 ns/op 69.09 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkString_EastAsian/rivo/uniseg-8 19339 ns/op 87.23 MB/s 0 B/op 0 allocs/op
|
||||
|
||||
BenchmarkString_Emoji/clipperhouse/displaywidth-8 3034 ns/op 238.61 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkString_Emoji/mattn/go-runewidth-8 4797 ns/op 150.94 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkString_Emoji/rivo/uniseg-8 6612 ns/op 109.50 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkString_Emoji/clipperhouse/displaywidth-8 3225 ns/op 224.51 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkString_Emoji/mattn/go-runewidth-8 4851 ns/op 149.25 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkString_Emoji/rivo/uniseg-8 6591 ns/op 109.85 MB/s 0 B/op 0 allocs/op
|
||||
|
||||
BenchmarkRune_Mixed/clipperhouse/displaywidth-8 3343 ns/op 504.67 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkRune_Mixed/mattn/go-runewidth-8 5414 ns/op 311.62 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkRune_Mixed/clipperhouse/displaywidth-8 3385 ns/op 498.34 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkRune_Mixed/mattn/go-runewidth-8 5354 ns/op 315.07 MB/s 0 B/op 0 allocs/op
|
||||
|
||||
BenchmarkRune_EastAsian/clipperhouse/displaywidth-8 3393 ns/op 497.17 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkRune_EastAsian/mattn/go-runewidth-8 15312 ns/op 110.17 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkRune_EastAsian/clipperhouse/displaywidth-8 3397 ns/op 496.56 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkRune_EastAsian/mattn/go-runewidth-8 15673 ns/op 107.64 MB/s 0 B/op 0 allocs/op
|
||||
|
||||
BenchmarkRune_ASCII/clipperhouse/displaywidth-8 256.9 ns/op 498.32 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkRune_ASCII/mattn/go-runewidth-8 265.7 ns/op 481.75 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkRune_ASCII/clipperhouse/displaywidth-8 255.7 ns/op 500.53 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkRune_ASCII/mattn/go-runewidth-8 261.5 ns/op 489.55 MB/s 0 B/op 0 allocs/op
|
||||
|
||||
BenchmarkRune_Emoji/clipperhouse/displaywidth-8 1336 ns/op 541.96 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkRune_Emoji/mattn/go-runewidth-8 2304 ns/op 314.23 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkRune_Emoji/clipperhouse/displaywidth-8 1371 ns/op 528.22 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkRune_Emoji/mattn/go-runewidth-8 2267 ns/op 319.43 MB/s 0 B/op 0 allocs/op
|
||||
|
||||
BenchmarkTruncateWithTail/clipperhouse/displaywidth-8 3229 ns/op 54.82 MB/s 192 B/op 14 allocs/op
|
||||
BenchmarkTruncateWithTail/mattn/go-runewidth-8 8408 ns/op 21.05 MB/s 192 B/op 14 allocs/op
|
||||
|
||||
BenchmarkTruncateWithoutTail/clipperhouse/displaywidth-8 3554 ns/op 64.43 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkTruncateWithoutTail/mattn/go-runewidth-8 11189 ns/op 20.47 MB/s 0 B/op 0 allocs/op
|
||||
```
|
||||
|
||||
Here are some notes on [how to make Unicode things fast](https://clipperhouse.com/go-unicode/).
|
||||
|
||||
+10
-11
@@ -1,7 +1,6 @@
|
||||
package displaywidth
|
||||
|
||||
import (
|
||||
"github.com/clipperhouse/stringish"
|
||||
"github.com/clipperhouse/uax29/v2/graphemes"
|
||||
)
|
||||
|
||||
@@ -9,8 +8,8 @@ import (
|
||||
//
|
||||
// Iterate using the Next method, and get the width of the current grapheme
|
||||
// using the Width method.
|
||||
type Graphemes[T stringish.Interface] struct {
|
||||
iter graphemes.Iterator[T]
|
||||
type Graphemes[T ~string | []byte] struct {
|
||||
iter *graphemes.Iterator[T]
|
||||
options Options
|
||||
}
|
||||
|
||||
@@ -44,10 +43,10 @@ func StringGraphemes(s string) Graphemes[string] {
|
||||
// Iterate using the Next method, and get the width of the current grapheme
|
||||
// using the Width method.
|
||||
func (options Options) StringGraphemes(s string) Graphemes[string] {
|
||||
return Graphemes[string]{
|
||||
iter: graphemes.FromString(s),
|
||||
options: options,
|
||||
}
|
||||
g := graphemes.FromString(s)
|
||||
g.AnsiEscapeSequences = options.ControlSequences
|
||||
|
||||
return Graphemes[string]{iter: g, options: options}
|
||||
}
|
||||
|
||||
// BytesGraphemes returns an iterator over grapheme clusters for the given
|
||||
@@ -65,8 +64,8 @@ func BytesGraphemes(s []byte) Graphemes[[]byte] {
|
||||
// Iterate using the Next method, and get the width of the current grapheme
|
||||
// using the Width method.
|
||||
func (options Options) BytesGraphemes(s []byte) Graphemes[[]byte] {
|
||||
return Graphemes[[]byte]{
|
||||
iter: graphemes.FromBytes(s),
|
||||
options: options,
|
||||
}
|
||||
g := graphemes.FromBytes(s)
|
||||
g.AnsiEscapeSequences = options.ControlSequences
|
||||
|
||||
return Graphemes[[]byte]{iter: g, options: options}
|
||||
}
|
||||
|
||||
+25
-22
@@ -2,8 +2,6 @@
|
||||
|
||||
package displaywidth
|
||||
|
||||
import "github.com/clipperhouse/stringish"
|
||||
|
||||
// property is an enum representing the properties of a character
|
||||
type property uint8
|
||||
|
||||
@@ -19,7 +17,7 @@ const (
|
||||
// lookup returns the trie value for the first UTF-8 encoding in s and
|
||||
// the width in bytes of this encoding. The size will be 0 if s does not
|
||||
// hold enough bytes to complete the encoding. len(s) must be greater than 0.
|
||||
func lookup[T stringish.Interface](s T) (v uint8, sz int) {
|
||||
func lookup[T ~string | []byte](s T) (v uint8, sz int) {
|
||||
c0 := s[0]
|
||||
switch {
|
||||
case c0 < 0x80: // is ASCII
|
||||
@@ -79,7 +77,7 @@ func lookup[T stringish.Interface](s T) (v uint8, sz int) {
|
||||
return 0, 1
|
||||
}
|
||||
|
||||
// stringWidthTrie. Total size: 17664 bytes (17.25 KiB). Checksum: c77d82ff2d69f0d2.
|
||||
// stringWidthTrie. Total size: 17664 bytes (17.25 KiB). Checksum: 220983462f26d765.
|
||||
// type stringWidthTrie struct { }
|
||||
|
||||
// func newStringWidthTrie(i int) *stringWidthTrie {
|
||||
@@ -1133,27 +1131,31 @@ var stringWidthValues = [15744]uint8{
|
||||
// Block 0xc2, offset 0x3080
|
||||
0x30a0: 0x0002, 0x30a1: 0x0002, 0x30a2: 0x0002, 0x30a3: 0x0002,
|
||||
0x30a4: 0x0001,
|
||||
0x30b0: 0x0002, 0x30b1: 0x0002,
|
||||
0x30b0: 0x0002, 0x30b1: 0x0002, 0x30b2: 0x0002, 0x30b3: 0x0002, 0x30b4: 0x0002, 0x30b5: 0x0002,
|
||||
0x30b6: 0x0002,
|
||||
// Block 0xc3, offset 0x30c0
|
||||
0x30c0: 0x0002, 0x30c1: 0x0002, 0x30c2: 0x0002, 0x30c3: 0x0002, 0x30c4: 0x0002, 0x30c5: 0x0002,
|
||||
0x30c6: 0x0002, 0x30c7: 0x0002, 0x30c8: 0x0002, 0x30c9: 0x0002, 0x30ca: 0x0002, 0x30cb: 0x0002,
|
||||
0x30cc: 0x0002, 0x30cd: 0x0002, 0x30ce: 0x0002, 0x30cf: 0x0002, 0x30d0: 0x0002, 0x30d1: 0x0002,
|
||||
0x30d2: 0x0002, 0x30d3: 0x0002, 0x30d4: 0x0002, 0x30d5: 0x0002, 0x30d6: 0x0002, 0x30d7: 0x0002,
|
||||
0x30d8: 0x0002, 0x30d9: 0x0002, 0x30da: 0x0002, 0x30db: 0x0002, 0x30dc: 0x0002, 0x30dd: 0x0002,
|
||||
0x30de: 0x0002, 0x30df: 0x0002, 0x30e0: 0x0002, 0x30e1: 0x0002, 0x30e2: 0x0002, 0x30e3: 0x0002,
|
||||
0x30e4: 0x0002, 0x30e5: 0x0002, 0x30e6: 0x0002, 0x30e7: 0x0002, 0x30e8: 0x0002, 0x30e9: 0x0002,
|
||||
0x30ea: 0x0002, 0x30eb: 0x0002, 0x30ec: 0x0002, 0x30ed: 0x0002, 0x30ee: 0x0002, 0x30ef: 0x0002,
|
||||
0x30f0: 0x0002, 0x30f1: 0x0002, 0x30f2: 0x0002, 0x30f3: 0x0002, 0x30f4: 0x0002, 0x30f5: 0x0002,
|
||||
0x30f6: 0x0002, 0x30f7: 0x0002,
|
||||
0x30d2: 0x0002, 0x30d3: 0x0002, 0x30d4: 0x0002, 0x30d5: 0x0002,
|
||||
0x30ff: 0x0002,
|
||||
// Block 0xc4, offset 0x3100
|
||||
0x3100: 0x0002, 0x3101: 0x0002, 0x3102: 0x0002, 0x3103: 0x0002, 0x3104: 0x0002, 0x3105: 0x0002,
|
||||
0x3106: 0x0002, 0x3107: 0x0002, 0x3108: 0x0002, 0x3109: 0x0002, 0x310a: 0x0002, 0x310b: 0x0002,
|
||||
0x310c: 0x0002, 0x310d: 0x0002, 0x310e: 0x0002, 0x310f: 0x0002, 0x3110: 0x0002, 0x3111: 0x0002,
|
||||
0x3112: 0x0002, 0x3113: 0x0002, 0x3114: 0x0002, 0x3115: 0x0002,
|
||||
0x313f: 0x0002,
|
||||
0x3112: 0x0002, 0x3113: 0x0002, 0x3114: 0x0002, 0x3115: 0x0002, 0x3116: 0x0002, 0x3117: 0x0002,
|
||||
0x3118: 0x0002, 0x3119: 0x0002, 0x311a: 0x0002, 0x311b: 0x0002, 0x311c: 0x0002, 0x311d: 0x0002,
|
||||
0x311e: 0x0002,
|
||||
// Block 0xc5, offset 0x3140
|
||||
0x3140: 0x0002, 0x3141: 0x0002, 0x3142: 0x0002, 0x3143: 0x0002, 0x3144: 0x0002, 0x3145: 0x0002,
|
||||
0x3146: 0x0002, 0x3147: 0x0002, 0x3148: 0x0002,
|
||||
0x3146: 0x0002, 0x3147: 0x0002, 0x3148: 0x0002, 0x3149: 0x0002, 0x314a: 0x0002, 0x314b: 0x0002,
|
||||
0x314c: 0x0002, 0x314d: 0x0002, 0x314e: 0x0002, 0x314f: 0x0002, 0x3150: 0x0002, 0x3151: 0x0002,
|
||||
0x3152: 0x0002, 0x3153: 0x0002, 0x3154: 0x0002, 0x3155: 0x0002, 0x3156: 0x0002, 0x3157: 0x0002,
|
||||
0x3158: 0x0002, 0x3159: 0x0002, 0x315a: 0x0002, 0x315b: 0x0002, 0x315c: 0x0002, 0x315d: 0x0002,
|
||||
0x315e: 0x0002, 0x315f: 0x0002, 0x3160: 0x0002, 0x3161: 0x0002, 0x3162: 0x0002, 0x3163: 0x0002,
|
||||
0x3164: 0x0002, 0x3165: 0x0002, 0x3166: 0x0002, 0x3167: 0x0002, 0x3168: 0x0002, 0x3169: 0x0002,
|
||||
0x316a: 0x0002, 0x316b: 0x0002, 0x316c: 0x0002, 0x316d: 0x0002, 0x316e: 0x0002, 0x316f: 0x0002,
|
||||
0x3170: 0x0002, 0x3171: 0x0002, 0x3172: 0x0002,
|
||||
// Block 0xc6, offset 0x3180
|
||||
0x31b0: 0x0002, 0x31b1: 0x0002, 0x31b2: 0x0002, 0x31b3: 0x0002, 0x31b5: 0x0002,
|
||||
0x31b6: 0x0002, 0x31b7: 0x0002, 0x31b8: 0x0002, 0x31b9: 0x0002, 0x31ba: 0x0002, 0x31bb: 0x0002,
|
||||
@@ -1443,7 +1445,7 @@ var stringWidthValues = [15744]uint8{
|
||||
0x3b40: 0x0002, 0x3b41: 0x0002, 0x3b42: 0x0002, 0x3b43: 0x0002, 0x3b44: 0x0002, 0x3b45: 0x0002,
|
||||
0x3b4c: 0x0002, 0x3b50: 0x0002, 0x3b51: 0x0002,
|
||||
0x3b52: 0x0002, 0x3b55: 0x0002, 0x3b56: 0x0002, 0x3b57: 0x0002,
|
||||
0x3b5c: 0x0002, 0x3b5d: 0x0002,
|
||||
0x3b58: 0x0002, 0x3b5c: 0x0002, 0x3b5d: 0x0002,
|
||||
0x3b5e: 0x0002, 0x3b5f: 0x0002,
|
||||
0x3b6b: 0x0002, 0x3b6c: 0x0002,
|
||||
0x3b74: 0x0002, 0x3b75: 0x0002,
|
||||
@@ -1482,8 +1484,8 @@ var stringWidthValues = [15744]uint8{
|
||||
0x3c7c: 0x0002,
|
||||
// Block 0xf2, offset 0x3c80
|
||||
0x3c80: 0x0002, 0x3c81: 0x0002, 0x3c82: 0x0002, 0x3c83: 0x0002, 0x3c84: 0x0002, 0x3c85: 0x0002,
|
||||
0x3c86: 0x0002, 0x3c87: 0x0002, 0x3c88: 0x0002, 0x3c89: 0x0002,
|
||||
0x3c8f: 0x0002, 0x3c90: 0x0002, 0x3c91: 0x0002,
|
||||
0x3c86: 0x0002, 0x3c87: 0x0002, 0x3c88: 0x0002, 0x3c89: 0x0002, 0x3c8a: 0x0002,
|
||||
0x3c8e: 0x0002, 0x3c8f: 0x0002, 0x3c90: 0x0002, 0x3c91: 0x0002,
|
||||
0x3c92: 0x0002, 0x3c93: 0x0002, 0x3c94: 0x0002, 0x3c95: 0x0002, 0x3c96: 0x0002, 0x3c97: 0x0002,
|
||||
0x3c98: 0x0002, 0x3c99: 0x0002, 0x3c9a: 0x0002, 0x3c9b: 0x0002, 0x3c9c: 0x0002, 0x3c9d: 0x0002,
|
||||
0x3c9e: 0x0002, 0x3c9f: 0x0002, 0x3ca0: 0x0002, 0x3ca1: 0x0002, 0x3ca2: 0x0002, 0x3ca3: 0x0002,
|
||||
@@ -1494,12 +1496,13 @@ var stringWidthValues = [15744]uint8{
|
||||
0x3cbc: 0x0002, 0x3cbd: 0x0002, 0x3cbe: 0x0002, 0x3cbf: 0x0002,
|
||||
// Block 0xf3, offset 0x3cc0
|
||||
0x3cc0: 0x0002, 0x3cc1: 0x0002, 0x3cc2: 0x0002, 0x3cc3: 0x0002, 0x3cc4: 0x0002, 0x3cc5: 0x0002,
|
||||
0x3cc6: 0x0002,
|
||||
0x3cce: 0x0002, 0x3ccf: 0x0002, 0x3cd0: 0x0002, 0x3cd1: 0x0002,
|
||||
0x3cc6: 0x0002, 0x3cc8: 0x0002,
|
||||
0x3ccd: 0x0002, 0x3cce: 0x0002, 0x3ccf: 0x0002, 0x3cd0: 0x0002, 0x3cd1: 0x0002,
|
||||
0x3cd2: 0x0002, 0x3cd3: 0x0002, 0x3cd4: 0x0002, 0x3cd5: 0x0002, 0x3cd6: 0x0002, 0x3cd7: 0x0002,
|
||||
0x3cd8: 0x0002, 0x3cd9: 0x0002, 0x3cda: 0x0002, 0x3cdb: 0x0002, 0x3cdc: 0x0002,
|
||||
0x3cdf: 0x0002, 0x3ce0: 0x0002, 0x3ce1: 0x0002, 0x3ce2: 0x0002, 0x3ce3: 0x0002,
|
||||
0x3ce4: 0x0002, 0x3ce5: 0x0002, 0x3ce6: 0x0002, 0x3ce7: 0x0002, 0x3ce8: 0x0002, 0x3ce9: 0x0002,
|
||||
0x3cea: 0x0002, 0x3cef: 0x0002,
|
||||
0x3cf0: 0x0002, 0x3cf1: 0x0002, 0x3cf2: 0x0002, 0x3cf3: 0x0002, 0x3cf4: 0x0002, 0x3cf5: 0x0002,
|
||||
0x3cf6: 0x0002, 0x3cf7: 0x0002, 0x3cf8: 0x0002,
|
||||
// Block 0xf4, offset 0x3d00
|
||||
@@ -1631,10 +1634,10 @@ var stringWidthIndex = [1920]uint8{
|
||||
0x440: 0x39, 0x441: 0x39, 0x442: 0x39, 0x443: 0x39, 0x444: 0x39, 0x445: 0x39, 0x446: 0x39, 0x447: 0x39,
|
||||
0x448: 0x39, 0x449: 0x39, 0x44a: 0x39, 0x44b: 0x39, 0x44c: 0x39, 0x44d: 0x39, 0x44e: 0x39, 0x44f: 0x39,
|
||||
0x450: 0x39, 0x451: 0x39, 0x452: 0x39, 0x453: 0x39, 0x454: 0x39, 0x455: 0x39, 0x456: 0x39, 0x457: 0x39,
|
||||
0x458: 0x39, 0x459: 0x39, 0x45a: 0x39, 0x45b: 0x39, 0x45c: 0x39, 0x45d: 0x39, 0x45e: 0x39, 0x45f: 0xc1,
|
||||
0x458: 0x39, 0x459: 0x39, 0x45a: 0x39, 0x45b: 0x39, 0x45c: 0x39, 0x45d: 0x39, 0x45e: 0x39, 0x45f: 0x39,
|
||||
0x460: 0x39, 0x461: 0x39, 0x462: 0x39, 0x463: 0x39, 0x464: 0x39, 0x465: 0x39, 0x466: 0x39, 0x467: 0x39,
|
||||
0x468: 0x39, 0x469: 0x39, 0x46a: 0x39, 0x46b: 0x39, 0x46c: 0x39, 0x46d: 0x39, 0x46e: 0x39, 0x46f: 0x39,
|
||||
0x470: 0x39, 0x471: 0x39, 0x472: 0x39, 0x473: 0xc2, 0x474: 0xc3,
|
||||
0x470: 0x39, 0x471: 0x39, 0x472: 0x39, 0x473: 0xc1, 0x474: 0xc2, 0x476: 0x39, 0x477: 0xc3,
|
||||
// Block 0x12, offset 0x480
|
||||
0x4bf: 0xc4,
|
||||
// Block 0x13, offset 0x4c0
|
||||
|
||||
+230
-30
@@ -1,23 +1,34 @@
|
||||
package displaywidth
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/clipperhouse/stringish"
|
||||
"github.com/clipperhouse/uax29/v2/graphemes"
|
||||
)
|
||||
|
||||
// Options allows you to specify the treatment of ambiguous East Asian
|
||||
// characters. When EastAsianWidth is false (default), ambiguous East Asian
|
||||
// characters are treated as width 1. When EastAsianWidth is true, ambiguous
|
||||
// East Asian characters are treated as width 2.
|
||||
// characters and ANSI escape sequences.
|
||||
type Options struct {
|
||||
// EastAsianWidth specifies whether to treat ambiguous East Asian characters
|
||||
// as width 1 or 2. When false (default), ambiguous East Asian characters
|
||||
// are treated as width 1. When true, they are width 2.
|
||||
EastAsianWidth bool
|
||||
|
||||
// ControlSequences specifies whether to ignore ECMA-48 escape sequences
|
||||
// when calculating the display width. When false (default), ANSI escape
|
||||
// sequences are treated as just a series of characters. When true, they are
|
||||
// treated as a single zero-width unit.
|
||||
//
|
||||
// Note that this option is about *sequences*. Individual control characters
|
||||
// are already treated as zero-width. With this option, ANSI sequences such as
|
||||
// "\x1b[31m" and "\x1b[0m" do not count towards the width of a string.
|
||||
ControlSequences bool
|
||||
}
|
||||
|
||||
// DefaultOptions is the default options for the display width
|
||||
// calculation, which is EastAsianWidth: false.
|
||||
var DefaultOptions = Options{EastAsianWidth: false}
|
||||
// calculation, which is EastAsianWidth false and ControlSequences false.
|
||||
var DefaultOptions = Options{EastAsianWidth: false, ControlSequences: false}
|
||||
|
||||
// String calculates the display width of a string,
|
||||
// by iterating over grapheme clusters in the string
|
||||
@@ -29,19 +40,43 @@ func String(s string) int {
|
||||
// String calculates the display width of a string, for the given options, by
|
||||
// iterating over grapheme clusters in the string and summing their widths.
|
||||
func (options Options) String(s string) int {
|
||||
// Optimization: no need to parse grapheme
|
||||
switch len(s) {
|
||||
case 0:
|
||||
return 0
|
||||
case 1:
|
||||
return asciiWidth(s[0])
|
||||
width := 0
|
||||
pos := 0
|
||||
|
||||
for pos < len(s) {
|
||||
// Try ASCII optimization
|
||||
asciiLen := printableASCIILength(s[pos:])
|
||||
if asciiLen > 0 {
|
||||
width += asciiLen
|
||||
pos += asciiLen
|
||||
continue
|
||||
}
|
||||
|
||||
// Not ASCII, use grapheme parsing
|
||||
g := graphemes.FromString(s[pos:])
|
||||
g.AnsiEscapeSequences = options.ControlSequences
|
||||
|
||||
start := pos
|
||||
|
||||
for g.Next() {
|
||||
v := g.Value()
|
||||
width += graphemeWidth(v, options)
|
||||
pos += len(v)
|
||||
|
||||
// Quick check: if remaining might have printable ASCII, break to outer loop
|
||||
if pos < len(s) && s[pos] >= 0x20 && s[pos] <= 0x7E {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
// Defensive, should not happen: if no progress was made,
|
||||
// skip a byte to prevent infinite loop. Only applies if
|
||||
// the grapheme parser misbehaves.
|
||||
if pos == start {
|
||||
pos++
|
||||
}
|
||||
}
|
||||
|
||||
width := 0
|
||||
g := graphemes.FromString(s)
|
||||
for g.Next() {
|
||||
width += graphemeWidth(g.Value(), options)
|
||||
}
|
||||
return width
|
||||
}
|
||||
|
||||
@@ -55,19 +90,43 @@ func Bytes(s []byte) int {
|
||||
// Bytes calculates the display width of a []byte, for the given options, by
|
||||
// iterating over grapheme clusters in the slice and summing their widths.
|
||||
func (options Options) Bytes(s []byte) int {
|
||||
// Optimization: no need to parse grapheme
|
||||
switch len(s) {
|
||||
case 0:
|
||||
return 0
|
||||
case 1:
|
||||
return asciiWidth(s[0])
|
||||
width := 0
|
||||
pos := 0
|
||||
|
||||
for pos < len(s) {
|
||||
// Try ASCII optimization
|
||||
asciiLen := printableASCIILength(s[pos:])
|
||||
if asciiLen > 0 {
|
||||
width += asciiLen
|
||||
pos += asciiLen
|
||||
continue
|
||||
}
|
||||
|
||||
// Not ASCII, use grapheme parsing
|
||||
g := graphemes.FromBytes(s[pos:])
|
||||
g.AnsiEscapeSequences = options.ControlSequences
|
||||
|
||||
start := pos
|
||||
|
||||
for g.Next() {
|
||||
v := g.Value()
|
||||
width += graphemeWidth(v, options)
|
||||
pos += len(v)
|
||||
|
||||
// Quick check: if remaining might have printable ASCII, break to outer loop
|
||||
if pos < len(s) && s[pos] >= 0x20 && s[pos] <= 0x7E {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
// Defensive, should not happen: if no progress was made,
|
||||
// skip a byte to prevent infinite loop. Only applies if
|
||||
// the grapheme parser misbehaves.
|
||||
if pos == start {
|
||||
pos++
|
||||
}
|
||||
}
|
||||
|
||||
width := 0
|
||||
g := graphemes.FromBytes(s)
|
||||
for g.Next() {
|
||||
width += graphemeWidth(g.Value(), options)
|
||||
}
|
||||
return width
|
||||
}
|
||||
|
||||
@@ -107,9 +166,123 @@ func (options Options) Rune(r rune) int {
|
||||
|
||||
const _Default property = 0
|
||||
|
||||
// TruncateString truncates a string to the given maxWidth, and appends the
|
||||
// given tail if the string is truncated.
|
||||
//
|
||||
// It ensures the visible width, including the width of the tail, is less than or
|
||||
// equal to maxWidth.
|
||||
//
|
||||
// When [Options.ControlSequences] is true, ANSI escape sequences that appear
|
||||
// after the truncation point are preserved in the output. This ensures that
|
||||
// escape sequences such as SGR resets are not lost, preventing color bleed
|
||||
// in terminal output.
|
||||
func (options Options) TruncateString(s string, maxWidth int, tail string) string {
|
||||
maxWidthWithoutTail := maxWidth - options.String(tail)
|
||||
|
||||
var pos, total int
|
||||
g := graphemes.FromString(s)
|
||||
g.AnsiEscapeSequences = options.ControlSequences
|
||||
|
||||
for g.Next() {
|
||||
gw := graphemeWidth(g.Value(), options)
|
||||
if total+gw <= maxWidthWithoutTail {
|
||||
pos = g.End()
|
||||
}
|
||||
total += gw
|
||||
if total > maxWidth {
|
||||
if options.ControlSequences {
|
||||
// Build result with trailing ANSI escape sequences preserved
|
||||
var b strings.Builder
|
||||
b.Grow(len(s) + len(tail)) // at most original + tail
|
||||
b.WriteString(s[:pos])
|
||||
b.WriteString(tail)
|
||||
rem := graphemes.FromString(s[pos:])
|
||||
rem.AnsiEscapeSequences = true
|
||||
for rem.Next() {
|
||||
v := rem.Value()
|
||||
if len(v) > 0 && v[0] == 0x1B {
|
||||
b.WriteString(v)
|
||||
}
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
return s[:pos] + tail
|
||||
}
|
||||
}
|
||||
// No truncation
|
||||
return s
|
||||
}
|
||||
|
||||
// TruncateString truncates a string to the given maxWidth, and appends the
|
||||
// given tail if the string is truncated.
|
||||
//
|
||||
// It ensures the total width, including the width of the tail, is less than or
|
||||
// equal to maxWidth.
|
||||
func TruncateString(s string, maxWidth int, tail string) string {
|
||||
return DefaultOptions.TruncateString(s, maxWidth, tail)
|
||||
}
|
||||
|
||||
// TruncateBytes truncates a []byte to the given maxWidth, and appends the
|
||||
// given tail if the []byte is truncated.
|
||||
//
|
||||
// It ensures the visible width, including the width of the tail, is less than or
|
||||
// equal to maxWidth.
|
||||
//
|
||||
// When [Options.ControlSequences] is true, ANSI escape sequences that appear
|
||||
// after the truncation point are preserved in the output. This ensures that
|
||||
// escape sequences such as SGR resets are not lost, preventing color bleed
|
||||
// in terminal output.
|
||||
func (options Options) TruncateBytes(s []byte, maxWidth int, tail []byte) []byte {
|
||||
maxWidthWithoutTail := maxWidth - options.Bytes(tail)
|
||||
|
||||
var pos, total int
|
||||
g := graphemes.FromBytes(s)
|
||||
g.AnsiEscapeSequences = options.ControlSequences
|
||||
|
||||
for g.Next() {
|
||||
gw := graphemeWidth(g.Value(), options)
|
||||
if total+gw <= maxWidthWithoutTail {
|
||||
pos = g.End()
|
||||
}
|
||||
total += gw
|
||||
if total > maxWidth {
|
||||
if options.ControlSequences {
|
||||
// Build result with trailing ANSI escape sequences preserved
|
||||
result := make([]byte, 0, len(s)+len(tail)) // at most original + tail
|
||||
result = append(result, s[:pos]...)
|
||||
result = append(result, tail...)
|
||||
rem := graphemes.FromBytes(s[pos:])
|
||||
rem.AnsiEscapeSequences = true
|
||||
for rem.Next() {
|
||||
v := rem.Value()
|
||||
if len(v) > 0 && v[0] == 0x1B {
|
||||
result = append(result, v...)
|
||||
}
|
||||
}
|
||||
return result
|
||||
}
|
||||
result := make([]byte, 0, pos+len(tail))
|
||||
result = append(result, s[:pos]...)
|
||||
result = append(result, tail...)
|
||||
return result
|
||||
}
|
||||
}
|
||||
// No truncation
|
||||
return s
|
||||
}
|
||||
|
||||
// TruncateBytes truncates a []byte to the given maxWidth, and appends the
|
||||
// given tail if the []byte is truncated.
|
||||
//
|
||||
// It ensures the total width, including the width of the tail, is less than or
|
||||
// equal to maxWidth.
|
||||
func TruncateBytes(s []byte, maxWidth int, tail []byte) []byte {
|
||||
return DefaultOptions.TruncateBytes(s, maxWidth, tail)
|
||||
}
|
||||
|
||||
// graphemeWidth returns the display width of a grapheme cluster.
|
||||
// The passed string must be a single grapheme cluster.
|
||||
func graphemeWidth[T stringish.Interface](s T, options Options) int {
|
||||
func graphemeWidth[T ~string | []byte](s T, options Options) int {
|
||||
// Optimization: no need to look up properties
|
||||
switch len(s) {
|
||||
case 0:
|
||||
@@ -118,6 +291,11 @@ func graphemeWidth[T stringish.Interface](s T, options Options) int {
|
||||
return asciiWidth(s[0])
|
||||
}
|
||||
|
||||
// Multi-byte grapheme clusters led by a C0 control (0x00-0x1F)
|
||||
if s[0] <= 0x1F {
|
||||
return 0
|
||||
}
|
||||
|
||||
p, sz := lookup(s)
|
||||
prop := property(p)
|
||||
|
||||
@@ -150,9 +328,31 @@ func asciiWidth(b byte) int {
|
||||
return 1
|
||||
}
|
||||
|
||||
// printableASCIILength returns the length of consecutive printable ASCII bytes
|
||||
// starting at the beginning of s.
|
||||
func printableASCIILength[T string | []byte](s T) int {
|
||||
i := 0
|
||||
for ; i < len(s); i++ {
|
||||
b := s[i]
|
||||
// Printable ASCII is 0x20-0x7E (space through tilde)
|
||||
if b < 0x20 || b > 0x7E {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
// If the next byte is non-ASCII (>= 0x80), back off by 1. The grapheme
|
||||
// parser may group the last ASCII byte with subsequent non-ASCII bytes,
|
||||
// such as combining marks.
|
||||
if i > 0 && i < len(s) && s[i] >= 0x80 {
|
||||
i--
|
||||
}
|
||||
|
||||
return i
|
||||
}
|
||||
|
||||
// isVS16 checks if the slice matches VS16 (U+FE0F) UTF-8 encoding
|
||||
// (EF B8 8F). It assumes len(s) >= 3.
|
||||
func isVS16[T stringish.Interface](s T) bool {
|
||||
func isVS16[T ~string | []byte](s T) bool {
|
||||
return s[0] == 0xEF && s[1] == 0xB8 && s[2] == 0x8F
|
||||
}
|
||||
|
||||
|
||||
@@ -1,2 +0,0 @@
|
||||
.DS_Store
|
||||
*.test
|
||||
-21
@@ -1,21 +0,0 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2025 Matt Sherman
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
-64
@@ -1,64 +0,0 @@
|
||||
# stringish
|
||||
|
||||
A small Go module that provides a generic type constraint for “string-like”
|
||||
data, and a utf8 package that works with both strings and byte slices
|
||||
without conversions.
|
||||
|
||||
```go
|
||||
type Interface interface {
|
||||
~[]byte | ~string
|
||||
}
|
||||
```
|
||||
|
||||
[](https://pkg.go.dev/github.com/clipperhouse/stringish/utf8)
|
||||
[](https://github.com/clipperhouse/stringish/actions/workflows/gotest.yml)
|
||||
|
||||
## Install
|
||||
|
||||
```
|
||||
go get github.com/clipperhouse/stringish
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
```go
|
||||
import (
|
||||
"github.com/clipperhouse/stringish"
|
||||
"github.com/clipperhouse/stringish/utf8"
|
||||
)
|
||||
|
||||
s := "Hello, 世界"
|
||||
r, size := utf8.DecodeRune(s) // not DecodeRuneInString 🎉
|
||||
|
||||
b := []byte("Hello, 世界")
|
||||
r, size = utf8.DecodeRune(b) // same API!
|
||||
|
||||
func MyFoo[T stringish.Interface](s T) T {
|
||||
// pass a string or a []byte
|
||||
// iterate, slice, transform, whatever
|
||||
}
|
||||
```
|
||||
|
||||
## Motivation
|
||||
|
||||
Sometimes we want APIs to accept `string` or `[]byte` without having to convert
|
||||
between those types. That conversion usually allocates!
|
||||
|
||||
By implementing with `stringish.Interface`, we can have a single API, and
|
||||
single implementation for both types: one `Foo` instead of `Foo` and
|
||||
`FooString`.
|
||||
|
||||
We have converted the
|
||||
[`unicode/utf8` package](https://github.com/clipperhouse/stringish/blob/main/utf8/utf8.go)
|
||||
as an example -- note the absence of`*InString` funcs. We might look at `x/text`
|
||||
next.
|
||||
|
||||
## Used by
|
||||
|
||||
- clipperhouse/uax29: [stringish trie](https://github.com/clipperhouse/uax29/blob/master/graphemes/trie.go#L27), [stringish iterator](https://github.com/clipperhouse/uax29/blob/master/internal/iterators/iterator.go#L9), [stringish SplitFunc](https://github.com/clipperhouse/uax29/blob/master/graphemes/splitfunc.go#L21)
|
||||
|
||||
- [clipperhouse/displaywidth](https://github.com/clipperhouse/displaywidth)
|
||||
|
||||
## Prior discussion
|
||||
|
||||
- [Consideration of similar by the Go team](https://github.com/golang/go/issues/48643)
|
||||
-5
@@ -1,5 +0,0 @@
|
||||
package stringish
|
||||
|
||||
type Interface interface {
|
||||
~[]byte | ~string
|
||||
}
|
||||
+8
-6
@@ -1,4 +1,4 @@
|
||||
An implementation of grapheme cluster boundaries from [Unicode text segmentation](https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries) (UAX 29), for Unicode version 15.0.0.
|
||||
An implementation of grapheme cluster boundaries from [Unicode text segmentation](https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries) (UAX 29), for Unicode 17.
|
||||
|
||||
[](https://pkg.go.dev/github.com/clipperhouse/uax29/v2/graphemes)
|
||||
|
||||
@@ -26,7 +26,7 @@ _A grapheme is a “single visible character”, which might be a simple as a si
|
||||
|
||||
## Conformance
|
||||
|
||||
We use the Unicode [test suite](https://unicode.org/reports/tr41/tr41-26.html#Tests29).
|
||||
We use the Unicode [test suite](https://unicode.org/reports/tr41/tr41-36.html#Tests29).
|
||||
|
||||
|
||||
|
||||
@@ -76,15 +76,17 @@ for tokens.Next() { // Next() returns true until end of data
|
||||
|
||||
### Benchmarks
|
||||
|
||||
On a Mac M2 laptop, we see around 200MB/s, or around 100 million graphemes per second, and no allocations.
|
||||
|
||||
```
|
||||
goos: darwin
|
||||
goarch: arm64
|
||||
pkg: github.com/clipperhouse/uax29/graphemes/comparative
|
||||
cpu: Apple M2
|
||||
BenchmarkGraphemes/clipperhouse/uax29-8 173805 ns/op 201.16 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkGraphemes/rivo/uniseg-8 2045128 ns/op 17.10 MB/s 0 B/op 0 allocs/op
|
||||
|
||||
BenchmarkGraphemesMixed/clipperhouse/uax29-8 142635 ns/op 245.12 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkGraphemesMixed/rivo/uniseg-8 2018284 ns/op 17.32 MB/s 0 B/op 0 allocs/op
|
||||
|
||||
BenchmarkGraphemesASCII/clipperhouse/uax29-8 8846 ns/op 508.73 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkGraphemesASCII/rivo/uniseg-8 366760 ns/op 12.27 MB/s 0 B/op 0 allocs/op
|
||||
```
|
||||
|
||||
### Invalid inputs
|
||||
|
||||
+119
@@ -0,0 +1,119 @@
|
||||
package graphemes
|
||||
|
||||
// ansiEscapeLength returns the byte length of a valid ANSI escape sequence at the
|
||||
// start of data, or 0 if none. Input is UTF-8; only 7-bit ESC sequences are
|
||||
// recognized (C1 0x80–0x9F can be UTF-8 continuation bytes).
|
||||
//
|
||||
// Recognized forms (ECMA-48 / ISO 6429):
|
||||
// - CSI: ESC [ then parameter bytes (0x30–0x3F), intermediate (0x20–0x2F), final (0x40–0x7E)
|
||||
// - OSC: ESC ] then payload until ST (ESC \) or BEL (0x07)
|
||||
// - DCS, SOS, PM, APC: ESC P / X / ^ / _ then payload until ST (ESC \)
|
||||
// - Two-byte: ESC + Fe (0x40–0x5F excluding above), or Fp (0x30–0x3F), or nF (0x20–0x2F then final)
|
||||
func ansiEscapeLength[T ~string | ~[]byte](data T) int {
|
||||
n := len(data)
|
||||
if n < 2 {
|
||||
return 0
|
||||
}
|
||||
if data[0] != esc {
|
||||
return 0
|
||||
}
|
||||
|
||||
b1 := data[1]
|
||||
switch b1 {
|
||||
case '[': // CSI
|
||||
body := csiLength(data[2:])
|
||||
if body == 0 {
|
||||
return 0
|
||||
}
|
||||
return 2 + body
|
||||
case ']': // OSC – allows BEL or ST as terminator
|
||||
body := oscLength(data[2:])
|
||||
if body == 0 {
|
||||
return 0
|
||||
}
|
||||
return 2 + body
|
||||
case 'P', 'X', '^', '_': // DCS, SOS, PM, APC – require ST (ESC \) only
|
||||
body := stSequenceLength(data[2:])
|
||||
if body == 0 {
|
||||
return 0
|
||||
}
|
||||
return 2 + body
|
||||
}
|
||||
if b1 >= 0x40 && b1 <= 0x5F {
|
||||
// Fe (C1) two-byte; [ ] P X ^ _ handled above
|
||||
return 2
|
||||
}
|
||||
if b1 >= 0x30 && b1 <= 0x3F {
|
||||
// Fp (private) two-byte
|
||||
return 2
|
||||
}
|
||||
if b1 >= 0x20 && b1 <= 0x2F {
|
||||
// nF: intermediates then one final (0x30–0x7E)
|
||||
i := 2
|
||||
for i < n && data[i] >= 0x20 && data[i] <= 0x2F {
|
||||
i++
|
||||
}
|
||||
if i < n && data[i] >= 0x30 && data[i] <= 0x7E {
|
||||
return i + 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// csiLength returns the length of the CSI body (param/intermediate/final bytes).
|
||||
// data is the slice after "ESC [".
|
||||
// Per ECMA-48, the CSI body has the form:
|
||||
//
|
||||
// parameters (0x30–0x3F)*, intermediates (0x20–0x2F)*, final (0x40–0x7E)
|
||||
//
|
||||
// Once an intermediate byte is seen, subsequent parameter bytes are invalid.
|
||||
func csiLength[T ~string | ~[]byte](data T) int {
|
||||
seenIntermediate := false
|
||||
for i := 0; i < len(data); i++ {
|
||||
b := data[i]
|
||||
if b >= 0x30 && b <= 0x3F {
|
||||
if seenIntermediate {
|
||||
return 0
|
||||
}
|
||||
continue
|
||||
}
|
||||
if b >= 0x20 && b <= 0x2F {
|
||||
seenIntermediate = true
|
||||
continue
|
||||
}
|
||||
if b >= 0x40 && b <= 0x7E {
|
||||
return i + 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// oscLength returns the length of the OSC body up to and including
|
||||
// the terminator. OSC accepts either BEL (0x07) or ST (ESC \) per
|
||||
// widespread terminal convention. data is the slice after "ESC ]".
|
||||
func oscLength[T ~string | ~[]byte](data T) int {
|
||||
for i := 0; i < len(data); i++ {
|
||||
b := data[i]
|
||||
if b == bel {
|
||||
return i + 1
|
||||
}
|
||||
if b == esc && i+1 < len(data) && data[i+1] == '\\' {
|
||||
return i + 2
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// stSequenceLength returns the length of a control-string body up to and
|
||||
// including the ST (ESC \) terminator. Used for DCS, SOS, PM, and APC, which
|
||||
// per ECMA-48 require ST and do not accept BEL. data is the slice after "ESC x".
|
||||
func stSequenceLength[T ~string | ~[]byte](data T) int {
|
||||
for i := 0; i < len(data); i++ {
|
||||
if data[i] == esc && i+1 < len(data) && data[i+1] == '\\' {
|
||||
return i + 2
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
+112
-16
@@ -1,12 +1,35 @@
|
||||
package graphemes
|
||||
|
||||
import (
|
||||
"github.com/clipperhouse/stringish"
|
||||
"github.com/clipperhouse/uax29/v2/internal/iterators"
|
||||
)
|
||||
import "unicode/utf8"
|
||||
|
||||
type Iterator[T stringish.Interface] struct {
|
||||
*iterators.Iterator[T]
|
||||
// FromString returns an iterator for the grapheme clusters in the input string.
|
||||
// Iterate while Next() is true, and access the grapheme via Value().
|
||||
func FromString(s string) *Iterator[string] {
|
||||
return &Iterator[string]{
|
||||
split: splitFuncString,
|
||||
data: s,
|
||||
}
|
||||
}
|
||||
|
||||
// FromBytes returns an iterator for the grapheme clusters in the input bytes.
|
||||
// Iterate while Next() is true, and access the grapheme via Value().
|
||||
func FromBytes(b []byte) *Iterator[[]byte] {
|
||||
return &Iterator[[]byte]{
|
||||
split: splitFuncBytes,
|
||||
data: b,
|
||||
}
|
||||
}
|
||||
|
||||
// Iterator is a generic iterator for grapheme clusters in strings or byte slices,
|
||||
// with an ASCII hot path optimization.
|
||||
type Iterator[T ~string | ~[]byte] struct {
|
||||
split func(T, bool) (int, T, error)
|
||||
data T
|
||||
pos int
|
||||
start int
|
||||
// AnsiEscapeSequences treats ANSI escape sequences (ECMA-48) as single grapheme
|
||||
// clusters when true. Default is false.
|
||||
AnsiEscapeSequences bool
|
||||
}
|
||||
|
||||
var (
|
||||
@@ -14,18 +37,91 @@ var (
|
||||
splitFuncBytes = splitFunc[[]byte]
|
||||
)
|
||||
|
||||
// FromString returns an iterator for the grapheme clusters in the input string.
|
||||
// Iterate while Next() is true, and access the grapheme via Value().
|
||||
func FromString(s string) Iterator[string] {
|
||||
return Iterator[string]{
|
||||
iterators.New(splitFuncString, s),
|
||||
const (
|
||||
esc = 0x1B
|
||||
cr = 0x0D
|
||||
bel = 0x07
|
||||
)
|
||||
|
||||
// Next advances the iterator to the next grapheme cluster.
|
||||
// Returns false when there are no more grapheme clusters.
|
||||
func (iter *Iterator[T]) Next() bool {
|
||||
if iter.pos >= len(iter.data) {
|
||||
return false
|
||||
}
|
||||
iter.start = iter.pos
|
||||
|
||||
if iter.AnsiEscapeSequences && iter.data[iter.pos] == esc {
|
||||
if a := ansiEscapeLength(iter.data[iter.pos:]); a > 0 {
|
||||
iter.pos += a
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
// ASCII hot path: any ASCII is one grapheme when next byte is ASCII or end.
|
||||
// Fall through on CR so splitfunc can handle CR+LF as a single cluster.
|
||||
b := iter.data[iter.pos]
|
||||
if b < utf8.RuneSelf && b != cr {
|
||||
if iter.pos+1 >= len(iter.data) || iter.data[iter.pos+1] < utf8.RuneSelf {
|
||||
iter.pos++
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
// Fall back to actual grapheme parsing
|
||||
remaining := iter.data[iter.pos:]
|
||||
advance, _, err := iter.split(remaining, true)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
if advance <= 0 {
|
||||
panic("splitFunc returned a zero or negative advance")
|
||||
}
|
||||
iter.pos += advance
|
||||
if iter.pos > len(iter.data) {
|
||||
panic("splitFunc advanced beyond end of data")
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// FromBytes returns an iterator for the grapheme clusters in the input bytes.
|
||||
// Iterate while Next() is true, and access the grapheme via Value().
|
||||
func FromBytes(b []byte) Iterator[[]byte] {
|
||||
return Iterator[[]byte]{
|
||||
iterators.New(splitFuncBytes, b),
|
||||
// Value returns the current grapheme cluster.
|
||||
func (iter *Iterator[T]) Value() T {
|
||||
return iter.data[iter.start:iter.pos]
|
||||
}
|
||||
|
||||
// Start returns the byte position of the current grapheme in the original data.
|
||||
func (iter *Iterator[T]) Start() int {
|
||||
return iter.start
|
||||
}
|
||||
|
||||
// End returns the byte position after the current grapheme in the original data.
|
||||
func (iter *Iterator[T]) End() int {
|
||||
return iter.pos
|
||||
}
|
||||
|
||||
// Reset resets the iterator to the beginning of the data.
|
||||
func (iter *Iterator[T]) Reset() {
|
||||
iter.start = 0
|
||||
iter.pos = 0
|
||||
}
|
||||
|
||||
// SetText sets the data for the iterator to operate on, and resets all state.
|
||||
func (iter *Iterator[T]) SetText(data T) {
|
||||
iter.data = data
|
||||
iter.start = 0
|
||||
iter.pos = 0
|
||||
}
|
||||
|
||||
// First returns the first grapheme cluster without advancing the iterator.
|
||||
func (iter *Iterator[T]) First() T {
|
||||
if len(iter.data) == 0 {
|
||||
return iter.data
|
||||
}
|
||||
|
||||
// Use a copy to leverage Next()'s ASCII optimization
|
||||
cp := *iter
|
||||
cp.pos = 0
|
||||
cp.start = 0
|
||||
cp.Next()
|
||||
return cp.Value()
|
||||
}
|
||||
|
||||
+39
-8
@@ -2,8 +2,6 @@ package graphemes
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
|
||||
"github.com/clipperhouse/stringish"
|
||||
)
|
||||
|
||||
// is determines if lookup intersects propert(ies)
|
||||
@@ -13,12 +11,22 @@ func (lookup property) is(properties property) bool {
|
||||
|
||||
const _Ignore = _Extend
|
||||
|
||||
// incbState tracks state for GB9c rule (Indic conjunct clusters)
|
||||
// Pattern: Consonant (Extend|Linker)* Linker (Extend|Linker)* × Consonant
|
||||
type incbState int
|
||||
|
||||
const (
|
||||
incbNone incbState = iota // initial/reset
|
||||
incbConsonant // seen Consonant, awaiting Linker
|
||||
incbLinker // seen Consonant and Linker (conjunct ready)
|
||||
)
|
||||
|
||||
// SplitFunc is a bufio.SplitFunc implementation of Unicode grapheme cluster segmentation, for use with bufio.Scanner.
|
||||
//
|
||||
// See https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries.
|
||||
var SplitFunc bufio.SplitFunc = splitFunc[[]byte]
|
||||
|
||||
func splitFunc[T stringish.Interface](data T, atEOF bool) (advance int, token T, err error) {
|
||||
func splitFunc[T ~string | ~[]byte](data T, atEOF bool) (advance int, token T, err error) {
|
||||
var empty T
|
||||
if len(data) == 0 {
|
||||
return 0, empty, nil
|
||||
@@ -30,6 +38,9 @@ func splitFunc[T stringish.Interface](data T, atEOF bool) (advance int, token T,
|
||||
var lastLastExIgnore property = 0 // "last one before that"
|
||||
var regionalIndicatorCount int
|
||||
|
||||
// GB9c state: tracking Indic conjunct clusters
|
||||
var incb incbState
|
||||
|
||||
// Rules are usually of the form Cat1 × Cat2; "current" refers to the first property
|
||||
// to the right of the ×, from which we look back or forward
|
||||
|
||||
@@ -76,6 +87,23 @@ func splitFunc[T stringish.Interface](data T, atEOF bool) (advance int, token T,
|
||||
lastExIgnore = last
|
||||
}
|
||||
|
||||
// Update GB9c state based on what we just advanced past
|
||||
if last.is(_InCBConsonant | _InCBLinker | _InCBExtend) {
|
||||
switch {
|
||||
case last.is(_InCBConsonant):
|
||||
if incb != incbLinker {
|
||||
incb = incbConsonant
|
||||
}
|
||||
case last.is(_InCBLinker):
|
||||
if incb >= incbConsonant {
|
||||
incb = incbLinker
|
||||
}
|
||||
// case last.is(_InCBExtend): stay in current state
|
||||
}
|
||||
} else {
|
||||
incb = incbNone
|
||||
}
|
||||
|
||||
current, w = lookup(data[pos:])
|
||||
if w == 0 {
|
||||
if atEOF {
|
||||
@@ -141,11 +169,14 @@ func splitFunc[T stringish.Interface](data T, atEOF bool) (advance int, token T,
|
||||
}
|
||||
|
||||
// https://unicode.org/reports/tr29/#GB9c
|
||||
// TODO(clipperhouse):
|
||||
// It appears to be added in Unicode 15.1.0:
|
||||
// https://unicode.org/versions/Unicode15.1.0/#Migration
|
||||
// This package currently supports Unicode 15.0.0, so
|
||||
// out of scope for now
|
||||
// Do not break within certain combinations with Indic_Conjunct_Break (InCB)=Linker.
|
||||
if incb == incbLinker && current.is(_InCBConsonant) {
|
||||
// After matching the pattern, reset state to start tracking a new pattern
|
||||
// The current Consonant becomes the start of the new pattern
|
||||
incb = incbConsonant
|
||||
pos += w
|
||||
continue
|
||||
}
|
||||
|
||||
// https://unicode.org/reports/tr29/#GB11
|
||||
if current.is(_ExtendedPictographic) && last.is(_ZWJ) && lastLastExIgnore.is(_ExtendedPictographic) {
|
||||
|
||||
+1323
-1015
File diff suppressed because it is too large
Load Diff
-100
@@ -1,100 +0,0 @@
|
||||
package iterators
|
||||
|
||||
import "github.com/clipperhouse/stringish"
|
||||
|
||||
type SplitFunc[T stringish.Interface] func(T, bool) (int, T, error)
|
||||
|
||||
// Iterator is a generic iterator for words that are either []byte or string.
|
||||
// Iterate while Next() is true, and access the word via Value().
|
||||
type Iterator[T stringish.Interface] struct {
|
||||
split SplitFunc[T]
|
||||
data T
|
||||
start int
|
||||
pos int
|
||||
}
|
||||
|
||||
// New creates a new Iterator for the given data and SplitFunc.
|
||||
func New[T stringish.Interface](split SplitFunc[T], data T) *Iterator[T] {
|
||||
return &Iterator[T]{
|
||||
split: split,
|
||||
data: data,
|
||||
}
|
||||
}
|
||||
|
||||
// SetText sets the text for the iterator to operate on, and resets all state.
|
||||
func (iter *Iterator[T]) SetText(data T) {
|
||||
iter.data = data
|
||||
iter.start = 0
|
||||
iter.pos = 0
|
||||
}
|
||||
|
||||
// Split sets the SplitFunc for the Iterator.
|
||||
func (iter *Iterator[T]) Split(split SplitFunc[T]) {
|
||||
iter.split = split
|
||||
}
|
||||
|
||||
// Next advances the iterator to the next token. It returns false when there
|
||||
// are no remaining tokens or an error occurred.
|
||||
func (iter *Iterator[T]) Next() bool {
|
||||
if iter.pos == len(iter.data) {
|
||||
return false
|
||||
}
|
||||
if iter.pos > len(iter.data) {
|
||||
panic("SplitFunc advanced beyond the end of the data")
|
||||
}
|
||||
|
||||
iter.start = iter.pos
|
||||
|
||||
advance, _, err := iter.split(iter.data[iter.pos:], true)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
if advance <= 0 {
|
||||
panic("SplitFunc returned a zero or negative advance")
|
||||
}
|
||||
|
||||
iter.pos += advance
|
||||
if iter.pos > len(iter.data) {
|
||||
panic("SplitFunc advanced beyond the end of the data")
|
||||
}
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
// Value returns the current token.
|
||||
func (iter *Iterator[T]) Value() T {
|
||||
return iter.data[iter.start:iter.pos]
|
||||
}
|
||||
|
||||
// Start returns the byte position of the current token in the original data.
|
||||
func (iter *Iterator[T]) Start() int {
|
||||
return iter.start
|
||||
}
|
||||
|
||||
// End returns the byte position after the current token in the original data.
|
||||
func (iter *Iterator[T]) End() int {
|
||||
return iter.pos
|
||||
}
|
||||
|
||||
// Reset resets the iterator to the beginning of the data.
|
||||
func (iter *Iterator[T]) Reset() {
|
||||
iter.start = 0
|
||||
iter.pos = 0
|
||||
}
|
||||
|
||||
func (iter *Iterator[T]) First() T {
|
||||
if len(iter.data) == 0 {
|
||||
return iter.data
|
||||
}
|
||||
advance, _, err := iter.split(iter.data, true)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
if advance <= 0 {
|
||||
panic("SplitFunc returned a zero or negative advance")
|
||||
}
|
||||
if advance > len(iter.data) {
|
||||
panic("SplitFunc advanced beyond the end of the data")
|
||||
}
|
||||
return iter.data[:advance]
|
||||
}
|
||||
Reference in New Issue
Block a user