build(deps): bump github.com/olekukonko/tablewriter from 1.1.0 to 1.1.1

Bumps [github.com/olekukonko/tablewriter](https://github.com/olekukonko/tablewriter) from 1.1.0 to 1.1.1. - [Commits](https://github.com/olekukonko/tablewriter/compare/v1.1.0...v1.1.1) --- updated-dependencies: - dependency-name: github.com/olekukonko/tablewriter dependency-version: 1.1.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
2025-11-20 11:50:20 +01:00
parent 9d173f0ea6
commit 28148d02bd
78 changed files with 7444 additions and 17015 deletions
@@ -0,0 +1 @@
+.DS_Store
@@ -0,0 +1,37 @@
+The goals and overview of this package can be found in the README.md file,
+start by reading that.
+
+The goal of this package is to determine the display (column) width of a
+string, UTF-8 bytes, or runes, as would happen in a monospace font, especially
+in a terminal.
+
+When troubleshooting, write Go unit tests instead of executing debug scripts.
+The tests can return whatever logs or output you need. If those tests are
+only for temporary troubleshooting, clean up the tests after the debugging is
+done.
+
+(Separate executable debugging scripts are messy, tend to have conflicting
+dependencies and are hard to cleanup.)
+
+If you make changes to the trie generation in internal/gen, it can be invoked
+by running `go generate` from the top package directory.
+
+## Pull Requests and branches
+
+For PRs (pull requests), you can use the gh CLI tool to retrieve details,
+or post comments. Then, compare the current branch with main. Reviewing a PR
+and reviewing a branch are about the same, but the PR may add context.
+
+Look for bugs. Think like GitHub Copilot or Cursor BugBot.
+
+Offer to post a brief summary of the review to the PR, via the gh CLI tool.
+
+## Comparisons to go-runewidth
+
+We originally attempted to make this package compatible with go-runewidth.
+However, we found that there were too many differences in the handling of
+certain characters and properties.
+
+We believe, preliminarily, that our choices are more correct and complete,
+by using more complete categories such as Unicode Cf (format) for zero-width
+and Mn (Nonspacing_Mark) for combining marks.
@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Matt Sherman
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
@@ -0,0 +1,115 @@
+# displaywidth
+
+A high-performance Go package for measuring the monospace display width of strings, UTF-8 bytes, and runes.
+
+[![Documentation](https://pkg.go.dev/badge/github.com/clipperhouse/displaywidth.svg)](https://pkg.go.dev/github.com/clipperhouse/displaywidth)
+[![Test](https://github.com/clipperhouse/displaywidth/actions/workflows/gotest.yml/badge.svg)](https://github.com/clipperhouse/displaywidth/actions/workflows/gotest.yml)
+[![Fuzz](https://github.com/clipperhouse/displaywidth/actions/workflows/gofuzz.yml/badge.svg)](https://github.com/clipperhouse/displaywidth/actions/workflows/gofuzz.yml)
+## Install
+```bash
+go get github.com/clipperhouse/displaywidth
+```
+
+## Usage
+
+```go
+package main
+
+import (
+    "fmt"
+    "github.com/clipperhouse/displaywidth"
+)
+
+func main() {
+    width := displaywidth.String("Hello, 世界!")
+    fmt.Println(width)
+
+    width = displaywidth.Bytes([]byte("🌍"))
+    fmt.Println(width)
+
+    width = displaywidth.Rune('🌍')
+    fmt.Println(width)
+}
+```
+
+### Options
+
+You can specify East Asian Width and Strict Emoji Neutral settings. If
+unspecified, the default is `EastAsianWidth: false, StrictEmojiNeutral: true`.
+
+```go
+options := displaywidth.Options{
+    EastAsianWidth:     true,
+    StrictEmojiNeutral: false,
+}
+
+width := options.String("Hello, 世界!")
+fmt.Println(width)
+```
+
+## Details
+
+This package implements the Unicode East Asian Width standard (UAX #11) and is
+intended to be compatible with `go-runewidth`. It operates on bytes without
+decoding runes for better performance.
+
+## Prior Art
+
+[mattn/go-runewidth](https://github.com/mattn/go-runewidth)
+
+[x/text/width](https://pkg.go.dev/golang.org/x/text/width)
+
+[x/text/internal/triegen](https://pkg.go.dev/golang.org/x/text/internal/triegen)
+
+## Benchmarks
+
+Part of my motivation is the insight that we can avoid decoding runes for better performance.
+
+```bash
+go test -bench=. -benchmem
+```
+
+```
+goos: darwin
+goarch: arm64
+pkg: github.com/clipperhouse/displaywidth
+cpu: Apple M2
+BenchmarkStringDefault/displaywidth-8      	     10537 ns/op	     160.10 MB/s	       0 B/op	       0 allocs/op
+BenchmarkStringDefault/go-runewidth-8      	     14162 ns/op	     119.12 MB/s	       0 B/op	       0 allocs/op
+BenchmarkString_EAW/displaywidth-8         	     10776 ns/op	     156.55 MB/s	       0 B/op	       0 allocs/op
+BenchmarkString_EAW/go-runewidth-8         	     23987 ns/op	      70.33 MB/s	       0 B/op	       0 allocs/op
+BenchmarkString_StrictEmoji/displaywidth-8 	     10892 ns/op	     154.88 MB/s	       0 B/op	       0 allocs/op
+BenchmarkString_StrictEmoji/go-runewidth-8 	     14552 ns/op	     115.93 MB/s	       0 B/op	       0 allocs/op
+BenchmarkString_ASCII/displaywidth-8       	      1116 ns/op	     114.72 MB/s	       0 B/op	       0 allocs/op
+BenchmarkString_ASCII/go-runewidth-8       	      1178 ns/op	     108.67 MB/s	       0 B/op	       0 allocs/op
+BenchmarkString_Unicode/displaywidth-8     	       896.9 ns/op	     148.29 MB/s	       0 B/op	       0 allocs/op
+BenchmarkString_Unicode/go-runewidth-8     	      1434 ns/op	      92.72 MB/s	       0 B/op	       0 allocs/op
+BenchmarkStringWidth_Emoji/displaywidth-8  	      3033 ns/op	     238.74 MB/s	       0 B/op	       0 allocs/op
+BenchmarkStringWidth_Emoji/go-runewidth-8  	      4841 ns/op	     149.56 MB/s	       0 B/op	       0 allocs/op
+BenchmarkString_Mixed/displaywidth-8       	      4064 ns/op	     124.74 MB/s	       0 B/op	       0 allocs/op
+BenchmarkString_Mixed/go-runewidth-8       	      4696 ns/op	     107.97 MB/s	       0 B/op	       0 allocs/op
+BenchmarkString_ControlChars/displaywidth-8	       320.6 ns/op	     102.93 MB/s	       0 B/op	       0 allocs/op
+BenchmarkString_ControlChars/go-runewidth-8	       373.8 ns/op	      88.28 MB/s	       0 B/op	       0 allocs/op
+BenchmarkRuneDefault/displaywidth-8        	       335.5 ns/op	     411.35 MB/s	       0 B/op	       0 allocs/op
+BenchmarkRuneDefault/go-runewidth-8        	       681.2 ns/op	     202.58 MB/s	       0 B/op	       0 allocs/op
+BenchmarkRuneWidth_EAW/displaywidth-8      	       146.7 ns/op	     374.80 MB/s	       0 B/op	       0 allocs/op
+BenchmarkRuneWidth_EAW/go-runewidth-8      	       495.6 ns/op	     110.98 MB/s	       0 B/op	       0 allocs/op
+BenchmarkRuneWidth_ASCII/displaywidth-8    	        63.00 ns/op	     460.33 MB/s	       0 B/op	       0 allocs/op
+BenchmarkRuneWidth_ASCII/go-runewidth-8    	        68.90 ns/op	     420.91 MB/s	       0 B/op	       0 allocs/op
+```
+
+I use a similar technique in [this grapheme cluster library](https://github.com/clipperhouse/uax29).
+
+## Compatibility
+
+`displaywidth` will mostly give the same outputs as `go-runewidth`, but there are some differences:
+
+- Unicode category Mn (Nonspacing Mark): `displaywidth` will return width 0, `go-runewidth` may return width 1 for some runes.
+- Unicode category Cf (Format): `displaywidth` will return width 0, `go-runewidth` may return width 1 for some runes.
+- Unicode category Mc (Spacing Mark): `displaywidth` will return width 1, `go-runewidth` may return width 0 for some runes.
+- Unicode category Cs (Surrogate): `displaywidth` will return width 0, `go-runewidth` may return width 1 for some runes. Surrogates are not valid UTF-8; some packages may turn them into the replacement character (U+FFFD).
+- Unicode category Zl (Line separator): `displaywidth` will return width 0, `go-runewidth` may return width 1.
+- Unicode category Zp (Paragraph separator): `displaywidth` will return width 0, `go-runewidth` may return width 1.
+- Unicode Noncharacters (U+FFFE and U+FFFF): `displaywidth` will return width 0, `go-runewidth` may return width 1.
+
+See `TestCompatibility` for more details.
@@ -0,0 +1,3 @@
+package displaywidth
+
+//go:generate go run -C internal/gen .
@@ -0,0 +1,159 @@
+package displaywidth
+
+import (
+	"unicode/utf8"
+
+	"github.com/clipperhouse/stringish"
+	"github.com/clipperhouse/uax29/v2/graphemes"
+)
+
+// String calculates the display width of a string
+// using the [DefaultOptions]
+func String(s string) int {
+	return DefaultOptions.String(s)
+}
+
+// Bytes calculates the display width of a []byte
+// using the [DefaultOptions]
+func Bytes(s []byte) int {
+	return DefaultOptions.Bytes(s)
+}
+
+func Rune(r rune) int {
+	return DefaultOptions.Rune(r)
+}
+
+type Options struct {
+	EastAsianWidth     bool
+	StrictEmojiNeutral bool
+}
+
+var DefaultOptions = Options{
+	EastAsianWidth:     false,
+	StrictEmojiNeutral: true,
+}
+
+// String calculates the display width of a string
+// for the given options
+func (options Options) String(s string) int {
+	if len(s) == 0 {
+		return 0
+	}
+
+	total := 0
+	g := graphemes.FromString(s)
+	for g.Next() {
+		// The first character in the grapheme cluster determines the width;
+		// modifiers and joiners do not contribute to the width.
+		props, _ := lookupProperties(g.Value())
+		total += props.width(options)
+	}
+	return total
+}
+
+// BytesOptions calculates the display width of a []byte
+// for the given options
+func (options Options) Bytes(s []byte) int {
+	if len(s) == 0 {
+		return 0
+	}
+
+	total := 0
+	g := graphemes.FromBytes(s)
+	for g.Next() {
+		// The first character in the grapheme cluster determines the width;
+		// modifiers and joiners do not contribute to the width.
+		props, _ := lookupProperties(g.Value())
+		total += props.width(options)
+	}
+	return total
+}
+
+func (options Options) Rune(r rune) int {
+	// Fast path for ASCII
+	if r < utf8.RuneSelf {
+		if isASCIIControl(byte(r)) {
+			// Control (0x00-0x1F) and DEL (0x7F)
+			return 0
+		}
+		// ASCII printable (0x20-0x7E)
+		return 1
+	}
+
+	// Surrogates (U+D800-U+DFFF) are invalid UTF-8 and have zero width
+	// Other packages might turn them into the replacement character (U+FFFD)
+	// in which case, we won't see it.
+	if r >= 0xD800 && r <= 0xDFFF {
+		return 0
+	}
+
+	// Stack-allocated to avoid heap allocation
+	var buf [4]byte // UTF-8 is at most 4 bytes
+	n := utf8.EncodeRune(buf[:], r)
+	// Skip the grapheme iterator and directly lookup properties
+	props, _ := lookupProperties(buf[:n])
+	return props.width(options)
+}
+
+func isASCIIControl(b byte) bool {
+	return b < 0x20 || b == 0x7F
+}
+
+const defaultWidth = 1
+
+// is returns true if the property flag is set
+func (p property) is(flag property) bool {
+	return p&flag != 0
+}
+
+// lookupProperties returns the properties for the first character in a string
+func lookupProperties[T stringish.Interface](s T) (property, int) {
+	if len(s) == 0 {
+		return 0, 0
+	}
+
+	// Fast path for ASCII characters (single byte)
+	b := s[0]
+	if b < utf8.RuneSelf { // Single-byte ASCII
+		if isASCIIControl(b) {
+			// Control characters (0x00-0x1F) and DEL (0x7F) - width 0
+			return _ZeroWidth, 1
+		}
+		// ASCII printable characters (0x20-0x7E) - width 1
+		// Return 0 properties, width calculation will default to 1
+		return 0, 1
+	}
+
+	// Use the generated trie for lookup
+	props, size := lookup(s)
+	return property(props), size
+}
+
+// width determines the display width of a character based on its properties
+// and configuration options
+func (p property) width(options Options) int {
+	if p == 0 {
+		// Character not in trie, use default behavior
+		return defaultWidth
+	}
+
+	if p.is(_ZeroWidth) {
+		return 0
+	}
+
+	if options.EastAsianWidth {
+		if p.is(_East_Asian_Ambiguous) {
+			return 2
+		}
+		if p.is(_East_Asian_Ambiguous|_Emoji) && !options.StrictEmojiNeutral {
+			return 2
+		}
+	}
+
+	if p.is(_East_Asian_Full_Wide) {
+		return 2
+	}
+
+	// Default width for all other characters
+	return defaultWidth
+}
@@ -0,0 +1,2 @@
+.DS_Store
+*.test
@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Matt Sherman
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
@@ -0,0 +1,64 @@
+# stringish
+
+A small Go module that provides a generic type constraint for “string-like”
+data, and a utf8 package that works with both strings and byte slices
+without conversions.
+
+```go
+type Interface interface {
+	~[]byte | ~string
+}
+```
+
+[![Go Reference](https://pkg.go.dev/badge/github.com/clipperhouse/stringish/utf8.svg)](https://pkg.go.dev/github.com/clipperhouse/stringish/utf8)
+[![Test Status](https://github.com/clipperhouse/stringish/actions/workflows/gotest.yml/badge.svg)](https://github.com/clipperhouse/stringish/actions/workflows/gotest.yml)
+
+## Install
+
+```
+go get github.com/clipperhouse/stringish
+```
+
+## Examples
+
+```go
+import (
+    "github.com/clipperhouse/stringish"
+    "github.com/clipperhouse/stringish/utf8"
+)
+
+s := "Hello, 世界"
+r, size := utf8.DecodeRune(s)   // not DecodeRuneInString 🎉
+
+b := []byte("Hello, 世界")
+r, size = utf8.DecodeRune(b)    // same API!
+
+func MyFoo[T stringish.Interface](s T) T {
+    // pass a string or a []byte
+    // iterate, slice, transform, whatever
+}
+```
+
+## Motivation
+
+Sometimes we want APIs to accept `string` or `[]byte` without having to convert
+between those types. That conversion usually allocates!
+
+By implementing with `stringish.Interface`, we can have a single API, and
+single implementation for both types: one `Foo` instead of `Foo` and
+`FooString`.
+
+We have converted the
+[`unicode/utf8` package](https://github.com/clipperhouse/stringish/blob/main/utf8/utf8.go)
+as an example -- note the absence of`*InString` funcs. We might look at `x/text`
+next.
+
+## Used by
+
+- clipperhouse/uax29: [stringish trie](https://github.com/clipperhouse/uax29/blob/master/graphemes/trie.go#L27), [stringish iterator](https://github.com/clipperhouse/uax29/blob/master/internal/iterators/iterator.go#L9), [stringish SplitFunc](https://github.com/clipperhouse/uax29/blob/master/graphemes/splitfunc.go#L21)
+
+- [clipperhouse/displaywidth](https://github.com/clipperhouse/displaywidth)
+
+## Prior discussion
+
+- [Consideration of similar by the Go team](https://github.com/golang/go/issues/48643)
@@ -0,0 +1,5 @@
+package stringish
+
+type Interface interface {
+	~[]byte | ~string
+}
@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2020 Matt Sherman
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
@@ -0,0 +1,82 @@
+An implementation of grapheme cluster boundaries from [Unicode text segmentation](https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries) (UAX 29), for Unicode version 15.0.0.
+
+## Quick start
+
+```
+go get "github.com/clipperhouse/uax29/v2/graphemes"
+```
+
+```go
+import "github.com/clipperhouse/uax29/v2/graphemes"
+
+text := "Hello, 世界. Nice dog! 👍🐶"
+
+tokens := graphemes.FromString(text)
+
+for tokens.Next() {                     // Next() returns true until end of data
+	fmt.Println(tokens.Value())         // Do something with the current grapheme
+}
+```
+
+[![Documentation](https://pkg.go.dev/badge/github.com/clipperhouse/uax29/v2/graphemes.svg)](https://pkg.go.dev/github.com/clipperhouse/uax29/v2/graphemes)
+
+_A grapheme is a “single visible character”, which might be a simple as a single letter, or a complex emoji that consists of several Unicode code points._
+
+## Conformance
+
+We use the Unicode [test suite](https://unicode.org/reports/tr41/tr41-26.html#Tests29). Status:
+
+![Go](https://github.com/clipperhouse/uax29/actions/workflows/gotest.yml/badge.svg)
+
+## APIs
+
+### If you have a `string`
+
+```go
+text := "Hello, 世界. Nice dog! 👍🐶"
+
+tokens := graphemes.FromString(text)
+
+for tokens.Next() {                     // Next() returns true until end of data
+	fmt.Println(tokens.Value())         // Do something with the current grapheme
+}
+```
+
+### If you have an `io.Reader`
+
+`FromReader` embeds a [`bufio.Scanner`](https://pkg.go.dev/bufio#Scanner), so just use those methods.
+
+```go
+r := getYourReader()                        // from a file or network maybe
+tokens := graphemes.FromReader(r)
+
+for tokens.Scan() {                         // Scan() returns true until error or EOF
+	fmt.Println(tokens.Text())              // Do something with the current grapheme
+}
+
+if tokens.Err() != nil {                    // Check the error
+	log.Fatal(tokens.Err())
+}
+```
+
+### If you have a `[]byte`
+
+```go
+b := []byte("Hello, 世界. Nice dog! 👍🐶")
+
+tokens := graphemes.FromBytes(b)
+
+for tokens.Next() {                     // Next() returns true until end of data
+	fmt.Println(tokens.Value())         // Do something with the current grapheme
+}
+```
+
+### Performance
+
+On a Mac M2 laptop, we see around 200MB/s, or around 100 million graphemes per second. You should see ~constant memory, and no allocations.
+
+### Invalid inputs
+
+Invalid UTF-8 input is considered undefined behavior. We test to ensure that bad inputs will not cause pathological outcomes, such as a panic or infinite loop. Callers should expect “garbage-in, garbage-out”.
+
+Your pipeline should probably include a call to [`utf8.Valid()`](https://pkg.go.dev/unicode/utf8#Valid).
@@ -0,0 +1,28 @@
+package graphemes
+
+import "github.com/clipperhouse/uax29/v2/internal/iterators"
+
+type Iterator[T iterators.Stringish] struct {
+	*iterators.Iterator[T]
+}
+
+var (
+	splitFuncString = splitFunc[string]
+	splitFuncBytes  = splitFunc[[]byte]
+)
+
+// FromString returns an iterator for the grapheme clusters in the input string.
+// Iterate while Next() is true, and access the grapheme via Value().
+func FromString(s string) Iterator[string] {
+	return Iterator[string]{
+		iterators.New(splitFuncString, s),
+	}
+}
+
+// FromBytes returns an iterator for the grapheme clusters in the input bytes.
+// Iterate while Next() is true, and access the grapheme via Value().
+func FromBytes(b []byte) Iterator[[]byte] {
+	return Iterator[[]byte]{
+		iterators.New(splitFuncBytes, b),
+	}
+}
@@ -0,0 +1,25 @@
+// Package graphemes implements Unicode grapheme cluster boundaries: https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries
+package graphemes
+
+import (
+	"bufio"
+	"io"
+)
+
+type Scanner struct {
+	*bufio.Scanner
+}
+
+// FromReader returns a Scanner, to split graphemes per
+// https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries.
+//
+// It embeds a [bufio.Scanner], so you can use its methods.
+//
+// Iterate through graphemes by calling Scan() until false, then check Err().
+func FromReader(r io.Reader) *Scanner {
+	sc := bufio.NewScanner(r)
+	sc.Split(SplitFunc)
+	return &Scanner{
+		Scanner: sc,
+	}
+}
@@ -0,0 +1,174 @@
+package graphemes
+
+import (
+	"bufio"
+
+	"github.com/clipperhouse/uax29/v2/internal/iterators"
+)
+
+// is determines if lookup intersects propert(ies)
+func (lookup property) is(properties property) bool {
+	return (lookup & properties) != 0
+}
+
+const _Ignore = _Extend
+
+// SplitFunc is a bufio.SplitFunc implementation of Unicode grapheme cluster segmentation, for use with bufio.Scanner.
+//
+// See https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries.
+var SplitFunc bufio.SplitFunc = splitFunc[[]byte]
+
+func splitFunc[T iterators.Stringish](data T, atEOF bool) (advance int, token T, err error) {
+	var empty T
+	if len(data) == 0 {
+		return 0, empty, nil
+	}
+
+	// These vars are stateful across loop iterations
+	var pos int
+	var lastExIgnore property = 0     // "last excluding ignored categories"
+	var lastLastExIgnore property = 0 // "last one before that"
+	var regionalIndicatorCount int
+
+	// Rules are usually of the form Cat1 × Cat2; "current" refers to the first property
+	// to the right of the ×, from which we look back or forward
+
+	current, w := lookup(data[pos:])
+	if w == 0 {
+		if !atEOF {
+			// Rune extends past current data, request more
+			return 0, empty, nil
+		}
+		pos = len(data)
+		return pos, data[:pos], nil
+	}
+
+	// https://unicode.org/reports/tr29/#GB1
+	// Start of text always advances
+	pos += w
+
+	for {
+		eot := pos == len(data) // "end of text"
+
+		if eot {
+			if !atEOF {
+				// Token extends past current data, request more
+				return 0, empty, nil
+			}
+
+			// https://unicode.org/reports/tr29/#GB2
+			break
+		}
+
+		/*
+			We've switched the evaluation order of GB1↓ and GB2↑. It's ok:
+			because we've checked for len(data) at the top of this function,
+			sot and eot are mutually exclusive, order doesn't matter.
+		*/
+
+		// Rules are usually of the form Cat1 × Cat2; "current" refers to the first property
+		// to the right of the ×, from which we look back or forward
+
+		// Remember previous properties to avoid lookups/lookbacks
+		last := current
+		if !last.is(_Ignore) {
+			lastLastExIgnore = lastExIgnore
+			lastExIgnore = last
+		}
+
+		current, w = lookup(data[pos:])
+		if w == 0 {
+			if atEOF {
+				// Just return the bytes, we can't do anything with them
+				pos = len(data)
+				break
+			}
+			// Rune extends past current data, request more
+			return 0, empty, nil
+		}
+
+		// Optimization: no rule can possibly apply
+		if current|last == 0 { // i.e. both are zero
+			break
+		}
+
+		// https://unicode.org/reports/tr29/#GB3
+		if current.is(_LF) && last.is(_CR) {
+			pos += w
+			continue
+		}
+
+		// https://unicode.org/reports/tr29/#GB4
+		// https://unicode.org/reports/tr29/#GB5
+		if (current | last).is(_Control | _CR | _LF) {
+			break
+		}
+
+		// https://unicode.org/reports/tr29/#GB6
+		if current.is(_L|_V|_LV|_LVT) && last.is(_L) {
+			pos += w
+			continue
+		}
+
+		// https://unicode.org/reports/tr29/#GB7
+		if current.is(_V|_T) && last.is(_LV|_V) {
+			pos += w
+			continue
+		}
+
+		// https://unicode.org/reports/tr29/#GB8
+		if current.is(_T) && last.is(_LVT|_T) {
+			pos += w
+			continue
+		}
+
+		// https://unicode.org/reports/tr29/#GB9
+		if current.is(_Extend | _ZWJ) {
+			pos += w
+			continue
+		}
+
+		// https://unicode.org/reports/tr29/#GB9a
+		if current.is(_SpacingMark) {
+			pos += w
+			continue
+		}
+
+		// https://unicode.org/reports/tr29/#GB9b
+		if last.is(_Prepend) {
+			pos += w
+			continue
+		}
+
+		// https://unicode.org/reports/tr29/#GB9c
+		// TODO(clipperhouse):
+		// It appears to be added in Unicode 15.1.0:
+		// https://unicode.org/versions/Unicode15.1.0/#Migration
+		// This package currently supports Unicode 15.0.0, so
+		// out of scope for now
+
+		// https://unicode.org/reports/tr29/#GB11
+		if current.is(_ExtendedPictographic) && last.is(_ZWJ) && lastLastExIgnore.is(_ExtendedPictographic) {
+			pos += w
+			continue
+		}
+
+		// https://unicode.org/reports/tr29/#GB12
+		// https://unicode.org/reports/tr29/#GB13
+		if (current & last).is(_RegionalIndicator) {
+			regionalIndicatorCount++
+
+			odd := regionalIndicatorCount%2 == 1
+			if odd {
+				pos += w
+				continue
+			}
+		}
+
+		// If we fall through all the above rules, it's a grapheme cluster break
+		break
+	}
+
+	// Return token
+	return pos, data[:pos], nil
+}
@@ -0,0 +1,85 @@
+package iterators
+
+type Stringish interface {
+	[]byte | string
+}
+
+type SplitFunc[T Stringish] func(T, bool) (int, T, error)
+
+// Iterator is a generic iterator for words that are either []byte or string.
+// Iterate while Next() is true, and access the word via Value().
+type Iterator[T Stringish] struct {
+	split SplitFunc[T]
+	data  T
+	start int
+	pos   int
+}
+
+// New creates a new Iterator for the given data and SplitFunc.
+func New[T Stringish](split SplitFunc[T], data T) *Iterator[T] {
+	return &Iterator[T]{
+		split: split,
+		data:  data,
+	}
+}
+
+// SetText sets the text for the iterator to operate on, and resets all state.
+func (iter *Iterator[T]) SetText(data T) {
+	iter.data = data
+	iter.start = 0
+	iter.pos = 0
+}
+
+// Split sets the SplitFunc for the Iterator.
+func (iter *Iterator[T]) Split(split SplitFunc[T]) {
+	iter.split = split
+}
+
+// Next advances the iterator to the next token. It returns false when there
+// are no remaining tokens or an error occurred.
+func (iter *Iterator[T]) Next() bool {
+	if iter.pos == len(iter.data) {
+		return false
+	}
+	if iter.pos > len(iter.data) {
+		panic("SplitFunc advanced beyond the end of the data")
+	}
+
+	iter.start = iter.pos
+
+	advance, _, err := iter.split(iter.data[iter.pos:], true)
+	if err != nil {
+		panic(err)
+	}
+	if advance <= 0 {
+		panic("SplitFunc returned a zero or negative advance")
+	}
+
+	iter.pos += advance
+	if iter.pos > len(iter.data) {
+		panic("SplitFunc advanced beyond the end of the data")
+	}
+
+	return true
+}
+
+// Value returns the current token.
+func (iter *Iterator[T]) Value() T {
+	return iter.data[iter.start:iter.pos]
+}
+
+// Start returns the byte position of the current token in the original data.
+func (iter *Iterator[T]) Start() int {
+	return iter.start
+}
+
+// End returns the byte position after the current token in the original data.
+func (iter *Iterator[T]) End() int {
+	return iter.pos
+}
+
+// Reset resets the iterator to the beginning of the data.
+func (iter *Iterator[T]) Reset() {
+	iter.start = 0
+	iter.pos = 0
+}