Initial QSfera import
This commit is contained in:
+6
@@ -0,0 +1,6 @@
|
||||
# See http://editorconfig.org
|
||||
|
||||
# In Go files we indent with tabs but still
|
||||
# set indent_size to control the GitHub web viewer.
|
||||
[*.go]
|
||||
indent_size=4
|
||||
@@ -0,0 +1,2 @@
|
||||
*.go text eol=lf
|
||||
*.peg text eol=lf
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
# Binaries for programs and plugins
|
||||
*.exe
|
||||
*.dll
|
||||
*.so
|
||||
*.dylib
|
||||
pigeon
|
||||
bin/
|
||||
bootstrap/cmd/bootstrap-pigeon/bootstrap-pigeon
|
||||
bootstrap/cmd/bootstrap-build/bootstrap-build
|
||||
bootstrap/cmd/pegscan/pegscan
|
||||
bootstrap/cmd/pegparse/pegparse
|
||||
|
||||
# Test binary, build with `go test -c`
|
||||
*.test
|
||||
|
||||
# Output of the go coverage tool, specifically when used with LiteIDE
|
||||
*.out
|
||||
|
||||
# Temporary and swap files
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
+37
@@ -0,0 +1,37 @@
|
||||
---
|
||||
version: 2
|
||||
|
||||
# This is an example .goreleaser.yml file with some sane defaults.
|
||||
# Make sure to check the documentation at http://goreleaser.com
|
||||
before:
|
||||
hooks:
|
||||
# You may remove this if you don't use go modules.
|
||||
- go mod tidy
|
||||
builds:
|
||||
- main: .
|
||||
binary: pigeon
|
||||
env:
|
||||
- CGO_ENABLED=0
|
||||
goos:
|
||||
- linux
|
||||
- windows
|
||||
- darwin
|
||||
archives:
|
||||
- name_template: >-
|
||||
{{- .Binary }}_
|
||||
{{- .Version }}_
|
||||
{{- title .Os }}_
|
||||
{{- if eq .Arch "amd64" }}x86_64
|
||||
{{- else if eq .Arch "386" }}i386
|
||||
{{- else }}{{ .Arch }}{{ end }}
|
||||
{{- if .Arm }}v{{ .Arm }}{{ end -}}
|
||||
snapshot:
|
||||
version_template: "{{ .Tag }}-next"
|
||||
changelog:
|
||||
disable: true
|
||||
release:
|
||||
github:
|
||||
owner: mna
|
||||
name: pigeon
|
||||
gomod:
|
||||
proxy: true
|
||||
+33
@@ -0,0 +1,33 @@
|
||||
# Contributing to pigeon
|
||||
|
||||
There are various ways to help support this open source project:
|
||||
|
||||
* if you use pigeon and find it useful, talk about it - that's probably the most basic way to help any open-source project: getting the word out that it exists and that it can be useful
|
||||
* if you use pigeon and find bugs, please [file an issue][0]
|
||||
* if something is poorly documented, or doesn't work as documented, this is also a bug, please [file an issue][0]
|
||||
* if you can fix the issue (whether it is documentation- or code-related), then [submit a pull-request][1] - but read on to see what should be done to get it merged
|
||||
* if you would like to see some new feature/behaviour being implemented, please first [open an issue][0] to discuss it because features are less likely to get merged compared to bug fixes
|
||||
|
||||
## Submitting a pull request
|
||||
|
||||
Assuming you already have a copy of the repository (either via `go get`, a github fork, a clone, etc.), you will also need `make` to regenerate all tools and files generated when a dependency changes. I use GNU make version 4.1, other versions of make may work too but haven't been tested.
|
||||
|
||||
Run `make` in the root directory of the repository. That will create the bootstrap builder, the bootstrap parser, and the final parser, along with some generated Go files. Once `make` is run successfully, run `go test ./...` in the root directory to make sure all tests pass.
|
||||
|
||||
Once this is done and tests pass, you can start implementing the bug fix (or the new feature provided **it has already been discussed and agreed in a github issue** first).
|
||||
|
||||
For a bug fix, the best way to proceed is to first write a test that proves the bug, then write the code that fixes the bug and makes the test pass. All other tests should still pass too (unless it relied on the buggy behaviour, in which case existing tests must be fixed).
|
||||
|
||||
For a new feature, it must be thoroughly tested. New code without new test(s) is unlikely to get merged.
|
||||
|
||||
Respect the coding style of the repository, which means essentially to respect the [coding guidelines of the Go community][2]. Use `gofmt` to format your code, and `goimports` to add and format the list of imported packages (or do it manually, but in a `goimports`-style).
|
||||
|
||||
Once all code is done and tests pass, regenerate the whole tree with `make`, run `make lint` (does need [`golangci-lint`](https://golangci-lint.run/))to make sure the code is correct, and run tests again. You are now ready to submit the pull request.
|
||||
|
||||
## Licensing
|
||||
|
||||
All pull requests that get merged will be made available under the BSD 3-Clause license (see the LICENSE file for details), as the rest of the pigeon repository. Do not submit pull requests if you do not want your contributions to be made available under those terms.
|
||||
|
||||
[0]: https://github.com/mna/pigeon/issues/new
|
||||
[1]: https://github.com/mna/pigeon/pulls
|
||||
[2]: https://github.com/golang/go/wiki/CodeReviewComments
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
Copyright (c) 2015, Martin Angers & Contributors
|
||||
All rights reserved.
|
||||
|
||||
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
|
||||
|
||||
* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
|
||||
|
||||
* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
|
||||
|
||||
* Neither the name of the author nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
|
||||
|
||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
||||
+255
@@ -0,0 +1,255 @@
|
||||
SHELL = /bin/bash
|
||||
|
||||
# force the use of go modules
|
||||
export GO111MODULE = on
|
||||
|
||||
# directories and source code lists
|
||||
ROOT = .
|
||||
ROOT_SRC = $(ROOT)/*.go
|
||||
BINDIR = ./bin
|
||||
EXAMPLES_DIR = $(ROOT)/examples
|
||||
TEST_DIR = $(ROOT)/test
|
||||
|
||||
# builder and ast packages
|
||||
BUILDER_DIR = $(ROOT)/builder
|
||||
BUILDER_SRC = $(BUILDER_DIR)/*.go
|
||||
AST_DIR = $(ROOT)/ast
|
||||
AST_SRC = $(AST_DIR)/*.go
|
||||
|
||||
# bootstrap tools variables
|
||||
BOOTSTRAP_DIR = $(ROOT)/bootstrap
|
||||
BOOTSTRAP_SRC = $(BOOTSTRAP_DIR)/*.go
|
||||
BOOTSTRAPBUILD_DIR = $(BOOTSTRAP_DIR)/cmd/bootstrap-build
|
||||
BOOTSTRAPBUILD_SRC = $(BOOTSTRAPBUILD_DIR)/*.go
|
||||
BOOTSTRAPPIGEON_DIR = $(BOOTSTRAP_DIR)/cmd/bootstrap-pigeon
|
||||
BOOTSTRAPPIGEON_SRC = $(BOOTSTRAPPIGEON_DIR)/*.go
|
||||
STATICCODEGENERATOR_DIR = $(BOOTSTRAP_DIR)/cmd/static_code_generator
|
||||
STATICCODEGENERATOR_SRC = $(STATICCODEGENERATOR_DIR)/*.go
|
||||
|
||||
# grammar variables
|
||||
GRAMMAR_DIR = $(ROOT)/grammar
|
||||
BOOTSTRAP_GRAMMAR = $(GRAMMAR_DIR)/bootstrap.peg
|
||||
PIGEON_GRAMMAR = $(GRAMMAR_DIR)/pigeon.peg
|
||||
|
||||
TEST_GENERATED_SRC = $(patsubst %.peg,%.go,$(shell echo ./{examples,test}/**/*.peg))
|
||||
|
||||
all: $(BUILDER_DIR)/generated_static_code.go $(BINDIR)/static_code_generator \
|
||||
$(BUILDER_DIR)/generated_static_code_range_table.go \
|
||||
$(BINDIR)/bootstrap-build $(BOOTSTRAPPIGEON_DIR)/bootstrap_pigeon.go \
|
||||
$(BINDIR)/bootstrap-pigeon $(ROOT)/pigeon.go $(BINDIR)/pigeon \
|
||||
$(TEST_GENERATED_SRC)
|
||||
|
||||
$(BINDIR)/static_code_generator: $(STATICCODEGENERATOR_SRC)
|
||||
go build -o $@ $(STATICCODEGENERATOR_DIR)
|
||||
|
||||
$(BINDIR)/bootstrap-build: $(BOOTSTRAPBUILD_SRC) $(BOOTSTRAP_SRC) $(BUILDER_SRC) \
|
||||
$(AST_SRC)
|
||||
go build -o $@ $(BOOTSTRAPBUILD_DIR)
|
||||
|
||||
$(BOOTSTRAPPIGEON_DIR)/bootstrap_pigeon.go: $(BINDIR)/bootstrap-build \
|
||||
$(BOOTSTRAP_GRAMMAR)
|
||||
$(BINDIR)/bootstrap-build $(BOOTSTRAP_GRAMMAR) > $@
|
||||
|
||||
$(BINDIR)/bootstrap-pigeon: $(BOOTSTRAPPIGEON_SRC) \
|
||||
$(BOOTSTRAPPIGEON_DIR)/bootstrap_pigeon.go
|
||||
go build -o $@ $(BOOTSTRAPPIGEON_DIR)
|
||||
|
||||
$(ROOT)/pigeon.go: $(BINDIR)/bootstrap-pigeon $(PIGEON_GRAMMAR)
|
||||
$(BINDIR)/bootstrap-pigeon $(PIGEON_GRAMMAR) > $@
|
||||
|
||||
$(BINDIR)/pigeon: $(ROOT_SRC) $(ROOT)/pigeon.go
|
||||
go build -o $@ $(ROOT)
|
||||
|
||||
$(BUILDER_DIR)/generated_static_code.go: $(BUILDER_DIR)/static_code.go $(BINDIR)/static_code_generator
|
||||
$(BINDIR)/static_code_generator $(BUILDER_DIR)/static_code.go $@ staticCode
|
||||
|
||||
$(BUILDER_DIR)/generated_static_code_range_table.go: $(BUILDER_DIR)/static_code_range_table.go $(BINDIR)/static_code_generator
|
||||
$(BINDIR)/static_code_generator $(BUILDER_DIR)/static_code_range_table.go $@ rangeTable0
|
||||
|
||||
$(BOOTSTRAP_GRAMMAR):
|
||||
$(PIGEON_GRAMMAR):
|
||||
|
||||
# surely there's a better way to define the examples and test targets
|
||||
$(EXAMPLES_DIR)/json/json.go: $(EXAMPLES_DIR)/json/json.peg $(EXAMPLES_DIR)/json/optimized/json.go $(EXAMPLES_DIR)/json/optimized-grammar/json.go $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint $< > $@
|
||||
|
||||
$(EXAMPLES_DIR)/json/optimized/json.go: $(EXAMPLES_DIR)/json/json.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint -optimize-parser -optimize-basic-latin $< > $@
|
||||
|
||||
$(EXAMPLES_DIR)/json/optimized-grammar/json.go: $(EXAMPLES_DIR)/json/json.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint -optimize-grammar $< > $@
|
||||
|
||||
$(EXAMPLES_DIR)/calculator/calculator.go: $(EXAMPLES_DIR)/calculator/calculator.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint $< > $@
|
||||
|
||||
$(EXAMPLES_DIR)/indentation/indentation.go: $(EXAMPLES_DIR)/indentation/indentation.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint $< > $@
|
||||
|
||||
$(TEST_DIR)/andnot/andnot.go: $(TEST_DIR)/andnot/andnot.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint $< > $@
|
||||
|
||||
$(TEST_DIR)/predicates/predicates.go: $(TEST_DIR)/predicates/predicates.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint $< > $@
|
||||
|
||||
$(TEST_DIR)/issue_1/issue_1.go: $(TEST_DIR)/issue_1/issue_1.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint $< > $@
|
||||
|
||||
$(TEST_DIR)/linear/linear.go: $(TEST_DIR)/linear/linear.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint $< > $@
|
||||
|
||||
$(TEST_DIR)/issue_18/issue_18.go: $(TEST_DIR)/issue_18/issue_18.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint $< > $@
|
||||
|
||||
$(TEST_DIR)/runeerror/runeerror.go: $(TEST_DIR)/runeerror/runeerror.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint $< > $@
|
||||
|
||||
$(TEST_DIR)/errorpos/errorpos.go: $(TEST_DIR)/errorpos/errorpos.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint $< > $@
|
||||
|
||||
$(TEST_DIR)/global_store/global_store.go: $(TEST_DIR)/global_store/global_store.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint $< > $@
|
||||
|
||||
$(TEST_DIR)/goto/goto.go: $(TEST_DIR)/goto/goto.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint $< > $@
|
||||
|
||||
$(TEST_DIR)/goto_state/goto_state.go: $(TEST_DIR)/goto_state/goto_state.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint $< > $@
|
||||
|
||||
$(TEST_DIR)/max_expr_cnt/maxexpr.go: $(TEST_DIR)/max_expr_cnt/maxexpr.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint $< > $@
|
||||
|
||||
$(TEST_DIR)/labeled_failures/labeled_failures.go: $(TEST_DIR)/labeled_failures/labeled_failures.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint $< > $@
|
||||
|
||||
$(TEST_DIR)/thrownrecover/thrownrecover.go: $(TEST_DIR)/thrownrecover/thrownrecover.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint $< > $@
|
||||
|
||||
$(TEST_DIR)/alternate_entrypoint/altentry.go: $(TEST_DIR)/alternate_entrypoint/altentry.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint -optimize-grammar -alternate-entrypoints Entry2,Entry3,C $< > $@
|
||||
|
||||
$(TEST_DIR)/state/state.go: $(TEST_DIR)/state/state.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint -optimize-grammar $< > $@
|
||||
|
||||
$(TEST_DIR)/stateclone/stateclone.go: $(TEST_DIR)/stateclone/stateclone.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint $< > $@
|
||||
|
||||
$(TEST_DIR)/statereadonly/statereadonly.go: $(TEST_DIR)/statereadonly/statereadonly.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint $< > $@
|
||||
|
||||
$(TEST_DIR)/staterestore/staterestore.go: $(TEST_DIR)/staterestore/staterestore.peg $(TEST_DIR)/staterestore/standard/staterestore.go $(TEST_DIR)/staterestore/optimized/staterestore.go $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint $< > $@
|
||||
|
||||
$(TEST_DIR)/staterestore/standard/staterestore.go: $(TEST_DIR)/staterestore/staterestore.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint $< > $@
|
||||
|
||||
$(TEST_DIR)/staterestore/optimized/staterestore.go: $(TEST_DIR)/staterestore/staterestore.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint -optimize-grammar -optimize-parser -alternate-entrypoints TestAnd,TestNot $< > $@
|
||||
|
||||
$(TEST_DIR)/emptystate/emptystate.go: $(TEST_DIR)/emptystate/emptystate.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint $< > $@
|
||||
|
||||
$(TEST_DIR)/issue_65/issue_65.go: $(TEST_DIR)/issue_65/issue_65.peg $(TEST_DIR)/issue_65/optimized/issue_65.go $(TEST_DIR)/issue_65/optimized-grammar/issue_65.go $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint $< > $@
|
||||
|
||||
$(TEST_DIR)/issue_65/optimized/issue_65.go: $(TEST_DIR)/issue_65/issue_65.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint -optimize-parser -optimize-basic-latin $< > $@
|
||||
|
||||
$(TEST_DIR)/issue_65/optimized-grammar/issue_65.go: $(TEST_DIR)/issue_65/issue_65.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint -optimize-grammar $< > $@
|
||||
|
||||
$(TEST_DIR)/issue_70/issue_70.go: $(TEST_DIR)/issue_70/issue_70.peg $(TEST_DIR)/issue_70/optimized/issue_70.go $(TEST_DIR)/issue_70/optimized-grammar/issue_70.go $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint $< > $@
|
||||
|
||||
$(TEST_DIR)/issue_70/optimized/issue_70.go: $(TEST_DIR)/issue_70/issue_70.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint -optimize-parser -optimize-basic-latin $< > $@
|
||||
|
||||
$(TEST_DIR)/issue_70/optimized-grammar/issue_70.go: $(TEST_DIR)/issue_70/issue_70.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint -optimize-grammar $< > $@
|
||||
|
||||
$(TEST_DIR)/issue_70b/issue_70b.go: $(TEST_DIR)/issue_70b/issue_70b.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint -optimize-grammar -support-left-recursion $< > $@
|
||||
|
||||
$(TEST_DIR)/issue_79/issue_79.go: $(TEST_DIR)/issue_79/issue_79.peg $(BINDIR)/pigeon
|
||||
@! $(BINDIR)/pigeon $< > $@ 2>/dev/null && exit 0 || echo "failure, expect build to fail due to left recursion!" && exit 1
|
||||
$(BINDIR)/pigeon -support-left-recursion $< > $@
|
||||
|
||||
$(TEST_DIR)/issue_80/issue_80.go: $(TEST_DIR)/issue_80/issue_80.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint $< > $@
|
||||
|
||||
$(TEST_DIR)/issue_115/issue_115.go: $(TEST_DIR)/issue_115/issue_115.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint $< > $@
|
||||
|
||||
$(TEST_DIR)/issue_134/issue_134.go: $(TEST_DIR)/issue_134/issue_134.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint $< > $@
|
||||
|
||||
$(TEST_DIR)/left_recursion/left_recursion.go: \
|
||||
$(TEST_DIR)/left_recursion/standart/leftrecursion/left_recursion.go \
|
||||
$(TEST_DIR)/left_recursion/optimized/leftrecursion/left_recursion.go \
|
||||
$(BINDIR)/pigeon
|
||||
|
||||
$(TEST_DIR)/left_recursion/standart/leftrecursion/left_recursion.go: \
|
||||
$(TEST_DIR)/left_recursion/left_recursion.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint -support-left-recursion $< > $@
|
||||
|
||||
$(TEST_DIR)/left_recursion/optimized/leftrecursion/left_recursion.go: \
|
||||
$(TEST_DIR)/left_recursion/left_recursion.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint -optimize-parser -support-left-recursion $< > $@
|
||||
|
||||
$(TEST_DIR)/left_recursion/without_left_recursion.go: \
|
||||
$(TEST_DIR)/left_recursion/standart/withoutleftrecursion/without_left_recursion.go \
|
||||
$(TEST_DIR)/left_recursion/optimized/withoutleftrecursion/without_left_recursion.go \
|
||||
$(BINDIR)/pigeon
|
||||
|
||||
$(TEST_DIR)/left_recursion/standart/withoutleftrecursion/without_left_recursion.go: \
|
||||
$(TEST_DIR)/left_recursion/without_left_recursion.peg \
|
||||
$(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint $< > $@
|
||||
|
||||
$(TEST_DIR)/left_recursion/optimized/withoutleftrecursion/without_left_recursion.go: \
|
||||
$(TEST_DIR)/left_recursion/without_left_recursion.peg \
|
||||
$(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint -optimize-parser $< > $@
|
||||
|
||||
$(TEST_DIR)/left_recursion_state/left_recursion_state.go: \
|
||||
$(TEST_DIR)/left_recursion_state/standart/left_recursion_state.go \
|
||||
$(TEST_DIR)/left_recursion_state/optimized/left_recursion_state.go \
|
||||
$(BINDIR)/pigeon
|
||||
|
||||
$(TEST_DIR)/left_recursion_state/standart/left_recursion_state.go: \
|
||||
$(TEST_DIR)/left_recursion_state/left_recursion_state.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint -support-left-recursion $< > $@
|
||||
|
||||
$(TEST_DIR)/left_recursion_state/optimized/left_recursion_state.go: \
|
||||
$(TEST_DIR)/left_recursion_state/left_recursion_state.peg $(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint -optimize-parser -support-left-recursion $< > $@
|
||||
|
||||
$(TEST_DIR)/left_recursion_labeled_failures/left_recursion_labeled_failures.go: \
|
||||
$(TEST_DIR)/left_recursion_labeled_failures/left_recursion_labeled_failures.peg \
|
||||
$(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint -support-left-recursion $< > $@
|
||||
|
||||
$(TEST_DIR)/left_recursion_thrownrecover/left_recursion_thrownrecover.go: \
|
||||
$(TEST_DIR)/left_recursion_thrownrecover/left_recursion_thrownrecover.peg \
|
||||
$(BINDIR)/pigeon
|
||||
$(BINDIR)/pigeon -nolint -support-left-recursion $< > $@
|
||||
|
||||
lint:
|
||||
golangci-lint run ./...
|
||||
|
||||
cmp:
|
||||
@boot=$$(mktemp) && $(BINDIR)/bootstrap-pigeon $(PIGEON_GRAMMAR) > $$boot && \
|
||||
official=$$(mktemp) && $(BINDIR)/pigeon $(PIGEON_GRAMMAR) > $$official && \
|
||||
cmp $$boot $$official && \
|
||||
unlink $$boot && \
|
||||
unlink $$official
|
||||
|
||||
test:
|
||||
go test -v ./...
|
||||
|
||||
clean:
|
||||
rm -f $(BUILDER_DIR)/generated_static_code.go $(BUILDER_DIR)/generated_static_code_range_table.go
|
||||
rm -f $(BOOTSTRAPPIGEON_DIR)/bootstrap_pigeon.go $(ROOT)/pigeon.go $(TEST_GENERATED_SRC) $(EXAMPLES_DIR)/json/optimized/json.go $(EXAMPLES_DIR)/json/optimized-grammar/json.go $(TEST_DIR)/staterestore/optimized/staterestore.go $(TEST_DIR)/staterestore/standard/staterestore.go $(TEST_DIR)/issue_65/optimized/issue_65.go $(TEST_DIR)/issue_65/optimized-grammar/issue_65.go
|
||||
rm -rf $(BINDIR)
|
||||
|
||||
.PHONY: all clean lint cmp test
|
||||
|
||||
+154
@@ -0,0 +1,154 @@
|
||||
# pigeon - a PEG parser generator for Go
|
||||
|
||||
[](https://pkg.go.dev/github.com/mna/pigeon)
|
||||
[](https://github.com/mna/pigeon/actions?query=workflow%3AGo%20Matrix)
|
||||
[](https://goreportcard.com/report/github.com/mna/pigeon)
|
||||
[](LICENSE)
|
||||
|
||||
The pigeon command generates parsers based on a [parsing expression grammar (PEG)][0]. Its grammar and syntax is inspired by the [PEG.js project][1], while the implementation is loosely based on the [parsing expression grammar for C# 3.0][2] article. It parses Unicode text encoded in UTF-8.
|
||||
|
||||
See the [godoc page][3] for detailed usage. Also have a look at the [Pigeon Wiki](https://github.com/mna/pigeon/wiki) for additional information about Pigeon and PEG in general.
|
||||
|
||||
## Releases
|
||||
|
||||
* v1.0.0 is the tagged release of the original implementation.
|
||||
* Work has started on v2.0.0 with some planned breaking changes.
|
||||
|
||||
Github user [@mna][6] created the package in April 2015, and [@breml][5] is the package's maintainer as of May 2017.
|
||||
|
||||
### Release policy
|
||||
|
||||
Starting of June 2023, the backwards compatibility support for `pigeon` is changed to follow the official [Go Security Policy](https://github.com/golang/go/security/policy).
|
||||
|
||||
Over time, the Go ecosystem is evolving.
|
||||
|
||||
On one hand side, packages like [golang.org/x/tools](https://pkg.go.dev/golang.org/x/tools), which are critical dependencies of `pigeon`, do follow the official Security Policy and with `pigeon` not following the same guidelines, it was no longer possible to include recent versions of these dependencies and with this it was no longer possible to include critical bugfixes.
|
||||
On the other hand there are changes to what is considered good practice by the greater community (e.g. change from `interface{}` to `any`). For users following (or even enforcing) these good practices, the code generated by `pigeon` does no longer meet the bar of expectations.
|
||||
Last but not least, following the Go Security Policy over the last years has been a smooth experience and therefore updating Go on a regular bases feels like duty that is reasonable to be put on users of `pigeon`.
|
||||
|
||||
This observations lead to the decision to follow the same Security Policy as Go.
|
||||
|
||||
## Installation
|
||||
|
||||
Provided you have Go correctly installed with the $GOPATH and $GOBIN environment variables set, run:
|
||||
|
||||
```
|
||||
$ go get -u github.com/mna/pigeon
|
||||
```
|
||||
|
||||
This will install or update the package, and the `pigeon` command will be installed in your $GOBIN directory. Neither this package nor the parsers generated by this command require any third-party dependency, unless such a dependency is used in the code blocks of the grammar.
|
||||
|
||||
## Basic usage
|
||||
|
||||
```
|
||||
$ pigeon [options] [PEG_GRAMMAR_FILE]
|
||||
```
|
||||
|
||||
By default, the input grammar is read from `stdin` and the generated code is printed to `stdout`. You may save it in a file using the `-o` flag.
|
||||
|
||||
## Example
|
||||
|
||||
Given the following grammar:
|
||||
|
||||
```
|
||||
{
|
||||
// part of the initializer code block omitted for brevity
|
||||
|
||||
var ops = map[string]func(int, int) int {
|
||||
"+": func(l, r int) int {
|
||||
return l + r
|
||||
},
|
||||
"-": func(l, r int) int {
|
||||
return l - r
|
||||
},
|
||||
"*": func(l, r int) int {
|
||||
return l * r
|
||||
},
|
||||
"/": func(l, r int) int {
|
||||
return l / r
|
||||
},
|
||||
}
|
||||
|
||||
func toAnySlice(v any) []any {
|
||||
if v == nil {
|
||||
return nil
|
||||
}
|
||||
return v.([]any)
|
||||
}
|
||||
|
||||
func eval(first, rest any) int {
|
||||
l := first.(int)
|
||||
restSl := toAnySlice(rest)
|
||||
for _, v := range restSl {
|
||||
restExpr := toAnySlice(v)
|
||||
r := restExpr[3].(int)
|
||||
op := restExpr[1].(string)
|
||||
l = ops[op](l, r)
|
||||
}
|
||||
return l
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
Input <- expr:Expr EOF {
|
||||
return expr, nil
|
||||
}
|
||||
|
||||
Expr <- _ first:Term rest:( _ AddOp _ Term )* _ {
|
||||
return eval(first, rest), nil
|
||||
}
|
||||
|
||||
Term <- first:Factor rest:( _ MulOp _ Factor )* {
|
||||
return eval(first, rest), nil
|
||||
}
|
||||
|
||||
Factor <- '(' expr:Expr ')' {
|
||||
return expr, nil
|
||||
} / integer:Integer {
|
||||
return integer, nil
|
||||
}
|
||||
|
||||
AddOp <- ( '+' / '-' ) {
|
||||
return string(c.text), nil
|
||||
}
|
||||
|
||||
MulOp <- ( '*' / '/' ) {
|
||||
return string(c.text), nil
|
||||
}
|
||||
|
||||
Integer <- '-'? [0-9]+ {
|
||||
return strconv.Atoi(string(c.text))
|
||||
}
|
||||
|
||||
_ "whitespace" <- [ \n\t\r]*
|
||||
|
||||
EOF <- !.
|
||||
```
|
||||
|
||||
The generated parser can parse simple arithmetic operations, e.g.:
|
||||
|
||||
```
|
||||
18 + 3 - 27 * (-18 / -3)
|
||||
|
||||
=> -141
|
||||
```
|
||||
|
||||
More examples can be found in the `examples/` subdirectory.
|
||||
|
||||
See the [package documentation][3] for detailed usage.
|
||||
|
||||
## Contributing
|
||||
|
||||
See the CONTRIBUTING.md file.
|
||||
|
||||
## License
|
||||
|
||||
The [BSD 3-Clause license][4]. See the LICENSE file.
|
||||
|
||||
[0]: http://en.wikipedia.org/wiki/Parsing_expression_grammar
|
||||
[1]: http://pegjs.org/
|
||||
[2]: http://www.codeproject.com/Articles/29713/Parsing-Expression-Grammar-Support-for-C-Part
|
||||
[3]: https://pkg.go.dev/github.com/mna/pigeon
|
||||
[4]: http://opensource.org/licenses/BSD-3-Clause
|
||||
[5]: https://github.com/breml
|
||||
[6]: https://github.com/mna
|
||||
+25
@@ -0,0 +1,25 @@
|
||||
----------------------------------------------------------------------
|
||||
License notice for github.com/stretchr/testify
|
||||
----------------------------------------------------------------------
|
||||
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2012-2020 Mat Ryer, Tyler Bunnell and contributors.
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
- refactor implementation as a VM to avoid stack overflow in pathological cases (and maybe better performance): in branch wip-vm
|
||||
? options like current receiver name read directly from the grammar file
|
||||
? type annotations for generated code functions
|
||||
+1131
File diff suppressed because it is too large
Load Diff
+469
@@ -0,0 +1,469 @@
|
||||
package ast
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type grammarOptimizer struct {
|
||||
rule string
|
||||
protectedRules map[string]struct{}
|
||||
rules map[string]*Rule
|
||||
ruleUsesRules map[string]map[string]struct{}
|
||||
ruleUsedByRules map[string]map[string]struct{}
|
||||
visitor func(expr Expression) Visitor
|
||||
optimized bool
|
||||
}
|
||||
|
||||
func newGrammarOptimizer(protectedRules []string) *grammarOptimizer {
|
||||
pr := make(map[string]struct{}, len(protectedRules))
|
||||
for _, nm := range protectedRules {
|
||||
pr[nm] = struct{}{}
|
||||
}
|
||||
|
||||
r := grammarOptimizer{
|
||||
protectedRules: pr,
|
||||
rules: make(map[string]*Rule),
|
||||
ruleUsesRules: make(map[string]map[string]struct{}),
|
||||
ruleUsedByRules: make(map[string]map[string]struct{}),
|
||||
}
|
||||
r.visitor = r.init
|
||||
return &r
|
||||
}
|
||||
|
||||
// Visit is a generic Visitor to be used with Walk
|
||||
// The actual function, which should be used during Walk
|
||||
// is held in ruleRefOptimizer.visitor.
|
||||
func (r *grammarOptimizer) Visit(expr Expression) Visitor {
|
||||
return r.visitor(expr)
|
||||
}
|
||||
|
||||
// init is a Visitor, which is used with the Walk function
|
||||
// The purpose of this function is to initialize the reference
|
||||
// maps rules, ruleUsesRules and ruleUsedByRules.
|
||||
func (r *grammarOptimizer) init(expr Expression) Visitor {
|
||||
switch expr := expr.(type) {
|
||||
case *Rule:
|
||||
// Keep track of current rule, which is processed
|
||||
r.rule = expr.Name.Val
|
||||
r.rules[expr.Name.Val] = expr
|
||||
case *RuleRefExpr:
|
||||
// Fill ruleUsesRules and ruleUsedByRules for every RuleRefExpr
|
||||
set(r.ruleUsesRules, r.rule, expr.Name.Val)
|
||||
set(r.ruleUsedByRules, expr.Name.Val, r.rule)
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
// Add element to map of maps, initialize the inner map
|
||||
// if necessary.
|
||||
func set(m map[string]map[string]struct{}, src, dst string) {
|
||||
if _, ok := m[src]; !ok {
|
||||
m[src] = make(map[string]struct{})
|
||||
}
|
||||
m[src][dst] = struct{}{}
|
||||
}
|
||||
|
||||
// optimize is a Visitor, which is used with the Walk function
|
||||
// The purpose of this function is to perform the actual optimizations.
|
||||
// See Optimize for a detailed list of the performed optimizations.
|
||||
func (r *grammarOptimizer) optimize(expr0 Expression) Visitor {
|
||||
switch expr := expr0.(type) {
|
||||
case *ActionExpr:
|
||||
expr.Expr = r.optimizeRule(expr.Expr)
|
||||
case *AndExpr:
|
||||
expr.Expr = r.optimizeRule(expr.Expr)
|
||||
case *ChoiceExpr:
|
||||
expr.Alternatives = r.optimizeRules(expr.Alternatives)
|
||||
|
||||
// Optimize choice nested in choice
|
||||
for i := 0; i < len(expr.Alternatives); i++ {
|
||||
if choice, ok := expr.Alternatives[i].(*ChoiceExpr); ok {
|
||||
r.optimized = true
|
||||
if i+1 < len(expr.Alternatives) {
|
||||
expr.Alternatives = append(expr.Alternatives[:i], append(choice.Alternatives, expr.Alternatives[i+1:]...)...)
|
||||
} else {
|
||||
expr.Alternatives = append(expr.Alternatives[:i], choice.Alternatives...)
|
||||
}
|
||||
}
|
||||
|
||||
// Combine sequence of single char LitMatcher to CharClassMatcher
|
||||
if i > 0 {
|
||||
l0, lok0 := expr.Alternatives[i-1].(*LitMatcher)
|
||||
l1, lok1 := expr.Alternatives[i].(*LitMatcher)
|
||||
c0, cok0 := expr.Alternatives[i-1].(*CharClassMatcher)
|
||||
c1, cok1 := expr.Alternatives[i].(*CharClassMatcher)
|
||||
|
||||
combined := false
|
||||
|
||||
switch {
|
||||
// Combine two LitMatcher to CharClassMatcher
|
||||
// "a" / "b" => [ab]
|
||||
case lok0 && lok1 && len([]rune(l0.Val)) == 1 && len([]rune(l1.Val)) == 1 && l0.IgnoreCase == l1.IgnoreCase:
|
||||
combined = true
|
||||
cm := CharClassMatcher{
|
||||
Chars: append([]rune(l0.Val), []rune(l1.Val)...),
|
||||
IgnoreCase: l0.IgnoreCase,
|
||||
posValue: l0.posValue,
|
||||
}
|
||||
expr.Alternatives[i-1] = &cm
|
||||
|
||||
// Combine LitMatcher with CharClassMatcher
|
||||
// "a" / [bc] => [abc]
|
||||
case lok0 && cok1 && len([]rune(l0.Val)) == 1 && l0.IgnoreCase == c1.IgnoreCase && !c1.Inverted:
|
||||
combined = true
|
||||
c1.Chars = append(c1.Chars, []rune(l0.Val)...)
|
||||
expr.Alternatives[i-1] = c1
|
||||
|
||||
// Combine CharClassMatcher with LitMatcher
|
||||
// [ab] / "c" => [abc]
|
||||
case cok0 && lok1 && len([]rune(l1.Val)) == 1 && c0.IgnoreCase == l1.IgnoreCase && !c0.Inverted:
|
||||
combined = true
|
||||
c0.Chars = append(c0.Chars, []rune(l1.Val)...)
|
||||
|
||||
// Combine CharClassMatcher with CharClassMatcher
|
||||
// [ab] / [cd] => [abcd]
|
||||
case cok0 && cok1 && c0.IgnoreCase == c1.IgnoreCase && c0.Inverted == c1.Inverted:
|
||||
combined = true
|
||||
c0.Chars = append(c0.Chars, c1.Chars...)
|
||||
c0.Ranges = append(c0.Ranges, c1.Ranges...)
|
||||
c0.UnicodeClasses = append(c0.UnicodeClasses, c1.UnicodeClasses...)
|
||||
}
|
||||
|
||||
// If one of the optimizations was applied, remove the second element from Alternatives
|
||||
if combined {
|
||||
r.optimized = true
|
||||
if i+1 < len(expr.Alternatives) {
|
||||
expr.Alternatives = append(expr.Alternatives[:i], expr.Alternatives[i+1:]...)
|
||||
} else {
|
||||
expr.Alternatives = expr.Alternatives[:i]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
case *Grammar:
|
||||
// Reset optimized at the start of each Walk.
|
||||
r.optimized = false
|
||||
for i := 0; i < len(expr.Rules); i++ {
|
||||
rule := expr.Rules[i]
|
||||
// Remove Rule, if it is no longer used by any other Rule and it is not the first Rule.
|
||||
_, used := r.ruleUsedByRules[rule.Name.Val]
|
||||
_, protected := r.protectedRules[rule.Name.Val]
|
||||
if !used && !protected {
|
||||
expr.Rules = append(expr.Rules[:i], expr.Rules[i+1:]...)
|
||||
// Compensate for the removed item
|
||||
i--
|
||||
|
||||
for k, v := range r.ruleUsedByRules {
|
||||
for kk := range v {
|
||||
if kk == rule.Name.Val {
|
||||
delete(r.ruleUsedByRules[k], kk)
|
||||
if len(r.ruleUsedByRules[k]) == 0 {
|
||||
delete(r.ruleUsedByRules, k)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
r.optimized = true
|
||||
continue
|
||||
}
|
||||
}
|
||||
case *LabeledExpr:
|
||||
expr.Expr = r.optimizeRule(expr.Expr)
|
||||
case *NotExpr:
|
||||
expr.Expr = r.optimizeRule(expr.Expr)
|
||||
case *OneOrMoreExpr:
|
||||
expr.Expr = r.optimizeRule(expr.Expr)
|
||||
case *Rule:
|
||||
r.rule = expr.Name.Val
|
||||
expr.Expr = r.optimizeRule(expr.Expr)
|
||||
case *SeqExpr:
|
||||
expr.Exprs = r.optimizeRules(expr.Exprs)
|
||||
|
||||
for i := 0; i < len(expr.Exprs); i++ {
|
||||
// Optimize nested sequences
|
||||
if seq, ok := expr.Exprs[i].(*SeqExpr); ok {
|
||||
r.optimized = true
|
||||
if i+1 < len(expr.Exprs) {
|
||||
expr.Exprs = append(expr.Exprs[:i], append(seq.Exprs, expr.Exprs[i+1:]...)...)
|
||||
} else {
|
||||
expr.Exprs = append(expr.Exprs[:i], seq.Exprs...)
|
||||
}
|
||||
}
|
||||
|
||||
// Combine sequence of LitMatcher
|
||||
if i > 0 {
|
||||
l0, ok0 := expr.Exprs[i-1].(*LitMatcher)
|
||||
l1, ok1 := expr.Exprs[i].(*LitMatcher)
|
||||
if ok0 && ok1 && l0.IgnoreCase == l1.IgnoreCase {
|
||||
r.optimized = true
|
||||
l0.Val += l1.Val
|
||||
expr.Exprs[i-1] = l0
|
||||
if i+1 < len(expr.Exprs) {
|
||||
expr.Exprs = append(expr.Exprs[:i], expr.Exprs[i+1:]...)
|
||||
} else {
|
||||
expr.Exprs = expr.Exprs[:i]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
case *ZeroOrMoreExpr:
|
||||
expr.Expr = r.optimizeRule(expr.Expr)
|
||||
case *ZeroOrOneExpr:
|
||||
expr.Expr = r.optimizeRule(expr.Expr)
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
func (r *grammarOptimizer) optimizeRules(exprs []Expression) []Expression {
|
||||
for i := 0; i < len(exprs); i++ {
|
||||
exprs[i] = r.optimizeRule(exprs[i])
|
||||
}
|
||||
return exprs
|
||||
}
|
||||
|
||||
func (r *grammarOptimizer) optimizeRule(expr Expression) Expression {
|
||||
// Optimize RuleRefExpr
|
||||
if ruleRef, ok := expr.(*RuleRefExpr); ok {
|
||||
if _, ok := r.ruleUsesRules[ruleRef.Name.Val]; !ok {
|
||||
r.optimized = true
|
||||
delete(r.ruleUsedByRules[ruleRef.Name.Val], r.rule)
|
||||
if len(r.ruleUsedByRules[ruleRef.Name.Val]) == 0 {
|
||||
delete(r.ruleUsedByRules, ruleRef.Name.Val)
|
||||
}
|
||||
delete(r.ruleUsesRules[r.rule], ruleRef.Name.Val)
|
||||
if len(r.ruleUsesRules[r.rule]) == 0 {
|
||||
delete(r.ruleUsesRules, r.rule)
|
||||
}
|
||||
// TODO: Check if reference exists, otherwise raise an error, which reference is missing!
|
||||
return cloneExpr(r.rules[ruleRef.Name.Val].Expr)
|
||||
}
|
||||
}
|
||||
|
||||
// Remove Choices with only one Alternative left
|
||||
if choice, ok := expr.(*ChoiceExpr); ok {
|
||||
if len(choice.Alternatives) == 1 {
|
||||
r.optimized = true
|
||||
return choice.Alternatives[0]
|
||||
}
|
||||
}
|
||||
|
||||
// Remove Sequence with only one Expression
|
||||
if seq, ok := expr.(*SeqExpr); ok {
|
||||
if len(seq.Exprs) == 1 {
|
||||
r.optimized = true
|
||||
return seq.Exprs[0]
|
||||
}
|
||||
}
|
||||
|
||||
return expr
|
||||
}
|
||||
|
||||
// cloneExpr takes an Expression and deep clones it (including all children)
|
||||
// This is necessary because referenced Rules are denormalized and therefore
|
||||
// have to become independent from their original Expression.
|
||||
func cloneExpr(expr Expression) Expression {
|
||||
switch expr := expr.(type) {
|
||||
case *ActionExpr:
|
||||
return &ActionExpr{
|
||||
Code: expr.Code,
|
||||
Expr: cloneExpr(expr.Expr),
|
||||
FuncIx: expr.FuncIx,
|
||||
p: expr.p,
|
||||
}
|
||||
case *AndExpr:
|
||||
return &AndExpr{
|
||||
Expr: cloneExpr(expr.Expr),
|
||||
p: expr.p,
|
||||
}
|
||||
case *AndCodeExpr:
|
||||
return &AndCodeExpr{
|
||||
Code: expr.Code,
|
||||
FuncIx: expr.FuncIx,
|
||||
p: expr.p,
|
||||
}
|
||||
case *CharClassMatcher:
|
||||
return &CharClassMatcher{
|
||||
Chars: append([]rune{}, expr.Chars...),
|
||||
IgnoreCase: expr.IgnoreCase,
|
||||
Inverted: expr.Inverted,
|
||||
posValue: expr.posValue,
|
||||
Ranges: append([]rune{}, expr.Ranges...),
|
||||
UnicodeClasses: append([]string{}, expr.UnicodeClasses...),
|
||||
}
|
||||
case *ChoiceExpr:
|
||||
alts := make([]Expression, 0, len(expr.Alternatives))
|
||||
for i := 0; i < len(expr.Alternatives); i++ {
|
||||
alts = append(alts, cloneExpr(expr.Alternatives[i]))
|
||||
}
|
||||
return &ChoiceExpr{
|
||||
Alternatives: alts,
|
||||
p: expr.p,
|
||||
}
|
||||
case *LabeledExpr:
|
||||
return &LabeledExpr{
|
||||
Expr: cloneExpr(expr.Expr),
|
||||
Label: expr.Label,
|
||||
p: expr.p,
|
||||
}
|
||||
case *NotExpr:
|
||||
return &NotExpr{
|
||||
Expr: cloneExpr(expr.Expr),
|
||||
p: expr.p,
|
||||
}
|
||||
case *NotCodeExpr:
|
||||
return &NotCodeExpr{
|
||||
Code: expr.Code,
|
||||
FuncIx: expr.FuncIx,
|
||||
p: expr.p,
|
||||
}
|
||||
case *OneOrMoreExpr:
|
||||
return &OneOrMoreExpr{
|
||||
Expr: cloneExpr(expr.Expr),
|
||||
p: expr.p,
|
||||
}
|
||||
case *SeqExpr:
|
||||
exprs := make([]Expression, 0, len(expr.Exprs))
|
||||
for i := 0; i < len(expr.Exprs); i++ {
|
||||
exprs = append(exprs, cloneExpr(expr.Exprs[i]))
|
||||
}
|
||||
return &SeqExpr{
|
||||
Exprs: exprs,
|
||||
p: expr.p,
|
||||
}
|
||||
case *StateCodeExpr:
|
||||
return &StateCodeExpr{
|
||||
p: expr.p,
|
||||
Code: expr.Code,
|
||||
FuncIx: expr.FuncIx,
|
||||
}
|
||||
case *ZeroOrMoreExpr:
|
||||
return &ZeroOrMoreExpr{
|
||||
Expr: cloneExpr(expr.Expr),
|
||||
p: expr.p,
|
||||
}
|
||||
case *ZeroOrOneExpr:
|
||||
return &ZeroOrOneExpr{
|
||||
Expr: cloneExpr(expr.Expr),
|
||||
p: expr.p,
|
||||
}
|
||||
}
|
||||
return expr
|
||||
}
|
||||
|
||||
// cleanupCharClassMatcher is a Visitor, which is used with the Walk function
|
||||
// The purpose of this function is to cleanup the redundancies created by the
|
||||
// optimize Visitor. This includes to remove redundant entries in Chars, Ranges
|
||||
// and UnicodeClasses of the given CharClassMatcher as well as regenerating the
|
||||
// correct content for the Val field (string representation of the CharClassMatcher).
|
||||
func (r *grammarOptimizer) cleanupCharClassMatcher(expr0 Expression) Visitor {
|
||||
// We are only interested in nodes of type *CharClassMatcher
|
||||
if chr, ok := expr0.(*CharClassMatcher); ok {
|
||||
// Remove redundancies in Chars
|
||||
chars := make([]rune, 0, len(chr.Chars))
|
||||
charsMap := make(map[rune]struct{})
|
||||
for _, c := range chr.Chars {
|
||||
if _, ok := charsMap[c]; !ok {
|
||||
charsMap[c] = struct{}{}
|
||||
chars = append(chars, c)
|
||||
}
|
||||
}
|
||||
if len(chars) > 0 {
|
||||
chr.Chars = chars
|
||||
} else {
|
||||
chr.Chars = nil
|
||||
}
|
||||
|
||||
// Remove redundancies in Ranges
|
||||
ranges := make([]rune, 0, len(chr.Ranges))
|
||||
rangesMap := make(map[string]struct{})
|
||||
for i := 0; i < len(chr.Ranges); i += 2 {
|
||||
rangeKey := string(chr.Ranges[i]) + "-" + string(chr.Ranges[i+1])
|
||||
if _, ok := rangesMap[rangeKey]; !ok {
|
||||
rangesMap[rangeKey] = struct{}{}
|
||||
ranges = append(ranges, chr.Ranges[i], chr.Ranges[i+1])
|
||||
}
|
||||
}
|
||||
if len(ranges) > 0 {
|
||||
chr.Ranges = ranges
|
||||
} else {
|
||||
chr.Ranges = nil
|
||||
}
|
||||
|
||||
// Remove redundancies in UnicodeClasses
|
||||
unicodeClasses := make([]string, 0, len(chr.UnicodeClasses))
|
||||
unicodeClassesMap := make(map[string]struct{})
|
||||
for _, u := range chr.UnicodeClasses {
|
||||
if _, ok := unicodeClassesMap[u]; !ok {
|
||||
unicodeClassesMap[u] = struct{}{}
|
||||
unicodeClasses = append(unicodeClasses, u)
|
||||
}
|
||||
}
|
||||
if len(unicodeClasses) > 0 {
|
||||
chr.UnicodeClasses = unicodeClasses
|
||||
} else {
|
||||
chr.UnicodeClasses = nil
|
||||
}
|
||||
|
||||
// Regenerate the content for Val
|
||||
var val bytes.Buffer
|
||||
val.WriteString("[")
|
||||
if chr.Inverted {
|
||||
val.WriteString("^")
|
||||
}
|
||||
for _, c := range chr.Chars {
|
||||
val.WriteString(escapeRune(c))
|
||||
}
|
||||
for i := 0; i < len(chr.Ranges); i += 2 {
|
||||
val.WriteString(escapeRune(chr.Ranges[i]))
|
||||
val.WriteString("-")
|
||||
val.WriteString(escapeRune(chr.Ranges[i+1]))
|
||||
}
|
||||
for _, u := range chr.UnicodeClasses {
|
||||
val.WriteString("\\p" + u)
|
||||
}
|
||||
val.WriteString("]")
|
||||
if chr.IgnoreCase {
|
||||
val.WriteString("i")
|
||||
}
|
||||
chr.posValue.Val = val.String()
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
func escapeRune(r rune) string {
|
||||
return strings.Trim(strconv.QuoteRune(r), `'`)
|
||||
}
|
||||
|
||||
// Optimize walks a given grammar and optimizes the grammar in regards
|
||||
// of parsing performance. This is done with several optimizations:
|
||||
// - removal of unreferenced rules
|
||||
// - replace rule references with a copy of the referenced Rule, if the
|
||||
// referenced rule it self has no references.
|
||||
// - resolve nested choice expressions
|
||||
// - resolve choice expressions with only one alternative
|
||||
// - resolve nested sequences expression
|
||||
// - resolve sequence expressions with only one element
|
||||
// - combine character class matcher and literal matcher, where possible
|
||||
func Optimize(g *Grammar, alternateEntrypoints ...string) {
|
||||
entrypoints := alternateEntrypoints
|
||||
if len(g.Rules) > 0 {
|
||||
entrypoints = append(entrypoints, g.Rules[0].Name.Val)
|
||||
}
|
||||
|
||||
r := newGrammarOptimizer(entrypoints)
|
||||
Walk(r, g)
|
||||
|
||||
r.visitor = r.optimize
|
||||
r.optimized = true
|
||||
for r.optimized {
|
||||
Walk(r, g)
|
||||
}
|
||||
|
||||
r.visitor = r.cleanupCharClassMatcher
|
||||
Walk(r, g)
|
||||
}
|
||||
+86
@@ -0,0 +1,86 @@
|
||||
package ast
|
||||
|
||||
import "fmt"
|
||||
|
||||
// A Visitor implements a Visit method, which is invoked for each Expression
|
||||
// encountered by Walk.
|
||||
// If the result visitor w is not nil, Walk visits each of the children
|
||||
// of Expression with the visitor w, followed by a call of w.Visit(nil).
|
||||
type Visitor interface {
|
||||
Visit(expr Expression) (w Visitor)
|
||||
}
|
||||
|
||||
// Walk traverses an AST in depth-first order: It starts by calling
|
||||
// v.Visit(expr); Expression must not be nil. If the visitor w returned by
|
||||
// v.Visit(expr) is not nil, Walk is invoked recursively with visitor
|
||||
// w for each of the non-nil children of Expression, followed by a call of
|
||||
// w.Visit(nil).
|
||||
func Walk(v Visitor, expr Expression) {
|
||||
if v = v.Visit(expr); v == nil {
|
||||
return
|
||||
}
|
||||
|
||||
switch expr := expr.(type) {
|
||||
case *ActionExpr:
|
||||
Walk(v, expr.Expr)
|
||||
case *AndCodeExpr:
|
||||
// Nothing to do
|
||||
case *AndExpr:
|
||||
Walk(v, expr.Expr)
|
||||
case *AnyMatcher:
|
||||
// Nothing to do
|
||||
case *CharClassMatcher:
|
||||
// Nothing to do
|
||||
case *ChoiceExpr:
|
||||
for _, e := range expr.Alternatives {
|
||||
Walk(v, e)
|
||||
}
|
||||
case *Grammar:
|
||||
for _, e := range expr.Rules {
|
||||
Walk(v, e)
|
||||
}
|
||||
case *LabeledExpr:
|
||||
Walk(v, expr.Expr)
|
||||
case *LitMatcher:
|
||||
// Nothing to do
|
||||
case *NotCodeExpr:
|
||||
// Nothing to do
|
||||
case *NotExpr:
|
||||
Walk(v, expr.Expr)
|
||||
case *OneOrMoreExpr:
|
||||
Walk(v, expr.Expr)
|
||||
case *Rule:
|
||||
Walk(v, expr.Expr)
|
||||
case *RuleRefExpr:
|
||||
// Nothing to do
|
||||
case *SeqExpr:
|
||||
for _, e := range expr.Exprs {
|
||||
Walk(v, e)
|
||||
}
|
||||
case *StateCodeExpr:
|
||||
// Nothing to do
|
||||
case *ZeroOrMoreExpr:
|
||||
Walk(v, expr.Expr)
|
||||
case *ZeroOrOneExpr:
|
||||
Walk(v, expr.Expr)
|
||||
default:
|
||||
panic(fmt.Sprintf("unknown expression type %T", expr))
|
||||
}
|
||||
}
|
||||
|
||||
type inspector func(Expression) bool
|
||||
|
||||
func (f inspector) Visit(expr Expression) Visitor {
|
||||
if f(expr) {
|
||||
return f
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Inspect traverses an AST in depth-first order: It starts by calling
|
||||
// f(expr); expr must not be nil. If f returns true, Inspect invokes f
|
||||
// recursively for each of the non-nil children of expr, followed by a
|
||||
// call of f(nil).
|
||||
func Inspect(expr Expression, f func(Expression) bool) {
|
||||
Walk(inspector(f), expr)
|
||||
}
|
||||
+842
@@ -0,0 +1,842 @@
|
||||
// Package builder generates the parser code for a given grammar. It makes
|
||||
// no attempt to verify the correctness of the grammar.
|
||||
package builder
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"fmt"
|
||||
"io"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"text/template"
|
||||
"unicode"
|
||||
|
||||
"github.com/mna/pigeon/ast"
|
||||
)
|
||||
|
||||
const codeGeneratedComment = "// Code generated by pigeon; DO NOT EDIT.\n\n"
|
||||
|
||||
// generated function templates
|
||||
var (
|
||||
onFuncTemplate = `func (%s *current) %s(%s) (any, error) {
|
||||
%s
|
||||
}
|
||||
`
|
||||
onPredFuncTemplate = `func (%s *current) %s(%s) (bool, error) {
|
||||
%s
|
||||
}
|
||||
`
|
||||
onStateFuncTemplate = `func (%s *current) %s(%s) (error) {
|
||||
%s
|
||||
}
|
||||
`
|
||||
callFuncTemplate = `func (p *parser) call%s() (any, error) {
|
||||
stack := p.vstack[len(p.vstack)-1]
|
||||
_ = stack
|
||||
return p.cur.%[1]s(%s)
|
||||
}
|
||||
`
|
||||
callPredFuncTemplate = `func (p *parser) call%s() (bool, error) {
|
||||
stack := p.vstack[len(p.vstack)-1]
|
||||
_ = stack
|
||||
return p.cur.%[1]s(%s)
|
||||
}
|
||||
`
|
||||
callStateFuncTemplate = `func (p *parser) call%s() error {
|
||||
stack := p.vstack[len(p.vstack)-1]
|
||||
_ = stack
|
||||
return p.cur.%[1]s(%s)
|
||||
}
|
||||
`
|
||||
)
|
||||
|
||||
// Option is a function that can set an option on the builder. It returns
|
||||
// the previous setting as an Option.
|
||||
type Option func(*builder) Option
|
||||
|
||||
// ReceiverName returns an option that specifies the receiver name to
|
||||
// use for the current struct (which is the struct on which all code blocks
|
||||
// except the initializer are generated).
|
||||
func ReceiverName(nm string) Option {
|
||||
return func(b *builder) Option {
|
||||
prev := b.recvName
|
||||
b.recvName = nm
|
||||
return ReceiverName(prev)
|
||||
}
|
||||
}
|
||||
|
||||
// Optimize returns an option that specifies the optimize option
|
||||
// If optimize is true, the Debug and Memoize code is completely
|
||||
// removed from the resulting parser
|
||||
func Optimize(optimize bool) Option {
|
||||
return func(b *builder) Option {
|
||||
prev := b.optimize
|
||||
b.optimize = optimize
|
||||
return Optimize(prev)
|
||||
}
|
||||
}
|
||||
|
||||
// SupportLeftRecursion returns an option that specifies the supportLeftRecursion option.
|
||||
// If supportLeftRecursion is true, LeftRecursion code is added to the resulting parser.
|
||||
func SupportLeftRecursion(support bool) Option {
|
||||
return func(b *builder) Option {
|
||||
prev := b.supportLeftRecursion
|
||||
b.supportLeftRecursion = support
|
||||
return SupportLeftRecursion(prev)
|
||||
}
|
||||
}
|
||||
|
||||
// Nolint returns an option that specifies the nolint option
|
||||
// If nolint is true, special '// nolint: ...' comments are added
|
||||
// to the generated parser to suppress warnings by gometalinter or golangci-lint.
|
||||
func Nolint(nolint bool) Option {
|
||||
return func(b *builder) Option {
|
||||
prev := b.nolint
|
||||
b.nolint = nolint
|
||||
return Optimize(prev)
|
||||
}
|
||||
}
|
||||
|
||||
// BasicLatinLookupTable returns an option that specifies the basicLatinLookup option
|
||||
// If basicLatinLookup is true, a lookup slice for the first 128 chars of
|
||||
// the Unicode table (Basic Latin) is generated for each CharClassMatcher
|
||||
// to increase the character matching.
|
||||
func BasicLatinLookupTable(basicLatinLookupTable bool) Option {
|
||||
return func(b *builder) Option {
|
||||
prev := b.basicLatinLookupTable
|
||||
b.basicLatinLookupTable = basicLatinLookupTable
|
||||
return BasicLatinLookupTable(prev)
|
||||
}
|
||||
}
|
||||
|
||||
// BuildParser builds the PEG parser using the provider grammar. The code is
|
||||
// written to the specified w.
|
||||
func BuildParser(w io.Writer, g *ast.Grammar, opts ...Option) error {
|
||||
b := &builder{w: w, recvName: "c"}
|
||||
b.setOptions(opts)
|
||||
return b.buildParser(g)
|
||||
}
|
||||
|
||||
type builder struct {
|
||||
w io.Writer
|
||||
err error
|
||||
|
||||
// options
|
||||
recvName string
|
||||
optimize bool
|
||||
basicLatinLookupTable bool
|
||||
globalState bool
|
||||
nolint bool
|
||||
supportLeftRecursion bool
|
||||
haveLeftRecursion bool
|
||||
|
||||
ruleName string
|
||||
exprIndex int
|
||||
argsStack [][]string
|
||||
|
||||
rangeTable bool
|
||||
}
|
||||
|
||||
func (b *builder) setOptions(opts []Option) {
|
||||
for _, opt := range opts {
|
||||
opt(b)
|
||||
}
|
||||
}
|
||||
|
||||
func (b *builder) buildParser(grammar *ast.Grammar) error {
|
||||
haveLeftRecursion, err := PrepareGrammar(grammar)
|
||||
if err != nil {
|
||||
return fmt.Errorf("incorrect grammar: %w", err)
|
||||
}
|
||||
if !b.supportLeftRecursion && haveLeftRecursion {
|
||||
return fmt.Errorf("incorrect grammar: %w", ErrHaveLeftRecursion)
|
||||
}
|
||||
b.haveLeftRecursion = haveLeftRecursion
|
||||
|
||||
b.writeInit(grammar.Init)
|
||||
b.writeGrammar(grammar)
|
||||
for _, rule := range grammar.Rules {
|
||||
b.writeRuleCode(rule)
|
||||
}
|
||||
b.writeStaticCode()
|
||||
|
||||
return b.err
|
||||
}
|
||||
|
||||
func (b *builder) writeInit(init *ast.CodeBlock) {
|
||||
if init == nil {
|
||||
return
|
||||
}
|
||||
|
||||
// remove opening and closing braces
|
||||
val := codeGeneratedComment + init.Val[1:len(init.Val)-1]
|
||||
b.writelnf("%s", val)
|
||||
}
|
||||
|
||||
func (b *builder) writeGrammar(g *ast.Grammar) {
|
||||
// transform the ast grammar to the self-contained, no dependency version
|
||||
// of the parser-generator grammar.
|
||||
b.writelnf("var g = &grammar {")
|
||||
b.writelnf("\trules: []*rule{")
|
||||
for _, r := range g.Rules {
|
||||
b.writeRule(r)
|
||||
}
|
||||
b.writelnf("\t},")
|
||||
b.writelnf("}")
|
||||
}
|
||||
|
||||
func (b *builder) writeRule(r *ast.Rule) {
|
||||
if r == nil || r.Name == nil {
|
||||
return
|
||||
}
|
||||
|
||||
b.exprIndex = 0
|
||||
b.ruleName = r.Name.Val
|
||||
|
||||
b.writelnf("{")
|
||||
b.writelnf("\tname: %q,", r.Name.Val)
|
||||
if r.DisplayName != nil && r.DisplayName.Val != "" {
|
||||
b.writelnf("\tdisplayName: %q,", r.DisplayName.Val)
|
||||
}
|
||||
pos := r.Pos()
|
||||
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
|
||||
b.writef("\texpr: ")
|
||||
b.writeExpr(r.Expr)
|
||||
if b.haveLeftRecursion {
|
||||
b.writelnf("\tleader: %t,", r.Leader)
|
||||
b.writelnf("\tleftRecursive: %t,", r.LeftRecursive)
|
||||
}
|
||||
b.writelnf("},")
|
||||
}
|
||||
|
||||
func (b *builder) writeExpr(expr ast.Expression) {
|
||||
b.exprIndex++
|
||||
switch expr := expr.(type) {
|
||||
case *ast.ActionExpr:
|
||||
b.writeActionExpr(expr)
|
||||
case *ast.AndCodeExpr:
|
||||
b.writeAndCodeExpr(expr)
|
||||
case *ast.AndExpr:
|
||||
b.writeAndExpr(expr)
|
||||
case *ast.AnyMatcher:
|
||||
b.writeAnyMatcher(expr)
|
||||
case *ast.CharClassMatcher:
|
||||
b.writeCharClassMatcher(expr)
|
||||
case *ast.ChoiceExpr:
|
||||
b.writeChoiceExpr(expr)
|
||||
case *ast.LabeledExpr:
|
||||
b.writeLabeledExpr(expr)
|
||||
case *ast.LitMatcher:
|
||||
b.writeLitMatcher(expr)
|
||||
case *ast.NotCodeExpr:
|
||||
b.writeNotCodeExpr(expr)
|
||||
case *ast.NotExpr:
|
||||
b.writeNotExpr(expr)
|
||||
case *ast.OneOrMoreExpr:
|
||||
b.writeOneOrMoreExpr(expr)
|
||||
case *ast.RecoveryExpr:
|
||||
b.writeRecoveryExpr(expr)
|
||||
case *ast.RuleRefExpr:
|
||||
b.writeRuleRefExpr(expr)
|
||||
case *ast.SeqExpr:
|
||||
b.writeSeqExpr(expr)
|
||||
case *ast.StateCodeExpr:
|
||||
b.writeStateCodeExpr(expr)
|
||||
case *ast.ThrowExpr:
|
||||
b.writeThrowExpr(expr)
|
||||
case *ast.ZeroOrMoreExpr:
|
||||
b.writeZeroOrMoreExpr(expr)
|
||||
case *ast.ZeroOrOneExpr:
|
||||
b.writeZeroOrOneExpr(expr)
|
||||
default:
|
||||
b.err = fmt.Errorf("builder: unknown expression type %T", expr)
|
||||
}
|
||||
}
|
||||
|
||||
func (b *builder) writeActionExpr(act *ast.ActionExpr) {
|
||||
if act == nil {
|
||||
b.writelnf("nil,")
|
||||
return
|
||||
}
|
||||
if act.FuncIx == 0 {
|
||||
act.FuncIx = b.exprIndex
|
||||
}
|
||||
b.writelnf("&actionExpr{")
|
||||
pos := act.Pos()
|
||||
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
|
||||
b.writelnf("\trun: (*parser).call%s,", b.funcName(act.FuncIx))
|
||||
b.writef("\texpr: ")
|
||||
b.writeExpr(act.Expr)
|
||||
b.writelnf("},")
|
||||
}
|
||||
|
||||
func (b *builder) writeAndCodeExpr(and *ast.AndCodeExpr) {
|
||||
if and == nil {
|
||||
b.writelnf("nil,")
|
||||
return
|
||||
}
|
||||
b.writelnf("&andCodeExpr{")
|
||||
pos := and.Pos()
|
||||
if and.FuncIx == 0 {
|
||||
and.FuncIx = b.exprIndex
|
||||
}
|
||||
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
|
||||
b.writelnf("\trun: (*parser).call%s,", b.funcName(and.FuncIx))
|
||||
b.writelnf("},")
|
||||
}
|
||||
|
||||
func (b *builder) writeAndExpr(and *ast.AndExpr) {
|
||||
if and == nil {
|
||||
b.writelnf("nil,")
|
||||
return
|
||||
}
|
||||
b.writelnf("&andExpr{")
|
||||
pos := and.Pos()
|
||||
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
|
||||
b.writef("\texpr: ")
|
||||
b.writeExpr(and.Expr)
|
||||
b.writelnf("},")
|
||||
}
|
||||
|
||||
func (b *builder) writeAnyMatcher(any *ast.AnyMatcher) {
|
||||
if any == nil {
|
||||
b.writelnf("nil,")
|
||||
return
|
||||
}
|
||||
b.writelnf("&anyMatcher{")
|
||||
pos := any.Pos()
|
||||
b.writelnf("\tline: %d, col: %d, offset: %d,", pos.Line, pos.Col, pos.Off)
|
||||
b.writelnf("},")
|
||||
}
|
||||
|
||||
func (b *builder) writeCharClassMatcher(ch *ast.CharClassMatcher) {
|
||||
if ch == nil {
|
||||
b.writelnf("nil,")
|
||||
return
|
||||
}
|
||||
b.writelnf("&charClassMatcher{")
|
||||
pos := ch.Pos()
|
||||
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
|
||||
b.writelnf("\tval: %q,", ch.Val)
|
||||
if len(ch.Chars) > 0 {
|
||||
b.writef("\tchars: []rune{")
|
||||
for _, rn := range ch.Chars {
|
||||
if ch.IgnoreCase {
|
||||
b.writef("%q,", unicode.ToLower(rn))
|
||||
} else {
|
||||
b.writef("%q,", rn)
|
||||
}
|
||||
}
|
||||
b.writelnf("},")
|
||||
}
|
||||
if len(ch.Ranges) > 0 {
|
||||
b.writef("\tranges: []rune{")
|
||||
for _, rn := range ch.Ranges {
|
||||
if ch.IgnoreCase {
|
||||
b.writef("%q,", unicode.ToLower(rn))
|
||||
} else {
|
||||
b.writef("%q,", rn)
|
||||
}
|
||||
}
|
||||
b.writelnf("},")
|
||||
}
|
||||
if len(ch.UnicodeClasses) > 0 {
|
||||
b.rangeTable = true
|
||||
b.writef("\tclasses: []*unicode.RangeTable{")
|
||||
for _, cl := range ch.UnicodeClasses {
|
||||
b.writef("rangeTable(%q),", cl)
|
||||
}
|
||||
b.writelnf("},")
|
||||
}
|
||||
if b.basicLatinLookupTable {
|
||||
b.writelnf("\tbasicLatinChars: %#v,", BasicLatinLookup(ch.Chars, ch.Ranges, ch.UnicodeClasses, ch.IgnoreCase))
|
||||
}
|
||||
b.writelnf("\tignoreCase: %t,", ch.IgnoreCase)
|
||||
b.writelnf("\tinverted: %t,", ch.Inverted)
|
||||
b.writelnf("},")
|
||||
}
|
||||
|
||||
// BasicLatinLookup calculates the decision results for the first 256 characters of the UTF-8 character
|
||||
// set for a given set of chars, ranges and unicodeClasses to speedup the CharClassMatcher.
|
||||
func BasicLatinLookup(chars, ranges []rune, unicodeClasses []string, ignoreCase bool) (basicLatinChars [128]bool) {
|
||||
for _, rn := range chars {
|
||||
if rn < 128 {
|
||||
basicLatinChars[rn] = true
|
||||
if ignoreCase {
|
||||
if unicode.IsLower(rn) {
|
||||
basicLatinChars[unicode.ToUpper(rn)] = true
|
||||
} else {
|
||||
basicLatinChars[unicode.ToLower(rn)] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
for i := 0; i < len(ranges); i += 2 {
|
||||
if ranges[i] < 128 {
|
||||
for j := ranges[i]; j < 128 && j <= ranges[i+1]; j++ {
|
||||
basicLatinChars[j] = true
|
||||
if ignoreCase {
|
||||
if unicode.IsLower(j) {
|
||||
basicLatinChars[unicode.ToUpper(j)] = true
|
||||
} else {
|
||||
basicLatinChars[unicode.ToLower(j)] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, cl := range unicodeClasses {
|
||||
rt := rangeTable(cl)
|
||||
for r := rune(0); r < 128; r++ {
|
||||
if unicode.Is(rt, r) {
|
||||
basicLatinChars[r] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
func (b *builder) writeChoiceExpr(ch *ast.ChoiceExpr) {
|
||||
if ch == nil {
|
||||
b.writelnf("nil,")
|
||||
return
|
||||
}
|
||||
b.writelnf("&choiceExpr{")
|
||||
pos := ch.Pos()
|
||||
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
|
||||
if len(ch.Alternatives) > 0 {
|
||||
b.writelnf("\talternatives: []any{")
|
||||
for _, alt := range ch.Alternatives {
|
||||
b.writeExpr(alt)
|
||||
}
|
||||
b.writelnf("\t},")
|
||||
}
|
||||
b.writelnf("},")
|
||||
}
|
||||
|
||||
func (b *builder) writeLabeledExpr(lab *ast.LabeledExpr) {
|
||||
if lab == nil {
|
||||
b.writelnf("nil,")
|
||||
return
|
||||
}
|
||||
b.writelnf("&labeledExpr{")
|
||||
pos := lab.Pos()
|
||||
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
|
||||
if lab.Label != nil && lab.Label.Val != "" {
|
||||
b.writelnf("\tlabel: %q,", lab.Label.Val)
|
||||
}
|
||||
b.writef("\texpr: ")
|
||||
b.writeExpr(lab.Expr)
|
||||
b.writelnf("},")
|
||||
}
|
||||
|
||||
func (b *builder) writeLitMatcher(lit *ast.LitMatcher) {
|
||||
if lit == nil {
|
||||
b.writelnf("nil,")
|
||||
return
|
||||
}
|
||||
b.writelnf("&litMatcher{")
|
||||
pos := lit.Pos()
|
||||
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
|
||||
if lit.IgnoreCase {
|
||||
b.writelnf("\tval: %q,", strings.ToLower(lit.Val))
|
||||
} else {
|
||||
b.writelnf("\tval: %q,", lit.Val)
|
||||
}
|
||||
b.writelnf("\tignoreCase: %t,", lit.IgnoreCase)
|
||||
ignoreCaseFlag := ""
|
||||
if lit.IgnoreCase {
|
||||
ignoreCaseFlag = "i"
|
||||
}
|
||||
b.writelnf("\twant: %q,", strconv.Quote(lit.Val)+ignoreCaseFlag)
|
||||
b.writelnf("},")
|
||||
}
|
||||
|
||||
func (b *builder) writeNotCodeExpr(not *ast.NotCodeExpr) {
|
||||
if not == nil {
|
||||
b.writelnf("nil,")
|
||||
return
|
||||
}
|
||||
b.writelnf("¬CodeExpr{")
|
||||
pos := not.Pos()
|
||||
if not.FuncIx == 0 {
|
||||
not.FuncIx = b.exprIndex
|
||||
}
|
||||
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
|
||||
b.writelnf("\trun: (*parser).call%s,", b.funcName(not.FuncIx))
|
||||
b.writelnf("},")
|
||||
}
|
||||
|
||||
func (b *builder) writeNotExpr(not *ast.NotExpr) {
|
||||
if not == nil {
|
||||
b.writelnf("nil,")
|
||||
return
|
||||
}
|
||||
b.writelnf("¬Expr{")
|
||||
pos := not.Pos()
|
||||
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
|
||||
b.writef("\texpr: ")
|
||||
b.writeExpr(not.Expr)
|
||||
b.writelnf("},")
|
||||
}
|
||||
|
||||
func (b *builder) writeOneOrMoreExpr(one *ast.OneOrMoreExpr) {
|
||||
if one == nil {
|
||||
b.writelnf("nil,")
|
||||
return
|
||||
}
|
||||
b.writelnf("&oneOrMoreExpr{")
|
||||
pos := one.Pos()
|
||||
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
|
||||
b.writef("\texpr: ")
|
||||
b.writeExpr(one.Expr)
|
||||
b.writelnf("},")
|
||||
}
|
||||
|
||||
func (b *builder) writeRecoveryExpr(recover *ast.RecoveryExpr) {
|
||||
if recover == nil {
|
||||
b.writelnf("nil,")
|
||||
return
|
||||
}
|
||||
b.writelnf("&recoveryExpr{")
|
||||
pos := recover.Pos()
|
||||
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
|
||||
|
||||
b.writef("\texpr: ")
|
||||
b.writeExpr(recover.Expr)
|
||||
b.writef("\trecoverExpr: ")
|
||||
b.writeExpr(recover.RecoverExpr)
|
||||
b.writelnf("\tfailureLabel: []string{")
|
||||
for _, label := range recover.Labels {
|
||||
b.writelnf("%q,", label)
|
||||
}
|
||||
b.writelnf("\t},")
|
||||
b.writelnf("},")
|
||||
}
|
||||
|
||||
func (b *builder) writeRuleRefExpr(ref *ast.RuleRefExpr) {
|
||||
if ref == nil {
|
||||
b.writelnf("nil,")
|
||||
return
|
||||
}
|
||||
b.writelnf("&ruleRefExpr{")
|
||||
pos := ref.Pos()
|
||||
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
|
||||
if ref.Name != nil && ref.Name.Val != "" {
|
||||
b.writelnf("\tname: %q,", ref.Name.Val)
|
||||
}
|
||||
b.writelnf("},")
|
||||
}
|
||||
|
||||
func (b *builder) writeSeqExpr(seq *ast.SeqExpr) {
|
||||
if seq == nil {
|
||||
b.writelnf("nil,")
|
||||
return
|
||||
}
|
||||
b.writelnf("&seqExpr{")
|
||||
pos := seq.Pos()
|
||||
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
|
||||
if len(seq.Exprs) > 0 {
|
||||
b.writelnf("\texprs: []any{")
|
||||
for _, e := range seq.Exprs {
|
||||
b.writeExpr(e)
|
||||
}
|
||||
b.writelnf("\t},")
|
||||
}
|
||||
b.writelnf("},")
|
||||
}
|
||||
|
||||
func (b *builder) writeStateCodeExpr(state *ast.StateCodeExpr) {
|
||||
if state == nil {
|
||||
b.writelnf("nil,")
|
||||
return
|
||||
}
|
||||
b.globalState = true
|
||||
b.writelnf("&stateCodeExpr{")
|
||||
pos := state.Pos()
|
||||
if state.FuncIx == 0 {
|
||||
state.FuncIx = b.exprIndex
|
||||
}
|
||||
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
|
||||
b.writelnf("\trun: (*parser).call%s,", b.funcName(state.FuncIx))
|
||||
b.writelnf("},")
|
||||
}
|
||||
|
||||
func (b *builder) writeThrowExpr(throw *ast.ThrowExpr) {
|
||||
if throw == nil {
|
||||
b.writelnf("nil,")
|
||||
return
|
||||
}
|
||||
b.writelnf("&throwExpr{")
|
||||
pos := throw.Pos()
|
||||
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
|
||||
b.writelnf("\tlabel: %q,", throw.Label)
|
||||
b.writelnf("},")
|
||||
}
|
||||
|
||||
func (b *builder) writeZeroOrMoreExpr(zero *ast.ZeroOrMoreExpr) {
|
||||
if zero == nil {
|
||||
b.writelnf("nil,")
|
||||
return
|
||||
}
|
||||
b.writelnf("&zeroOrMoreExpr{")
|
||||
pos := zero.Pos()
|
||||
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
|
||||
b.writef("\texpr: ")
|
||||
b.writeExpr(zero.Expr)
|
||||
b.writelnf("},")
|
||||
}
|
||||
|
||||
func (b *builder) writeZeroOrOneExpr(zero *ast.ZeroOrOneExpr) {
|
||||
if zero == nil {
|
||||
b.writelnf("nil,")
|
||||
return
|
||||
}
|
||||
b.writelnf("&zeroOrOneExpr{")
|
||||
pos := zero.Pos()
|
||||
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
|
||||
b.writef("\texpr: ")
|
||||
b.writeExpr(zero.Expr)
|
||||
b.writelnf("},")
|
||||
}
|
||||
|
||||
func (b *builder) writeRuleCode(rule *ast.Rule) {
|
||||
if rule == nil || rule.Name == nil {
|
||||
return
|
||||
}
|
||||
|
||||
// keep trace of the current rule, as the code blocks are created
|
||||
// in functions named "on<RuleName><#ExprIndex>".
|
||||
b.ruleName = rule.Name.Val
|
||||
b.pushArgsSet()
|
||||
b.writeExprCode(rule.Expr)
|
||||
b.popArgsSet()
|
||||
}
|
||||
|
||||
func (b *builder) pushArgsSet() {
|
||||
b.argsStack = append(b.argsStack, nil)
|
||||
}
|
||||
|
||||
func (b *builder) popArgsSet() {
|
||||
b.argsStack = b.argsStack[:len(b.argsStack)-1]
|
||||
}
|
||||
|
||||
func (b *builder) addArg(arg *ast.Identifier) {
|
||||
if arg == nil {
|
||||
return
|
||||
}
|
||||
ix := len(b.argsStack) - 1
|
||||
b.argsStack[ix] = append(b.argsStack[ix], arg.Val)
|
||||
}
|
||||
|
||||
func (b *builder) writeExprCode(expr ast.Expression) {
|
||||
switch expr := expr.(type) {
|
||||
case *ast.ActionExpr:
|
||||
b.writeExprCode(expr.Expr)
|
||||
b.writeActionExprCode(expr)
|
||||
|
||||
case *ast.AndCodeExpr:
|
||||
b.writeAndCodeExprCode(expr)
|
||||
|
||||
case *ast.LabeledExpr:
|
||||
b.addArg(expr.Label)
|
||||
b.pushArgsSet()
|
||||
b.writeExprCode(expr.Expr)
|
||||
b.popArgsSet()
|
||||
|
||||
case *ast.NotCodeExpr:
|
||||
b.writeNotCodeExprCode(expr)
|
||||
|
||||
case *ast.AndExpr:
|
||||
b.pushArgsSet()
|
||||
b.writeExprCode(expr.Expr)
|
||||
b.popArgsSet()
|
||||
|
||||
case *ast.ChoiceExpr:
|
||||
for _, alt := range expr.Alternatives {
|
||||
b.pushArgsSet()
|
||||
b.writeExprCode(alt)
|
||||
b.popArgsSet()
|
||||
}
|
||||
|
||||
case *ast.NotExpr:
|
||||
b.pushArgsSet()
|
||||
b.writeExprCode(expr.Expr)
|
||||
b.popArgsSet()
|
||||
|
||||
case *ast.OneOrMoreExpr:
|
||||
b.pushArgsSet()
|
||||
b.writeExprCode(expr.Expr)
|
||||
b.popArgsSet()
|
||||
|
||||
case *ast.RecoveryExpr:
|
||||
b.pushArgsSet()
|
||||
b.writeExprCode(expr.Expr)
|
||||
b.writeExprCode(expr.RecoverExpr)
|
||||
b.popArgsSet()
|
||||
|
||||
case *ast.SeqExpr:
|
||||
for _, sub := range expr.Exprs {
|
||||
b.writeExprCode(sub)
|
||||
}
|
||||
|
||||
case *ast.StateCodeExpr:
|
||||
b.writeStateCodeExprCode(expr)
|
||||
|
||||
case *ast.ZeroOrMoreExpr:
|
||||
b.pushArgsSet()
|
||||
b.writeExprCode(expr.Expr)
|
||||
b.popArgsSet()
|
||||
|
||||
case *ast.ZeroOrOneExpr:
|
||||
b.pushArgsSet()
|
||||
b.writeExprCode(expr.Expr)
|
||||
b.popArgsSet()
|
||||
}
|
||||
}
|
||||
|
||||
func (b *builder) writeActionExprCode(act *ast.ActionExpr) {
|
||||
if act == nil {
|
||||
return
|
||||
}
|
||||
if act.FuncIx > 0 {
|
||||
b.writeFunc(act.FuncIx, act.Code, callFuncTemplate, onFuncTemplate)
|
||||
act.FuncIx = 0 // already rendered, prevent duplicates
|
||||
}
|
||||
}
|
||||
|
||||
func (b *builder) writeAndCodeExprCode(and *ast.AndCodeExpr) {
|
||||
if and == nil {
|
||||
return
|
||||
}
|
||||
if and.FuncIx > 0 {
|
||||
b.writeFunc(and.FuncIx, and.Code, callPredFuncTemplate, onPredFuncTemplate)
|
||||
and.FuncIx = 0 // already rendered, prevent duplicates
|
||||
}
|
||||
}
|
||||
|
||||
func (b *builder) writeNotCodeExprCode(not *ast.NotCodeExpr) {
|
||||
if not == nil {
|
||||
return
|
||||
}
|
||||
if not.FuncIx > 0 {
|
||||
b.writeFunc(not.FuncIx, not.Code, callPredFuncTemplate, onPredFuncTemplate)
|
||||
not.FuncIx = 0 // already rendered, prevent duplicates
|
||||
}
|
||||
}
|
||||
|
||||
func (b *builder) writeStateCodeExprCode(state *ast.StateCodeExpr) {
|
||||
if state == nil {
|
||||
return
|
||||
}
|
||||
if state.FuncIx > 0 {
|
||||
b.writeFunc(state.FuncIx, state.Code, callStateFuncTemplate, onStateFuncTemplate)
|
||||
state.FuncIx = 0 // already rendered, prevent duplicates
|
||||
}
|
||||
}
|
||||
|
||||
func (b *builder) writeFunc(funcIx int, code *ast.CodeBlock, callTpl, funcTpl string) {
|
||||
if code == nil {
|
||||
return
|
||||
}
|
||||
val := strings.TrimSpace(code.Val)[1 : len(code.Val)-1]
|
||||
if len(val) > 0 && val[0] == '\n' {
|
||||
val = val[1:]
|
||||
}
|
||||
if len(val) > 0 && val[len(val)-1] == '\n' {
|
||||
val = val[:len(val)-1]
|
||||
}
|
||||
var args bytes.Buffer
|
||||
ix := len(b.argsStack) - 1
|
||||
if ix >= 0 {
|
||||
for i, arg := range b.argsStack[ix] {
|
||||
if i > 0 {
|
||||
args.WriteString(", ")
|
||||
}
|
||||
args.WriteString(arg)
|
||||
}
|
||||
}
|
||||
if args.Len() > 0 {
|
||||
args.WriteString(" any")
|
||||
}
|
||||
|
||||
fnNm := b.funcName(funcIx)
|
||||
b.writelnf(funcTpl, b.recvName, fnNm, args.String(), val)
|
||||
|
||||
args.Reset()
|
||||
if ix >= 0 {
|
||||
for i, arg := range b.argsStack[ix] {
|
||||
if i > 0 {
|
||||
args.WriteString(", ")
|
||||
}
|
||||
args.WriteString(fmt.Sprintf(`stack[%q]`, arg))
|
||||
}
|
||||
}
|
||||
b.writelnf(callTpl, fnNm, args.String())
|
||||
}
|
||||
|
||||
func (b *builder) writeStaticCode() {
|
||||
buffer := bytes.NewBufferString("")
|
||||
params := struct {
|
||||
Optimize bool
|
||||
BasicLatinLookupTable bool
|
||||
GlobalState bool
|
||||
LeftRecursion bool
|
||||
Nolint bool
|
||||
}{
|
||||
Optimize: b.optimize,
|
||||
BasicLatinLookupTable: b.basicLatinLookupTable,
|
||||
GlobalState: b.globalState,
|
||||
LeftRecursion: b.haveLeftRecursion,
|
||||
Nolint: b.nolint,
|
||||
}
|
||||
t := template.Must(template.New("static_code").Parse(staticCode))
|
||||
|
||||
err := t.Execute(buffer, params)
|
||||
if err != nil {
|
||||
// This is very unlikely to ever happen
|
||||
panic("executing template: " + err.Error())
|
||||
}
|
||||
|
||||
// Clean the ==template== comments from the generated parser
|
||||
lines := strings.Split(buffer.String(), "\n")
|
||||
buffer.Reset()
|
||||
re := regexp.MustCompile(`^\s*//\s*(==template==\s*)+$`)
|
||||
reLineEnd := regexp.MustCompile(`//\s*==template==\s*$`)
|
||||
for _, line := range lines {
|
||||
if !re.MatchString(line) {
|
||||
line = reLineEnd.ReplaceAllString(line, "")
|
||||
_, err := buffer.WriteString(line + "\n")
|
||||
if err != nil {
|
||||
// This is very unlikely to ever happen
|
||||
panic("unable to write to byte buffer: " + err.Error())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
b.writeln(buffer.String())
|
||||
if b.rangeTable {
|
||||
b.writeln(rangeTable0)
|
||||
}
|
||||
}
|
||||
|
||||
func (b *builder) funcName(ix int) string {
|
||||
return "on" + b.ruleName + strconv.Itoa(ix)
|
||||
}
|
||||
|
||||
func (b *builder) writef(f string, args ...any) {
|
||||
if b.err == nil {
|
||||
_, b.err = fmt.Fprintf(b.w, f, args...)
|
||||
}
|
||||
}
|
||||
|
||||
func (b *builder) writelnf(f string, args ...any) {
|
||||
b.writef(f+"\n", args...)
|
||||
}
|
||||
|
||||
func (b *builder) writeln(f string) {
|
||||
if b.err == nil {
|
||||
_, b.err = fmt.Fprint(b.w, f+"\n")
|
||||
}
|
||||
}
|
||||
+1592
File diff suppressed because it is too large
Load Diff
+21
@@ -0,0 +1,21 @@
|
||||
// Code generated by static_code_generator with go generate; DO NOT EDIT.
|
||||
|
||||
package builder
|
||||
|
||||
var rangeTable0 = `
|
||||
func rangeTable(class string) *unicode.RangeTable {
|
||||
if rt, ok := unicode.Categories[class]; ok {
|
||||
return rt
|
||||
}
|
||||
if rt, ok := unicode.Properties[class]; ok {
|
||||
return rt
|
||||
}
|
||||
if rt, ok := unicode.Scripts[class]; ok {
|
||||
return rt
|
||||
}
|
||||
|
||||
// cannot happen
|
||||
panic(fmt.Sprintf("invalid Unicode class: %s", class))
|
||||
}
|
||||
|
||||
`
|
||||
+132
@@ -0,0 +1,132 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
|
||||
"github.com/mna/pigeon/ast"
|
||||
)
|
||||
|
||||
var (
|
||||
// ErrNoLeader is no leader error.
|
||||
ErrNoLeader = errors.New(
|
||||
"SCC has no leadership candidate (no element is included in all cycles)")
|
||||
// ErrHaveLeftRecursion is recursion error.
|
||||
ErrHaveLeftRecursion = errors.New("grammar contains left recursion")
|
||||
)
|
||||
|
||||
// PrepareGrammar evaluates parameters associated with left recursion.
|
||||
func PrepareGrammar(grammar *ast.Grammar) (bool, error) {
|
||||
mapRules := make(map[string]*ast.Rule, len(grammar.Rules))
|
||||
for _, rule := range grammar.Rules {
|
||||
mapRules[rule.Name.Val] = rule
|
||||
}
|
||||
ComputeNullables(mapRules)
|
||||
haveLeftRecursion, err := ComputeLeftRecursives(mapRules)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("error compute left recursive: %w", err)
|
||||
}
|
||||
return haveLeftRecursion, nil
|
||||
}
|
||||
|
||||
// ComputeNullables evaluates nullable nodes.
|
||||
func ComputeNullables(rules map[string]*ast.Rule) {
|
||||
// Compute which rules in a grammar are nullable
|
||||
for _, rule := range rules {
|
||||
rule.NullableVisit(rules)
|
||||
}
|
||||
}
|
||||
|
||||
func findLeader(
|
||||
graph map[string]map[string]struct{}, scc map[string]struct{},
|
||||
) (string, error) {
|
||||
// Try to find a leader such that all cycles go through it.
|
||||
leaders := make(map[string]struct{}, len(scc))
|
||||
for k := range scc {
|
||||
leaders[k] = struct{}{}
|
||||
}
|
||||
for start := range scc {
|
||||
cycles, err := FindCyclesInSCC(graph, scc, start)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("error find cycles: %w", err)
|
||||
}
|
||||
for _, cycle := range cycles {
|
||||
mapCycle := make(map[string]struct{}, len(cycle))
|
||||
for _, k := range cycle {
|
||||
mapCycle[k] = struct{}{}
|
||||
}
|
||||
for k := range scc {
|
||||
if _, okCycle := mapCycle[k]; !okCycle {
|
||||
delete(leaders, k)
|
||||
}
|
||||
}
|
||||
if len(leaders) == 0 {
|
||||
return "", ErrNoLeader
|
||||
}
|
||||
}
|
||||
}
|
||||
// Pick an arbitrary leader from the candidates.
|
||||
var leader string
|
||||
for k := range leaders {
|
||||
leader = k // The only element.
|
||||
break
|
||||
}
|
||||
return leader, nil
|
||||
}
|
||||
|
||||
// ComputeLeftRecursives evaluates left recursion.
|
||||
func ComputeLeftRecursives(rules map[string]*ast.Rule) (bool, error) {
|
||||
graph := MakeFirstGraph(rules)
|
||||
vertices := make([]string, 0, len(graph))
|
||||
haveLeftRecursion := false
|
||||
for k := range graph {
|
||||
vertices = append(vertices, k)
|
||||
}
|
||||
sccs := StronglyConnectedComponents(vertices, graph)
|
||||
for _, scc := range sccs {
|
||||
if len(scc) > 1 {
|
||||
for name := range scc {
|
||||
rules[name].LeftRecursive = true
|
||||
haveLeftRecursion = true
|
||||
}
|
||||
leader, err := findLeader(graph, scc)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("error find leader %v: %w", scc, err)
|
||||
}
|
||||
rules[leader].Leader = true
|
||||
} else {
|
||||
var name string
|
||||
for k := range scc {
|
||||
name = k // The only element.
|
||||
break
|
||||
}
|
||||
if _, ok := graph[name][name]; ok {
|
||||
rules[name].LeftRecursive = true
|
||||
rules[name].Leader = true
|
||||
haveLeftRecursion = true
|
||||
}
|
||||
}
|
||||
}
|
||||
return haveLeftRecursion, nil
|
||||
}
|
||||
|
||||
// MakeFirstGraph compute the graph of left-invocations.
|
||||
// There's an edge from A to B if A may invoke B at its initial position.
|
||||
// Note that this requires the nullable flags to have been computed.
|
||||
func MakeFirstGraph(rules map[string]*ast.Rule) map[string]map[string]struct{} {
|
||||
graph := make(map[string]map[string]struct{})
|
||||
vertices := make(map[string]struct{})
|
||||
for rulename, rule := range rules {
|
||||
names := rule.InitialNames()
|
||||
graph[rulename] = names
|
||||
for name := range names {
|
||||
vertices[name] = struct{}{}
|
||||
}
|
||||
}
|
||||
for vertex := range vertices {
|
||||
if _, ok := graph[vertex]; !ok {
|
||||
graph[vertex] = make(map[string]struct{})
|
||||
}
|
||||
}
|
||||
return graph
|
||||
}
|
||||
+151
@@ -0,0 +1,151 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// ErrInvalidParameters is parameters error.
|
||||
var ErrInvalidParameters = errors.New("invalid parameters passed to function")
|
||||
|
||||
func min(a1 int, a2 int) int {
|
||||
if a1 <= a2 {
|
||||
return a1
|
||||
}
|
||||
return a2
|
||||
}
|
||||
|
||||
// StronglyConnectedComponents compute strongly сonnected сomponents of a graph.
|
||||
// Tarjan's strongly connected components algorithm.
|
||||
func StronglyConnectedComponents(
|
||||
vertices []string, edges map[string]map[string]struct{},
|
||||
) []map[string]struct{} {
|
||||
// Tarjan's strongly connected components algorithm
|
||||
var (
|
||||
identified = map[string]struct{}{}
|
||||
stack = []string{}
|
||||
index = map[string]int{}
|
||||
lowlink = map[string]int{}
|
||||
dfs func(v string) []map[string]struct{}
|
||||
)
|
||||
|
||||
dfs = func(vertex string) []map[string]struct{} {
|
||||
index[vertex] = len(stack)
|
||||
stack = append(stack, vertex)
|
||||
lowlink[vertex] = index[vertex]
|
||||
|
||||
sccs := []map[string]struct{}{}
|
||||
for w := range edges[vertex] {
|
||||
if _, ok := index[w]; !ok {
|
||||
sccs = append(sccs, dfs(w)...)
|
||||
lowlink[vertex] = min(lowlink[vertex], lowlink[w])
|
||||
} else if _, ok := identified[w]; !ok {
|
||||
lowlink[vertex] = min(lowlink[vertex], lowlink[w])
|
||||
}
|
||||
}
|
||||
|
||||
if lowlink[vertex] == index[vertex] {
|
||||
scc := map[string]struct{}{}
|
||||
for _, v := range stack[index[vertex]:] {
|
||||
scc[v] = struct{}{}
|
||||
}
|
||||
stack = stack[:index[vertex]]
|
||||
for v := range scc {
|
||||
identified[v] = struct{}{}
|
||||
}
|
||||
sccs = append(sccs, scc)
|
||||
}
|
||||
return sccs
|
||||
}
|
||||
|
||||
sccs := []map[string]struct{}{}
|
||||
for _, v := range vertices {
|
||||
if _, ok := index[v]; !ok {
|
||||
sccs = append(sccs, dfs(v)...)
|
||||
}
|
||||
}
|
||||
return sccs
|
||||
}
|
||||
|
||||
func contains(s []string, e string) bool {
|
||||
for _, a := range s {
|
||||
if a == e {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func reduceGraph(
|
||||
graph map[string]map[string]struct{}, scc map[string]struct{},
|
||||
) map[string]map[string]struct{} {
|
||||
reduceGraph := map[string]map[string]struct{}{}
|
||||
for src, dsts := range graph {
|
||||
if _, ok := scc[src]; !ok {
|
||||
continue
|
||||
}
|
||||
reduceGraph[src] = map[string]struct{}{}
|
||||
for dst := range dsts {
|
||||
if _, ok := scc[dst]; !ok {
|
||||
continue
|
||||
}
|
||||
reduceGraph[src][dst] = struct{}{}
|
||||
}
|
||||
}
|
||||
return reduceGraph
|
||||
}
|
||||
|
||||
// FindCyclesInSCC find cycles in SCC emanating from start.
|
||||
// Yields lists of the form ['A', 'B', 'C', 'A'], which means there's
|
||||
// a path from A -> B -> C -> A. The first item is always the start
|
||||
// argument, but the last item may be another element, e.g. ['A',
|
||||
// 'B', 'C', 'B'] means there's a path from A to B and there's a
|
||||
// cycle from B to C and back.
|
||||
func FindCyclesInSCC(
|
||||
graph map[string]map[string]struct{}, scc map[string]struct{}, start string,
|
||||
) ([][]string, error) {
|
||||
// Basic input checks.
|
||||
if _, ok := scc[start]; !ok {
|
||||
return nil, fmt.Errorf(
|
||||
"%w: scc %v does not contain %q", ErrInvalidParameters, scc, start)
|
||||
}
|
||||
extravertices := []string{}
|
||||
for k := range scc {
|
||||
if _, ok := graph[k]; !ok {
|
||||
extravertices = append(extravertices, k)
|
||||
}
|
||||
}
|
||||
if len(extravertices) != 0 {
|
||||
return nil, fmt.Errorf(
|
||||
"%w: graph does not contain scc. %v",
|
||||
ErrInvalidParameters, extravertices)
|
||||
}
|
||||
|
||||
// Reduce the graph to nodes in the SCC.
|
||||
graph = reduceGraph(graph, scc)
|
||||
if _, ok := graph[start]; !ok {
|
||||
return nil, fmt.Errorf(
|
||||
"%w: graph %v does not contain %q",
|
||||
ErrInvalidParameters, graph, start)
|
||||
}
|
||||
|
||||
// Recursive helper that yields cycles.
|
||||
var dfs func(node string, path []string) [][]string
|
||||
dfs = func(node string, path []string) [][]string {
|
||||
ret := [][]string{}
|
||||
if contains(path, node) {
|
||||
t := make([]string, 0, len(path)+1)
|
||||
t = append(t, path...)
|
||||
t = append(t, node)
|
||||
ret = append(ret, t)
|
||||
return ret
|
||||
}
|
||||
path = append(path, node) // TODO: Make this not quadratic.
|
||||
for child := range graph[node] {
|
||||
ret = append(ret, dfs(child, path)...)
|
||||
}
|
||||
return ret
|
||||
}
|
||||
|
||||
return dfs(start, []string{}), nil
|
||||
}
|
||||
+1608
File diff suppressed because it is too large
Load Diff
+24
@@ -0,0 +1,24 @@
|
||||
//go:generate go run ../bootstrap/cmd/static_code_generator/main.go -- $GOFILE generated_$GOFILE rangeTable0
|
||||
|
||||
package builder
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"unicode"
|
||||
)
|
||||
|
||||
// IMPORTANT: All code below this line is added to the parser as static code
|
||||
func rangeTable(class string) *unicode.RangeTable {
|
||||
if rt, ok := unicode.Categories[class]; ok {
|
||||
return rt
|
||||
}
|
||||
if rt, ok := unicode.Properties[class]; ok {
|
||||
return rt
|
||||
}
|
||||
if rt, ok := unicode.Scripts[class]; ok {
|
||||
return rt
|
||||
}
|
||||
|
||||
// cannot happen
|
||||
panic(fmt.Sprintf("invalid Unicode class: %s", class))
|
||||
}
|
||||
+618
@@ -0,0 +1,618 @@
|
||||
/*
|
||||
Command pigeon generates parsers in Go from a PEG grammar.
|
||||
|
||||
From Wikipedia [0]:
|
||||
|
||||
A parsing expression grammar is a type of analytic formal grammar, i.e.
|
||||
it describes a formal language in terms of a set of rules for recognizing
|
||||
strings in the language.
|
||||
|
||||
Its features and syntax are inspired by the PEG.js project [1], while
|
||||
the implementation is loosely based on [2]. Formal presentation of the
|
||||
PEG theory by Bryan Ford is also an important reference [3]. An introductory
|
||||
blog post can be found at [4].
|
||||
|
||||
[0]: http://en.wikipedia.org/wiki/Parsing_expression_grammar
|
||||
[1]: http://pegjs.org/
|
||||
[2]: http://www.codeproject.com/Articles/29713/Parsing-Expression-Grammar-Support-for-C-Part
|
||||
[3]: http://pdos.csail.mit.edu/~baford/packrat/popl04/peg-popl04.pdf
|
||||
[4]: http://0value.com/A-PEG-parser-generator-for-Go
|
||||
|
||||
Command-line usage
|
||||
|
||||
The pigeon tool must be called with PEG input as defined
|
||||
by the accepted PEG syntax below. The grammar may be provided by a
|
||||
file or read from stdin. The generated parser is written to stdout
|
||||
by default.
|
||||
|
||||
pigeon [options] [GRAMMAR_FILE]
|
||||
|
||||
The following options can be specified:
|
||||
|
||||
-cache : cache parser results to avoid exponential parsing time in
|
||||
pathological cases. Can make the parsing slower for typical
|
||||
cases and uses more memory (default: false).
|
||||
|
||||
-debug : boolean, print debugging info to stdout (default: false).
|
||||
|
||||
-nolint: add '// nolint: ...' comments for generated parser to suppress
|
||||
warnings by gometalinter (https://github.com/alecthomas/gometalinter) or
|
||||
golangci-lint (https://golangci-lint.run/).
|
||||
|
||||
-no-recover : boolean, if set, do not recover from a panic. Useful
|
||||
to access the panic stack when debugging, otherwise the panic
|
||||
is converted to an error (default: false).
|
||||
|
||||
-o=FILE : string, output file where the generated parser will be
|
||||
written (default: stdout).
|
||||
|
||||
-optimize-basic-latin : boolean, if set, a lookup table for the first 128
|
||||
characters of the Unicode table (Basic Latin) is generated for each character
|
||||
class matcher. This speeds up the parsing, if parsed data mainly consists
|
||||
of characters from this range (default: false).
|
||||
|
||||
-optimize-grammar : boolean, (EXPERIMENTAL FEATURE) if set, several performance
|
||||
optimizations on the grammar are performed, with focus to the reduction of the
|
||||
grammar depth.
|
||||
Optimization:
|
||||
* removal of unreferenced rules
|
||||
* replace rule references with a copy of the referenced Rule, if the
|
||||
referenced rule it self has no references.
|
||||
* resolve nested choice expressions
|
||||
* resolve choice expressions with only one alternative
|
||||
* resolve nested sequences expression
|
||||
* resolve sequence expressions with only one element
|
||||
* combine character class matcher and literal matcher, where possible
|
||||
The resulting grammar is usually more memory consuming, but faster for parsing.
|
||||
The optimization of the grammar is done in multiple rounds (optimize until no
|
||||
more optimizations have applied). This process takes some time, depending on the
|
||||
optimization potential of the grammar.
|
||||
|
||||
-optimize-parser : boolean, if set, the options Debug, Memoize and Statistics are
|
||||
removed from the resulting parser. The global "state" is optimized as well by
|
||||
either removing all related code if no state change expression is present in the
|
||||
grammar or by removing the restoration of the global "state" store after action
|
||||
and predicate code blocks. This saves a few cpu cycles, when using the generated
|
||||
parser (default: false).
|
||||
|
||||
-x : boolean, if set, do not build the parser, just parse the input grammar
|
||||
(default: false).
|
||||
|
||||
-receiver-name=NAME : string, name of the receiver variable for the generated
|
||||
code blocks. Non-initializer code blocks in the grammar end up as methods on the
|
||||
*current type, and this option sets the name of the receiver (default: c).
|
||||
|
||||
-alternate-entrypoints=RULE[,RULE...] : string, comma-separated list of rule names
|
||||
that may be used as alternate entrypoints for the parser, in addition to the
|
||||
default entrypoint (the first rule in the grammar) (default: none).
|
||||
Such entrypoints can be specified in the call to Parse by passing an
|
||||
Entrypoint option that specifies the alternate rule name to use. This is only
|
||||
necessary if the -optimize-parser flag is set, as some rules may be optimized
|
||||
out of the resulting parser.
|
||||
|
||||
-support-left-recursion : boolean, (EXPERIMENTAL FEATURE) if set, add support
|
||||
for left recursion rules, including those with indirect recursion
|
||||
(default: false).
|
||||
E.g.:
|
||||
expr = expr '*' term / expr '+' term
|
||||
|
||||
If the code blocks in the grammar (see below, section "Code block") are golint-
|
||||
and go vet-compliant, then the resulting generated code will also be golint-
|
||||
and go vet-compliant.
|
||||
|
||||
The generated code doesn't use any third-party dependency unless code blocks
|
||||
in the grammar require such a dependency.
|
||||
|
||||
PEG syntax
|
||||
|
||||
The accepted syntax for the grammar is formally defined in the
|
||||
grammar/pigeon.peg file, using the PEG syntax. What follows is an informal
|
||||
description of this syntax.
|
||||
|
||||
Identifiers, whitespace, comments and literals follow the same
|
||||
notation as the Go language, as defined in the language specification
|
||||
(http://golang.org/ref/spec#Source_code_representation):
|
||||
|
||||
// single line comment*/
|
||||
// /* multi-line comment */
|
||||
/* 'x' (single quotes for single char literal)
|
||||
"double quotes for string literal"
|
||||
`backtick quotes for raw string literal`
|
||||
RuleName (a valid identifier)
|
||||
|
||||
The grammar must be Unicode text encoded in UTF-8. New lines are identified
|
||||
by the \n character (U+000A). Space (U+0020), horizontal tabs (U+0009) and
|
||||
carriage returns (U+000D) are considered whitespace and are ignored except
|
||||
to separate tokens.
|
||||
|
||||
Rules
|
||||
|
||||
A PEG grammar consists of a set of rules. A rule is an identifier followed
|
||||
by a rule definition operator and an expression. An optional display name -
|
||||
a string literal used in error messages instead of the rule identifier - can
|
||||
be specified after the rule identifier. E.g.:
|
||||
RuleA "friendly name" = 'a'+ // RuleA is one or more lowercase 'a's
|
||||
|
||||
The rule definition operator can be any one of those:
|
||||
=, <-, ← (U+2190), ⟵ (U+27F5)
|
||||
|
||||
Expressions
|
||||
|
||||
A rule is defined by an expression. The following sections describe the
|
||||
various expression types. Expressions can be grouped by using parentheses,
|
||||
and a rule can be referenced by its identifier in place of an expression.
|
||||
|
||||
Choice expression
|
||||
|
||||
The choice expression is a list of expressions that will be tested in the
|
||||
order they are defined. The first one that matches will be used. Expressions
|
||||
are separated by the forward slash character "/". E.g.:
|
||||
ChoiceExpr = A / B / C // A, B and C should be rules declared in the grammar
|
||||
|
||||
Because the first match is used, it is important to think about the order
|
||||
of expressions. For example, in this rule, "<=" would never be used because
|
||||
the "<" expression comes first:
|
||||
BadChoiceExpr = "<" / "<="
|
||||
|
||||
Sequence expression
|
||||
|
||||
The sequence expression is a list of expressions that must all match in
|
||||
that same order for the sequence expression to be considered a match.
|
||||
Expressions are separated by whitespace. E.g.:
|
||||
SeqExpr = "A" "b" "c" // matches "Abc", but not "Acb"
|
||||
|
||||
Labeled expression
|
||||
|
||||
A labeled expression consists of an identifier followed by a colon ":"
|
||||
and an expression. A labeled expression introduces a variable named with
|
||||
the label that can be referenced in the code blocks in the same scope.
|
||||
The variable will have the value of the expression that follows the colon.
|
||||
E.g.:
|
||||
LabeledExpr = value:[a-z]+ {
|
||||
fmt.Println(value)
|
||||
return value, nil
|
||||
}
|
||||
|
||||
The variable is typed as an empty interface, and the underlying type depends
|
||||
on the following:
|
||||
|
||||
For terminals (character and string literals, character classes and
|
||||
the any matcher), the value is []byte. E.g.:
|
||||
Rule = label:'a' { // label is []byte }
|
||||
|
||||
For predicates (& and !), the value is always nil. E.g.:
|
||||
Rule = label:&'a' { // label is nil }
|
||||
|
||||
For a sequence, the value is a slice of empty interfaces, one for each
|
||||
expression value in the sequence. The underlying types of each value
|
||||
in the slice follow the same rules described here, recursively. E.g.:
|
||||
Rule = label:('a' 'b') { // label is []any }
|
||||
|
||||
For a repetition (+ and *), the value is a slice of empty interfaces, one for
|
||||
each repetition. The underlying types of each value in the slice follow
|
||||
the same rules described here, recursively. E.g.:
|
||||
Rule = label:[a-z]+ { // label is []any }
|
||||
|
||||
For a choice expression, the value is that of the matching choice. E.g.:
|
||||
Rule = label:('a' / 'b') { // label is []byte }
|
||||
|
||||
For the optional expression (?), the value is nil or the value of the
|
||||
expression. E.g.:
|
||||
Rule = label:'a'? { // label is nil or []byte }
|
||||
|
||||
Of course, the type of the value can be anything once an action code block
|
||||
is used. E.g.:
|
||||
RuleA = label:'3' {
|
||||
return 3, nil
|
||||
}
|
||||
RuleB = label:RuleA { // label is int }
|
||||
|
||||
And and not expressions
|
||||
|
||||
An expression prefixed with the ampersand "&" is the "and" predicate
|
||||
expression: it is considered a match if the following expression is a match,
|
||||
but it does not consume any input.
|
||||
|
||||
An expression prefixed with the exclamation point "!" is the "not" predicate
|
||||
expression: it is considered a match if the following expression is not
|
||||
a match, but it does not consume any input. E.g.:
|
||||
AndExpr = "A" &"B" // matches "A" if followed by a "B" (does not consume "B")
|
||||
NotExpr = "A" !"B" // matches "A" if not followed by a "B" (does not consume "B")
|
||||
|
||||
The expression following the & and ! operators can be a code block. In that
|
||||
case, the code block must return a bool and an error. The operator's semantic
|
||||
is the same, & is a match if the code block returns true, ! is a match if the
|
||||
code block returns false. The code block has access to any labeled value
|
||||
defined in its scope. E.g.:
|
||||
CodeAndExpr = value:[a-z] &{
|
||||
// can access the value local variable...
|
||||
return true, nil
|
||||
}
|
||||
|
||||
Repeating expressions
|
||||
|
||||
An expression followed by "*", "?" or "+" is a match if the expression
|
||||
occurs zero or more times ("*"), zero or one time "?" or one or more times
|
||||
("+") respectively. The match is greedy, it will match as many times as
|
||||
possible. E.g.
|
||||
ZeroOrMoreAs = "A"*
|
||||
|
||||
Literal matcher
|
||||
|
||||
A literal matcher tries to match the input against a single character or a
|
||||
string literal. The literal may be a single-quoted single character, a
|
||||
double-quoted string or a backtick-quoted raw string. The same rules as in Go
|
||||
apply regarding the allowed characters and escapes.
|
||||
|
||||
The literal may be followed by a lowercase "i" (outside the ending quote)
|
||||
to indicate that the match is case-insensitive. E.g.:
|
||||
LiteralMatch = "Awesome\n"i // matches "awesome" followed by a newline
|
||||
|
||||
Character class matcher
|
||||
|
||||
A character class matcher tries to match the input against a class of characters
|
||||
inside square brackets "[...]". Inside the brackets, characters represent
|
||||
themselves and the same escapes as in string literals are available, except
|
||||
that the single- and double-quote escape is not valid, instead the closing
|
||||
square bracket "]" must be escaped to be used.
|
||||
|
||||
Character ranges can be specified using the "[a-z]" notation. Unicode
|
||||
classes can be specified using the "[\pL]" notation, where L is a
|
||||
single-letter Unicode class of characters, or using the "[\p{Class}]"
|
||||
notation where Class is a valid Unicode class (e.g. "Latin").
|
||||
|
||||
As for string literals, a lowercase "i" may follow the matcher (outside
|
||||
the ending square bracket) to indicate that the match is case-insensitive.
|
||||
A "^" as first character inside the square brackets indicates that the match
|
||||
is inverted (it is a match if the input does not match the character class
|
||||
matcher). E.g.:
|
||||
NotAZ = [^a-z]i
|
||||
|
||||
Any matcher
|
||||
|
||||
The any matcher is represented by the dot ".". It matches any character
|
||||
except the end of file, thus the "!." expression is used to indicate "match
|
||||
the end of file". E.g.:
|
||||
AnyChar = . // match a single character
|
||||
EOF = !.
|
||||
|
||||
Code block
|
||||
|
||||
Code blocks can be added to generate custom Go code. There are three kinds
|
||||
of code blocks: the initializer, the action and the predicate. All code blocks
|
||||
appear inside curly braces "{...}".
|
||||
|
||||
The initializer must appear first in the grammar, before any rule. It is
|
||||
copied as-is (minus the wrapping curly braces) at the top of the generated
|
||||
parser. It may contain function declarations, types, variables, etc. just
|
||||
like any Go file. Every symbol declared here will be available to all other
|
||||
code blocks. Although the initializer is optional in a valid grammar, it is
|
||||
usually required to generate a valid Go source code file (for the package
|
||||
clause). E.g.:
|
||||
{
|
||||
package main
|
||||
|
||||
func someHelper() {
|
||||
// ...
|
||||
}
|
||||
}
|
||||
|
||||
Action code blocks are code blocks declared after an expression in a rule.
|
||||
Those code blocks are turned into a method on the "*current" type in the
|
||||
generated source code. The method receives any labeled expression's value
|
||||
as argument (as any) and must return two values, the first being
|
||||
the value of the expression (an any), and the second an error.
|
||||
If a non-nil error is returned, it is added to the list of errors that the
|
||||
parser will return. E.g.:
|
||||
RuleA = "A"+ {
|
||||
// return the matched string, "c" is the default name for
|
||||
// the *current receiver variable.
|
||||
return string(c.text), nil
|
||||
}
|
||||
|
||||
Predicate code blocks are code blocks declared immediately after the and "&"
|
||||
or the not "!" operators. Like action code blocks, predicate code blocks
|
||||
are turned into a method on the "*current" type in the generated source code.
|
||||
The method receives any labeled expression's value as argument (as any)
|
||||
and must return two opt, the first being a bool and the second an error.
|
||||
If a non-nil error is returned, it is added to the list of errors that the
|
||||
parser will return. E.g.:
|
||||
RuleAB = [ab]i+ &{
|
||||
return true, nil
|
||||
}
|
||||
|
||||
State change code blocks are code blocks starting with "#". In contrast to
|
||||
action and predicate code blocks, state change code blocks are allowed to
|
||||
modify values in the global "state" store (see below).
|
||||
State change code blocks are turned into a method on the "*current" type
|
||||
in the generated source code.
|
||||
The method is passed any labeled expression's value as an argument (of type
|
||||
any) and must return a value of type error.
|
||||
If a non-nil error is returned, it is added to the list of errors that the
|
||||
parser will return, note that the parser does NOT backtrack if a non-nil
|
||||
error is returned.
|
||||
E.g:
|
||||
Rule = [a] #{
|
||||
c.state["a"]++
|
||||
if c.state["a"] > 5 {
|
||||
return fmt.Errorf("we have seen more than 5 a's") // parser will not backtrack
|
||||
}
|
||||
return nil
|
||||
}
|
||||
The "*current" type is a struct that provides four useful fields that can be
|
||||
accessed in action, state change, and predicate code blocks: "pos", "text",
|
||||
"state" and "globalStore".
|
||||
|
||||
The "pos" field indicates the current position of the parser in the source
|
||||
input. It is itself a struct with three fields: "line", "col" and "offset".
|
||||
Line is a 1-based line number, col is a 1-based column number that counts
|
||||
runes from the start of the line, and offset is a 0-based byte offset.
|
||||
|
||||
The "text" field is the slice of bytes of the current match. It is empty
|
||||
in a predicate code block.
|
||||
|
||||
The "state" field is a global store, with backtrack support, of type
|
||||
"map[string]any". The values in the store are tied to the parser's
|
||||
backtracking, in particular if a rule fails to match then all updates to the
|
||||
state that occurred in the process of matching the rule are rolled back. For a
|
||||
key-value store that is not tied to the parser's backtracking, see the
|
||||
"globalStore".
|
||||
The values in the "state" store are available for read access in action and
|
||||
predicate code blocks, any changes made to the "state" store will be reverted
|
||||
once the action or predicate code block is finished running. To update values
|
||||
in the "state" use state change code blocks ("#{}").
|
||||
|
||||
IMPORTANT:
|
||||
- In order to properly roll back the state if a rule fails to match the
|
||||
parser must clone the state before trying to match a rule.
|
||||
- The default clone mechanism makes a "shallow" copy of each value in the
|
||||
"state", this implies that pointers, maps, slices, channels, and structs
|
||||
containing any of the previous types are not properly copied.
|
||||
- To support theses cases pigeon offers the "Cloner" interface which
|
||||
consists of a single method "Clone". If a value stored in the "state"
|
||||
store implements this interface, the "Clone" method is used to obtain a
|
||||
proper copy.
|
||||
- If a general solution is needed, external libraries which provide deep
|
||||
copy functionality may be used in the "Clone" method
|
||||
(e.g. https://github.com/mitchellh/copystructure).
|
||||
|
||||
The "globalStore" field is a global store of type "map[string]any",
|
||||
which allows to store arbitrary values, which are available in action and
|
||||
predicate code blocks for read as well as write access.
|
||||
It is important to notice, that the global store is completely independent from
|
||||
the backtrack mechanism of PEG and is therefore not set back to its old state
|
||||
during backtrack.
|
||||
The initialization of the global store may be achieved by using the GlobalStore
|
||||
function (http://godoc.org/github.com/mna/pigeon/test/predicates#GlobalStore).
|
||||
Be aware, that all keys starting with "_pigeon" are reserved for internal use
|
||||
of pigeon and should not be used nor modified. Those keys are treated as
|
||||
internal implementation details and therefore there are no guarantees given in
|
||||
regards of API stability.
|
||||
|
||||
Left recursion
|
||||
|
||||
With options -support-left-recursion pigeon supports left recursion. E.g.:
|
||||
expr = expr '*' term
|
||||
Supports indirect recursion:
|
||||
A = B / D
|
||||
B = A / C
|
||||
The implementation is based on the [Left-recursive PEG Grammars][9] article that
|
||||
links to [Left Recursion in Parsing Expression Grammars][10] and
|
||||
[Packrat Parsers Can Support Left Recursion][11] papers.
|
||||
|
||||
References:
|
||||
|
||||
[9]: https://medium.com/@gvanrossum_83706/left-recursive-peg-grammars-65dab3c580e1
|
||||
[10]: https://arxiv.org/pdf/1207.0443.pdf
|
||||
[11]: http://web.cs.ucla.edu/~todd/research/pepm08.pdf
|
||||
|
||||
Failure labels, throw and recover
|
||||
|
||||
pigeon supports an extension of the classical PEG syntax called failure labels,
|
||||
proposed by Maidl et al. in their paper "Error Reporting in Parsing Expression Grammars" [7].
|
||||
The used syntax for the introduced expressions is borrowed from their lpeglabel [8]
|
||||
implementation.
|
||||
|
||||
This extension allows to signal different kinds of errors and to specify, which
|
||||
recovery pattern should handle a given label.
|
||||
|
||||
With labeled failures it is possible to distinguish between an ordinary failure
|
||||
and an error. Usually, an ordinary failure is produced when the matching of a
|
||||
character fails, and this failure is caught by ordered choice. An error
|
||||
(a non-ordinary failure), by its turn, is produced by the throw operator and
|
||||
may be caught by the recovery operator.
|
||||
|
||||
In pigeon, the recovery expression consists of the regular expression, the recovery
|
||||
expression and a set of labels to be matched. First, the regular expression is tried.
|
||||
If this fails with one of the provided labels, the recovery expression is tried. If
|
||||
this fails as well, the error is propagated. E.g.:
|
||||
FailureRecoveryExpr = RegularExpr //{FailureLabel1, FailureLabel2} RecoveryExpr
|
||||
|
||||
To signal a failure condition, the throw expression is used. E.g.:
|
||||
ThrowExpr = %{FailureLabel1}
|
||||
|
||||
For concrete examples, how to use throw and recover, have a look at the examples
|
||||
"labeled_failures" and "thrownrecover" in the "test" folder.
|
||||
|
||||
The implementation of the throw and recover operators work as follows:
|
||||
The failure recover expression adds the recover expression for every failure label
|
||||
to the recovery stack and runs the regular expression.
|
||||
The throw expression checks the recovery stack in reversed order for the provided
|
||||
failure label. If the label is found, the respective recovery expression is run. If
|
||||
this expression is successful, the parser continues the processing of the input. If
|
||||
the recovery expression is not successful, the parsing fails and the parser starts
|
||||
to backtrack.
|
||||
|
||||
If throw and recover expressions are used together with global state, it is the
|
||||
responsibility of the author of the grammar to reset the global state to a valid
|
||||
state during the recovery operation.
|
||||
|
||||
[7]: https://arxiv.org/pdf/1405.6646v3.pdf
|
||||
[8]: https://github.com/sqmedeiros/lpeglabel
|
||||
|
||||
Using the generated parser
|
||||
|
||||
The parser generated by pigeon exports a few symbols so that it can be used
|
||||
as a package with public functions to parse input text. The exported API is:
|
||||
- Parse(string, []byte, ...Option) (any, error)
|
||||
- ParseFile(string, ...Option) (any, error)
|
||||
- ParseReader(string, io.Reader, ...Option) (any, error)
|
||||
- AllowInvalidUTF8(bool) Option
|
||||
- Debug(bool) Option
|
||||
- Entrypoint(string) Option
|
||||
- GlobalStore(string, any) Option
|
||||
- MaxExpressions(uint64) Option
|
||||
- Memoize(bool) Option
|
||||
- Recover(bool) Option
|
||||
- Statistics(*Stats) Option
|
||||
|
||||
See the godoc page of the generated parser for the test/predicates grammar
|
||||
for an example documentation page of the exported API:
|
||||
http://godoc.org/github.com/mna/pigeon/test/predicates.
|
||||
|
||||
Like the grammar used to generate the parser, the input text must be
|
||||
UTF-8-encoded Unicode.
|
||||
|
||||
The start rule of the parser is the first rule in the PEG grammar used
|
||||
to generate the parser. A call to any of the Parse* functions returns
|
||||
the value generated by executing the grammar on the provided input text,
|
||||
and an optional error.
|
||||
|
||||
Typically, the grammar should generate some kind of abstract syntax tree (AST),
|
||||
but for simple grammars it may evaluate the result immediately, such as in
|
||||
the examples/calculator example. There are no constraints imposed on the
|
||||
author of the grammar, it can return whatever is needed.
|
||||
|
||||
Error reporting
|
||||
|
||||
When the parser returns a non-nil error, the error is always of type errList,
|
||||
which is defined as a slice of errors ([]error). Each error in the list is
|
||||
of type *parserError. This is a struct that has an "Inner" field that can be
|
||||
used to access the original error.
|
||||
|
||||
So if a code block returns some well-known error like:
|
||||
{
|
||||
return nil, io.EOF
|
||||
}
|
||||
|
||||
The original error can be accessed this way:
|
||||
_, err := ParseFile("some_file")
|
||||
if err != nil {
|
||||
list := err.(errList)
|
||||
for _, err := range list {
|
||||
pe := err.(*parserError)
|
||||
if pe.Inner == io.EOF {
|
||||
// ...
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
By default the parser will continue after an error is returned and will
|
||||
cumulate all errors found during parsing. If the grammar reaches a point
|
||||
where it shouldn't continue, a panic statement can be used to terminate
|
||||
parsing. The panic will be caught at the top-level of the Parse* call
|
||||
and will be converted into a *parserError like any error, and an errList
|
||||
will still be returned to the caller.
|
||||
|
||||
The divide by zero error in the examples/calculator grammar leverages this
|
||||
feature (no special code is needed to handle division by zero, if it
|
||||
happens, the runtime panics and it is recovered and returned as a parsing
|
||||
error).
|
||||
|
||||
Providing good error reporting in a parser is not a trivial task. Part
|
||||
of it is provided by the pigeon tool, by offering features such as
|
||||
filename, position, expected literals and rule name in the error message,
|
||||
but an important part of good error reporting needs to be done by the grammar
|
||||
author.
|
||||
|
||||
For example, many programming languages use double-quotes for string literals.
|
||||
Usually, if the opening quote is found, the closing quote is expected, and if
|
||||
none is found, there won't be any other rule that will match, there's no need
|
||||
to backtrack and try other choices, an error should be added to the list
|
||||
and the match should be consumed.
|
||||
|
||||
In order to do this, the grammar can look something like this:
|
||||
|
||||
StringLiteral = '"' ValidStringChar* '"' {
|
||||
// this is the valid case, build string literal node
|
||||
// node = ...
|
||||
return node, nil
|
||||
} / '"' ValidStringChar* !'"' {
|
||||
// invalid case, build a replacement string literal node or build a BadNode
|
||||
// node = ...
|
||||
return node, errors.New("string literal not terminated")
|
||||
}
|
||||
|
||||
This is just one example, but it illustrates the idea that error reporting
|
||||
needs to be thought out when designing the grammar.
|
||||
|
||||
Because the above mentioned error types (errList and parserError) are not
|
||||
exported, additional steps have to be taken, ff the generated parser is used as
|
||||
library package in other packages (e.g. if the same parser is used in multiple
|
||||
command line tools).
|
||||
One possible implementation for exported errors (based on interfaces) and
|
||||
customized error reporting (caret style formatting of the position, where
|
||||
the parsing failed) is available in the json example and its command line tool:
|
||||
http://godoc.org/github.com/mna/pigeon/examples/json
|
||||
|
||||
API stability
|
||||
|
||||
Generated parsers have user-provided code mixed with pigeon code
|
||||
in the same package, so there is no package
|
||||
boundary in the resulting code to prevent access to unexported symbols.
|
||||
What is meant to be implementation
|
||||
details in pigeon is also available to user code - which doesn't mean
|
||||
it should be used.
|
||||
|
||||
For this reason, it is important to precisely define what is intended to be
|
||||
the supported API of pigeon, the parts that will be stable
|
||||
in future versions.
|
||||
|
||||
The "stability" of the version 1.0 API attempts to make a similar guarantee
|
||||
as the Go 1 compatibility [5]. The following lists what part of the
|
||||
current pigeon code falls under that guarantee (features may be added in
|
||||
the future):
|
||||
|
||||
- The pigeon command-line flags and arguments: those will not be removed
|
||||
and will maintain the same semantics.
|
||||
|
||||
- The explicitly exported API generated by pigeon. See [6] for the
|
||||
documentation of this API on a generated parser.
|
||||
|
||||
- The PEG syntax, as documented above.
|
||||
|
||||
- The code blocks (except the initializer) will always be generated as
|
||||
methods on the *current type, and this type is guaranteed to have
|
||||
the fields pos (type position) and text (type []byte). There are no
|
||||
guarantees on other fields and methods of this type.
|
||||
|
||||
- The position type will always have the fields line, col and offset,
|
||||
all defined as int. There are no guarantees on other fields and methods
|
||||
of this type.
|
||||
|
||||
- The type of the error value returned by the Parse* functions, when
|
||||
not nil, will always be errList defined as a []error. There are no
|
||||
guarantees on methods of this type, other than the fact it implements the
|
||||
error interface.
|
||||
|
||||
- Individual errors in the errList will always be of type *parserError,
|
||||
and this type is guaranteed to have an Inner field that contains the
|
||||
original error value. There are no guarantees on other fields and methods
|
||||
of this type.
|
||||
|
||||
The above guarantee is given to the version 1.0 (https://github.com/mna/pigeon/releases/tag/v1.0.0)
|
||||
of pigeon, which has entered maintenance mode (bug fixes only). The current
|
||||
master branch includes the development toward a future version 2.0, which
|
||||
intends to further improve pigeon.
|
||||
While the given API stability should be maintained as far as it makes sense,
|
||||
breaking changes may be necessary to be able to improve pigeon.
|
||||
The new version 2.0 API has not yet stabilized and therefore changes to the API
|
||||
may occur at any time.
|
||||
|
||||
References:
|
||||
|
||||
[5]: https://golang.org/doc/go1compat
|
||||
[6]: http://godoc.org/github.com/mna/pigeon/test/predicates
|
||||
|
||||
*/
|
||||
package main
|
||||
+301
@@ -0,0 +1,301 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"golang.org/x/tools/imports"
|
||||
|
||||
"github.com/mna/pigeon/ast"
|
||||
"github.com/mna/pigeon/builder"
|
||||
)
|
||||
|
||||
// exit function mockable for tests
|
||||
var exit = os.Exit
|
||||
|
||||
// ruleNamesFlag is a custom flag that parses a comma-separated
|
||||
// list of rule names. It implements flag.Value.
|
||||
type ruleNamesFlag []string
|
||||
|
||||
func (r *ruleNamesFlag) String() string {
|
||||
return fmt.Sprint(*r)
|
||||
}
|
||||
|
||||
func (r *ruleNamesFlag) Set(value string) error {
|
||||
names := strings.Split(value, ",")
|
||||
*r = append(*r, names...)
|
||||
return nil
|
||||
}
|
||||
|
||||
func main() {
|
||||
fs := flag.NewFlagSet(os.Args[0], flag.ExitOnError)
|
||||
|
||||
// define command-line flags
|
||||
var (
|
||||
cacheFlag = fs.Bool("cache", false, "cache parsing results")
|
||||
dbgFlag = fs.Bool("debug", false, "set debug mode")
|
||||
shortHelpFlag = fs.Bool("h", false, "show help page")
|
||||
longHelpFlag = fs.Bool("help", false, "show help page")
|
||||
nolint = fs.Bool("nolint", false, "add '// nolint: ...' comments to suppress warnings by gometalinter or golangci-lint")
|
||||
noRecoverFlag = fs.Bool("no-recover", false, "do not recover from panic")
|
||||
outputFlag = fs.String("o", "", "output file, defaults to stdout")
|
||||
optimizeBasicLatinFlag = fs.Bool("optimize-basic-latin", false, "generate optimized parser for Unicode Basic Latin character sets")
|
||||
optimizeGrammar = fs.Bool("optimize-grammar", false, "optimize the given grammar (EXPERIMENTAL FEATURE)")
|
||||
optimizeParserFlag = fs.Bool("optimize-parser", false, "generate optimized parser without Debug and Memoize options")
|
||||
recvrNmFlag = fs.String("receiver-name", "c", "receiver name for the generated methods")
|
||||
noBuildFlag = fs.Bool("x", false, "do not build, only parse")
|
||||
supportLeftRecursion = fs.Bool("support-left-recursion", false, "add support left recursion (EXPERIMENTAL FEATURE)")
|
||||
|
||||
altEntrypointsFlag ruleNamesFlag
|
||||
)
|
||||
fs.Var(&altEntrypointsFlag, "alternate-entrypoints", "comma-separated list of rule names that may be used as entrypoints")
|
||||
|
||||
fs.Usage = usage
|
||||
err := fs.Parse(os.Args[1:])
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, "args parse error:\n", err)
|
||||
exit(6)
|
||||
}
|
||||
|
||||
if *shortHelpFlag || *longHelpFlag {
|
||||
fs.Usage()
|
||||
exit(0)
|
||||
}
|
||||
|
||||
if fs.NArg() > 1 {
|
||||
argError(1, "expected one argument, got %q", strings.Join(fs.Args(), " "))
|
||||
}
|
||||
|
||||
// get input source
|
||||
infile := ""
|
||||
if fs.NArg() == 1 {
|
||||
infile = fs.Arg(0)
|
||||
}
|
||||
nm, rc := input(infile)
|
||||
defer func() {
|
||||
err = rc.Close()
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, "close file error:\n", err)
|
||||
}
|
||||
if r := recover(); r != nil {
|
||||
panic(r)
|
||||
}
|
||||
if err != nil {
|
||||
exit(7)
|
||||
}
|
||||
}()
|
||||
|
||||
// parse input
|
||||
g, err := ParseReader(nm, rc, Debug(*dbgFlag), Memoize(*cacheFlag), Recover(!*noRecoverFlag))
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, "parse error(s):\n", err)
|
||||
exit(3)
|
||||
}
|
||||
|
||||
// validate alternate entrypoints
|
||||
grammar := g.(*ast.Grammar)
|
||||
rules := make(map[string]struct{}, len(grammar.Rules))
|
||||
for _, rule := range grammar.Rules {
|
||||
rules[rule.Name.Val] = struct{}{}
|
||||
}
|
||||
for _, entrypoint := range altEntrypointsFlag {
|
||||
if entrypoint == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
if _, ok := rules[entrypoint]; !ok {
|
||||
fmt.Fprintf(os.Stderr, "argument error:\nunknown rule name %s used as alternate entrypoint\n", entrypoint)
|
||||
exit(9)
|
||||
}
|
||||
}
|
||||
|
||||
if !*noBuildFlag {
|
||||
if *optimizeGrammar {
|
||||
ast.Optimize(grammar, altEntrypointsFlag...)
|
||||
}
|
||||
|
||||
// generate parser
|
||||
out := output(*outputFlag)
|
||||
defer func() {
|
||||
err := out.Close()
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, "close file error:\n", err)
|
||||
exit(8)
|
||||
}
|
||||
}()
|
||||
|
||||
outBuf := bytes.NewBuffer([]byte{})
|
||||
|
||||
curNmOpt := builder.ReceiverName(*recvrNmFlag)
|
||||
optimizeParser := builder.Optimize(*optimizeParserFlag)
|
||||
basicLatinOptimize := builder.BasicLatinLookupTable(*optimizeBasicLatinFlag)
|
||||
nolintOpt := builder.Nolint(*nolint)
|
||||
leftRecursionSupporter := builder.SupportLeftRecursion(*supportLeftRecursion)
|
||||
if err := builder.BuildParser(
|
||||
outBuf, grammar, curNmOpt, optimizeParser, basicLatinOptimize,
|
||||
nolintOpt, leftRecursionSupporter); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "build error: ", err)
|
||||
exit(5)
|
||||
}
|
||||
|
||||
// Defaults from golang.org/x/tools/cmd/goimports
|
||||
options := &imports.Options{
|
||||
TabWidth: 8,
|
||||
TabIndent: true,
|
||||
Comments: true,
|
||||
Fragment: true,
|
||||
}
|
||||
|
||||
formattedBuf, err := imports.Process("filename", outBuf.Bytes(), options)
|
||||
if err != nil {
|
||||
if _, err := out.Write(outBuf.Bytes()); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "write error: ", err)
|
||||
exit(7)
|
||||
}
|
||||
fmt.Fprintln(os.Stderr, "format error: ", err)
|
||||
exit(6)
|
||||
}
|
||||
|
||||
if _, err := out.Write(formattedBuf); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "write error: ", err)
|
||||
exit(7)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
var usagePage = `usage: %s [options] [GRAMMAR_FILE]
|
||||
|
||||
Pigeon generates a parser based on a PEG grammar.
|
||||
|
||||
By default, pigeon reads the grammar from stdin and writes the
|
||||
generated parser to stdout. If GRAMMAR_FILE is specified, the
|
||||
grammar is read from this file instead. If the -o flag is set,
|
||||
the generated code is written to this file instead.
|
||||
|
||||
-cache
|
||||
cache parser results to avoid exponential parsing time in
|
||||
pathological cases. Can make the parsing slower for typical
|
||||
cases and uses more memory.
|
||||
-debug
|
||||
output debugging information while parsing the grammar.
|
||||
-h -help
|
||||
display this help message.
|
||||
-nolint
|
||||
add '// nolint: ...' comments for generated parser to suppress
|
||||
warnings by gometalinter (https://github.com/alecthomas/gometalinter) or
|
||||
golangci-lint (https://golangci-lint.run/).
|
||||
-no-recover
|
||||
do not recover from a panic. Useful to access the panic stack
|
||||
when debugging, otherwise the panic is converted to an error.
|
||||
-o OUTPUT_FILE
|
||||
write the generated parser to OUTPUT_FILE. Defaults to stdout.
|
||||
-optimize-basic-latin
|
||||
generate optimized parser for Unicode Basic Latin character set
|
||||
-optimize-grammar
|
||||
perform several performance optimizations on the grammar (EXPERIMENTAL FEATURE)
|
||||
-optimize-parser
|
||||
generate optimized parser without Debug and Memoize options and
|
||||
with some other optimizations applied.
|
||||
-receiver-name NAME
|
||||
use NAME as for the receiver name of the generated methods
|
||||
for the grammar's code blocks. Defaults to "c".
|
||||
-x
|
||||
do not generate the parser, only parse the grammar.
|
||||
-alternate-entrypoints RULE[,RULE...]
|
||||
comma-separated list of rule names that may be used as alternate
|
||||
entrypoints for the parser, in addition to the first rule in the
|
||||
grammar.
|
||||
-support-left-recursion
|
||||
add support left recursion (EXPERIMENTAL FEATURE)
|
||||
|
||||
See https://godoc.org/github.com/mna/pigeon for more information.
|
||||
`
|
||||
|
||||
// usage prints the help page of the command-line tool.
|
||||
func usage() {
|
||||
fmt.Printf(usagePage, os.Args[0])
|
||||
}
|
||||
|
||||
// argError prints an error message to stderr, prints the command usage
|
||||
// and exits with the specified exit code.
|
||||
func argError(exitCode int, msg string, args ...any) {
|
||||
fmt.Fprintf(os.Stderr, msg, args...)
|
||||
fmt.Fprintln(os.Stderr)
|
||||
usage()
|
||||
exit(exitCode)
|
||||
}
|
||||
|
||||
// input gets the name and reader to get input text from.
|
||||
func input(filename string) (nm string, rc io.ReadCloser) {
|
||||
nm = "stdin"
|
||||
inf := os.Stdin
|
||||
if filename != "" {
|
||||
f, err := os.Open(filename)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
exit(2)
|
||||
}
|
||||
inf = f
|
||||
nm = filename
|
||||
}
|
||||
r := bufio.NewReader(inf)
|
||||
return nm, makeReadCloser(r, inf)
|
||||
}
|
||||
|
||||
// output gets the writer to write the generated parser to.
|
||||
func output(filename string) io.WriteCloser {
|
||||
out := os.Stdout
|
||||
if filename != "" {
|
||||
f, err := os.Create(filename)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
exit(4)
|
||||
}
|
||||
out = f
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// create a ReadCloser that reads from r and closes c.
|
||||
func makeReadCloser(r io.Reader, c io.Closer) io.ReadCloser {
|
||||
rc := struct {
|
||||
io.Reader
|
||||
io.Closer
|
||||
}{r, c}
|
||||
return io.ReadCloser(rc)
|
||||
}
|
||||
|
||||
// astPos is a helper method for the PEG grammar parser. It returns the
|
||||
// position of the current match as an ast.Pos.
|
||||
func (c *current) astPos() ast.Pos {
|
||||
return ast.Pos{Line: c.pos.line, Col: c.pos.col, Off: c.pos.offset}
|
||||
}
|
||||
|
||||
// toAnySlice is a helper function for the PEG grammar parser. It converts
|
||||
// v to a slice of empty interfaces.
|
||||
func toAnySlice(v any) []any {
|
||||
if v == nil {
|
||||
return nil
|
||||
}
|
||||
return v.([]any)
|
||||
}
|
||||
|
||||
// validateUnicodeEscape checks that the provided escape sequence is a
|
||||
// valid Unicode escape sequence.
|
||||
func validateUnicodeEscape(escape, errMsg string) (any, error) {
|
||||
r, _, _, err := strconv.UnquoteChar("\\"+escape, '"')
|
||||
if err != nil {
|
||||
return nil, errors.New(errMsg)
|
||||
}
|
||||
if 0xD800 <= r && r <= 0xDFFF {
|
||||
return nil, errors.New(errMsg)
|
||||
}
|
||||
return nil, nil
|
||||
}
|
||||
+4681
File diff suppressed because it is too large
Load Diff
+71
@@ -0,0 +1,71 @@
|
||||
package main
|
||||
|
||||
var reservedWords = map[string]bool{
|
||||
// Go keywords http://golang.org/ref/spec#Keywords
|
||||
"break": true,
|
||||
"case": true,
|
||||
"chan": true,
|
||||
"const": true,
|
||||
"continue": true,
|
||||
"default": true,
|
||||
"defer": true,
|
||||
"else": true,
|
||||
"fallthrough": true,
|
||||
"for": true,
|
||||
"func": true,
|
||||
"goto": true,
|
||||
"go": true,
|
||||
"if": true,
|
||||
"import": true,
|
||||
"interface": true,
|
||||
"map": true,
|
||||
"package": true,
|
||||
"range": true,
|
||||
"return": true,
|
||||
"select": true,
|
||||
"struct": true,
|
||||
"switch": true,
|
||||
"type": true,
|
||||
"var": true,
|
||||
|
||||
// predeclared identifiers http://golang.org/ref/spec#Predeclared_identifiers
|
||||
"bool": true,
|
||||
"byte": true,
|
||||
"complex64": true,
|
||||
"complex128": true,
|
||||
"error": true,
|
||||
"float32": true,
|
||||
"float64": true,
|
||||
"int8": true,
|
||||
"int16": true,
|
||||
"int32": true,
|
||||
"int64": true,
|
||||
"int": true,
|
||||
"rune": true,
|
||||
"string": true,
|
||||
"uint8": true,
|
||||
"uint16": true,
|
||||
"uint32": true,
|
||||
"uint64": true,
|
||||
"uintptr": true,
|
||||
"uint": true,
|
||||
"true": true,
|
||||
"false": true,
|
||||
"iota": true,
|
||||
"nil": true,
|
||||
"append": true,
|
||||
"cap": true,
|
||||
"close": true,
|
||||
"complex": true,
|
||||
"copy": true,
|
||||
"delete": true,
|
||||
"imag": true,
|
||||
"len": true,
|
||||
"make": true,
|
||||
"new": true,
|
||||
"panic": true,
|
||||
"println": true,
|
||||
"print": true,
|
||||
"real": true,
|
||||
"recover": true,
|
||||
}
|
||||
+200
@@ -0,0 +1,200 @@
|
||||
// This file is generated by the misc/cmd/unicode-classes tool.
|
||||
// Do not edit.
|
||||
|
||||
package main
|
||||
|
||||
var unicodeClasses = map[string]bool{
|
||||
"ASCII_Hex_Digit": true,
|
||||
"Arabic": true,
|
||||
"Armenian": true,
|
||||
"Avestan": true,
|
||||
"Balinese": true,
|
||||
"Bamum": true,
|
||||
"Bassa_Vah": true,
|
||||
"Batak": true,
|
||||
"Bengali": true,
|
||||
"Bidi_Control": true,
|
||||
"Bopomofo": true,
|
||||
"Brahmi": true,
|
||||
"Braille": true,
|
||||
"Buginese": true,
|
||||
"Buhid": true,
|
||||
"C": true,
|
||||
"Canadian_Aboriginal": true,
|
||||
"Carian": true,
|
||||
"Caucasian_Albanian": true,
|
||||
"Cc": true,
|
||||
"Cf": true,
|
||||
"Chakma": true,
|
||||
"Cham": true,
|
||||
"Cherokee": true,
|
||||
"Co": true,
|
||||
"Common": true,
|
||||
"Coptic": true,
|
||||
"Cs": true,
|
||||
"Cuneiform": true,
|
||||
"Cypriot": true,
|
||||
"Cyrillic": true,
|
||||
"Dash": true,
|
||||
"Deprecated": true,
|
||||
"Deseret": true,
|
||||
"Devanagari": true,
|
||||
"Diacritic": true,
|
||||
"Duployan": true,
|
||||
"Egyptian_Hieroglyphs": true,
|
||||
"Elbasan": true,
|
||||
"Ethiopic": true,
|
||||
"Extender": true,
|
||||
"Georgian": true,
|
||||
"Glagolitic": true,
|
||||
"Gothic": true,
|
||||
"Grantha": true,
|
||||
"Greek": true,
|
||||
"Gujarati": true,
|
||||
"Gurmukhi": true,
|
||||
"Han": true,
|
||||
"Hangul": true,
|
||||
"Hanunoo": true,
|
||||
"Hebrew": true,
|
||||
"Hex_Digit": true,
|
||||
"Hiragana": true,
|
||||
"Hyphen": true,
|
||||
"IDS_Binary_Operator": true,
|
||||
"IDS_Trinary_Operator": true,
|
||||
"Ideographic": true,
|
||||
"Imperial_Aramaic": true,
|
||||
"Inherited": true,
|
||||
"Inscriptional_Pahlavi": true,
|
||||
"Inscriptional_Parthian": true,
|
||||
"Javanese": true,
|
||||
"Join_Control": true,
|
||||
"Kaithi": true,
|
||||
"Kannada": true,
|
||||
"Katakana": true,
|
||||
"Kayah_Li": true,
|
||||
"Kharoshthi": true,
|
||||
"Khmer": true,
|
||||
"Khojki": true,
|
||||
"Khudawadi": true,
|
||||
"L": true,
|
||||
"Lao": true,
|
||||
"Latin": true,
|
||||
"Lepcha": true,
|
||||
"Limbu": true,
|
||||
"Linear_A": true,
|
||||
"Linear_B": true,
|
||||
"Lisu": true,
|
||||
"Ll": true,
|
||||
"Lm": true,
|
||||
"Lo": true,
|
||||
"Logical_Order_Exception": true,
|
||||
"Lt": true,
|
||||
"Lu": true,
|
||||
"Lycian": true,
|
||||
"Lydian": true,
|
||||
"M": true,
|
||||
"Mahajani": true,
|
||||
"Malayalam": true,
|
||||
"Mandaic": true,
|
||||
"Manichaean": true,
|
||||
"Mc": true,
|
||||
"Me": true,
|
||||
"Meetei_Mayek": true,
|
||||
"Mende_Kikakui": true,
|
||||
"Meroitic_Cursive": true,
|
||||
"Meroitic_Hieroglyphs": true,
|
||||
"Miao": true,
|
||||
"Mn": true,
|
||||
"Modi": true,
|
||||
"Mongolian": true,
|
||||
"Mro": true,
|
||||
"Myanmar": true,
|
||||
"N": true,
|
||||
"Nabataean": true,
|
||||
"Nd": true,
|
||||
"New_Tai_Lue": true,
|
||||
"Nko": true,
|
||||
"Nl": true,
|
||||
"No": true,
|
||||
"Noncharacter_Code_Point": true,
|
||||
"Ogham": true,
|
||||
"Ol_Chiki": true,
|
||||
"Old_Italic": true,
|
||||
"Old_North_Arabian": true,
|
||||
"Old_Permic": true,
|
||||
"Old_Persian": true,
|
||||
"Old_South_Arabian": true,
|
||||
"Old_Turkic": true,
|
||||
"Oriya": true,
|
||||
"Osmanya": true,
|
||||
"Other_Alphabetic": true,
|
||||
"Other_Default_Ignorable_Code_Point": true,
|
||||
"Other_Grapheme_Extend": true,
|
||||
"Other_ID_Continue": true,
|
||||
"Other_ID_Start": true,
|
||||
"Other_Lowercase": true,
|
||||
"Other_Math": true,
|
||||
"Other_Uppercase": true,
|
||||
"P": true,
|
||||
"Pahawh_Hmong": true,
|
||||
"Palmyrene": true,
|
||||
"Pattern_Syntax": true,
|
||||
"Pattern_White_Space": true,
|
||||
"Pau_Cin_Hau": true,
|
||||
"Pc": true,
|
||||
"Pd": true,
|
||||
"Pe": true,
|
||||
"Pf": true,
|
||||
"Phags_Pa": true,
|
||||
"Phoenician": true,
|
||||
"Pi": true,
|
||||
"Po": true,
|
||||
"Ps": true,
|
||||
"Psalter_Pahlavi": true,
|
||||
"Quotation_Mark": true,
|
||||
"Radical": true,
|
||||
"Rejang": true,
|
||||
"Runic": true,
|
||||
"S": true,
|
||||
"STerm": true,
|
||||
"Samaritan": true,
|
||||
"Saurashtra": true,
|
||||
"Sc": true,
|
||||
"Sharada": true,
|
||||
"Shavian": true,
|
||||
"Siddham": true,
|
||||
"Sinhala": true,
|
||||
"Sk": true,
|
||||
"Sm": true,
|
||||
"So": true,
|
||||
"Soft_Dotted": true,
|
||||
"Sora_Sompeng": true,
|
||||
"Sundanese": true,
|
||||
"Syloti_Nagri": true,
|
||||
"Syriac": true,
|
||||
"Tagalog": true,
|
||||
"Tagbanwa": true,
|
||||
"Tai_Le": true,
|
||||
"Tai_Tham": true,
|
||||
"Tai_Viet": true,
|
||||
"Takri": true,
|
||||
"Tamil": true,
|
||||
"Telugu": true,
|
||||
"Terminal_Punctuation": true,
|
||||
"Thaana": true,
|
||||
"Thai": true,
|
||||
"Tibetan": true,
|
||||
"Tifinagh": true,
|
||||
"Tirhuta": true,
|
||||
"Ugaritic": true,
|
||||
"Unified_Ideograph": true,
|
||||
"Vai": true,
|
||||
"Variation_Selector": true,
|
||||
"Warang_Citi": true,
|
||||
"White_Space": true,
|
||||
"Yi": true,
|
||||
"Z": true,
|
||||
"Zl": true,
|
||||
"Zp": true,
|
||||
"Zs": true,
|
||||
}
|
||||
Reference in New Issue
Block a user