Currently after an explicit flow key '?' in a flow sequence, an immediately
following closing ] is ignored by the parser:
% echo '[ ? ]' | ./tests/run-parser-test-suite --flow keep
+STR
+DOC
+SEQ []
+MAP {}
=VAL :
=VAL :
-MAP
Parse error: did not find expected ',' or ']'
Line: 2 Column: 1
% echo '[ ? ] ]' | ./tests/run-parser-test-suite --flow keep
+STR
+DOC
+SEQ []
+MAP {}
=VAL :
=VAL :
-MAP
-SEQ
-DOC
-STR
It is read correctly by the scanner as a YAML_FLOW_SEQUENCE_END_TOKEN, and
the flow_level is decreased. Then it is passed to the parser where it gets
ignored.
This leads to invalid YAML being accepted, and valid YAML resulting in an
error.
Also the flow_level is incorrectly decreased, so you can nest sequences
like that without running in to the MAX_NESTING_LEVEL.
Each nesting level increases the stack and the number of previous
starting events that the parser has to check.
The default maximum is 1000 and can be set via yaml_set_max_nest_level()
I also added new options to run-parser and run-parser-test-suite:
* --max-level: you can now try out this feature on the command line
* --show-error: By default, run-parser doesn't show errors. The new option
helps with debugging
See http://yaml.org/spec/1.1/#id907281
The question mark isn't mentioned as something special here, only
,[]{}
This commit will allow
[foo?bar]
[foo ? bar]
The PR does only change the behaviour when the question mark is in the middle or
at the end of the string, not at the beginning, e.g. [?foo] is handled by
a different part of the code.
Before:
a: &b x
*b: c
Now:
a: &b x
*b : c
Passing tests:
* 26DV: Whitespace around colon in mappings
* E76Z: Aliases in Implicit Block Mapping
* X38W: Aliases in Flow Objects
Test manually because `make test-suite` will overwrite changes in the skiplist:
./tests/run-parser-test-suite tests/run-test-suite/data/26DV/in.yaml | ./tests/run-emitter-test-suite
./tests/run-parser-test-suite tests/run-test-suite/data/E76Z/in.yaml | ./tests/run-emitter-test-suite
./tests/run-parser-test-suite tests/run-test-suite/data/X38W/in.yaml | ./tests/run-emitter-test-suite
Or edit tests/run-test-suite/test/blacklist/libyaml-emitter and do:
(cd tests/run-test-suite; prove -lv test)
Also I added some newlines to yaml.h to help identifying states by number.
With `--flow (keep|on)` run-parser-test-suite will output:
+MAP {}
+SEQ []
run-emitter-test-suite will then emit flow style collections if requested:
echo 'foo: [bar, {x: y}]' | ./tests/run-parser-test-suite | ./tests/run-emitter-test-suite
echo 'foo: [bar, {x: y}]' | ./tests/run-parser-test-suite \
--flow keep | ./tests/run-emitter-test-suite --flow keep
Also: add that yaml_private.h include again that I had thrown out. Needed
for printing directives.
Wonder if there is a way to create a directive without using the private api.
Seems this was an oversight in d050fe3f3006b55edf33a2ef91019a67d6c3fb10.
We don't want to ignore all those file, but only the compiled executables.
Also added '/' to several files.
open_ended can have three states now:
0: The previous document was ended explicitly with '...'
1: The previous document wasn't ended with '...'
2: The last scalar event was a block scalar with trailing empty lines |+, and
last document wasn't ended with '...'.
Important at stream end.
This was broken in the past, and fixed in fa1293a.
With my last PR #162 I added the faulty behaviour again.
The problematic behaviour showed only when all of the following conditions were
true:
* writing YAML directives
* writing unquoted top level scalars
* writing more than one document
================== BROKEN ==============================
The first example shows that the document end marker is not emitted before
the next document. This would be valid in YAML 1.1 if the scalar was quoted,
but not if it is plain.
This commit fixes this.
echo '--- foo
--- bar
' | ./tests/run-parser-test-suite | ./tests/run-emitter-test-suite --directive 1.1
%YAML 1.1
--- foo
%YAML 1.1
--- bar
================== FIXED ==============================
echo '--- foo
--- bar
' | ./tests/run-parser-test-suite | ./tests/run-emitter-test-suite --directive 1.1
%YAML 1.1
--- foo
...
%YAML 1.1
--- bar
=======================================================
Other examples which should look like this (and were correct already before
this fix):
Open ended scalars like |+ need '...' at the end of the stream:
echo '--- |+
a
--- |+
a
' | ./tests/run-parser-test-suite | ./tests/run-emitter-test-suite
--- |+
a
--- |+
a
...
=======================================================
If a document is ended with an explicit '...', the code should not
print '...' twice:
echo '--- foo
...
--- bar
' | ./tests/run-parser-test-suite | ./tests/run-emitter-test-suite --directive 1.1
%YAML 1.1
--- foo
...
%YAML 1.1
--- bar
==========================================================
Before `1.1` was hardcoded in the emitter.
* Also add --directive option to run-emitter-test-suite
Allows to easily test how output looks like if %YAML directives
are output.
(provided by msvc compiler) instead of WIN32 (which is user configurable, and
omitted by default on some x64 builds); this fixes an issue with 64 bit windows
static library builds (#66)
Co-authored-by: Mark Sheahan <mark.sheahan@upguard.com>
This will simply allow `%YAML 1.2` directives additionally to `%YAML 1.1`.
There is no change in behaviour.
See also #20
No changes are needed regarding tag directives.
In YAML 1.2 tag directives are for the following document only.
This is already implemented like that in libyaml.
We would rather have to fix the code if we want to have the correct behaviour
(global directives) in YAML 1.1. This would be a bit more complicated, as we
would have to save the default version and the current version in the parser
object.
New passing parser tests:
* 27NA: Spec Example 5.9. Directive Indicator
* 6ZKB: Spec Example 9.6. Stream
* 9DXL: Spec Example 9.6. Stream [1.3]
* RTP8: Spec Example 9.2. Document Markers
New failing error parser tests (before they were errors for the wrong reason):
* EB22: Missing document-end marker before directive
* RHX7: YAML directive without document end marker
The document loading API (yaml_parser_load) was susseptable to a
stack overflow issue when given input data which opened many
mappings and/or sequences without closing them.
This was due to the use of recurion in the implementation.
With this change, we avoid recursion, and maintain our own loader
context stack on the heap.
The loader context contains a stack of document node indexes.
Each time a sequence or mapping start event is encountered,
the node index corrasponding to the event is pushed to the
stack. Each time a sequence or mapping end event is encountered,
the corrasponding node's index is popped from the stack.
The yaml_parser_load_nodes() function sits on the event stream,
issuing events to the appropriate handlers by type.
When an event handler function constructs a node, it needs to
connect the new node to its parent (unless it's the root node).
This is where the loader context stack is used to find the
parent node. The way that the new node is added to the tree
depends on whether the parent node is a mapping (with a
yaml_node_pair_t to fill), or a sequence (with a yaml_node_item_t).
Fixes: https://github.com/yaml/libyaml/issues/107
In YAML 1.1, the document end marker `...` is optional even if the next document starts with a directive:
https://github.com/yaml/pyyaml/blob/master/tests/data/spec-07-09.canonical
```
%YAML 1.1
---
!!str "foo"
%YAML 1.1
---
!!str "bar"
%YAML 1.1
---
!!str "baz"
```
It is only required if the scalar is "open ended", for example for plain scalars.
In YAML 1.2 the `...` marker is always required before a directive.
My suggestion would be to make the output 1.2 compatible. It will still be 1.1 compatible, so that shouldn't be a problem.
I believe this will also make it easier to fix#123 which was introduced with the last fixes regarding `open_ended`. I think I can make a fix for this soon after this issue is fixed.
Fixing #123 without this would be a bit more complicated.
If we do this, we also need to adjust PyYAML to behave the same.
Related issues/commits:
- https://github.com/yaml/libyaml/issues/60
- https://github.com/yaml/libyaml/pull/122
- 56400d9, 8ee83c0, 56f4b17
example-deconstructor-alt.c: In function ‘main’:
example-deconstructor-alt.c:649:51: warning:
comparison between ‘yaml_sequence_style_t {aka enum yaml_sequence_style_e}’
and ‘enum yaml_mapping_style_e’ [-Wenum-compare]
example-deconstructor-alt.c:650:36: warning:
comparison between ‘yaml_sequence_style_t {aka enum yaml_sequence_style_e}’
and ‘enum yaml_mapping_style_e’ [-Wenum-compare]
I suspect this this was a bit of an oversight when first setting up
appveyor for Windows/msvc, but 'release' does not match any target
cmake knows about; 'Release', however, does.
Signed-off-by: Marty E. Plummer <hanetzer@protonmail.com>
