Fixes#463 and #268
When using the --noline flag or the %option noline, flex was failing to
suppress the %top line directives.
There was an if-guard missing in the code that adds the line directives
to the top_buf. This commit fixes it.
With sufficient parallelism and `make --shuffle` to reorder builds the
test suite occasionally fails to build with errors like this:
tests/bison_yylval_parser.y:36:10: fatal error: bison_yylval_scanner.h: No such file or directory
36 | #include "bison_yylval_scanner.h"
Solve this by adding the missing dependencies to the *_parser.o targets
on the _scanner.h files.
When generating the flex man page, fix the program name in the SYNOPSIS
section to 'flex' and not 'stage1flex' or 'flex.exe'. This make it
possible to generate the man page deterministically with a recent enough
'help2man' tool (1.47.1 or later, which supports SOURCE_DATE_EPOCH that
eliminated the last nondeterministic information).
Signed-off-by: Kang-Che Sung <explorer09@gmail.com>
Avoid the GNU Make specific "| order-only-prerequisite" syntax and use a
portable alternative of having doc/Makefile invoke src/Makefile before
generating the man page.
Note that 'flex.1' man page target does not depend on '$(FLEX)' program
directly - the direct dependency would confuse 'make' and it is a
limitation of the current "recursive makefile" structure.
Signed-off-by: Kang-Che Sung <explorer09@gmail.com>
In src/Makefile.am, remove flex_CFLAGS variable, and append
$(WARNINGFLAGS) to AM_CFLAGS. The use of flex_CFLAGS has forced
Automake to rename flex's object files (.o) to have a 'flex-' prefix.
By preventing the rename, the generated src/Makefile.in can be about
37 kB smaller.
Signed-off-by: Kang-Che Sung <explorer09@gmail.com>
They were either 'int[]' or 'bool[]'. In C, it is suboptimal to use
'bool' types in arrays as the arrays would not be optimized for size or
speed.
Redefine the boolean array variables to use 'char' array instead. As we
are allocating them in 'char' array type, the macros
reallocate_bool_array() and allocate_bool_array() would draw confusion.
Replace the use of reallocate_bool_array() to reallocate_array().
It meant "exclusive flag" but was abbreviated poorly.
We don't have tight constraints on variable naming now, so rename it to
'sc_is_exclusive'.
Also use 'bool' type for the parameter.
We expect to have room for 2 EOB chars, but in this particular realloc
path this only works because yy_n_chars tends to be large enough for
1.5x its value to have room for our 2 EOB chars.
That said, clang-6's static analyzer does produce a warning about this
("Use of zero-allocated memory", meaning realloc with size=0) because it
goes by the assumption that yy_n_chars and number_to_move could be 0.
While this is somewhat bogus, it does make sense to just put the "+ 2"
here and be done with it. AFAICS, this is the correct thing to do here
after all.
Signed-off-by: David Lamparter <equinox@diac24.net>
Move the job of strdup() from cclinstal(), ndinstal() and scinstal() to
addsym(). This means ccltab[], sctbl[] and ndtbl[], respectively, will
assume ownership of the duplicated string buffers, and no more garbage
will be left in cclinstal(), ndinstal() or scinstal().
Because of the ownership assumption of sctbl[], strings in scname[] are
now READ-ONLY as they point to the "name" members within sctbl[].
Fixes: #340
Signed-off-by: Kang-Che Sung <explorer09@gmail.com>
With stage1scan.c being deterministic, it's possible to add a bootstrap
check that ensures the generated flex program can build the scanner of
itself flawlessly. The check is performed on native build only.
Code is partly inspired from GCC's makefile.
Signed-off-by: Kang-Che Sung <explorer09@gmail.com>
Change the generating rules of scan.c (in dist-hook) and stage1scan.c
so that the source and output file names in #line directives are fixed.
This would allow the scanner results be comparable to each other.
Also in --disable-bootstrap build, the flex binary will now be built
with scan.c directly - no more sedding.
Signed-off-by: Kang-Che Sung <explorer09@gmail.com>
These skeleton header files can be easily regenerated.
(The things that "mostlyclean" shouldn't delete are files that are more
expensive to rebuild, e.g. library archives and linked executables.)
Signed-off-by: Kang-Che Sung <explorer09@gmail.com>
State the dependencies of '*-flex.h' skeleton files properly and avoid
the 'BUILT_SOURCES' Automake variable, which often does more harm than
good (building files that are not needed for a particular target).
(Regression from commit e6472493e20fc1e604592b7a698f85e515d97856)
Also remove a GNU make wildcard function call as it's no longer needed.
Signed-off-by: Kang-Che Sung <explorer09@gmail.com>
The '*-flex.h' files are included by skeletons.c and not by other source
files. Correct the dependencies in the makefile.
(The error was introduced by 3603f033509d5a214b603f60ce4dd6ed7819843e)
Signed-off-by: Kang-Che Sung <explorer09@gmail.com>
This should be enough to fix the main bug as reported in #314.
Other cross-compile limitations may still exist.
Signed-off-by: Kang-Che Sung <explorer09@gmail.com>
Re-add the fallback definition of 'bool', which was deleted in commit
3efb40df0ec5cddf0cb63a7900eb88258c478514. Also amend the preprocessor
conditionals to turn off the fallback in C++ mode or C23 or later
standard.
This is for backward compatibility to compilers that do not yet support
C99 stdbool.h.
Signed-off-by: Kang-Che Sung <explorer09@gmail.com>
Replace the $'\r' quoting in commit
c674b5faa3419032a6c41cc150488866f3b3cff9 with a portable alternative
`` cr=`printf '\r'` ``
Although $' ' (dollar-single-quote) quoting is proposed for inclusion in
POSIX, we have not seen it in a published standard yet, so it is best to
avoid it for now.
Signed-off-by: Kang-Che Sung <explorer09@gmail.com>
The proper way to use a pure parser since 2.7 (2012-12-12) is "%define
api.pure full". The proper way to pass params since 3.0 (2013-07-25)
is "%param".
The argument is that it's easy to regenerate stage1scan.c comparing to
the stage 1 flex program, which takes time to rebuild.
Signed-off-by: Kang-Che Sung <explorer09@gmail.com>
Clarify and expand tool dependency info in INSTALL.md, particularly with respect to GNU tool variants.
Describe minimal troubleshooting steps for failed builds on non-Debian systems.
Refs: #632
tests/prefix_c99.l calls a function before its definition from within
a dead code block. This is treated as an error in newer compilers. The
test isn't needed as the scanner's main() repeats the call.
Refs: #632
Correct fullspeed table size computation from 'tblend + numecs + 1' to
'tblend + 2 + 1' to avoid allocating extra space that is never initialized.
Refs: #628
This makes the test much less fragile as now all that is need is for
@LINE@ and __LINE__ to be on the same line (the comment gets substituted
too, but it's a comment...)
An indented code block would cause the subsequent all subsequent code
block line numbers (including its own) to be out by one.
For example, the following would cause the line number for the `BEGIN`
code block to have the line number for the `warning` code block. This
would also affect the code in the final section after the second `%%`.
%%
yyflexdebug = 0;
grab_frame = GRAB_FRAME;
grab_other = GRAB_OTHER;
grab_write = GRAB_WRITE;
yylval->pointer = 0;// ensure pointer vals are null
"/*" { BEGIN (COMMENT); }
<COMMENT>"/*" { warning (0, "nested /* in comment"); }
Currently, when the NULL-terminated variadic function
filter_create_ext() is invoked, the value "0" is passed as
the last argument to act as a terminator. However, this is
an integer value, which is incompatible with the pointer
data type expected by filter_create_ext().
This is undefined behavior in C, correct operation is not
guaranteed. In fact, it causes flex to crash on Apple M1
when GCC is used - the loop is not terminated when it should,
instead, it keeps running, corrupting the argument list for
invoking m4. As a result, it creates the following error:
> flex: fatal internal error, exec of gm4 failed
This commit fixes the problem by explicitly casting the value 0 to
the correct pointer type (char *).
Signed-off-by: Yifeng Li <tomli@tomli.me>
`gcount()` is of type `streamsize` which has a system-dependent size with at least `int`. Because of the `int max_size` limit no bigger value will be returned, the explicit cast fixes warnings from compilers/static analyzers.
When complied with 'gcc -O3', the yy_init_buffer call can confuse gcc,
thinking the 'b' pointer may be NULL. (gcc would warn that if
'-Wnull-dereference' is used). Fix the confusion by never pass a NULL
constant to the function.
Fixes: #377
Signed-off-by: Kang-Che Sung <explorer09@gmail.com>
This reverts commit c34590c4bf067d4e749e5298a8ce9f90072d6332
and all search code patches that followed it.
The execvp() would do the $PATH searching of m4 for flex. There's no
need to duplicate the effort.
Another reason for the removal is to remove dependency on stat()
system call, which would cause a problem in a 32-bit OS accessing a
filesystem with 64-bit inode number (see #413).
Signed-off-by: Kang-Che Sung <explorer09@gmail.com>
Override the TEXI2DVI and TEXI2PDF automake variables to add an include directory so VPATH builds work.
I'm not thrilled with doing this. The right thing is probably to move the included examples to doc/ and do the
VPATH magic in examples/manual/ where LaTeX is better able to deal with it.
Automake attempts to compute default _SOURCES for the test-yydecl-*.sh runners generated by test-yydecl-gen.sh. Providing an intentionally empty _SOURCES variable for each script fixes the attendant errors.
Also removed EXTRA_DIST lines for deleted test inputs.
Begin the Go back end. Right now it's a fake that generates C...
...but this commit puts all the build and testnmachinery in place
for when it's not. All tests pass.
We a ruleset test for BOL matching and a test for the array case of yymore.
Improve yymore testing by adding tests that require a BOL match in
both array and pointer cases.
Also, convert lineno sporadic test to be ruleset-based so it's
applied on all back ends.
No cide changes.
Remove two dependencies that the C99 back end no longer has.
Repair to force in-time building of skel.h files. This way of doing
it reduces where a new skel dependency has to be declared in the
vuilld recipe to the one spot where the flex.h file is made from the
flex.skl file.
The posixly correct test only needs to be done once, since it only
affects how the tables are genetated rather than setting code
conditionals in the back end.
Also, some comment typo fixes.
There's only one left. in the prefix code. We leave that in place
because ity will be needed if we ever want to replace the default
C back end with C99.
The tradeoff is a bit delicate here; yy_set_bol() did not strictly
speaking need to change. On the other hand, squeezing the underscores
out of YY_AT_BOL() but not yy_set_bol() would have been exactly the
kind of irregularity that people trip over and curse.
The underlying issue is of course languages like Go and Python with
validators that get upset by all caps and embedded underscores. On the
one hand, we want to not be churning the Flex API gratuitously. On
the other hand it is desirable if as few entry point names as possible
need to change in different target languages.
The approach I'm taking to deciding this question in individual cases
is this: if an entry point name is likely to appear in actions, and
must be recognized as magic by Flex itself in order to have a final
context argument added or an object reference prepended, then it must
be multilanguage-safe - not all-caps and no underscores.
Any name that does *not* need to be wired into Flex can be left
alone. Users may have to change it when they port.
...to get to the scanner internals structure. While this was a valiant
effort at information hiding, it won't port out of C to any other plausible
target language. That being the case, we cut the LOC.
...in the flex scanner. Presently this works for yyecho(), yyinput(),
yystart(), yybegin(), and yyunput(). These no longer need to be nacros.
A few more remain to be done, notably yyless() and yymore().
Along the way, deprecate YY_FATAL_ERROR and add a replacement option.
..left there from a weird compromise when the default C back end also
supported C++. Add a yyunput ruleset testing it on all back ends to
verify that the change is good.
These served a purpose when the code had to handle both reentrant and
non-reentrant cases. Now, while they should still be available in user
actions, the generated code is much less confusing if the yyg pointer
in the references is explicit.
Also, drop VMS support and consolidate platform-specific references
at one spot. This is in the same commit with the macro cleanup because
it included getting rid of the last stray yyin/yyout references.
Gone: C++ support, M4_YY_IN_HEADER. M4_YY_REENTRANT,
and their negation symbols. The strange macrology
requitred to makecunction signatures conditionally
compile for npn-rentrant or reentrant is gone.
The default C back end is not touched by these changes.
Also, document what the posix and posixlycorrect rulesets are doing
netter. I lost a day because the dependency on setting POSIXLY_CORRECT
omn the envoronment was not obvious.
The quotes test is backend-independent and doesn't need
to be run anywhere but C.
Also, remove some debris from the C99 skeleton,
it injects noise into C output comparisons.
If you drop a file with the extension ".rules" in tests/,
"make check" will now automatically make and run tests
for all back ends - presently nr, r, and c99.
It would be trivial to change ruleset.sh so all table-compression
options are checked for all rulesets, and probably a good idea.
It's easy enough to tell if a testfile wants a table by looking at its
name. Getting rid of this option is a step towards putting all
tableopts tests in the generated set so they are trivially portable to
new back ends.
Still to be folded in: all the reject and tableopts tests.
Note: I understand VPATH builds just well enough to suspect that this
commit and my previous broke them, but not well enough to test
or fix it. Mea culpa.
The existing method of generating multiple tests from l4 files is not
flexible enough to handle multilanguage - reliance on the C
preprocessor prevents that. This commit creates replacement machinery
that's more flexible and will greatly reduce the complexity of adding
new back ends.
The idea is to generate test .l files by combining a
backend-independent ruleset with backend-dependent boilerplate for
test mains and preambles.
This commit creates the rules files, the m4 to template them into
tests, and a shell wrapper to drive the m4 expansion. The autoconf
integration works OK, but requires additional rules at the
end of tests/Makefile.am that probably ought to be autogenerated.
There was one other change forced by this. In order for the new
testmaker to work, it needs to be possible to analyze test file names
into stems and option modifiers. Having underscores in the stem
would make the analysis slower and bug-prone.
I wanted to change them to "-" rather than deleting them, but this
gives autoconf fits - it seems to not like filenames with embedded
dashes in certain contexts.
At this commit, only one test - array_nr.l - is generated using the new
system.
With a minor tweak to testwrapper we can allow all the versions of
a test for different back ends (counting _r and _nr as back ends)
to share a common input file.
alloc_extra -> alloc_extra_cpp
ccl -> ccl_cpp
These rena,es meam that the tests for all backends are in either two
parallel series (*_cpp, *_c99) or three (*_nr, *_r, *_c99).
The idea behind this is to exploit the existing test suite - write a
stripped-down, cleaned up C skeleton that only does reentrant scanners
and tosses out features like loadable tables and header generation
that can't be ported to other languages.
The C99 skeleton can then serve both as a launch point for the Go back
end and a model for back ends in procedural languages without OO. The
initial version is identical to the cpp skeleton except for different
name and suffix properties abd having M4_MODE_C_OLY forcrd on. The
latter means it blows up on the C++ tests.
The way this is done is by adding a capabiiliy to mine string-valued
properties out of skel files in Flex's binary.
Also, some reorganization to improve the encapsulation of
knowledge about skel files.
No changes in generated test files from this patch.
To make this work, use line_directive_out() had to be servered from
its deoendence on hidden globaks. Which was a good idea anyway,
No diffs in generated test code from this commit.
Yet another well-hidden C assumption. I begin to woner if I will
expire of old age before fkushing out all of these.
Before this commit, "rm cpp-skel.h" may be required in addition
to "make clean".
Since there no longer any pusging or poping operations, the sko
stack goes away.
Because having two different consitionalization syntaxes was ugly
and conducive to error. After this change there are %% breakpoints
and %# comments and nothing else. Comments on %if lines have been
preserved.
I used the following script:
========================================================================
import sys
flagstack = []
for line in sys.stdin:
if (not line.startswith("%if")) and (not line.startswith("%endif")) and (not line.startswith("%ok-for")) and (not line.startswith("%not-for")):
sys.stdout.write(line)
continue
elif line.startswith("%endif"):
if flagstack.pop():
sys.stdout.write("]])\n")
continue
fields = line.strip().split()
trailer = " ".join(fields[1:]) + "\n"
if line.startswith("%if-bison-bridge"):
sys.stdout.write("m4_ifdef([[M4_YY_BISON_LVAL]], [[\n")
flagstack.append(True)
elif line.startswith("%if-c++-only"):
sys.stdout.write("m4_ifdef([[M4_MODE_CXX_ONLY]], [[\n")
flagstack.append(True)
elif line.startswith("%if-c-only"):
sys.stdout.write("m4_ifdef([[M4_MODE_C_ONLY]], [[\n")
flagstack.append(True)
elif line.startswith("%if-c-or-c++"):
flagstack.append(False)
elif line.startswith("%if-not-reentrant"):
sys.stdout.write("m4_ifdef([[M4_YY_NOT_REENTRANT]], [[\n")
flagstack.append(True)
elif line.startswith("%if-reentrant"):
sys.stdout.write("m4_ifdef([[M4_YY_REENTRANT]], [[\n")
flagstack.append(True)
elif line.startswith("%if-tables-serialization"):
sys.stdout.write("m4_ifdef([[M4_MODE_TABLESEXT]], [[\n")
flagstack.append(True)
elif line.startswith("%not-for-header"):
sys.stdout.write("m4_ifdef( [[M4_YY_IN_HEADER]],,[[m4_dnl\n")
elif line.startswith("%ok-for-header"):
sys.stdout.write("]])\n")
else:
sys.stderr.write("depercent: unrecognized %r\n" % line)
raise SystemExit(1)
if trailer.strip():
sys.stdout.write("%# " + trailer)
========================================================================
One nasty little abuse if markup around yyread_r had to be fixed up by hand.
The bad commit was:
"Rename unput() to yyunput(); follow through in option switches."
I shouldn't have missed this, but te build has normal warnings
that obscured the waning from Flex.
Good modern style in C is to write it as though it were a strongly typed
language even though it isn't; thius, it's no l;onger considered good practice
to apply ! to non-booleans.
Also, fix some indentation annd vraxe usage for uniformity.
Replace YY_FLUSH_BUFFER with yy_flush_current_buffer()
Replace YY_CURRENT_BUFFER with yy_current_buffer().
Replace YY_BUFFER_STATE with yybuffer.
The following are the all-caps definitions still exposed:
YY_FLEX_MAJOR_VERSION
YY_FLEX_MINOR_VERSION
YY_FLEX_SUBMINOR_VERSION
YY_NULL
YY_END_OF_BUFFER_CHAR
YY_BUF_SIZE
YYLMAX
It is not idea for this list to be nonempty, but
at least these are all actual comple-time constants
rather than function calls or type names.
This corrects a minor mistake I made earlier because I did not yet understand
the full generality of the option syntax.
Also fixes some minor markup errors in the manual.
This change was not strictly necessary to sever a preprocessor
dependency nor make the API uniform aross both C and C++. But it
cried out to be made, because now *all* the rule hooks are in the yy*
namespace. This makes the API easier to document and remember.
unput() is left in place as a compatibility macro for existing
users, but only documented as a legacy interface. The "unput"
variants of switches and options have also been retained.
...leaving an "input" macro in place for legacy compatibility.
input() had already become yyinput() in the C++ back end in order to
avoid collision with predefibed C++ inoput. In a multi-language world,
this is good policy in general. There's no real reason for C to
be different, and excellent reason to pull all possible entry
points into the yy namespace.
Non-C/C++ back ends won't have macros, so the documentation should
treat macro-ness as an implementation detail when it has to be
mentioned at all - and usually it doesn't.
The key thing about rule hooks like yyless(), yymore() etc. isn't that
they're macros, it's that they can only br used in rule actions (e.g.
inside the body of the genetated tyylex code.
This documentation patch removes the term "macro" where it isn't needed.
...rather than splicing a bunch of exposed guts into the middle of
yylex(). yyread() is put in the set of functions that gets
prefix-modified.
This means buffer refill can be documented without C-specific
references to YY_INPUT.
It should also enable actually having a non-macro replacement
for YY_INPUT, with a bit more work.
No specific thing can be said about a non-C/C++ backend yet, but
this patch prepares the way by explaining which features and aspects
of the Flex interface are specific to C/C++.
It also fixes one pre-ANSI prototype - that of non-reentrant yylex(),
which should be declared yylex(void) in this day and age.
These changes make one new commitment. Observing that the YY_INPUT
macro is impossible to port out of the C/C++ context, I have observed that
it is probably extinct in the wild (due to the later introduction of
the multi-buffer primitives, though I don't say that explicitly). The
text says tat people who really need the equivalent of this capability
in a non-C/C++ back end should file an issue with the Flex maintainers.
I don't actually expect this to happen.
This patch implements and documents a yyreject() macro to replace
argumentless REJECT. It does not remove REJECT, but warns that this
macro will not be supported in non-C languages and deprecates it.
This commit begins a new appendix in the Flex manual, to list
deprecated interfaces and explain why they have been superseded.
Flex has a strategy of packing its arrays with in32 or imt32 depending
on length, but it wasn't applied consistently. While I don't thinlk this
kind of space optimization matters a lot in 2020, if we're going to do it at
all we should do it thoroughly.
No need for it, since the skel content is in core
and the relevant hook can be searched for.
This is a postscript to the retargeting series. It's
not necessary, but it imoroves the code slightly.
It will doubtless need expansion and revision when we actually
write one.
No diffs in generated test code.
#70 and last in the retargeting patch series
Everything Flex ships to the skeleton-file expansion phase is
now either a macro expansion or a macro call. This almost finishes
the retargeting patch series; the wrapup will be documentation.
Sadky, this does not get us *all* the way to target-syntax independence.
The probem is the inclusion of tables_shared.c when table serialization is
enabled. Which ,eans table serialization is not practical to support
outside the C/C++ back end.
No diffs in generated test code from this commit.
#69 in the retargeting patch series
Produces a tediously large diff in generated test code that
is all table moving around. This is due to them being shipped
as macros and being substitured in a fixed order determined
by the calls in the skell file, rather than veing generated as
the functrions originally emitting the tables are called.
#68 in the retargeting patch series
What it used to do is now handled entirely by macro conditionals.
Besides being a good complexoty reduction in itself, this is one
of the last steps in turning C backend methods into macro deliveries.
Order of the yydmap table is perturbed. No other non-whitespace
diffs and no logic changes,
#67 in the retargeting patch series
This commit collects several minor changes:
* Fix a minor type specification bug in a tablesext initializer.
* macroize the trans_offset, mkctbl, and mkftbl methods.
* Fix a bug in footprint computation.
This commit oroduces no code diffs in the generated test code, but the
footprint reports change due to the bug fix.
#66 in the retargeting patch series
...to have an indent style uniform with the rest of the code,
and one that makes it easier noy to miss the trailing table delimiters.
Not all tables are generated this way yet. I'm working on it.
Is isolated in its own commit so the format change can't confuse a
reviewer's eyeballs out of noticing real mutations in the table data.
#65 in the retargeting patch series
Also macro-generate yydmap entry for the yymeta table.
We're npw about 75% of the way through pushing all
C syntax out of the method table.
Permutes table order in the generated code.
#64 in the retargeting patch series
This required addin a new 0.0 breakpoint right after the
M4_HOOK_* definitions so they will be visible early.
Produces no diffs in generated test code.
#62 in the retargeting patch series
They were: geneol, fulltable, eecs, and debug.
To accomplish this, dataend's emission of trailing } needed to be
suppressable.
Also, remove a %% mark that is no longer required.
This doesn't change any of the generated tables, but does change the
orer in which they're generated, froducing large diffs in the
generated test code that don't actually mean anything. The reason for
this is that tables used to come out in a variable order as functions
like geneecs were called ar variable times depending on the
compressuion mode. Now, instead, the order is fixed by where the
tanle-body macros these functions define are expanded.
More methods remain to be turnerd into macro generators.
#61 in the retargeting patch series, following an unnumbered
bugfix patch that I shipped in too much of a hurry.
Do this for table opener/closer/continuation syntax, the trace-format
string, the state entry string, constant definitions, the state-dyad
format, and the three pieces of EOF state syntax. The documentation
appendix on how to write a back end is also updated.
There are comment diffs because I decided generating an
explicit fallthrough marker and some other new explanatory comments
was a good idea.
#58 in the retargeting patch series
All symbols except a handful dependent on nultrans and the number of
backups are now written in one visible group right at the start of m4
generation. The exception are exceptions because their values
are not known until after DFA computation.
Has comment diffs in generated test code due to one symbol rename and
symbols beoming visible. Should be the last time the latter happens.
#57 in the retargeting patch series
Includes handling of --nounistd, --always_interactive, --never_interactive, --stack,
their corresponding lexer items, and and noinput.
An unavoidable side effect is that the place where "#define
YY_NO_INPUT 1" is inserted, if it's inserted. has to move because it's
done by a different route - m4 expansion rather than the action_define
function (which is now gone - this was the last use). I have put the
new insertion point just iin time for the first reference to the macro.
Otherwise the only diffs in generarted test code are symbol
definitions becoming visible.
#56 in the retargeting patch series
Formerly, Flex's own lexer and the logic for pocessing command-line
options both did calls to write M4 conditionals to a buffer that was
later dumped into the befinning of the text that m4 expands, before
the body of the skel file.
This was bad layering. Instead, both these places now set flags in
the ctrl structure. Later, (almost) all the generated m4 conditionals
are shipped at once.
It's "almost" because there are a couple of awkward cases to be
cleaned up. Again, this was the part that could be done
simply via almost mechanical cut and paste.
In generated code, there are some comment diffs because symbols that
used to be invisibly set are now visibly set - that is, shown at the
beginning of the generated C.
#55 in the retargeting patch series
This is separate from the big reorganization in commit #52 because
there's a comment about this variable in flexdef.h that makes me
nervous. According to the comment this variable is a trit, but
it looks to me like flexinit sets it to false and I can't find
anywhere in the code that sets it to a non-boolean value.
This commit asumes that the comment is stale and the member
can be typed boolean. Should be audited.
As I was working on some layer separation. I realized that I
was getting confused a lot by the huge pile of globals that
control this program.
In particular, I need to be able to clearly distinguish those that set
m4 conditional symbols from those that don't. So I've done something
about it. Almost all globals that can be set by options are now
bundled into two context structures, "ctrl" for options that have
corresponding m4 symbols and "env" for options that don't.
The few I haven't moved have sufficiently tricky interdependencies
that I'm going to break out any changes related to them into smaller
patches that can be easier to review. In this one I did only the bulk
of straightforward changes that could be done mechanically with search
and replace.
I changed one varuable name to reflext its senantics better;
the performance_report global is now env.performance_hint.
Ideally there ought to be a third structure that bundles all the
shared state used by DFA/NDFSA table computation, so all globals would
live in one of three context structures. I may do that in a later
commit, but this patch is already unpleasantly large as it is.
No diffs in generated test code, nor any logic changes.
#52 in the retargeting patch series.
There were only two left, for YY_MAIN, and that definition
was moved so it's in the visible controls.
This is a step towards making *all* conditionalization symbols
viible in generated comments.
This commit also cleans up some misnamed mode symbols. There are
still a couple of duplicative pairs, to be cleaned up in a later
commit.
We can now report generated M4 symbols with values in the "m4
controls" part of a generated file. Partly as a result, the following
symbols become visible in generare code from the tests:
M4_MODE_PREFIX, M4_YY_TABLES_VERIFY, M4_YY_REENTRANT, and
M4_MODE_PREFIX.
No other diffs.
#51 in the retargeting patch series. #50 was accidentally
unnumbered.
It was a no-op anyway in the C version, there as a placeholder
in case other languages needed it. But in the new organization
of things, with everything being done by conditional expansion in
the skeleton file, there's no point.
No diffs at all in generated test code.
Thios does remove some cpde that was conditioned out, an abandoned
attempt to undefine all #defines at the end of code generation.
Now that all the mode conditionals are visible early, wverything that
used to be done in the prolog can be done as conditionalized code in the
skeleton.
Whitespace and comment diffs only.
#49 in the retargeting patch series
Also, clean up some unused and duplicative symbols.
In generated test code, comment and whitespace diffs only
except for YY_INT_ALIGNED going away.
#48 in the retargeting patch series
The point of this change is to move the setting of the M4_MODE_*
controls up to the front of the generated code so that they can be
used for conditionalization earlier, notably in replacing the prolog
method. I tried to do this in #46 but dididn't move the mode
setting far enough up.
(Also, rename instances of a duplicated mode switch.)
In generated code, the m4 controls move but nothing else changes.
#47 in the retargeting patch series
This should make it possible to eliminate much of the C-specific
prolog code.
Sadly, because of the moves of the generated comments this makes
a rather noisy diff. All comments and whitespace, though;
what looks like being other than that is pieces of generated code
being shifteed around.
#46 in the retargeting patch series
Produces only whitespace diffs in generated code for tests, except the
order of items in the initializer for table serialization changes.
#45 in the retargeting patch series
This feature is better implemented with m4 macroexpansion;
that way skelout() does not have to know that #define is a thing.
Also in skelout(), use the backend comment method rather than
embedding knowledge about /* and */, and int_format_define
to factor out knowedge about #define.
Produces only comment diffs in the generated test code.
#44 in the retargeting patch series
This patch is a pure refactoring step. It changes the
interface between gen.c and the back end so that the
method table can shed a number of methods and no headers
are generated in gen.c any more.
Most methods now return the amount of memory they
allocate. Eventually this will be used to add
a report on this to the generated code.
No diffs in generated code, even without ignoring whitespace.
#43 in the retargeting patch series, which turned out
not to be finished after all. These is ugly magic in skelout()
that needs to be factored out.
Pull the C mode setting for M4_YYL_BASE into m4.
Use if/then/else extension of m4_ifdef to simplify how yy_flex_debug
is set. In the process, fix a bug introduced earlier in this patchest
where yy_flex_debug initialization was inverted.
Remove unused skel breakpoints and renumber those left.
Diffs have only commenmt and whitespace changes *except* for the
corrected initialzation of yy_flex_debug.
#40 in the retargeting patch series
I did some indentation of the end-of-file case statements with tabs to
make the generated code a bit more readable. So there are trivial
whitespace diffs in generated code.
#39 in the retargeting patch series
Finally, scavenge trailing skel breakpoints that are no
longer used, as theey no longer carry information.
Diffs in generated code are all comments and whitesoace, as usual
#37 in the retargeting patch series
Turned out to be mildly trick because %if/%endif pairs have push/pop
behavior rather than combining.
In generated code, the location of the debug declarations moves,
and that's all.
#34 in the retargeting patch series
Presently only C can do this. Otherwise we fall through to storing
numeric indices for a tiny loss of performance.
No generated-code changes from this.
#33 in the retargeting patch series
Moves one table initializer in the generated code, but doesn't alter
it.
Leaves one FIXME. The computation of M4_YYL_BASE really ought to be
done in m4, but for some unknown reason I couldn't beat it into
dping the right thing. Do be revisited, bu right now finishing
off the last bits of code generation is more important.
#32 in the retargeting patch series
Generated-code diffs are blank lines with ] continuations and
shifted %% commends made by skel breakpoint directives. I could garbage
collect those, but I plan to do that all at once in a commit of its
own for clarity's sake.
#30 in the retargeting patch series
Also, remove two insertion points in the skel file
that are no longer used because we're now using m4
conditionals instead.
In the generated C, two magic comments from the insertion points go
away.
#27 in the retargeting patch series
Required defining a new method table entry and shuffling the order in
which things are generated a bit, but the generated-code diffs are all
trivial. Some comments from the previous C code are now in the
skeleton.
#26 in the retargeting patch series
...and case statements. Add guidance to backend.adoc. This is the
"eventually" promised in the comment on #10 of this series.
No diffs in generated code.
#25 in the retargeting patch series
...and then inline the code of gen_NUL_trans() at its callsite. This
contains all remaining procedural code generation in make_tables(),
on the way to moving it to m4.
Generated code diffs are like those of #20 and #22, for the same
reasons. They add comments, whitespace changes, and uses of the
YY_NUL_EC and YY+JAMSTATE macros equivalent to the old numeric
literals.
#24 in the retargeting patch series
Also, move the yymatch label to the skeleton. Remove an unused
prototype. And fix a typo.
In generated C, comment and whitespace changes only.
#23 in the retargeting patch series
As with patch #20 in this series, the changes in generated code fall in the classes:
1. Addition of comments. I elected to copy the comments from the
generator functions into their corresponding m4 macros in hopes of
making the generated code less nasty to read.
2. Lines like "if ( yy_current_state >= 12 )" changing so the numeric
literal is replaced by YY_JAMSTATE + 1. This was a consequence of the
change in #15 to simplify the C code generators so they could be
translated into static nacros.
3. "YY_CHAR yy_c = (*yy_cp ? yy_ec[YY_SC_TO_UI(*yy_cp)] : YY_NUL_EC);:
being replaced by "YY_CHAR yy_c = (*yy_cp ?
*(yy_ec+YY_SC_TO_UI(*yy_cp)) : YY_NUL_EC);" The strange way of
indexing yy_ec avoids a syntactic collision with the use of [] as
m4 quotes.
The gen_next_state(false) case was translated to m4 at patch #20#22 in the retargeting patch series
Also, two GEN_START_STATE calls get inlined in the skeleton file.
Changes in generated C from the tests are all comments and whitespace.
#21 in the retargeting patch series
To verify this patch, notice that the non-whitespace changes in the
generated C from the tests are of only three kinds:
1. Addition of comments. I elected to copy the comments from the
generator functions into their corresponding m4 macros in hopes of
making the generated code less nasty to read.
2. Lines like "if ( yy_current_state >= 12 )" changing so the numeric
literal is replaced by YY_JAMSTATE + 1. This was a consequence of the
change in #15 to simplify the C code generators so they could be
translated into static nacros.
3. "YY_CHAR yy_c = yy_ec[YY_SC_TO_UI(*yy_cp)]" being replaced by
"YY_CHAR yy_c = *(yy_ec+YY_SC_TO_UI(*yy_cp));". The strange way of indexing yy_ec
avoids a syntactic collision with the use of [] as m4 quotes.
#20 in the retargeting patch series
Generated-code changes consist of whitespace and a comment.
Thus patch also fixes the computation of two guard symbols
that weren't actually used yet, prospectively introduced in
patch #15 of this series. At the time I wrote that I didn't
understand what num_backing_up was doing and failed to
translate it properly.
#19 in the retargeting patch series
The pieces are BOL checking, yylineno updating, and YY_RULE_SETUP
generation.
No generated code changes at all from this.
#18 in the retargeting patch series
Differences in generated code are trivial. Some integer
literals turn into symbol expressions as a side effect
of the previous patch.
#16 in the retargeting patch series
Changes the generated C sources from the tests, but only by making
them easier to read. In future patches this is going to make them
easier to compare by eyeball as well.
#15 in the retargeting patch series
This is the first patch in the retargeting series to substantively
modify the skeleton file. (The only previous one to touch it just
renamed a mode switch.)
Adds some comments and whitespace to the generated code in the tests,
but produces no logic changes.
#14 in the retargeting patch series
Presently this is used to pass -L, making it easier to compare changes
in generared C code cross revisions.
To perform such comparisons, do this at a base (before) revision:
# Make generated code
make clean; (cd tests; make clean); make check
# Save a smashot of it
rm -fr snapshot && mkdir snapshot && cp tests/*.c snapshot
Then rebuild at the later revision and do this:
# Compare gemerated code to snapshot
do diff -b -B -u $x tests/`basename $x`; done
#13 in the retargeting patch series
These aren't used yet. They will be as we push code feneration out
of gen.c into the epansion of the m4 skeleton.
This is the first revision to touch the C skeleton file, changing
M4_YY_USES_REJECT to M4_MODE_USES_REJECT for uniformity with the other
mode switches.
C code generated from the tests is still unchanged at this revision.
#11 in the retargeting patch series
...and one additional method.
This almost finushes off all language-specfic detauks outside of
gen.c. The one exception is just below the comment "add terminator
for initialization" in dfa.c; we'll get to this eventually.
#10 in the retargeting patch series
A large patch, but conceptually simple. Takes a bunch of functions and
callsites that knew details of C initializer and declaration syntax
and moves them into methods in the C backend table.
This is a pure refactoring step. At this point in the patch series,
not only do all tests pass but nothing in the lex-generated C code
from the tests has changed.
#7 in the retargeting patch series
This means it's possible to have multiple language-specific skel
files. All go through the same double m4 macroexpansion.
#5 in the retargeting patch series
A build test is done to determine the minimum versions of the configure
tools (Autoconf, Automake, Libtool & gettext) that can successfully
build flex in the current state of the code. This commit would help
builders to figure out whether their build tools are still compatible
for building flex, or they need tools upgrade, especially when they
patch flex and need to regenerate configure script for any reason.
This commit resolves version requirement debates as in #374.
A summary of version requirements: Autoconf 2.62, Libtool 2.2,
Automake 1.11.3 and gettext 0.19.6.
Signed-off-by: Kang-che Sung <explorer09@gmail.com>
* Don't silence `libtoolize --version` error message. It would be
useful for diagnosing any problem when `libtoolize --version` fails.
(libtoolize not found? No permission to execute? Now you know which
error it is.)
* Change the guessing code of libtoolize program name to a loop.
It'll perform silently.
* Silence the `libtoolize --version` output when it succeeds.
Signed-off-by: Kang-che Sung <explorer09@gmail.com>
* Let autogen.sh error message go to stderr.
* Don't quote the variable $LIBTOOLIZE when invoking it. This would
allow invocation when LIBTOOLIZE='a-future-libtoolize --gnu-compat'
Signed-off-by: Kang-che Sung <explorer09@gmail.com>
This prevents compiler's "redefinition of typedef" warnings or errors
when we could include flexint_shared.h twice, when building flex's own
scanner (scan.c) or in the "multiple_scanners" test in the testsuite.
(Depending on the compiler's flags this might or might not produce
errors, but it's good to guard it anyway.)
Fixes: #329
Signed-off-by: Kang-Che Sung <explorer09@gmail.com>
Since automake takes a hands off approach to the po/ directory, we
need to list update_linguas.sh in the top Makefile.am. We list it as a
dist_noinst_SCRIPT so that it will get distributed with executeable
permissions. We also list autogen.sh in the same way for the same
reason.
For an _empty_ state stack the top of the state stack defaults to the state as if no state stack were present.
NB: sanity check for `yy_start_stack_ptr` could be added in `yy_top_state()`.
Visual C++ 2010 does not define __STDC_VERSION__, and (before Visual
C++ 2013) has <stdint.h> and not <inttypes.h>.
ANSI/ISO C only requires at least 16 bits for int type (not 32 bits as
required by POSIX since SUSv2), so check the bit length before the
typedef.
Signed-off-by: Kang-Che Sung <explorer09@gmail.com>
The [U]INT{8,16,32}_{MIN,MAX} macros are never used in skeleton code.
Having them in skeleton just increases the chance of conflicts in case
that user defines them in non-C99 environment (see issue #307, when
flex code is built in Visual C++ (before VS2013)).
flexint.h is now split in two files. Only "flexint_shared.h" will be
included in skeleton now, which defines flex integral types.
flexint.h contains integer limits macros that would be used in flex
only.
Signed-off-by: Kang-Che Sung <explorer09@gmail.com>
Commit a17d79e9c722a6735b6d2a8f152287404f27df32 defines _POSIX_C_SOURCE
to the minimum of 1 if it's not defined in the user's scanner code or
the compiling environment. However in glibc the macros are not yet set
up until one of the libc headers is included. This unfortunately have
made us overwrite the default _POSIX_C_SOURCE value that would be
defined by glibc (200809L at the time of writing), causing regressions
in user code.
Now in this patch:
1. Ensure feature test macros have been set up in glibc before checking
or defining any of them in our skeleton code.
2. Have a more conservative logic when determining the need to define
_POSIX_C_SOURCE (required for fileno()).
Fixes: #313
Note:
It could be tricky for application code to ensure feature test macros
have been set up in glibc, since <features.h> is no portable header and
not meant to be included directly by applications. The way to do it is
to include a libc header which in turn includes <features.h>. However,
many of the glibc headers check __USE_POSIX (a glibc internal macro
defined if _POSIX_C_SOURCE is defined) and determine which interfaces
to expose already, making the headers inappropriate for our goal.
Those which don't depend on _POSIX_C_SOURCE, and are also available
since ANSI C89, are only <assert.h>, <errno.h> and <math.h>.
<assert.h> is finally favored due to other considerations:
- <math.h> check for __USE_XOPEN in glibc, making a dependency on
_XOPEN_SOURCE, besides it exposes much more interfaces than we need.
- In djgpp, <errno.h> depends on _POSIX_SOURCE to hide definitions of
some errno values when it's defined.
- <assert.h> exposes the fewest interfaces among the 3 headers and, at
the time of writing, checks for only C99 (for __func__), C11 (for
_Static_assert), and _GNU_SOURCE when needed.
Signed-off-by: Kang-Che Sung <explorer09@gmail.com>
xgettext was not very clever at interpreting lex patterns and can get
confused when seeing unquoted quotation marks, and emit warnings for
them. Now fix the warnings by properly quoting the quotation marks in
lex regex patterns.
Example
Original: \"[^"\n]*\" -> "warning: unterminated string literal"
Fixed: "\""[^""\n]*"\"" -> OK
My basic build test shows that the generated stage1scan.c is
bit-identical to the original.
Signed-off-by: Kang-Che Sung <explorer09@gmail.com>
GPG signatures require an external keyserver which might be offline,
which is undesirable for build server use. It's equally secure to just
hard-code the hashes, provided they're trusted (i.e. you verify a hash
against a GPG signature once).
Fixes: #311.
As a side note: the original two signatures
(gettext-0.19.8.1.tar.lz.sig and automake-1.15.1.tar.gz.sig) are signed
against the files' SHA-1 hash.
Due to a bug, autoheader (2.69) will treat M4 dnl comments in a quoted
argument of AC_CHECK_FUNCS as function tokens and generate a lot of
redundant and useless HAVE_* macros in config.h.in.
(Examples: HAVE_DNL, HAVE_AVAILABLE_, HAVE_BY)
It seems to be this commit dbb4e94dc7bacbcfd4acef4f085ef752fe1aa03f of
mine that revealed this autoheader bug, and the affected config.h.in
had been shipped within flex-2.6.4 release tarball.
I have reported the autoheader bug here:
<https://lists.gnu.org/archive/html/bug-autoconf/2018-02/msg00005.html>
As a workaround, let's move comments out of AC_CHECK_FUNCS.
Signed-off-by: Kang-Che Sung <explorer09@gmail.com>
Slightly rewrite the logic in yytbl_data_load() and yytbl_write_pad64()
so they simply check if the bytes read/written are in 8-byte boundary.
No need to calculate how many bytes we need to pad. (Incidentally this
makes smaller code in x86_64.)
For yytbl_hdr_init(), just expand the calculation from the macro.
For ccladd(), if cclp given is a non-last ccl, adding a char into it
will overflow the buffer and overwrite the first char in the next ccl.
For now, add a temporary detection and protection code. (Not sure if
this could happen in user input, but if it could, then you can expect
some "corrupted" behavior for generated scanners.)
Because ENABLE_NLS may be defined to 0 (manually, not through
autoconf) and it's semantically incorrect to only check whether it's
defined.
This is a correction to commit
661d603b65385f62f372acd2017e5af2e0f0cd50.
This would, e.g. define _GNU_SOURCE in config.h, enabling the
reallocarray() prototype in glibc 2.26+ on Linux systems with that
version of glibc.
Fixes#241.
config.h may define macros that alter the API of the standard library
funtions, and so it should be included before any other standard
header, even before the skeleton's standard header inclusion.
For example: config.h may #define _GNU_SOURCE that would expose the
reallocarray() prototype from <stdlib.h> on glibc 2.26+ systems. If we
include <stdlib.h> before config.h, reallocarray() would not be
available for use in lex file since the second include doesn't help
due to header guard.
For now our config.h might `#define malloc rpl_malloc` -- this
substitution must work before including stdlib.h, or else the compiler
will complain about missing prototypes, and may result in incorrect
code in scan.l (gcc warning: return makes pointer from integer without
a cast [-Wint-conversion]).
Fixes#247.
These are wrappers around automake- and libtool-generated targets,
allowing users to build libfl only, without the main flex program.
See GH-256 for discussion.
Signed-off-by: Kang-Che Sung <explorer09@gmail.com>
Don't use program_name in the description of -T/--trace or -V/--version
option. It's ugly when user invokes flex with a long path like
"/home/username/tools/bin/my-custom-built-flex".
This solution is not long term. If possible, the help text should be
modified so that the "flex" name is no longer needed below the first
"Usage:" line. All translations of help text will need to be updated
as well.
There's no technical need of stripping path from program_name. I think
the users should be fine if they see the path they use to invoke flex
is diagnostic messages and help texts.
Yes, users will see "Usage: ../flex [OPTIONS]..." now if they invoke
flex with the path "../flex".
The --version output has been changed so that the name field will be
always "flex" or "flex++". If the flex program has been renamed to
"lex" (for compatibility or other reason) this will allow identifying
the implementation name ("flex"). (And it's a recommended practice in
GNU Coding Standards)
In skeleton files comments are indicated by leading `%#` and when
directly read in using `flex -S <skeleton.skl>` they should be
ignored. Example: `flex.skl`.
Amending commit 2f21edac99b5efc432417233e6e53326d630e08f which removed
this conditional branch.
It's simply to return (regexec(®ex_blank_line, str, 0, NULL, 0) == 0);
The reason for encapsulation is to allow replacing this with a
non-regex method if necessary.
**Issue:**
Scanner generation leaks memory for transition tables when invoked
without `--tables-file` option.
**Root cause:**
`gentabs` function has different memory acquire/release conditions.
**Solution:**
Reclaim memory at the same scope where it was alloacated.
**Issue:**
Scanner postprocessing leaks memory during correction of `#line`
directives values and generation of C header file.
**Root cause:**
`filter_fix_linedirs` and `filter_tee_header` functions do not
dispose allocated memory.
**Solution:**
Automatically reclaim affected memory by allocating it on stack
insted of heap. Stack allocation should not be a problem as its
only 512 bytes and there is no recursive calls.
NetBSD had a crash during build. Since the provided substitute for
reallocarray() wasn't working, use NetBSD's reallocarr(). Let
configure choose that function whenever it is available. Use
reallocarray if available. Still fallback if neither is available.
Fixes#219
A #line pre-processor directive specifies the line number and source
file of the following lines. If the source file _is_ the current file,
the line number should be that of the line following the directive. So
the specified line number should be the current line number plus 1.
-C*f and -C*F option-specific test files collide on case-insensitive
file systems, and cause tests to either not be run at all, or to
overwrite each other's files at build time. So rename -C*F ones to
-C*_F.
When running 'make V=0 check', some compile and link steps are made
less verbose and some are not. This cleans up most steps that are
not.
These have no effect if V=0 is not specified.
Files in lib/ are picked up and built using makefile directives in
src/Makefile.am. Remove the need to generate lib/Makefile.in and the
stub lib/Makefile.am.
* Don't call isascii() here. It's deprecated in POSIX and not needed
for myesc's case.
* The check of the character class and range here should match what's
defined as {ESCSEQ} in scan.l, so for [[:xdigit:]] we use isxdigit();
for [0-7] we check '0' <= c <= '7' (not isdigit(c) because isdigit is
locale-dependant in standard's sense)
* Add missing length limit for "\x<hex>" (<hex> is at most 2 digits)
Previously, "make dist" and similar commands would cause the po files
to be updated, which is incorrect. The more recent version now used
should not do this.
Replaced Rules-getpo by extra script to manually update and rebuild all po files.
Listing the flex and bison sources -- as the gettext manual suggests
-- causes warnings. Listing the generated C files for the lexer and
parser removes the warnings. Since both files contain line directives,
any translator who needs to refer back to those files should be able
to find the context in which strings are needed.
Previously, we would call libtoolize and if that failed, we would call
glibtoolize since some platforms have the libtoolize program under
that name.
Instead, we check to see if the environment variable LIBTOOLIZE is set
and use that if so. Otherwise, we try to find a program "libtoolize"
on the path and failing that glibtoolize. We alert the user if no
suitable program can be found.
Check for cross compiling. If cross compiling, build stage1flex using a
custom link command.
We also override LDADD since that adds the replacement implementations
that are cross compiled, and instead always use the replacement library
implementations.
We don't use BUILD_OBJEXT and BUILD_EXEEXT since it seems that automake
does not support these.
Fixes#78.
strings.h (not string.h) is the standard-defined include header for
strcasecmp(). Include <strings.h> for portability (even though glibc
exposes strcasecmp() declaration also in <string.h> by default).
`configure` will now error if a required header or function is not
found on the system.
Also add comments on optional functions checks.
Add sys/stats.h and sys/wait.h to list of required headers in
configure.
This fixes issue #180.
Replace the naive "for" loop in determining power of two with a clever
bitwise solution. This code is around the Internet already and is in
Public Domain.
"Unlisting" the pthread test when libpthread is not available is never
a good idea. Should let it compile into a stub program that returns
the "skip" status.
configure.ac and tests/Makefile.am are edited so that pthread test can
be built anyway, but only works (not as the stub) when both pthread.h
and libpthread are detected on the system.
Also fix a small typo in tests/pthread.l comments.
Revert change to copy scan.c when building the distribution
archive. Move it instead. This satisfies automake's notions of what
should be clean and ensures that the distribution archive contains a
just-built copy of scan.c.
Fixes#186
This will ensure the flex scanner included in release tarball being
generated by exactly the flex version to be released.
Here's one catch after this change: You may not run "make dist" or
"make distdir" if you configure the flex source to be cross-compiled.
Revert commit e02c3d27eeb6b53ec82532ab080c23ce26813fd4
("include the intl/ subdirectory when searching for include files")
Flex doesn't come with libintl source code, so inclusion of
$(top_srcdir)/intl subdirectory during compiling is unnecessary.
This flag seems to be left after "remove intl from dist"
(commit 8f45da06adbd7dbcdfdbdd5c20ff4c47fc504f35) without someone
cleaning this up.
Users who need to build with gettext functionality from libintl would
use the '--with-libintl-prefix' configure option.
Fix unneeded double malloc - let regerror(). Write its message after
our "regcomp for (regex) failed: " string.
Also, errbuf cannot be free()'d because of flexfatal containing a
longjmp.
AC_PROG_YACC already detects the bison program. We can just use it,
and make notice if [ $YACC != 'bison -y' ]
Remove unused AC_SUBST([BISON], ...).
Add AM_CONDITIONAL([HAVE_BISON], ...) so that we can skip bison tests
in testsuite when bison is absent.
We don't make any attempt to detect other yacc implementations since
flex is developed with bison. While it would be possible, the need
isn't there right now. It would be possible to add a --with-parser
option that defaulted to bison if such a need cropped up.
The current clauses stating "main.c: parse.h" and "yylex.c: parse.h" do
not work as expected. Make did not try to build parse.h upon building
flex-main.o as it would think main.c exist already and ignore the
clause. Fix this by explicitly stating that the .o files depend on
parse.h instead.
This dependency bug only happens if user builds flex from a
checked-out repository.
If a ${foo}_scanner.c exists but the respective ${foo}_scanner.h is
missing, then 'make' can halt. Fix this by deleting the
${foo}_scanner.c file and then remake it. This rule is the same as the
automake-generated rule for .c files that require LEX or YACC (except
automake will use ylwrap).
Without BUILT_SOURCES in Makefile, no test sources in tests/ will be
built when using the default "make all" target. This is desireable for
people who are just building flex but who do not want to run, and
therefore, build any part of the test suite.
Also fix missing sources dependencies:
bison_nr_scanner.$(OBJEXT): bison_nr_parser.h
bison_yylloc_scanner.$(OBJEXT): bison_yylloc_parser.h
bison_yylval_scanner.$(OBJEXT): bison_yylval_parser.h
Use reallocarray() when we have it (i.e. in OpenBSD system). When we
don't, use equivalent overflow detection for our allocate_array and
reallocate_array functions.
Remove lib/reallocarray.c from our LIBOBJS as we no longer need it.
Provide a fallback SIZE_MAX macro definition in flexint.h (not
preprocessor friendly, but enough for our reallocate_array use case).
BSD make doesn't like $< with more than one source
specified. Additionally, make sure to specify srcdir when referring to
source files since vpath builds depend on this.
This allows user to use a different shell if /bin/sh has problems.
The use of $SHELL (variable) to run scripts is already an Automake
convention. All shell scripts in build-aux are run by $SHELL.
They now all work under a POSIX compliant sh. Specifically, 'dash' can
now be used for running "make check" tests.
Note that none of these scripts has ever been using a pipe, so
`set -o pipefail` is unnecessary and removed.
If a '-1' test program returns a non-zero status, don't go further and
compare it's (numeric) output values. This allows a '-1' test to
indicate a "skip" status or an otherwise error. (Currently no such '-1'
test would skip like this, though.)
Shell syntax note: `set -e` will not exit with the syntax like this
`test $(false) = $(false);`. The exit statuses of `false` in the
example will be ignored instead. But putting the output in a variable,
such as `VAR=$(false)`, DOES exit.
This 'make indent' target has not been working since the directories
reorganization in flex 2.6.0. Now make it work again.
Note that the current indent profile breaks many styles of existing
code. The indent target should not be used until the .indent.pro
options are reviewed for desireability.
One place emitting a #line directive to the generated header was
missed in commit 647a92b9f4 when resolving #55. Fix it to respect
gen_line_dirs as well.
Binding a reference to a dereferenced null pointer is invalid and
compilers optimise away the &file == 0 check. We need a real stream.
yyin is available already, and yyrestart(NULL) is only supported when
yyin will not be used, so there is no harm in just passing in that.
Since we now always have a valid stream, we can skip the null check
too.
Fixes#98.
Removed a newline that caused a problem building the scanner in some
circumstances. Specifically:
'bad character' error when executing
/bin/sh ../build-aux/ylwrap scan.l lex.yy.c scan.c -- flex
If configure is run with '--disable-bootstrap', then stage1flex won't
be built and stage1scan.c will be generated by sed'ing scan.c.
This option is intended to workaround bootstrap bugs rather than to
fix the bootstrapping issues which are proving subtler and harder to
fix than anyone would like.
For various reasons, we may wish to build libfl explicitly even when
configure has been run with the --disable-libfl option. This is
possible, now, via 'make -C src libfl.la'.
Flex did not check the length of the m4 path which could lead to a
buffer overflow in some cases. Additionally, not all platforms believe
in PATH_MAX, so stop relying on it.
Fixes#138
The libcompat.la library was small and less friendly to bootstrapping
and cross compilation. Now, we will simply link individual object
files as needed, which is simpler.
HELP2MAN and TEXI2DVI (or the corresponding ac_prog variables) will
never be zero length as they fall back to the missing script; check for
the fall back and warn on that instead of always warning.
flex.1 is pre-generated in release tarball. If we delete it, the next
"configure and make" on the source directory will then require help2man
unnecessarily.
Make the flex binary an order-only prerequisite, and add back the
prerequisites from before 7cfb440. This prevents rebuilding the man page
whenever the flex binary is rebuilt, which causes problems if help2man
is not installed and will never work when cross compiling.
Fixes#108.
The PIC version of libfl was not being built correctly. From the lack
of bug reports around this problem, we conclude that the PIC version
of libfl is not used and so we drop it from the build build targets
for flex.
* Wrap everything in the raw document in 72 char per line limit.
* Proper casing for terms "Git" and "GitHub" (don't try to look lazy).
* Add unordered list marks when needed.
* Say `configure && make && make install` and quoted for fixed-width
font.
Signed-off-by: Kang-Che Sung <explorer09@gmail.com>
This fixes M4 quotation of certain strings beginning with `yy` (in
section 3 of the input file only) and character literals. The new
quotation method is also less brittle and faster.
Tests that relied on the old behavior were fixed.
Also, `yyconst` is no longer defined; use `const` (which it
unconditionally was defined to) instead.
This fixes M4 quoting of section 3 of the input file, including escape
sequences and character constants.
Tests were added to verify the behavior in section 3 with respect to
quoting. Both escaping of quotes and quoting of potential macro-start
characters are tested. Existing tests were also fixed to account for the new -- and now correct -- behavior. Many tests relied on the old behavior of expanding M4 macros
in section 3. They needed to be updated for the new behavior.
Previously, `[[` and `]]` were not escaped in strings, which led
to bad interactions with m4.
Also, don't break strings on newline, as GCC et al support whitespace
between a backslash and the subsequent newline.
Currently, every call to sf_push() realloc()'s _sf_stack, even if the
maximum size _sf_max wasn't changed. As the indentation beneath the
"if" clause already indicates, the realloc() should only be executed
if _sf_max was increased.
Found by compiling flex with the -Wmisleading-indentation flags of gcc,
which leads to the following warning:
scanflags.c: In function ‘sf_push’:
scanflags.c:42:5: warning: this ‘if’ clause does not guard... [-Wmisleading-indentation]
if (_sf_top_ix + 1 >= _sf_max)
^~
scanflags.c:44:9: note: ...this statement, but the latter is misleadingly indented as if it is guarded by the ‘if’
_sf_stk = realloc(_sf_stk, sizeof(scanflags_t) * _sf_max);
^~~~~~~
git clean -fdx && ./autogen.sh && \
mkdir -p ../build && cd ../build && ../flex/configure && make -j10 && \
make dist
failed with:
> help2man: can't get `--help' info from ../../flex/src/flex
Note:
There is no dependency except for the flex binary.
git clean -fdx && ./autogen.sh && \
mkdir -p ../build && cd ../build && ../flex/configure && make -j10
failed with:
> ../src/stage1flex -o stage1scan.c stage1scan.l
> stage1flex: can't open stage1scan.l
Note:
stage1scan.c is not necessairy in the "make dist" generated tar.gz file.
stage1flex will be build from scan.c (which is distributed),
and this will then generate stage1scan.c from scan.l
`WARNINGFLAGS' can be passed when invoking `configure'
and when invoking `make'
if configure switch `--enable-warnings' was given
then default to something useful if we have `GCC'
`WARNINGFLAGS' is not used when compiling `stage1flex'
to avoid unnecessary clutter
When using clang/llvm 3.8 to compile flex, the following warning is
emitted:
main.c:378:27: warning: the value of the size argument in 'strncat' is too large, might lead to a buffer overflow [-Wstrncat-size]
strncat(m4_path, m4, sizeof(m4_path));
^~~~~~~~~~~~~~~
main.c:378:27: note: change the argument to be the free space in the destination buffer minus the terminating null byte
strncat(m4_path, m4, sizeof(m4_path));
^~~~~~~~~~~~~~~
sizeof(m4_path) - strlen(m4_path) - 1
Fix it up by using the solution proposed by the warning message.
The variable yy_n_chars had been of type yy_size_t which is incorrect
given its use in read(). While it might be adviseable to look at
defining a yy_ssize_t, there might be some issues doing this and so, for
now, at least, we'll punt back to int.
The value of n_alloc was a count, not a size. Multiplying the value by the element size was incorrect. That multiplication was already being done and having it done twice was incorrect.
Allow specifying '%option noline' in the input file, leading to the same
effect as calling flex with the command line option --noline.
Signed-off-by: Tobias Klauser <tklauser@distanz.ch>
There are two instances in the code which will print a #line directive
to the resulting lexer, regardless of the value of gen_line_dirs. Fix
them, so they also respect gen_line_dirs.
Signed-off-by: Tobias Klauser <tklauser@distanz.ch>
The sparse static checker warns about using plain integer 0 as NULL
pointers in the generated lexer code. Fix this by using NULL
consistently for pointers.
Only the declaration of yy_fatal_error is marked with
__attribute__((__noreturn__)) in case GCC >= 3 is used, but not the
definition. This leads to the sparse static checker to complain about
function declaration mismatch.
Fix it by defining a macro yynoreturn and using it for both the
declaration and the definition of yy_fatal_error.
The prototype declares yy_fatal_error parameter as "const char msg[]"
while the definition uses "const char* msg" (introduced by commit
e9d5fc713f61b) which causes the sparse static checkers to produce an
error.
Fix this by adjusting the definition to use "const char* msg" as well.
Also change the C++ version accordingly so it matches the declaration in
FlexLexer.hpp.
Given the following program:
\#include <libgen.h>
\#include <stdio.h>
/* extracts basename from path, optionally stripping the extension "\.*"
* (same concept as /bin/sh `basename`, but different handling of extension). */
static char *basename2 (char *path)
{
char *b;
for (b = path; *path; path++)
if (*path == '/')
b = path + 1;
return b;
}
static void basename_compare(char *path)
{
printf("basename: %s\n", basename(path));
printf("basename2: %s\n\n", basename2(path));
}
int main (int argc, char *argv[])
{
// From http://pubs.opengroup.org/onlinepubs/9699919799/
// ``Sample Input and Output Strings''
basename_compare("/usr/lib");
basename_compare("/usr/");
basename_compare("/");
basename_compare("///");
basename_compare("//usr//lib//");
return 0;
}
... and the program's output:
basename: lib
basename2: lib
basename: usr
basename2:
basename: /
basename2:
basename: /
basename2:
basename: lib
basename2:
... we can see that basename2() behaves the same as basename(3) in the
average use case, but messes up pretty severely in others. Besides
that, basename(3) is mandated by POSIX so should be present on modern
Unix-like systems, so we shouldn't define it ourselves.
Some notes:
- it doesn't appear to be mentioned in POSIX, but OpenBSD's basename(3)
returns NULL if the returned path componenet is > PATH_MAX, so add a
check for that
- basename(3) shouldn't return an empty string, so remove the
program_name[0] != '\0' check
Just increment or decrement the indentation counter. That's less to
remember and makes the code more readable.
w# Please enter the commit message for your changes. Lines starting
The current test wrapper works only when the inputs are specified using
relative paths. If they're specified with absolute paths, the driver
fails to detect the inputs because it always prepends the input dir name
which itself is a relative path:
$ cd tests
$ ./testwrapper.sh -d . -i $PWD/reject.txt -t ./reject_ver.table
<fails to open inputs>
This normally doesn't show up because people run `./configure` or, for
out of tree builds, `../configure`. But if you happen to run configure
with an absolute path, then automake tends to generate absolute paths
as well leading to test failures.
Fix all of this by dropping the implicit input directory prepending.
- INPUT_NAME is often a list of files, not just a single one
- the input directory is used to find the testname tables which are
usually generated, so it's impossible to use files from both source
and build directories
- most of the time, the full/correct path is already specified
The [] characters are used for quoting in m4, so the attempt to use
them in place of `test` fails yielding the warning at build time:
.../flex/configure: line 20222: no: command not found
This is taken from OpenSSH Portable, which in turn takes it from
OpenBSD.
reallocarray wraps the stdlib's realloc function. It takes two size
arguments and checks for overflow, like calloc, but doesn't zero the
memory. Therefore, it allows us to do overflow-safe array reallocations
and overflow-safe unzeroed array allocations, which the stdlib
allocation functions don't.
We have a bunch of specific array allocation macros, none of
which check for overflow. reallocarray should be able to replace them.
Given the age of the MS-DOS and VMS platforms, it's likely that no one is building flex on them any more. Additionally, the preferred approach is to test for particular platform features rather than to test for particular platforms.
Removed version numbers for build tools. Noted that version requirements for build tools will be noted in configure.ac. Expanded documentation of building texinfo based docs.
The function flex_alloc() was just a wrapper around malloc(). Since this only added unclarity, and the flex_alloc() function is likely a legacy of olden times, remove it in favor of calls to malloc() directly.
Style elements cleaned up:
* superfluous spacing around parentheses
* non-constant initialization in variable declarations
* needless casts
* almost all uses of assignments as subexpressions
Changes in scan.l need to be built into flex with the same version of
flex in some cases. Since this build requirement is minimal, we simply
bootstrap flex unconditionally.
We intentionally exclude from version control the bootstrap artifacts as
the extra copy of the lexer, the intermediate scanner and the bootstrap
executable are not of interest.
Like copy_string(), copy_unsigned_string() is just a clone of the
stlib's strdup(). We only use it twice. I'm pretty confident that char
signedness is irrelevant in this case.
Rewrote the target for skel.c to explicitly mention the srcdir. This
should help when building flex from a directory outside the flex
tree. Spread the rule out over several lines to enhance readability.
On Mac OS X, libtoolize is known as glibtoolize. In cases where libtoolize is not present, then calling glibtoolize when bootstrapping the build system gives more folks a shot at getting flex built from the ground up.
Added test for presence of (g)texi2dvi program. Gave notice if texi2dvi
is unavailable and set TEXI2DVI=: to avoid giving users headaches.
Enhanced tests for bison and help2man with notices when the programs
aren't found. Set their program variables to use the missing script
in build-aux since it's compatible with them.
copy_string() was a clone of the stdlib's strdup(). For safety,
simplicity, and speed, we should use that instead. We introduce xstrdup() which wraps strdup() in a failure upon memory allocation errors.
Thanks to Michael McConville for pointing out that the old Char macro
causes problems with static analysis. The macro has been removed and
replaced with 'unsigned char' throughout the flex sources. The macro is
not needed at best and was confusing at worst. It was not used in any of
the example files nor was it mentioned in the manual at all.
flex was using K&R function definitions for some functions and
ANSI C style in others, sometimes even in the same file. Change
the code to consistently use ANSI C.
Signed-off-by: Stefan Reinauer <stefan.reinauer@coreboot.org>
# Consider using larger runners or machines with greater resources for possible analysis time improvements.
runs-on:ubuntu-22.04
timeout-minutes:240
permissions:
# required for all workflows
security-events:write
# required to fetch internal or private CodeQL packs
packages:read
# only required for workflows in private repositories
actions:read
contents:read
strategy:
fail-fast:false
matrix:
include:
- language:c-cpp
build-mode:autobuild
# CodeQL supports the following values keywords for 'language': 'c-cpp', 'csharp', 'go', 'java-kotlin', 'javascript-typescript', 'python', 'ruby', 'swift'
# Use `c-cpp` to analyze code written in C, C++ or both
# Use 'java-kotlin' to analyze code written in Java, Kotlin or both
# Use 'javascript-typescript' to analyze code written in JavaScript, TypeScript or both
# To learn more about changing the languages that are analyzed or customizing the build mode for your analysis,
# see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/customizing-your-advanced-setup-for-code-scanning.
# If you are analyzing a compiled language, you can modify the 'build-mode' for that language to customize how
# your codebase is analyzed, see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages
# If you wish to specify custom queries, you can do so here or in a config file.
# By default, queries listed here will override any specified in a config file.
# Prefix the list here with "+" to use these queries and those in the config file.
# For more details on CodeQL's query packs, refer to: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
Thank you for your interest in contributing to Flex! Flex's [issue
tracker](https://github.com/westes/flex/issues) is on GitHub. That's
the best place to find a task that needs attention. It's also the best
place to check whether a problem you've found is known to the
community.
## Baseline Build Environment
The maintenance baseline build environment is Ubuntu 20.04 LTS.
Flex needs a previous version of itself in order to build a new
version of itself. You can either install an operating system package
of flex, or you can build flex from one of the release tar balls.
Configuring your Flex development environment to match the maintenance
baseline will help you collaborate and review contributions with the
other developers working on Flex. See the
[INSTALL](https://github.com/westes/flex/blob/master/INSTALL.md) for
programs required to build flex from scratch.
## Branching Convention
Flex source code is maintained in Git at GitHub. Each Flex developer
forks their own copy of the main repository at
[westes/flex](https://github.com/westes/flex). Flex development occurs
in feature branches of the forks. PRs will eventually be submitted
from the feature branches of the forks to westes/flex.
Flex development forks are not required to reside on GitHub. However:
- GitHub's pull request (PR) process only works with repositories
hosted on GitHub;
- Flex's automated testing pipeline depends on GitHub's Actions
infrastructure and may not function on other Git servers.
Therefore, if your fork of flex resides outside of github, you may
wish to submit your patches via email, using standard git mechanisms.
## Preparing Your Commits
Small PRs are easier to review and merge. Minimize the scope of the
changes in a single PR within reason. Changes that touch one or two
files are likely to be reviewed and accepted more quickly than changes
that touch every file in Flex. Your pull request should do one atomic
thing unless there is a really good reason for your pull request to do
more than one thing.
Format your commit messages following [Conventional Commits
1.0.0](https://www.conventionalcommits.org/en/v1.0.0/) (CC BY 3.0
netlify.com). Briefly:
```
<type>[(optional scope)][!]: <description>
[BLANK LINE]
[optional body]
[BLANK LINE]
[optional footer(s)]
```
A typical commit message might look like:
```
docs: Add CONTRIBUTING manual
Refs: #1234
```
The description should summarize the changes in a single, short
sentence. The description should be written in the imperative mood,
like the descriptions of the types below.
The primary types are:
- build: Change the build system, including tracking external dependencies
- ci: Change the CI/CD system
- docs: Update the package documentation
- feat: Add a new feature
- fix: Fix a bug or incorrect behavior
- perf: Improve performance of a feature
- refactor: Simplify maintainability without affecting performance or behavior
- revert: Revert a previous commit
- style: Correct a style mismatch (indentation, etc.)
- test: Change or add to the test suite
The optional scope must appear in parentheses and must consist of a
noun describing the section of the code base that was changed.
```
docs(contrib): Update URLs in CONTRIBUTING manual
```
The optional `!` must appear before the description whenever the
commit introduces a breaking change - one that breaks compatibility
with previous versions of the code base. Whenever the `!` marker
appears, a BREAKING-CHANGE footer should also be included.
The optional body may be included to provid additional information
or context about the change. If it appears, it must be preceded by a
blank line.
The optional footers may be included to provide additional referential
information and links to tracked issues, PRs, etc. Each footer must be
preceded by a blank line. Footers must be formatted as `<footer-type>:
<description>`, similar to the first line of commit messages. The
description's format depends upon the footer-type. Known footer-types
include:
- Refs: A comma-separated list of commit hashes and/or issue and PR numbers. Issue and PR numbers must be prefixed with a `#`.
- BREAKING-CHANGE: A description of the change that breaks compatibility with previous versions of the code, including the version number at which compatibility is broken.
A breaking change commit message might look like:
```
feat!: Switch to quantum scanner
BREAKING-CHANGE: New scanner runs in constant time but doesn't support any existing hardware.
Breaks compatibility with 1.0.0.
```
Squash your commits so that each commit is an atomic change. This aids
discussion and revision during the review process. In particular,
there should be no commits that fixup prior commits.
## Submitting PRs
Before submitting a PR to flex, test your changes either locally or
using the GitHub Actions pipeline.
### Testing locally
1. Commit and/or stash your changes.
1. Clean your working copy using
```
git clean -xdf
```
1. Build flex following the directions in [INSTALL](https://www.github.com/westes/flex/blob/master/INSTALL.md).
1. Run both the `make check` and `make distcheck` targets.
### Testing with GitHub Actions
If you created a fork of Flex on GitHub, any PR you make to your own
main branch will trigger the build and test pipeline.
1. Commit your changes.
1. Push your local branch to your remote at GitHub. Assuming your GitHub remote is called origin:
```
git push origin feature_branch
```
To submit a pull request through GitHub's web interface:
1. Open your GitHub Flex repository in your web browser.
1. Click the 'Pull requests' link.
1. Click the 'New pull request' button.
1. Change the 'base repository' from 'westes/flex' to your fork.
1. Change the 'base' branch to 'master' if it isn't already set.
1. Change the 'compare' branch to your feature branch.
1. Click the 'Create pull request' button.
1. Click the 'Actions' link to monitor the progress of your build and test job.
### Submitting PRs to westes/flex
Sending a PR to westes/flex follows nearly the same process as sending one to your own main branch.
1. Commit your changes.
1. Push your local branch to your remote at GitHub. Assuming your GitHub remote is called origin:
```
git push origin feature_branch
```
To use GitHub's web interface:
1. Open your GitHub Flex repository in your web browser.
1. Click the 'Pull requests' link.
1. Click the 'New pull request' button.
1. Change the 'base repository' to 'westes/flex'.
1. Change the 'base' branch to 'master' if it isn't already set.
1. Change the 'compare' branch to your feature branch.
1. Click the 'Create pull request' button.
1. Add notes to your PR including
- A title or commit-like message
- A summary of the commits in your pull request
- Issue numbers your PR covers
- Links to your GitHub Actions test results or a copy of the last few lines of output from your local test results.
If this is your first contribution to Flex, execution of the Actions
pipeline will have to be manually approved by the maintainer. If you
are a returning contributor, you can click the Actions link to watch
your job run.
Keep an eye on your PR's discussion page and your email for review
This is the file NEWS for the flex package. It records user -visible
changes between releases of flex.
flex NEWS
See the file COPYING for copying conditions.
* Noteworthy changes in release ?.? (????-??-??) [?]
** build
*** Flex now includes its own <config.h> header before including
system headers so that any system specific features detected by
configure are taken into account during compilation of flex
itself.
*** The flex build system now includes Makefile targets at the top
level to allow just building and installing libfl.
*** The flex distribution now includes a file, src/libfl.pc, to allow
using pkgconfig to find out what flags to use when building
against libfl from flex.
*** Various edge cases, mostly involving out-of-tree builds have been
accounted for in the autotools build system.
*** A crash during building on NetBSD has been fixed.
*** Flex is now automatically built by travis-ci. That should increase
the visibility of bugs and help prevent regressions.
** documentation
*** new bg, eo, ka, pt, uk translations from the translation project
** scanner
*** Some memory leaks have been fixed.
*** A long standing bug that effected expressions of the form c{i,j}
where 'c' is a character and {i,j} describes the number of times
to match against 'c' when case sensitivity was turned on has been
fixed.
*** New option: --backup-file allows setting the name of the file
written containing backing up information. Useful if you need
backing up information from multiple scanners in the same
directory.
*** flex emits correct line number directives when line numbers refer
to the file containing the line number directives.
*** The options {no,}yy{get,set}_column are now supported.
** test
*** Generating the various tableoptions make rules is now more portable.
*** Tests of the options -C*f and -C*F will now run correctly on
filesystems that are not case sensitive.
* Noteworthy changes in release 2.6.4 (2017-05-06) [stable]
** build
*** The indent target now knows about flex's new (as of 2.6.0)
layout. The indent rules it would apply are not correct and do
need to be fixed.
*** The files included in the flex distribution are now built by the
version of flex that is included in the distribution.
*** The configure script has a better idea of which headers are
required to build flex. It will also error when missing functions
are detected.
*** We have lowered the versions of automake and gettext that
configure.ac lists as required for building flex. In autogen.sh,
we now check for how to call libtoolize and use what we find in
the rest of the script.
*** Since files in lib/ are picked up as needed by src/, we no longer
generate a Makefile for that directory.
*** Flex can be cross compiled.
** documentation
*** Some typos were removed from the manual.
** scanner
*** Some minor performance enhancements.
*** We honor user defined yy_* macros again. We are also more careful
to not leak macro definitions into header files.
*** A number of portability fixes were introduced so building flex is
more reliable on more platforms. Additionally, outdated function
calls were removed.
*** When building the flex executable itself, %# comments from
flex.skl are removed when generating the C source code array. This
reduces the size of flex.
** test suite
*** All scripts in the test suite are now run by $(SHELL) and the
needed portability fixes have been included.
*** Test suite dependencies are handled much better. This only matters
if you are actively developing flex or its test suite.
*** Tests that depend on platform dependent features now properly skip
when those platforms are not present.
*** When running "make check", you can now pas V=0 to silence more of
the build. This is useful when you're less concerned about the
details of building and linking the test programs themselves.
* Noteworthy changes in release 2.6.3 (2016-12-30) [stable]
** scanner
*** several bug fixes resolved problems introduced in recent flex
versions regarding processing of comments, literals and various
quoting scenarios.
*** If the path to m4 was sufficiently long, a buffer overflow could
occur. This has been resolved. The fix also removes dependence on
the constant PATH_MAX.
** build
*** A new configure option --disable-bootstrap changes the behavior of
the build system when building flex. The default
"--enable-bootstrap" behavior is to build flex, then to use that
flex to build flex again. With --disable-bootstrap, the scanner is
simply built by sedding the scanner source. This is friendlier to
cross compilation.
*** The compatibility functions in lib/ are no longer built as a
library. Instead, they are built as $(LIBOBJ) objects. This is
simpler and friendlier to cross compilation.
*** It is now possible to build flex without building the accompanying
libfl. This is friendlier to cross compilation. See the
--disable-libfl option to configure. Resolves #99.
*** the PIC version of libfl was not correctly built. It is no longer
included in the build/installation targets of flex since it was
unused.
*** the distributed man page is only rebuilt when the relevant source
files change or when the binary doesn't exist. In particular, this
is friendlier to cross compilation. Resolves #108
** test
*** the shell scripts in the test suite are more portable across different shell implementations.
* version 2.6.2 released 2016-10-24
** flex internals
*** a segfault involving yyrestart(NULL) has been fixed
*** flex should now handle quoting when mixed with m4 processing correctly
*** flex handles `[[' and `]]' correctly
*** flex no longer generates non-ANSI code
*** more compilation warnings were squashed in generated scanners
*** prevented a buffer overflow that could occur when input buffers were the exact wrong size
** test suite
*** input filenames on MSWindows are now calculated correctly
*** general code cleanups in a number of tests now make the test suite compile much more cleanly
** build system
*** the xz archive has been replaced with an lzip archive
*** a new option to configure --enable-warnings to encapsulate passing
of warning-related flags which is useful in testing flex
*** make indent now works for out of source builds
*** Portability warnings when generating Makefile.in files are now suppressed; they were just noise and the use of GNU extensions in Makefile.{am,in,} was intentional and well known.
** bugs
*** resolved gh#67
** new sv translation from the translation project
* version 2.6.1 released 2016-03-01
** flex resources
*** The flex project is now hosted at github. Consider this a "period of transition". In particular, you should start at https://github.com/westes/flex for the flex codebase, issue tracking and pull requests.
*** New releases of flex are to be found at https://github.com/westes/flex/releases.
** flex internals
*** Flex now uses more modern and more standard names for variable types. There's more work to be done on that front yet, though.
*** A number of compiler warnings have been remedied.
*** Line directives should now work as expected and be absent when that is expected.
** test suite
*** When running the test suite, c++ files are compiled with the c++ header inside the flex distribution, rather than relying on the build system's flex header , which might not be installed yet or which might be out of date with respect to what flex tests expect.
*** Some portability fixes in the test suite such as opening files for reading in binary mode
** Building flex
*** The file src/scan.c asdistributed with flex source is now built with the current version of flex. Occasionally this had to be done manually to pick up new flex features. It's now just a part of flex's build system.
*** The pdf version of the manual is no longer distributed with flex, although if you have the texinfo package installed, you can still build it.
*** lots of general build system cleanup
*** the build system tries a bit harder to find libtoolize and texi2dvi.
*** When help2man and texi2dvi are missing, the error messages are now much more helpful.
This is flex, the fast lexical analyzer generator.
flex is a tool for generating scanners: programs which recognize
lexical patterns in text.
The flex codebase is kept in
[Git on GitHub.](https://github.com/westes/flex) Source releases of flex with some intermediate files already built can be found on [the github releases page.](https://github.com/westes/flex/releases)
Use GitHub's [issues](https://github.com/westes/flex/issues) and
[pull request](https://github.com/westes/flex) features to file bugs
and submit patches.
There are several mailing lists available as well:
* flex-announce@lists.sourceforge.net - where posts will be made
announcing new releases of flex.
* flex-help@lists.sourceforge.net - where you can post questions about
using flex
* flex-devel@lists.sourceforge.net - where you can discuss development
of flex itself
Find information on subscribing to the mailing lists or search in the
dnl AC_PROG_LEX requires an argument in autoconf 2.70, but we cannot
dnl specify it through AM_PROG_LEX until automake 1.17.
AC_PROG_LEX([noyywrap])
AM_PROG_LEX
AC_PROG_CC
AX_PROG_CC_FOR_BUILD
AC_PROG_CXX
AM_PROG_CC_C_O
AC_PROG_LN_S
AC_PROG_AWK
AC_PROG_INSTALL
AC_PATH_PROG(BISON, bison,bison)
AC_PATH_PROG(HELP2MAN, help2man, help2man)
AS_IF([test "x${BUILD_OBJEXT-}" = x],
[AS_IF([test "$cross_compiling" = no],
[BUILD_EXEEXT="$EXEEXT"
BUILD_OBJEXT="$OBJEXT"],
[BUILD_EXEEXT="${ac_cv_build_exeext-}"
BUILD_OBJEXT="${ac_cv_build_objext-}"])])
AS_IF([test "x${BUILD_OBJEXT-}" = x && test "x${enable_bootstrap-yes}" = xyes],
[AC_MSG_ERROR([BUILD_OBJEXT is invalid; please regenerate 'configure' with a newer ax_prog_cc_for_build.m4 (serial 23 or later) or configure with '--disable-bootstrap'])])
pkgconfigdir=${libdir}/pkgconfig
AC_SUBST(pkgconfigdir)
# allow passing a variable `WARNINGFLAGS',
# either when invoking `configure', or when invoking `make'
# default to something useful if GCC was detected
AC_ARG_ENABLE([warnings],
[AS_HELP_STRING([--enable-warnings],
[enable a bunch of compiler warning flags (defaults to GCC warning flags).])],
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
