This function is described in its comments as 'terrifying', and by its
original author, Larry Wall, as "truly awful". As a result, it has been
mostly untouched since its introduction in 1993. That means it has not
been updated as new language features have been added.
As an example, it does not know about lexical variables, so the code it
has for globals just doesn't work on the vast majority of modern day
coding practices.
Another example is it knows nothing of UTF-8, and as a result simply
changing the input encoding from Latin1 to UTF-8 can result in its
outcome being the opposite result.
And it is buggy.
An example of how hard this can be to get right is this fairly common
use in our test suite:
[$A-Z]
That looks like a character class matching 27 characters. But wait,
what if there exists a $A and a parameterless subroutine 'Z'. Then this
could instead be an expression for a subcript.
A few years ago, I set out to try to understand it. I added commentary
and simplified some overly complicated expressions, but left its
behavior unchanged.
Now, I set out to make some changes, and found many more issues than I
had earlier. This commit adds commentary about those. Hopefully this
will lead to some discussion and a consensus on the way forward.
-CA (also known as PERL_UNICODE=A) tells perl to assume the command-line
arguments to be UTF-8. This did not apply to the global variables
implicitly created by -s, however, only to the elements of @ARGV.
Logically speaking it does not make sense to have half a command line
treated as UTF-8, so this patch makes -CA apply to everything.
Fixes#23377.
Local, non-static constant arrays may get re-initialised at runtime (by
pushing their contents on the stack) every time they are reached. This
somewhat surprising behaviour is caused by the fact that the C standard
requires (almost) every object to have a unique address.
In theory, a compiler may eliminate the re-initialisation, if it can
prove that the address of an array is never examined. However:
1. This often isn't possible to prove. The compiler has no way to know
what will happen to an array pointer once it leaves the compilation
unit.
2. Unlike Clang, GCC and Visual C++ don't even attempt this optimisation
at all. GCC has an open ticket for this:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59863
Making those arrays static avoids the problem, because static arrays are
guaranteed to be initialised only once, prior the to the program
startup.
As it's not required to pre-allocate PL_strtab hash before
perl_construct() and PL_strtab is not initialized with NULL by
default, SIGSEGV could happen in certain circumstances.
Particularly, when application links libperl and runs perl several
times. After the first run PL_strtab remains unitialized in
perl_construct() as it was not cleared in perl_destruct() and it
contains garbage and the next initialization in perl_construct() is
skipped because !PL_strtab check is skipped.
See also: c82f4881ef (Allow custom PL_strtab hash in perl_construct.)
Replace trivial uses of
if(ckWARN(WARN_FOO))
warner(packWARN(WARN_FOO), ...);
with
ck_warner(packWARN(WARN_FOO), ...);
This does mean that the format string arguments get evaluated even if
the warning category isn't enabled, but the most expensive thing I
could see was Strerror(), which I woudn't worry about.
Note, non-standard used of "static link" below, I am using it to refer
to static importing funtions/data symbols from another DLL, using the
PE import table. Opposite of "LoadLibrary()"/"GetProcAddress()"
linking. I am NOT using "static link" in typical usage of fully including
a copy of a library at compile time, through a .a/.lib/.o/.obj file.
Since commit
Revision: af2f850fb5b3bd37dab6742ca16cce6aa5482fb9 10/19/2015 5:47:16 PM
const vtables in win32/perlhost.h
the vtables have been stored in read-only memory. There have been no bug
tickets or complaints since, of any users, wanting or depending on this
runtime instrumentation system.
All Win32 perl builds, are static DLL linked to a specific MSVCRT (LIBC)
at interp C compile build time. No matter the name of the CRT DLL,
msvcrt.dll, msvcrt120.dll, ucrtbase.dll, etc. Runtime swapping the
libperl MSVCRT DLL by an embedder, to his favorite CRT DLL, has never
been supported, and wouldn't even work, since perlhost.h's hooking isn't
perfect, and often Win32 Perl uses "win32_*()" functions by accident, or
explictly, and those static-link call into the hard coded CRTs. Plus
prototypes of CRT posix-ish functions have changed over the years.
What is time_t? stat_t? etc. While func symbol name stays the same.
The original commit for all this complexity, was from 5.0/5.6 era, where
it was assumed, perl 5 maint/stable releases will be abandoned by P5P
in favor of Perl 6, and all this complexity were provisions and APIs,
to fix, upgrade and improve Win32 Perl, on Microsoft's/ActiveState's
rapid release schedule, without any dependency on
P5P devs/pumpking/P5P policy, about releasing a new perl5 .tar.gz.
0f4eea8fa1779e08575278392ed398ffeda6dcd2 6/19/1998 6:59:50 AM
commoit title "applied patch, along with many changes:"
"The features of this are:
1. All OS dependant code is in the Perl Host and not the Perl Core.
(At least this is the holy grail goal of this work)
2. The Perl Host (see perl.h for description) can provide a new and
improved interface to OS functionality if required.
3. Developers can easily hook into the OS calls for instrumentation
or diagnostic purposes."
None of these provisions and APIs, have ever been used. CPerlHost:: never
became a separate DLL. Perl >= 5.12 has a "rapid release" policy.
ActiveState dropped sponsorship/product interest in Win32 Perl, many years
ago. Strawberry Perl took over the market. CPerlHost:: is way too
over engineereed for perl's ithreads/psuedofork, which only requires
"1 OS process" and 2 %ENVs, and 2 CWDs, functionality. Most of the
CPerlHost::* methods are jump stubs to "win32_*();" anyways, and the
hooking is redundant runtime overhead, but that is for another commit.
This commit is about removing the pointless malloc() and memcpy() of the
plain C to C++ "thunk funcs" vtables, from the RO const master copy in
perl5**.dll to each "my_perl" instance at runtime.
On x64, copying the vtables to malloc memory, wasted the following amounts
of malloc() memory. These are the actual integers passed to malloc() by
CPerlHost::/perl. malloc() secret internal headers not included in these
numbers.
perlMem, 0x38
perlMemShared, 0x38
perlMemParse, 0x38
perlEnv, 0x70
perlStdIO, 0x138
perlLIO, 0xE0
perlDir, 0x58
perlSock, 0x160
perlProc, 0x108
The old design of malloc-ed vtables, seems to have been, from the
original devs not knowing, or a poor understanding, how MS COM
(C++ obj in plain C) and MSVC ISO C++ objects (almost same ABI), are
layed out in memory. The original devs realized, if they use a ptr to
global vtable struct, they can't "cast" from the child class like
VDir:: or VMem::, back to a CPerlHost:: obj which is a design
requirement here.
But they wanted to pass around child class ptrs like VMem::* instead of one
CPerlHost:: obj ptr, and those VMem:: ptrs must be seen in 'extern "C"'
land by plain C perl, since my_perl keeps 9 of these C++ obj *s as seperate
ptrs in the my_perl "plain C" struct. So instead they made malloced copies
of the vtables, and put those copies in the CPerlHost:: obj, so from a
child class ptrs, they can C++ cast to the base class CPerlHost:: obj if
needed.
This is just wrong. Almost universally, vtables are stored in const
RO memory. Monkey-patching at runtime is a Perl lang thing, and rare
to never in C/C++land.
The ptr to "plain C to C++ func thunk vtable", CAN NOT be stored
per VDir::* or per VMem::* ptr. You can't store them, per C++ tradition,
as the 1st member/field of a VDir::/VMem:: object.
The reason is, VDir::/VMem:: objects can have refcounts, and multiple
CPerlHost:: ptrs, hold refs to one VMem:: ptr. So there is no way to
reverse a random VMem:: ptr, back to a CPerlHost:: ptr.
Main examples are VMem:: "MemShared" and VMem:: "MemParse".
Also the C->C++ thunk funcs must pick/separate, between 3 VMem:: obj ptrs.
Which are "Mem", "MemShared" and "MemParse" and stored at different
offsets in CPerlHost::*, but all 3 VMem:: derived "classes",
must have the same plain-C vtable layout with 7 extern "C" func thunk ptrs.
B/c my minimal C++ knowledge and also not wanting to add even more C++
classes to iperlsys.h perlhost.h and perllib.c, and those new C++ classes
may or may not inline-away. Don't fix this with more C++ classes.
So fix all of this, by each CPerlHost:: obj storing a ptr to the RO
vtable instead of a huge RW inlined copy of the vtable.
To keep all previous design requirements, just use
"&cperlhost_obj->vmem_whatever_vtable" as the plain-C representation
of a VMem::* ptr, instead of
"&cperlhost_obj->IPerlWhateverMem.pMalloc".
The 1 extra pointer de-ref CPU machine op, in each perl core and perl xs
caller, that executes in "iperlsys.h" family of macros I think is
irrelavent compared to the savings of having RO vtables. It is the same
machine code length on x86/x64 in each caller, comparing old vs new.
This extra ptr deref to reach the vtable can be removed, and I will
probably do it in a future commit. Not done here for bisect/small patch
reasons.
"iperlsys.h" family of macros is for example, the macro
"PerlEnv_getenv(str);"
Specific example, for macro PerlMem_free() in Perl_safesysfree()
old before this commit
----
mov rax, [rax+0CE8h]
mov rcx, rax
call qword ptr [rax+10h]
-----
new after this commit
-----
mov rcx, [rax+0CE8h]
mov rax, [rcx]
call qword ptr [rax+10h]
----
"mov rcx, rax" is "0x48 0x8B 0xC8" compared to
"mov rax, [rcx]" which is "0x48 0x8B 0x01".
No extra machine code "bloat" in any callers. The extra 1 memory read
is irrelavent if we are about to call malloc() or any of these other
WinOS kernel32.dll syscalls. iperlsys.h/perlhost.h does NOT hook anything
super perf critical such as "memcmp()" or "memcpy()".
There is no point in using a separate format string if the whole error
message is written right next to it. Not only does this change lead to
simpler code (passing one argument instead of two), it also exposes more
error messages to t/porting/diag.t, which relies on croak's first
argument to supply the message template.
Also make an equivalent change to S_open_script, which passes a constant
string in the form of an err variable that is not used anywhere else.
(It used to be, but that code was deleted in commit 5bc7d00e3e.)
Background:
-----------
Before perl 5.004, the perl parser would skip over CR (carriage return)
between tokens in source files, treating it as whitespace, but would
retain CRs in quoted constructs such as multi-line strings and
here-documents.
In 5.004, the behavior was changed to make CR in source files a fatal
error ("Illegal character %s (carriage return)") to avoid surprises with
unexpected literal CRs in string constants when scripts were copied from
DOS/Windows without newline conversion.
In 5.005, the behavior changed again. Now CR was back to being ignored,
but harder: Even in quoted constructs, CR was ignored when immediately
followed by a newline. However, the 5.004 behavior could be restored by
compiling perl with the `PERL_STRICT_CR` macro defined (e.g. with
`./Configure -A ccflags=-DPERL_STRICT_CR ...`). This option was
undocumented except for a brief note in perl5005delta. (Also, the
"Illegal character ..." error was changed to a warning, but perldiag
wasn't updated and so still listed the message as "fatal" (F).)
And that's how things have been ever since 1998.
Foreground:
-----------
This patch removes all checks for PERL_STRICT_CR entirely, treating it
as always off.
Rationale: It simplifies the code and reduces clutter. (Plus I don't see
the need to perpetually maintain an undocumented configuration option
that enables compatibility with an ancient perl version used sometime
around 1997-1998.)
References:
-----------
- 4fdae80067 ("Make \r in script an error (per Larry)")
- ff0cee690d ("Fix carriage-return message")
- 54310121b4 ("Improve diagnostic on \r in program text")
- 2db4f57cd9
- f63a84b229
- 637e912262
- b8957cf14d
- 6a27c1886b
This is only done for pthreads, Win32 already uses something like
my suggestion from #22530 and unlike POSIX doesn't have a way to
asynchronously interrupt a thread that I'm aware of. It's also
complicated by pseudo-processes.
Fixes#22487
Create specific macros for incrementing the refcount of an AV, CV, GV or
HV, returning a typed pointer to avoid the caller needing to cast the
result. This yields neater easier-to-read code.
While not implemented here, this gives an opportunity to add an
`assert()` check on the SvTYPE of the structure being adjusted, for
extra debug checking during debugging builds.
This allows a debugging perl to be built with the high water mark
checks disabled, or a non-debugging perl to be built with the
high water marks enabled.
This should allow Debian, the reporter for #16607 to build both their
normal perl and debugperl with the same state of high water mark
checks and avoid the mismatch between a debugperl and non-debug
dynamic extension.
Fixes#16607
I believe I finally understand the difference between these, and this
commit is the result.
Under MULTIPLICITY, for locale.c, it:
1) Updates comments to reflect reality
2) displays the aTHX in DEBUGGING statements.
3) Uses reentrant mbrtowc vs non-reentrant mbtowc
Under multiplicity, for perl.c, it avoids a leak. PL_cur_locale_obj is
created whenever there is MULTIPLICITY, not just when threaded, so needs
to be destroyed whenever MULTIPLICITY.
Prior to this commit, the code implementing the langinfo interface used
malloc'd buffers to store results in.
However, I discovered that perl has never implemented one langinfo item
properly, because its documentation is misleading. And that item is not
a standard C string, but instead contains embedded NUL characters.
I started to implement an extension to the existing buffer interface to
handle that. But then I realized it would need the equivalent of SvCUR
and SvLEN.
And then, I realized that the glibc extensions to nl_langinfo() return
values that are better interpreted as UVs than PVs.
To cope with all that, my implementation would be starting to smell like
perl's already-existing SV implementation.
So this commit changes to use SVs behind the scenes.
This presents some advantages here and in the next commit, due to the
maturity of the perl SV code. The same SV can be reused for scratch
with potential for PV stealing, COW, fewer mallocs, etc. The explicit
freeing of that SV can be delayed until global destruction.
This produced warnings like:
g++ -c -D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing \
-pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 \
-xc++ -I/usr/lib/llvm-18/include -std=c++17 -fno-exceptions \
-funwind-tables -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS \
-D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -fexceptions \
-I/usr/lib/x86_64-linux-gnu/perl5/5.36/Imager/include -std=c++23 \
-O2 -g -DVERSION=\"1.000\" -DXS_VERSION=\"1.000\" -fPIC \
"-I/usr/lib/x86_64-linux-gnu/perl/5.36/CORE" LLVM.c
In file included from /usr/lib/x86_64-linux-gnu/perl/5.36/CORE/perl.h:5798,
from LLVM.xs:15:
/usr/lib/x86_64-linux-gnu/perl/5.36/CORE/proto.h:392:65: warning: ‘volatile’-qualified parameter is deprecated [-Wvolatile]
392 | PERL_CALLCONV I32 Perl_call_sv(pTHX_ SV* sv, volatile I32 flags);
| ~~~~~~~~~~~~~^~~~~
I haven't been able to produce this when building perl itself as C++
since it requires a modern -std argument to reproduce, and that breaks
the build in other ways. It is reproducible when building a C++
XS module with gcc and -std=c++20 or later.
In a strict standard sense flags does not need to be volatile since
flags is not modified between setjmp() and longjmp(), but due to a
bug in gcc, gcc complains flags could be clobbered.
GCC bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=21161
By default it will use I32 for backward compatibility, but you
can select SSize_t during configuration with:
-Accflags=-DPERL_STACK_OFFSET_SSIZET
which may one day become the default.
The actual offsets throughout the code continue to be SSize_t.
This object, only on POSIX 2008 systems, is no longer used. I've been
keeping it around in case I could figure out if it had any remaining
utility, but don't see any.
call_sv() used I32 for a saved mark and the return value.
The XS::APItest call_sv() wrapper used an I32 index to reposition
the argument list supplied to call_sv().
I haven't created additional tests for call_argv(), call_pv() nor
call_method(), these simply pass the return value on, I don't think
it worth the extra tests.
eval_sv() had two problems, it saved the starting stack position
in an I32, so a call with a deep stack would save a negative or
otherwise invalid depth, and the return value was an I32, so
if the eval returned a large list the return count would overflow.
I'm not entirely happy with this, since we're changing the return
type and the types of the items and ax values passed via va_args.
But an excuse: the return and the fetching of ax and items via
va_args both happen after we've validated the handshake key, which
would prevent the va_arg() fetches and the return.
But but: this is fine for va_args, but purely the call to the
function with the different return type is technically undefined
behaviour.
I'm not sure it's worth trying to workaround that, since it will
require code duplication for a rare case, that I expect to just
work on the platforms involved (64-bit platforms).
The argument to op_free() is declared NULLOK, and the first thing it
does is return if the op is null or already freed, so there's no point
in callers checking for it.
This macro was disabled a few commits ago. Now remove all use of it
from the core.
It was just a temporary crutch that disabled actually adjusting the
reference count of things being pushed and popped off the stack, even on
PERL_RC_STACK builds. It allowed the core to be converted to support a
reference-counted stack as a bunch of incremental changes.
When built with PERL_RC_STACK, this substantial commit:
* makes stacks AvREAL() by default.
* adds the si_stack_nonrc_base field to the stackinfo structure.
* adds the runops_wrap() wrapper
It also adds rpp_obliterate_stack_to() for clearing the stack on
exceptions
Collectively, this allows each stack to be in one of three states:
1) AvREAL(PL_curstack) && PL_curstackinfo->si_stack_nonrc_base == 0
(the new default) all the SVs pointed to from the stack are
reference-counted.
2) AvREAL(PL_curstack) && PL_curstackinfo->si_stack_nonrc_base > 0
items on the stack are reference-counted only below si_stack_nonrc_base.
3) !AvREAL(PL_curstack)
(existing behaviour) no items on the stack are reference-counted.
The way it generally works is that runops loops assume that all the PP
functions they call out to are reference-count-aware. Where this isn't
yet the case, the recently-added pp_wrap() and xs_wrap() functions
assume a reference-counted stack, but use the si_stack_nonrc_base
mechanism to call out to PP or XS functions which aren't aware of a
reference-counted stack.
Conversely, the runops_wrap() function added by this commit wraps calls
to runops loops, since such loops work only with an RC stack. So if
called with a non-RC or partially-RC stack, the wrapper temporarily
fixes up the stack to be fully-RC, calls the real runops loop, then on
return reverts the stack back to it's non-RC ways, mortalising any
return values if necessary.
This downgrading and upgrading by pp_wrap()/xs_wrap() and
runops_wrap() allows handling of multiple nesting of when, for
example, perl calls pp_entersub(), which calls XS code, which calls
call_sv(), which calls a runops loop, which calls pp_entersub(), which
calls XS code, and so on.
For si_stack_nonrc_base, this index value marks the lowest position on
the argument stack which is not reference-counted. The special (and
normal) value of 0 indicates that *all* items on the stack are
reference-counted.
The new function rpp_obliterate_stack_to() is a bit like
rpp_popfree_to(), except that it can handle stacks in all three of the
states listed above. It's intended to be used when dying, throwing
exceptions or exiting, since the stack could be in any state when that
happens.
Note that as of this commit, PERL_XXX_TMP_NORC is still defined, which
means that even in the presence of AvREAL(PL_curstack) etc, the stack
isn't yet actually reference counted. So with this commit, perl goes
through all the motions, (calling wrappers like runops_wrap()
etc), but skips actually manipulating any reference counts. There will
be a few more commits, fixing up a few more things, before
PERL_XXX_TMP_NORC will be removed.
Add a _flags() variant of new_stackinfo() which indicates whether the
new stack AV should be created real or not.
Modify the new push_stackinfo() function to have a similar flag.
Then make the backcompat macros like PUSHSTACKi() still push a non-real
stack, while functions which have been updated to use the new
push_stackinfo() etc will be able get a real AV. The next commit makes
use of that.
This means that existing code (core and XS) which hasn't been updated to
the new ref-counted stack regime can do stuff like:
PUSHSTACKi(FOO);
PUSHMARK(sp)
XPUSHs(sv);
call_sv();
where call_sv() (or rather, the runops loop it invokes) will be able to
determine that it's been called from a non-RC environment and that the
args on the stack aren't reference-counted.
The next commit will update the runops loops etc to do exactly that.
Similar to the previous commit which fixed up call_sv() and similar,
the commit updates eval_sv() to handle a reference-counted stack
environment.
Note that these functions are slightly unusual in that they can
be called from either a reference-counted or non-RC stack environment,
so must be able to handle either case. This is done mostly by relying on
CALLRUNOPS() to fix things up, but for anything pushed/pulled around
that, by checking rpp_stack_is_rc(): which currently always returns
false, but once ref-counting is enabled in a few commits' time, may
start returning true.
Update the functions which directly or indirectly call a CV, to handle
a reference-counted stack environment.
Note that call_sv(), eval_pv() etc are slightly unusual in that they can
be called from either a reference-counted or non-RC stack environment,
so must be able to handle either case. This is done mostly by relying on
CALLRUNOPS() to fix things up, but for anything pushed/pulled around
that, by checking rpp_stack_is_rc(): which currently always returns
false, but once ref-counting is enabled in a few commits' time, may
start returning true.
call_sv(cv, G_EVAL) pushes a CXt_EVAL context, fakes up an entersub op,
then runs it, letting entersub consume the arguments pushed onto the current
stack frame by call_sv()'s caller.
So the C-level
PUSHMARK():
PUSHs(arg1);
...
PUSHs(argN);
call_sv(cv, G_EVAL);
is about equivalent to the perl-level:
eval { cv($arg1,..$argN) }
Except that in the latter, when the CXt_EVAL is pushed, it is done
*before* any of the arguments have been pushed, so the eval context's
blk_oldsp points to just before the arguments. Conversely in the case of
call_sv(), blk_oldsp gets set to just *after* the arguments.
During an exception, the context stack gets unwound until the CXt_EVAL
is reached, then PL_stack_sp is restored to the blk_oldsp value.
The difference currently makes no real functional difference, as
call_sv() resets PL_stack_sp immediately after an exception anyway, so
the bad just-restored value is discarded. But it does mean that
PL_stack_sp briefly points to junk on the stack. When the stack becomes
reference-counted, this will start to become important. So fix now.
This commit changes it so that call_sv()'s blk_oldsp gets set to just
before the arguments too. It also asserts that the stack was cleared
back to blk_oldsp.
Rather than always using PL_stack_sp to pass to cx_pushblock(),
allow the caller of create_eval_scope() to specify what stack position
is to be saved.
This will be used shortly to fix a bug.
This macro turns on stack reference counting.
Add the macro, but commented out for now.
In addition, temporarily define and enable PERL_XXX_TMP_NORC.
This is a temporary transition measure: even in the presence of
PERL_RC_STACK, do not actually modify reference counts yet. So
PERL_RC_STACK will still define all the new static functions and code
behaviour, except for the bits that actually modify reference counts in
ways that assume the stack is reference counted. It allows this branch
to have many commits which incrementally deal with adding support for
PERL_RC_STACK, with each individual commit still working while half of
the core is still not yet "ref count aware".
