A big hindrance for full integration are the CPAN Modules and XS Code: Compiling CPAN modules and XS (C-extensions) is challenging due to dependencies on native code and system libraries. What can be done about it? Should they be rewritten?

Yes, the current plan is to port CPAN modules to be JVM-native. When you consider the intricacies of Perl's XS system and the various ways Perl has been ported across different environments (as detailed in Perl's documentation), it's clear that a direct port of XS modules to the JVM is not straightforward. The challenge lies in XS's dependence on native code and system libraries, which don’t map cleanly to the JVM's architecture.

While the Java Native Interface (JNI) could theoretically bridge some of this gap, using JNI introduces its own complexities and overhead, which could hinder performance and ease of use. Rather than taking on the overhead of adapting or wrapping native code, the best path forward is to directly port these modules to the JVM using the PerlOnJava internal API.

This approach has significant advantages. By implementing Perl modules and operators natively in Java, we gain access to a wealth of powerful Java libraries, such as ICU4J for internationalization or NIO for non-blocking I/O. These JVM-native implementations offer better performance, easier integration, and long-term maintainability compared to relying on high-overhead adapters like JNI.

By embracing the JVM’s ecosystem directly, we can not only replicate the functionality of existing CPAN modules but also unlock new capabilities, making PerlOnJava an even more powerful tool for modern application development.

Have you considered targeting GraalVM which is capable of running programs in different languages other than Java, allowing direct interoperation among them? As such you can write polyglot programs, like calling JavaScript from within Java or  Ruby from Python, etc, the so called “Truffle” languages, see box below.

In an environment like this Perl could for instance call into Python's deep learning libraries to take advantage of them. Another option with GraalVM would also be creating native executables through AOT, which would be platform specific with the advantage of faster execution rather than waiting to spin up the interpreter.

GraalVM and the Truffle framework are indeed impressive. They enable polyglot programming, allowing seamless interoperation between languages like JavaScript, Ruby, Python, and more. This flexibility opens exciting possibilities, such as Perl calling into Python's deep learning libraries or Ruby interacting with Perl’s system utilities. The performance improvements seen with GraalPy and TruffleRuby are particularly notable, especially in benchmarks where there’s time for warm-up and JIT optimization.

However, for PerlOnJava, we chose to use the ASM library to generate static bytecode rather than Truffle for a few key reasons. Perl programs often involve frequent eval statements, small compilation units, and dynamic behavior such as AUTOLOAD and BEGIN blocks. These traits benefit from the fast startup and lower overhead that ASM-generated bytecode offers, making it ideal for the typical Perl workflow where the warm-up time required by GraalVM's JIT optimizations might be too costly.

While GraalVM’s ahead-of-time (AOT) compilation and polyglot features are attractive, PerlOnJava's focus is on improving runtime efficiency without introducing the overhead that comes with optimizing longer-running applications. In this current phase of development, we prioritize efficient, dynamic code execution, rather than AOT compilation or deep polyglot capabilities. Our choice of ASM allows us to maintain the agility and responsiveness Perl developers expect, while still providing an easy path for integrating with the JVM ecosystem.

**********************************************************************

Truffle

Truffle is a library for building programming language implementations expressed as self-optimizing Abstract Syntax Tree (AST) interpreters. In other words, it makes it easy to develop an interpreter for your custom programming language as the implementer has just to write a Java based AST interpreter for his language which subsequently gets JIT compiled by Graal into machine code. It's important to note that Graal can compile any language that is implemented as a Truffle interpreter.

The given language interpreter takes the program as input and works on the generated abstract syntax tree (AST) which it modifies and optimizes during the interpretation process to incorporate type feedback, substitute a node at its parent with a different node during runtime and rewrite itself by collecting language specific profiling information;as such the self-optimizing part. When the AST is stabilized (when no specialization/ optimization occurs anymore) Graal compiles the interpreter using Partial evaluation forming one combined compilation unit for an entire tree to produce Graal IR (not Java byte code). Then, as stated above, this is translated to LIR for the target processor architecture and then to the final machine code.