Some questions

[satsen]

First of all, thank you for this great library. It is so much simpler than ASM and I can actually do things with it.

I have 2 questions, and I hope that you will be able to help me.

How do I obtain a "Variable" (I think it's not the right thing for this use-case, but I did not find an alternative) for a constant/literal value? Say if I have a boolean variable that I want to assign the expression 1L < 4L to, the only way I found is to do mm.var(long.class).set(1L).lt(4L) which generates:

long var2 = 1L;
boolean var3 = var2 < 4L;

but I tried compiling a class with javac and I got the much simpler

boolean var1 = 1L < 4L;

(javac optimizes this but I tried with a more difficult expression, it still did not generate a single-use variable).

And then that var3 is used in a method invocation, so there's 2 redundant variables. I would love to get to know how I could for example do a simple method(1L < 4L) without 2 redundant variables.

How do I perform logical operations? Like, how can I get a var1 && var2 or a var1 || var2 expression? I only found the bitwise options in the API.

If these could be added to the wiki it would be great. Thanks again for the library.

[broneill]

1. You can do mm.var(boolean.class).set(1L < 4L), but that might not be what you really meant. The code that gets generated sometimes has extra variables, but that's an artifact of not exposing the operand stack. When the bytecode gets compiled by the JVM, these extra variables go away. The old (and deprecated) version of Cojen performed liveness analysis to clean things up, but this was slow and yielded no benefit because HotSpot did the same thing anyhow.
2. I added a new section: https://github.com/cojen/Maker/wiki/Coding-patterns#short-circuit-logical-operations

[broneill]

By the way, some work is performed to reduce local variable usage already, but only for simple cases. The lt operation is actually quite complicated, and as a result, the reduction doesn't work. Simpler operations like add do produce instructions that you would expect -- no extra variable is used.

[satsen]

1. Ok then I guess what I was already doing is the right way (the values are obtained from expressions, so I don't evaluate it on compile-time, 1L and 4L were examples)
2. It is pretty confusing, my compiler is going through the AST with a recursive approach, so I work on specific values, which can be embedded quite deep. Not sure how the if statements would work but I'll try to figure it out

[satsen]

Regarding logical operations, the bitwise AND and OR seems to do the same job as logical AND and OR when used with booleans? I could not figure out what to do with labels in my procedure so I guess I could use that instead.

[broneill]

If you have an AST, generating short-circuit logic code using goto statements is typical, and javac does this too. If the Cojen/Maker library supported short-circuit operations directly, it would need do perform a bunch of reverse engineering to reconstruct an AST first.

You can use the bitwise operations with boolean types, but this doesn't perform short-circuit logic, and so it's not a good substitute.

[satsen]

I honestly have no idea what short-circuit logic even is in this context.

After some more tinkering I figured I could create a boolean Variable that is false to begin with, then have ifFalse for both the left-hand argument and right-hand argument (which at the point where a && or || comparison is being handled, have already been turned into boolean Variables or are boolean Objects-in the case of literals) to skip to a label which is right after a variable = true; statement.

Thanks for the help. The decompiled code isn't "pretty", but it gets the work done.

[satsen]

Cojen/Maker is a great library, thanks again for making it. I'll go work on the other parts of my language (method selection-overloads and varargs can be tough, type promotion, etc.).

If you use cryptocurrency, I'd love to send you a tip in the currency of your choice. As a thank you for the library and the help :)

[satsen]

Hey I ran into another issue, I hope you can help me. So I did a lot of things, and now I need to pass a lambda to a method call.

The method I want to call is public static DoubleBinding createDoubleBinding(Callable<Double>, Observable...) (Observable is a class of the toolkit, don't mind that)

I've done all the LambdaMetafactory things, but when I try to invoke this method, it says Unsupported constant type: class [Lorg.cojen.maker.Variable;

I invoke it like this: mm.var(Bindings.class).invoke("createDoubleBinding", function, observables.toArray()) the observables List contains Variables.

---

I suspect that even if I get past this point I will run into other issues, due to this comment that I read in the wiki
// Note: Extra params are needed if lambda implementation method isn't static.
The lambda accesses a local variable (which is made using mm.var), I'm not sure how I would pass my local variable from the current method to the lambda method, and also not sure whether the lambda method should be static or not.

[broneill]

The function param and the observables need to be passed as combined Object array. The varags param doesn't understand that's what you intend and so it sees the function as the first param, and the array of variables as the second param.

[broneill]

...or do you intend for the Observable... param to be varargs? In that case, you need to pass an Observable array.

var observablesVar = mm.new_(Observable[].class, observables.size());
foreach observable {
    observablesVar.aset(i, observable);
}
mm.var(Bindings.class).invoke("createDoubleBinding", function, observablesVar);

[satsen]

Got it, thanks. I figured it after shutting down the computer but would have no idea how to do the array creation. I get confused by the boundary between the current VM and the target VM sometimes...

[satsen]

I suspect that even if I get past this point I will run into other issues, due to this comment that I read in the wiki
// Note: Extra params are needed if lambda implementation method isn't static.
The lambda accesses a local variable (which is made using mm.var), I'm not sure how I would pass my local variable from the current method to the lambda method, and also not sure whether the lambda method should be static or not.

My suspicion was correct, the lambda method obviously does not have access to the local variables of the other method. Do you know how I would pass them?

[broneill]

Sounds like a cool project. I don't do anything with JavaFX (or Swing etc), but I have always found GUI construction be quite difficult. FXML doesn't look all that approachable. Thankfully, the world has moved on from XML, but it still lingers in a few places.

The risk when creating a DSL is not knowing when to stop. It looks like the original JavaFX script was evolving into a first class programming language, and so switching to XML creates a natural limit. However, XML didn't stop the Jelly project from creating a first class programming language.

[satsen]

Thankfully, the world has moved on from XML, but it still lingers in a few places.

It lingers in quite a few, I guess, haha: HTML, maven pom.xml, Android interface, and I'm pretty sure those horrible "modern" web frameworks embed XML-like syntax inside their javascript files.

The risk when creating a DSL is not knowing when to stop. It looks like the original JavaFX script was evolving into a first class programming language, and so switching to XML creates a natural limit.

Indeed, I fully agree. At least I am lucky because I have another format to compare with, FXML. (FXML actually supports inline scripting, but I have never seen anybody use that and it recently got hidden behind a system property as well) So, anything that FXML does not support is something that I have to think extra carefully on. And I am not going to focus on backwards compatibility to begin with so if it gets too bloated I can easily remove things.

---

The focus will be on a declarative approach, I like to think that a user interface designing tool (such as Scene Builder) should be able to display what the layout should look like without executing any code. Just from the AST (or likely a higher level model), so to say.

[broneill]

Are you still stuck on the LambdaMetafactory thing? If you provide a Java example with a lambda function that you want supported I can try to figure out the translation strategy.

[satsen]

Yeah I tried some things but it was too complicated for me. Here is an example in Java code:

// (this code is in a method body)
HBox hbox = new HBox();
hbox.spacingProperty().bind(Bindings.createDoubleBinding(() -> {
    return hbox.getHeight() / 2;
}, hbox.heightProperty());

There are other ways to achieve simple bindings like this without all that ceremony, but I need to support every possible expression and this is the generic way where it can depend on multiple properties and prepare the return value in any way. (otherwise it would have just been hbox.spacingProperty().bind(hbox.heightProperty().divide(2)) or hbox.spacingProperty().bind(hbox.heightProperty().map(height -> height / 2));, just saying this because I don't want to give the impression that JavaFX is super verbose)

This is what I have in my compiler:

// Make the method which will contain the binding expression. propValueType can for example be double
MethodMaker lambda = cm.addMethod(propValueType, "lambda$" + mm.name() + "$0", obj.classType()).private_();
lit.tempHackyReference = obj;
lambda.return_(transformValue(lambda, binding));
// Get a method handle that targets the lambda method
Variable implMethod = mm.this_().methodHandle(propValueType, lambda.name(), obj.classType());
// Do....stuff?
Bootstrap bootstrap = mm.var(LambdaMetafactory.class)
		.indy("metafactory",
				// This is the type of the lambda that is implemented. Callable<Double> for example would be methodType(Double.class)
				MethodType.methodType(Double.class),
				implMethod,
				// This is the more complicated type, because the method signature differs from the lambda interface signature if the lambda uses local vars
				MethodType.methodType(Double.class, obj.classType()));
// Make the invocation of the lambda
var function = bootstrap.invoke(Callable.class, "call", new Object[]{obj.classType()}, obj);

I think another issue that I will run into as soon as I solve this is that propValueType = double.class while a Callable<Double> should return a Double not a double.

[broneill]

I'm using this as an opportunity to figure out how to add the ability to define lambdas without the complexity. Ideally, you'd want something like this in the MethodMaker: Variable lambda(Object type, Consumer<MethodMaker> body)

You'd pass in the type of Callable.class, and you'd pass in a block of code for making the lambda. You'd write a lambda to make a lambda, so let that sink in. The parameters for the body method conform to the functional method of the Callable interface (it has none), and the return type from the method is inferred rather than fully declared. This means you can return double and the full interface type becomes Callable<Double>. There's no way to define this with a class literal. Finally, the Variable returned by the lambda method is the Callable instance that you'd pass to the bind createDoubleBinding method.

[broneill]

Here's the ugly code to make it work manually:

import java.lang.invoke.*;

import java.util.concurrent.*;

import javafx.beans.binding.*;
import javafx.scene.layout.*;

import org.cojen.maker.*;

public class Lambda {
    public static void main(String[] args) throws Exception {
        test();
    }

    private static void test() throws Exception {
        var cm = ClassMaker.begin().public_();
        cm.addConstructor().public_();

        // Define the desugared method.
        MethodMaker mm = cm.addMethod(Double.class, "lambda$foo$0", HBox.class);
        mm.private_().static_();
        mm.return_(mm.param(0).invoke("getHeight").div(2));

        mm = cm.addMethod(Object.class, "test").public_();
        var hboxVar = mm.new_(HBox.class);

        var implHandleVar = mm.var(cm).methodHandle(Double.class, "lambda$foo$0", HBox.class);

        var indy = mm.var(LambdaMetafactory.class).indy
            ("altMetafactory",
             MethodType.methodType(Double.class),
             implHandleVar,
             MethodType.methodType(Double.class),
             LambdaMetafactory.FLAG_BRIDGES, 1, MethodType.methodType(Object.class));

        var callableVar = indy.invoke(Callable.class, "call", null, hboxVar);

        var bindingVar = mm.var(Bindings.class).invoke
            ("createDoubleBinding", callableVar, hboxVar.invoke("heightProperty"));

        hboxVar.invoke("spacingProperty").invoke("bind", bindingVar);

        mm.return_(hboxVar);

        var clazz = cm.finish();
        var test = clazz.getConstructor().newInstance();
        var hbox = (HBox) clazz.getMethod("test").invoke(test);

        System.out.println(hbox);
    }

    /*
    private static Object foo() {
        HBox hbox = new HBox();
        hbox.spacingProperty().bind(Bindings.createDoubleBinding(() -> {
            return hbox.getHeight() / 2;
        }, hbox.heightProperty()));
        return hbox;
    }
    */
}

[satsen]

// Make the method which will contain the binding expression. propValueType can for example be double
MethodMaker lambda = cm.addMethod(propValueType, "lambda$" + mm.name() + "$0", obj.classType()).private_();
lit.tempHackyReference = lambda.param(0);
lambda.return_(transformValue(lambda, binding));
// Get a method handle that targets the lambda method
Variable implHandleVar = mm.this_().methodHandle(propValueType, lambda.name(), obj.classType());
// Do....stuff?
Bootstrap indy = mm.var(LambdaMetafactory.class).indy
		("metafactory",
				MethodType.methodType(Object.class),
				implHandleVar,
				MethodType.methodType(Double.class));

// Make the invocation of the lambda
Variable function = indy.invoke(Callable.class, "call", new Object[]{obj.classType()}, obj);

I've done this, but the decompiled (Fernflower) code looks like this:

    public void main() {
        ...
        Label var3 = new Label();
        Callable var4 = var3::lambda$main$0;
        DoubleProperty var5 = var3.maxHeightProperty();
        Observable[] var6 = new Observable[]{var5};
        DoubleBinding var7 = Bindings.createDoubleBinding(var4, var6);
        var3.maxWidthProperty().bind(var7);
        ...
    }

    private double lambda$main$0(Label var1) {
        return var1.getMaxHeight();
    }

Same thing when using the altMetafactory version.

[satsen]

Making the lambda method static seems to have fixed it

Edit: Not really..

[satsen]

Ok it seems that the code works but something about the bytecode makes the decompiler very confused.

CFR even decompiles it to Callable<Double> callable = () -> ClassFile2.lambda$main$0(label); lol, so there's definitely something strange with the bytecode. I think it might be relevant to investigate (if you want to)

[broneill]

The decompiler is confused by the extra local variables that get created. It's expecting a certain pattern of stack pushes in order to re-create an inline lambda definition. As I indicated earlier, without direct stack manipulation support, extra local variables can be created.

If you take the decompiled code generated by Fernflower and manually move some code around you get this:

        Label var3 = new Label();
        var3.maxWidthProperty().bind(Bindings.createDoubleBinding(var3::lambda$main$0, new Observable[]{var3.maxHeightProperty()}));

Then change Observable[] to varargs syntax:

        Label var3 = new Label();
        var3.maxWidthProperty().bind(Bindings.createDoubleBinding(var3::lambda$main$0, var3.maxHeightProperty()));

It's much closer to what you might expect to see, but not quite. There's no special tag to indicate that the main$0 method is supposed to be inlined, but if I call synthetic() on the lambda MethodMaker instance, then Fernflower generates this snippet:

        Callable var4 = () -> {
            return var3.getMaxHeight();
        };

[satsen]

Thank you, it is good to know that it's just extra local variables that caused it. Marking as synthetic finally fixes it for Fernflower but completely breaks CFR. I'm not sure if you care about decompiler support (but decompilers failing likely means there's something non-standard with the generation, ....or my code) so I won't bother you with that unless you're interested

[broneill]

I don't care about decompiler support. They're looking for code generation patterns specifically generated by javac, and they're confused by any deviation from these patterns. Obfuscaters are designed to confuse decompilers by deliberately moving code around and changing the patterns.

I've found Fernflower to be the best at working around this, even when decompiling code which represents illegal Java. For example, if you generate code which branches into the middle of a loop, Fernflower can sometimes produce the equivalent Java code by unwinding loops and duplicating code.