Suggest using standard scripting language like Lua

[Beirlis]

I think that it's best to have multiple script-type objects, and here, I'm using the term "script" a bit loosely. For instance, during a conversation, when you choose a particular item, we like to be able to have a change to the game, like the standard "Fight!" option, where a character becomes attackable, etc. I think that it's best to just be able to fire off an abject "script" object (a Java object) triggered by:

• a conversation choice
• stepping on a certain tile
• attacking a certain monster or fleeing from them in combat
• "Use"-ing an inventory item (drinking a potion, rubbing a magic stone, whatever)
• etc.

I addition, other trigger object could be created as needed, like if you come near a particular NPC, attack with a cursed sword (e.g., maybe attacking with a cursed sword has a 2% to summon annoying little imps that start beating on you while you're already in combat), etc.

When firing off an event, you pass the actor who initiated the event (usually the player, but maybe an NPC) and the target (the other NPC, map tile, item, etc.). These objects can usually be Java, but you can also have one that will run a Lua script. The reason for my suggestion is that it would enable a greater number of community members to participate actively (rather than just posting suggestions) in the game's development process without needing to learn the script disciplines of a language like Java. However, it is also a *very* useful tool for adding flexibility to the game engine without hundreds of custom Java functions for different type of behavior that you want to have for a particular area, NPC or quest.

As usual with an interpreted scripting language, care must be taken to prevent it from turning into a monster (see Wesnoth's WML for an example ), it is inherently slower and not suitable for large-scale processing. However, by allowing the flexibility to implement events in either Java or Lua, if you discover something that you guys really want that was written in Lua, but is too slow there, it can be ported into a Java event class.

Implementation
I've never used Lua with Java, but I'm guessing that the process is similar to what you do in C++. You define what classes you want exposed to the Lua subsystem and then you export the objects of those types that you want exposed. I think you usually would just export your primary objects (World, Map, Player, etc.) and then a custom object dedicated to locating other Java objects so that the remaining connectors only happen as-needed. Then, the process of expanding different parts of the game as being "scriptable" can be as simple as writing the code to export that class type and locate and export that object's interface. (Of course, I know that sometimes the challenge is more than just that. )

[Zukero]

Lua can easily be integrated into standard Java, using a C interface to the Lua runtime.
However, with android, netive-code integration is a pain to maintain, and pure java implementations of Lua (and all other scripting languages like python for that matter) are all based on Java introspection (java.lang.reflect package), which is highly inefficient CPU-wise.

This is why I am developping a dedicated scripting engine for AT. It is far from complete (I'd say about 30/40%), but it is evolving quickly (starting end of October, I plan to have an alpha version by end of January '14 at the latest). One of the goals is to rewrite the whole skill system using it, but open those same possibilities to almost all game objects (player, item, maps, npcs...).

However, to avoid Java introspection, every field and every method exposed is done case by case. This also allows me to expose things in a way that seems more natural to the scripters than what they are in the game engine.

Having a mostly Java background, and AT being written in Java, my scripting language uses a C-style syntax. By the way, I've had a look at Lua in the past, and while it is rather compact (low memory footprint for the interpreter), I find it rather hard to remember.

I am documenting my scripting engine at the same time as writing it, and you can learn all there is to know about it in the AT Scripting Language documentation thread.
If you have ideas, make sure to reply in that thread, as I am eager to find new features to implement that I may not have thought about yet. Please bear with the fact that it is far from complete, adn that some obvious features are lacking, and thus undocumented, but in the works anyway.

[Beirlis]

I was aware of some type of AT-specific scripting language, but I haven't studied it. In my experience (almost 2 decades in software engineering), it's always better to "buy vs build" when possible. Now, I haven't studied the Lua pure-java implementation or its performance, and I would certainly suggest a pure-java approach (let the JIT optimize it), but I'm quite curious about the actual overhead of java introspection. Indeed, introspection is slow. Even on C++ (and especially since RTTI was implemented using strings instead of unique numerical IDs) it is cumbersome and slow. However, how often does this introspection overhead come into play? I guess I'm pretty curious about that. When I was looking into the Lua to C++ wrappers, they seemed very tight and efficient.

The second thing that I have to wonder is that even if the introspection overhead is significant, is there a way to extend the pure-java Lua library with custom code to eliminate the introspection overhead (for objects of known types) and if so, would that be less effort? For instance, isn't there a way to design some Lua objects that have both a Lua declaration and a Java implementation that bypass introspection and that you can use to interact with the rest of the game's objects?

Designing a scripting language is no small endeavor. Designing and implementing one is an even greater task. Designing and implementing a good scripting language is exponentially more so. The idea of using "off-the-shelf" libraries in projects is to keep us focused on our project and spending less time on implementing the underpinnings. Mature scripting languages have faced the test of time and been debugged, debated, refined and improved many times over. I have to wonder if this is the design anti-pattern known as NIH ("Not Invented Here") -- one that I've fallen into a number of times in the past.

[Zukero]

You do raise some valid points here, so I'll try to answer them one by one.

However, how often does this introspection overhead come into play? I guess I'm pretty curious about that. When I was looking into the Lua to C++ wrappers, they seemed very tight and efficient.

It comes into play at pretty much every word. A simple line like player.addQuestProgress(new QuestProgress("quest_one", 10) would translates roughly to :

Code: Select all

Object param1  = "quest_one";
Object param2 = 10;
Object param3 = Class.forName("QuestProgress").newInstance(new Object[]{param1, param2});
Method m = player.getClass().getMethod("addQuestProgress", param3.getClass());
m.invoke(player, Object[]{param3});

And I removed the long list of try-catch blocks.

The C++ wrappers for Lua are indeed very efficient, but this is Java, and actually, it is more the interpreter that suffers than the wrappers.

The second thing that I have to wonder is that even if the introspection overhead is significant, is there a way to extend the pure-java Lua library with custom code to eliminate the introspection overhead (for objects of known types) and if so, would that be less effort?

IMHO, that would be even more work, as you'd have to a lot of regression testing, and try to understand the often optimized code, and the amount of code to write would be quite similar to the one needed to do it from scratch. Moreover, you would need to add special checks in the langugage to treat the special class differently than unknown ones, which would make using the unkown ones even worse.

Designing a scripting language is no small endeavor. Designing and implementing one is an even greater task. Designing and implementing a good scripting language is exponentially more so. The idea of using "off-the-shelf" libraries in projects is to keep us focused on our project and spending less time on implementing the underpinnings. Mature scripting languages have faced the test of time and been debugged, debated, refined and improved many times over. I have to wonder if this is the design anti-pattern known as NIH ("Not Invented Here") -- one that I've fallen into a number of times in the past.

Actually, I WANTED to use an exisiting language, but simply found nothing suitable. Lua & Jython on android all use the same "android script" project, itself relying heavily on introspection. I also looked at beanshell, which is rather nice from a user perspective, but once again, implementation is based on introspection. What's more, a language is just a norm, what needs to have faced the tests of time & volume is the interpreter, and in this regard, the android versions of Lua & Jython are still very young, and not heavily used.

Designing and implementing a language is actually rather easy, using the right tools (I'm using JavaCC, similar in many ways to Yacc or Bison, except JavaCC is LL(k) while Yacc and Bison are LALR()). The hard part IMHO is building all the in-game hooks, especially in terms of what triggers a script, and how do you exchange data between Java and script. I am currently expanding the possibilities in this regard, as well as allowing wider access to Java objects' fields & methods, but generally providing a more natural API, hiding the technical details.

If you have time, check out the thread I mentioned, or even better, the code itself, and tell me what you think of it !

[Beirlis]

OK, I'm so sorry for my delay in responding! Ok, so I think we have two main issues here.

1. I think you may misunderstand the actual level of overhead involved in introspection and
2. when I talk about a "wrapper", and you talk about a level of effort similar to writing a scripting language from scratch, it's clear that we aren't talking about the same thing

So let me just cover the introspection thing first.

It comes into play at pretty much every word. A simple line like player.addQuestProgress(new QuestProgress("quest_one", 10) would translates roughly to :

Code: Select all

Object param1  = "quest_one";
Object param2 = 10;
Object param3 = Class.forName("QuestProgress").newInstance(new Object[]{param1, param2});
Method m = player.getClass().getMethod("addQuestProgress", param3.getClass());
m.invoke(player, Object[]{param3});

And I removed the long list of try-catch blocks.

OK, so this makes sense to me and if Java were a purely interpreted or statically compiled language, I would say "well screw that!" However, it is a dynamically compiled language which gives it a distinct advantage in situations like this: it has the data at the time it compiles. Normally when a JVM executes a method, it will interpret it. This is actually a loose statement, what really happens is usually quite complex, depends upon several factors (how much CPU the programs threads are using and how much is available for background compiling, etc.) and is also vendor-specific to the JVM. But pretty much, if it's only going to run the method once or twice, it's going to interpret it. However, if it runs the function several times, it's going to optimize it. If it sees that a function is often called with the same data, it will create an optimized version of that function for that specific data set.

This can leave the JVM with several different compiled versions of a function, which can often take up more memory. The nice thing about this facet of the JIT in relation to introspection is that it can just eliminate the introspection entirely and just return the java method, field, class or object directly. Thus, if we ran some performance tests on this, I would bet it would end up surprisingly fast because the entire snippet above could end up being optimized as player.addQuestProgress(new QuestProgress(param1, param2)), moreso if it's called often or the JVM has been running for a non-trivial amount of time. Of course, this is where you insert the disclaimers "Actual mileage may vary. Void where prohibited.", etc. It depends upon the quality of the JVM and I would suspect that Google, with 50 billion-ish cash on hand has invested in writing a good JVM. Heck, they even decided to break with the standard to eliminate the one bottleneck that we've been able to criticize the JVM design with: being stack-based instead of register-based.

In reality though, this lua function would be represented as a object of a type from the Lua library who's "executeFunction()" method (or whatever it's called) would be the one most likely to receive these optimizations.

The C++ wrappers for Lua are indeed very efficient, but this is Java, and actually, it is more the interpreter that suffers than the wrappers.

The second thing that I have to wonder is that even if the introspection overhead is significant, is there a way to extend the pure-java Lua library with custom code to eliminate the introspection overhead (for objects of known types) and if so, would that be less effort?

IMHO, that would be even more work, as you'd have to a lot of regression testing, and try to understand the often optimized code, and the amount of code to write would be quite similar to the one needed to do it from scratch. Moreover, you would need to add special checks in the langugage to treat the special class differently than unknown ones, which would make using the unkown ones even worse.

So this is where I don't think we're talking about the same thing. I guess I'm going to have to actually dig into the java Lua implementations to confirm that the mechanisms that I would *assume* any "bolt-on" type of scripting language should have are indeed available. The idea is that you write some custom Java code to export some custom Lua objects to bypass the introspection. To me, your original example demonstrates the flexibility of a well-implemented bolt-on scripting language, by utilizing introspection at the cost of speed, code compactness, etc. So very, very loosely (since I don't know a thing about the pure-java Lua library yet), I would think this would happen something like this:

Code: Select all

package com.gpl.rpg.AndorsTrail.lua;
import org.lua.or.whatever;

class ATLuaInterface extends SomeLuaClassThatAllowsYouToDefineAndExportObjectsIntoLua {
    private static ATLuaInterface singleton = null; // unless you'll have more than one lua environment at a time.

    public static void exportClass(SomeLuaEnvironemntClass lua) throws SomeLuaException {
        if (singleton != null)
            throw new Something();

        lua.exportClass() // describe this class type to the Lua environment and construct an ATLuaInterface type in Lua that connects to this one.
        singleton = new ATLuaInterface();
        lua.exportObject(singleton);
    }

    public int someMethods() throws Whatever {...}
    public com.gpl.rpg.AndorsTrail.World getWorld() throws Whatever {...}
    // elsewhere you export the World class similarly so that no introspection is required
}

Again, I need to dig into the java Lua implementations to make sure that I'm not talking in vapor-ware, "pie in the sky" "I want a pony!"-isms. (yes, "I want a pony!"-isms is a plural noun to me )

Designing a scripting language is no small endeavor. Designing and implementing one is an even greater task. Designing and implementing a good scripting language is exponentially more so. The idea of using "off-the-shelf" libraries in projects is to keep us focused on our project and spending less time on implementing the underpinnings. Mature scripting languages have faced the test of time and been debugged, debated, refined and improved many times over. I have to wonder if this is the design anti-pattern known as NIH ("Not Invented Here") -- one that I've fallen into a number of times in the past.

Actually, I WANTED to use an exisiting language, but simply found nothing suitable. Lua & Jython on android all use the same "android script" project, itself relying heavily on introspection. I also looked at beanshell, which is rather nice from a user perspective, but once again, implementation is based on introspection. What's more, a language is just a norm, what needs to have faced the tests of time & volume is the interpreter, and in this regard, the android versions of Lua & Jython are still very young, and not heavily used.

Designing and implementing a language is actually rather easy, using the right tools (I'm using JavaCC, similar in many ways to Yacc or Bison, except JavaCC is LL(k) while Yacc and Bison are LALR()).

Well, lexicographic parsers is admittedly not my forte and I've only worked with yacc/bison on a trivial level. None the less, I'm talking about a full-feature robust language with lots of libraries to add support for JUNK like splitting and manipulating strings and arrays, and doing the common types of things that you'll find in the assortment of Lua libraries that others have written over the last 20 years. I seriously don't consider designing and implementing such a full-featured scripting language anything near trivial and anybody who does I would have to say is naive.

The hard part IMHO is building all the in-game hooks, especially in terms of what triggers a script, and how do you exchange data between Java and script. I am currently expanding the possibilities in this regard, as well as allowing wider access to Java objects' fields & methods, but generally providing a more natural API, hiding the technical details.

w00t! I agree with you for once!! While having the ability to directly access the game objects *can* be good sometimes, it's certainly too darn complex and we often don't want to do that because we expect those writing scripts to not understand the nuances of our objects or the impact of changes they make on the game. Best if it can be made so that it's hard to crash the game from the script. Indeed, that is challenging because we design and implement the classes of the game for a different purpose than what we want exposed in the scripting environment.

So even if it turns out that Lua, Jython or some such is indeed a viable and attractive alternative to an AT-specific language and I manage to convince you guys of that, then it sounds like there is still a lot of value in your current endeavour that could be ported.

EDIT: Ok, so this is definitely possible for simple Lua functions, I'm not yet certain about doing so for Lua objects (complete with methods, etc.): http://www.keplerproject.org/luajava/API/index.html

EDIT #2: Here we go, it looks like kahlua will do what I'm talking about. And even better, they use annotations! https://code.google.com/p/kahlua/wiki/K ... ct_methods. So instead of my example above, it would be something more like:

Code: Select all

package com.gpl.rpg.AndorsTrail.lua;

import com.gpl.rpg.AndorsTrail.context.ControllerContext;
import com.gpl.rpg.AndorsTrail.context.WorldContext;
import com.gpl.rpg.AndorsTrail.model.actor.Player;
//import se.krka.kahlua.*;

@LuaClass
public class AT {
    private ControllerContext controller;
    private WorldContext world;
    private Player player;

    @LuaConstructor(name="NewAT")
    @Desc("returns a new Andor's Trail Interface object")
    public AT() {
        // populate fields
     }

    @LuaMethod(name="GetPlayer", global=false)
    public Player healPlayer() {
        return player; // Player also has to have annotations for exposed methods and fields.
    }

    // or a more practical, but limited method
    @LuaMethod(name="HealPlayer", global=false)
    public void healPlayer(@Desc("range 0-10000") int amount) {
        player.hp += amount;
    }
}