Squirrel

As I'm working on the next release of Sqrat (sorry it's taking so long guys! I haven't forgotten about it!) I've run into a concept that would be very helpful to me personally and am curious if anyone else in the community would use it.

Essentially, it's just a scheduler like the one in Stackless Python. Very simple, round robin, geared towards coroutines. I've got a working example below, coded in pure squirrel, but I think that the final product would be in the form of native routines. I'd also like to get a system like stackless' channels in there too.

I would probably package them and any other utilities/libraries like this that I build in the future as part of Sqrat but keep it separate from the binding libs (maybe SqratEx?) Honestly, I would really like to see this and a couple of other concepts become part of the core language, but I think Alberto has his hands full with Squirrel 3 right now :)

In any case, I'd like some feedback on 1) if anyone else thinks they would use this, 2) if you have opinions about how the interface should be set up or how the internals should work, and 3) if there are any other utilities like this that you feel would be a good fit.

Thanks!

====================================================================

//
// Scheduling Example
//

sqrat <- {

    taskList = [] // Array of active or scheduled threads
    sleepCount = 0 // How many threads are currently sleeping
    running = false;
    
    // Add a closure to be scheduled
    function schedule(closure) {
        local t = {
            thread = ::newthread(closure)
            vars = []
            started = false
        }
        
        taskList.push(t);
        
        return function(...) : (t) {
            for(local i = 0; i< vargc; i++) {
                t.vars.push(vargv);
            }
            
        }
    }
    
    // Start Scheduler (Simple Round Robin)
    function run() {
        if(running) { return; } // Prevent the scheduler from running twice in different threads
        running = true;
        do {
            foreach(idx, task in taskList) {
                if(!task.started) {
                    task.started = true;
                    
                    // TODO: Native function that converts array to var args, allow for any count of variables.
                    switch(task.vars.len()) {
                        case 0: task.thread.call(); break;
                        case 1: task.thread.call(task.vars[0]); break;
                        case 2: task.thread.call(task.vars[0], task.vars[1]); break;
                        case 3: task.thread.call(task.vars[0], task.vars[1], task.vars[2]); break;
                        case 4: task.thread.call(task.vars[0], task.vars[1], task.vars[2], task.vars[3]); break;
                        case 5: task.thread.call(task.vars[0], task.vars[1], task.vars[2], task.vars[3], task.vars[4]); break;
                        case 6: task.thread.call(task.vars[0], task.vars[1], task.vars[2], task.vars[3], task.vars[4], task.vars[5]); break;
                        case 7: task.thread.call(task.vars[0], task.vars[1], task.vars[2], task.vars[3], task.vars[4], task.vars[5], task.vars); break;
                        case 8: task.thread.call(task.vars[0], task.vars[1], task.vars[2], task.vars[3], task.vars[4], task.vars[5], task.vars, task.vars[7]); break;
                        case 9: task.thread.call(task.vars[0], task.vars[1], task.vars[2], task.vars[3], task.vars[4], task.vars[5], task.vars, task.vars[7], task.vars); break;
                    }
                } else {
                    if(task.thread.getstatus() == "suspended") {
                        task.thread.wakeup();
                    } else {
                        taskList.remove(idx); // Remove a thread from the scheduler when done
                    }
                }
            }
            if(sleepCount >= taskList.len()) {
                // If all threads are sleeping, we probably want to yield some time to the OS, such as a sleep(1)
                //yieldToSystem(); // This function doesn't exist yet!
            }
            sleepCount = 0;
        } while( taskList.len() > 0 ); // Exit when we've run out of threads to schedule
        running = false;
    }
    
    // Suspend a thread for at least "timeout" seconds
    function sleep(timeout) {
        local begin = clock(); // I would like to measure time in ms
        local now;
        
        do {
            sleepCount += 1;
            ::suspend();
            now = clock();
        } while( (now - begin) < timeout ); // Loop an suspend until at least the specified time has passed (may be more!)
    }
    
    function newchannel() {
        local c = {
            msgQueue = []
            rcvQueue = []
            
            function send(msg) {
                msgQueue.push(msg);
                if(rcvQueue.len()) {
                    rcvr = rcvQueue.top();
                    rcvQueue.remove(0);
                    rcvr.wakeup(msg);
                }
            }
            
            function receive() {
            
            }
        }
        return c;
    }
}

//
// Usage
//

function timedPrint(str, timeout) {
    for(local i = 0; i < 100; i++) {
        ::print(str + ": " + i + "\n");
        sqrat.sleep(timeout); // Sleep requires regular wakeups to be at all accurate. Works best when used with the scheduler
    }
}

function ping() {
    ::print("Ping\n");
    sqrat.sleep(1);
    sqrat.schedule(pong); // New closures can be scheduled while the scheduler is running
}

function pong() {
    ::print("Pong\n");
    sqrat.sleep(1);
    sqrat.schedule(ping);
}

// Works with anonymous functions too!
sqrat.schedule(function(x) {
    ::print("Countdown!\n");
    for(local i = 0; i < x; i++) {
        ::print(i + "...\n");
        ::suspend();
    }
})(10);

sqrat.schedule(timedPrint)("Tick", 5);
sqrat.schedule(ping);

sqrat.run(); // Will run indefinately due to the infinate ping/pong

are squirrel threads "true" threads in the sense of pthreads or Win32 threads?

One of the simultaneous design limitations and advantages of stackless python is that it operates in a single hardware thread. This means that it can't reap the benefit of multicore systems. But is also means that the overhead for inter-tasklet communication is comparatively lower.

Do the same issues apply to squirrel thread primitives?

Regardless, this seems really cool and something I'll definitely be keeping my eyes open for.

It seems Squirrel coroutines are not "true" threads but constructs inside the language VM

Honestly, I would really like to see this and a couple of other concepts become part of the core language, but I think Alberto has his hands full with Squirrel 3 right now :)

What are these "other concepts"?  Do you have implementations that can be patches against the standard Squirrel...  I think many people may be interested...

Well, my wishlist is pretty small for now but it grows every time I sit down and seriously use the language. :) Right now I would really like to see:

- Scheduling (as above)
- Channels
- Serialization of threads (save state to file and resume later)
- Possibly a "property" system like C#

If you haven't noticed, I like some of the concepts of Stackless Python :) I find Squirrel far more appropriate for embedded scripting, though, and can see a clear path for implementing all of these concepts using the Squirrel C api.

And no, I don't have full implementations (yet). It may be better, though, to start out with them as "extensions" to the language anyway to test functionality and acceptance before making an effort to integrate into the core language (kinda like Boost). In any case, I'm just testing the waters to see if anyone else even cares at this point.

pfox:

...Do the same issues apply to squirrel thread primitives?

Squirrel "threads" are very similar to Stackless' threads (technically "coroutines"). They all operate on a single OS thread, and must be suspended and resumed manually (or, of course, by a scheduler :) This gives exactly the benefits/limitations that you are talking about.

BTW, Toji, why don't you use acall instead of big switch - Squirrel alredy supports what you need?

[Squirrel Help -> The Language -> Built-in functions -> Default delegates -> Function]

acall(array_args)

calls the function with the specified environment object(’this’) and parameters. The function accepts an array containing the parameters that will be passed to the called function.

Ooh! I didn't even know that existed! And here I was all ready to go and code my own. :)

Excellent advice, and it makes me wonder how many other "hidden features" like this I'm missing that would make my life easier...

Thank you very much!

mmh threads don't have a acall() yet. I will add that :)

Alberto