XI4N - eXtensible Insim client 4 Nodejs [Archive]

the_angry_angel

11th January 2012, 19:37

Ladies and gents,

I present the first stable-ish, but not-yet-complete, version of the eXtensible InSim client 4 Nodejs (https://github.com/theangryangel/XI4N). As usual I'm throwing it out there early partly incase any other ****er has been working on it, and partly because it may be interesting to someone else.

It maybe used as a standalone "executable" (still requires nodejs (http://www.nodejs.org)) or as a library. Theoretically if the node sockets are swapped out, for socket.io it could also be used within a web browser - altho that would mean it leaks your admin password in a very public way.

Call it a "developer preview" rather than a finished product.

It was my 2 days off between Christmas and New Year project, but it's kinda grown a bit :D

I still plan on working on it for a bit as it's proving to be a useful distraction from work in the evenings at the moment.

As a bit of background, it's much like luaLFS (http://www.lfsforum.net/showthread.php?t=19832) was with "subscriptions" or hooks to functions, but more complete at this point. It also takes some inspiration from the awesome PRISM (http://www.lfsforum.net/forumdisplay.php?f=312), but with some Javascript orientated bits slapped on top.

README has a lot of info it in at the moment, including some basic usage. I've not documented much other than that yet. If you are planning on playing with it at this stage that main thing not mentioned is that I take explicit advantage of being able to manipulate "this" when plugins functions are called, giving easy access to the client connection, insim packet definitions, and the logger.

I just wanna say a quick bit of thanks to the guys who run the cargame.nl servers as I've been testing xi4n locally whilst connected to that, and doing 90 second sprints of furious debugging between races :D Also thanks for reminding me how much fun LFS can be :D

Dygear

11th January 2012, 21:06

That's unbelievable! Having the option to swap out NodeJS with socket.io to have it run in a browser is just awesome. Must play around with this, possibilities are endless!

cargame.nl

12th January 2012, 09:55

Great, awesome, ground breaking!

Eehhh.. What does it do? :x :)

I´ve also read that Lua thread a bit (its 5 years old :/ ). But I do not get it.

I just wanna say a quick bit of thanks to the guys who run the cargame.nl servers as I've been testing xi4n locally whilst connected to that, and doing 90 second sprints of furious debugging between races :D
Hhmm must have been a short track otherwise you get more seconds :) :thumb:

the_angry_angel

12th January 2012, 18:08

Cheers for running the cargame servers dude :D I really do enjoy playing on them, it's been a fun few weeks :D

Eehhh.. What does it do? :x :)Basically it's a multi-connection insim library written in javascript, but one you can also choose to run by pointing it at a directory with a config file and bunch of plugins.

It runs on nodejs which is a platform built on Google's (chrome) v8 javascript engine "for easily building fast, scalable network applications".

What xi4n does is it gives you a basic framework of socket maintenance, packet parsing, etc. and exposes an API that allows you to write plugins without worrying about how InSim really works - you just say what packets you want and what function(s) to call when that packet comes in - pre-parsed into an easy to use javascript object.

Once I've finished work for the day I'll try and write some documentation tonight and give a quick demo somewhere :)

The main benefit that I see over things like PRISM and pyinsim is that it's non-blocking by default, meaning you dont have to worry about yielding, or threads and the sync issues you get. By default all plugins can support as many servers as you've got in your config file. This combined with the other features that nodejs supports out of the box means you can very easily write things like an entirely in-memory live tracker (i.e. no DB, ActiveMQ, or other message queue or RPC), including InSim connection(s) and HTTP server, all in less than a few hundred lines of plugin :) That's the bit I find really cool :D

The other interesting thing, is that as its written in javascript it would be possible to do a little fiddling and port it to use websockets, which means given a sufficiently modern browser, you can get an insim connection directly from your browser, to your server. Could be very interesting for a control panel, perhaps (something I hadn't thought about.. hmmm).

I've got the nodejs bug at the moment after using it for a demo project at work and wanted something to play with at home :D

DarkTimes

12th January 2012, 20:24

The main benefit that I see over things like PRISM and pyinsim is that it's non-blocking by default, meaning you dont have to worry about yielding, or threads and the sync issues you get.

Just a small correction in that as of version 2.0 pyinsim has been non-blocking and runs everything on a single thread.

Dygear

13th January 2012, 05:43

And PRISM has always been non-blocking, thanks to Victor's network code (found in PRISM's core (https://github.com/Dygear/PRISM/blob/master/PHPInSimMod.php#L223) and in the modules (https://github.com/Dygear/PRISM/blob/master/modules/prism_hosts.php) he (https://github.com/Dygear/PRISM/blob/master/modules/prism_http.php) made (https://github.com/Dygear/PRISM/blob/master/modules/prism_telnet.php)).

the_angry_angel

13th January 2012, 12:44

Epic TAA fail. Boooo. Thats what happens when I don't pay attention :(

the_angry_angel

21st January 2012, 20:15

It took me a few extra days to get around to putting together a demo of what xi4n is exactly, and what it can do right now.

I figured since I'm not running it against a populated server at the moment I'd video something.

http://www.youtube.com/watch?v=U_SiwqAuzFw

Excuse the quality of the voice over. I don't like "public speaking", even pre-recorded and over the internet.

The entire xi4n install and livemap plugin, plus a few connected people weighs in at about 10-20MB of ram usage total, depending on the node garbage collection. Of that, about 4MB is the node environment. So it's reasonably slim for a 100% live tracker, in comparison to running your InSim client plus apache/nginx/IIS/whateveer and PHP/python/whatever and some form of database/RPC to communicate between the two.

Obviously what I've demo'd isn't 100% feature complete. I need to write some iconv stuff to deal with LFS' codepage nonsense and so forth, etc.

A couple of other things I'm interested in working on is a re-implementation of the LFSW relay that you can run yourself and allow anyone to connect in but not send destructive commands. I'm also interested in arsing about with a live control panel which may be an extension of the livemap plugin.

cargame.nl

21st January 2012, 20:40

I can't watch the video yet, text browsing atm.. But it sounds pretty cool.

Javascript is an easy 'language' and with that AJAX stuff nowadays not bad at all.

cargame.nl

23rd January 2012, 00:19

OK, nice!

So... SO4.pth ... OK

But... Can you do it with a custom pth as well?

Found here; http://www.airio.eu/Downloads.aspx

And... Can the system detect easily if a car is off track?

the_angry_angel

23rd January 2012, 08:59

But... Can you do it with a custom pth as well?At the moment it relies on insim's IS_STA packets to determine the track, and for a custom track that's only ever something like BL1X, but I'm sure it could be figured out some how - maybe from the IS_MSO packets when the track is loaded.. I'll see what I can do :)

And... Can the system detect easily if a car is off track?It could be added, again not something I've done - yet :)

mcgas001

24th January 2012, 21:32

Holy Shit... Just watched the youtube video. Thats pretty smart man. Well done! :thumbsup:

cargame.nl

25th January 2012, 00:21

but I'm sure it could be figured out some how
Yeah it's possible... With PRISM I look at the track code and then the name of the layout.

The problem with open configs is, is that there is no node information, but you say it doesn't matter ... Well.. OK.. Sounds good :tilt:

the_angry_angel

25th January 2012, 21:06

Holy Shit... Just watched the youtube video. Thats pretty smart man. Well done! :thumbsup:The example there isn't really doing anything but translating the insim packets into javascript objects and then basically proxying them to the browser and then theres some drawing going on - it's really not that hard, but thanks for the sentiment :)

I'm tempted to switch away from drawing using SVG at the moment, but I'll probably save that for much, much later. Maybe when webgl is a bit less crappy.

Yeah it's possible... With PRISM I look at the track code and then the name of the layout. Aah, yes, so I see - there's the layout name in IS_AXI. Hadn't noticed that :D That makes life a lot easier :D Cheers for the pointer dude :D

The problem with open configs is, is that there is no node information, but you say it doesn't matter ... Well.. OK.. Sounds good :tilt:I cheat and don't use any node info - only the X an Y coordinates from IS_MCI - you still seem to get them for custom layouts still. I've not tried drawing a custom track yet, but I don't see why it would be any different.. Or at least I hope not :D

Tonight I've been fiddling with translating LFS' strings into utf8, which is annoyingly tricky when javascript doesn't have anything suitable out of the box, and you start not really knowing anything about multi-byte character encoding. I've more or less got that sorted (although I really do want to murder someone who was involved with character encoding right now :tilt:) and I want to pop that into the packet handler so you can easily translate from LFS' munged mess to utf8 - useful for the livemap to deal with LFS' mixed character sets on the web interface in standard HTML. I've got a bit of work to get it to that stage, but the basics seem to work now, which is nice.

Next on my list, is setting up a general state manager that multiple plugins can use and then I'll get the livemap ported over to use that instead. It'll be a bit lighter on the bandwidth than the current version - which will be nice :) Hopefully it'll help out with the browser CPU usage too :)

Hoping to have a couple more hours before the weekend to try and get some of that done. Hopefully in a week or so time there'll be a nice, lower bandwidth, non-buggy version that can display most of the character sets LFS supports :)

Dygear

25th January 2012, 23:19

Took a look at the video, how did you get color into the CMD prompt, very jealous that you got this working in such a short amount of time. Keep on rocking on dude!

the_angry_angel

25th January 2012, 23:45

how did you get color into the CMD promptCheating really. The node runtime executable for windows is kind enough to add support for the normal ANSI escape codes that work on everything else, so I basically get ANSI colour support on Windows for free.

I don't really understand why Microsoft never brought that across from Windows 9x into NT land :(

There is ansihack, but I don't know if it works on modern versions of Windows. There is also ansicon (http://adoxa.110mb.com/ansicon/index.html) (also on github (https://github.com/adoxa/ansicon)), which I use for a couple of other projects both personal and at work, and is still updated

Keep on rocking on dude! Cheers for the encouragement dude :) It's definitely appreciated :thumb:

the_angry_angel

28th January 2012, 18:31

Hi guys,

Just an update. If you'd rather this in video form there's another on the youtubes (http://www.youtube.com/watch?v=ABstuvDAcOA). It's not necessary to watch as it's just the same as what I've written here and visually there's not much difference :)

I've bumped the version to 0.0.2 because of an API change.
I've completely rewritten the livemap plugin - the original is now gone.
Livemap2 depends on a generic state plugin, which makes livemap2 a lot simplier
Livemap2 now understands the concept of layouts, and should attempt to load them - I've not yet managed to test that yet, but it should work :)
Livemap2 is lower bandwidth
Really basic LFS string support (translating codepages and colours)

Translation to UTF8 is mostly working, but there are few characters that aren't quite right - I'm working on those :)
Colour support, but needs tidying massively
I want to add multiple output types, so you can auto-colour for the web, or for the console, or strip completely

Livemap2 has highlightable players (pretty sure that wasn't in the original)
There are a couple of problems with either the livemap2 or state plugin, such as it occasionally not picking up players that have left, but I'm working on them.

The video discusses a couple of ideas I have for the future for this project, such as reverse chat (i.e. chat from the web interface to the server). One I've not mentioned in that I've got a couple of ideas for a longer 3/6/12/24 hour event with a live stream, and live interactivity from non-racers in a few months time - but we'll see how things go as a lot of it will depend on xi4n's progress and how much of what I've got in mind is realistically implementable :)

misiek08

4th May 2012, 23:07

The node.js server makes 1 InSim connection and every web client is connecting through node.js server so it's still 1 InSim connection?

If yes - you are my JS master! :) It's incredible that we have sockets in JS.

the_angry_angel

7th May 2012, 15:53

The node.js server makes 1 InSim connection and every web client is connecting through node.js server so it's still 1 InSim connection?Correct :) The livemap plugin works pretty well underload - I've had a couple of thousand concurrent connections before there are CPU issues on a small VM.

The web client itself is a little hacky and could really benefit from being ported to use angular.js or backbone, or something similar, but it's totally usable as it is right now.

As an fyi I've not worked on anything that's in the github repo at the moment (the repo on my home network has a few additions that I've not pushed up), as I've been using my evenings for a work project for a few weeks, however I do have plans to continue with xi4n as I've got a couple of competition plans that totally rely on it :)

Edit: The video above is out of date, in terms of the look and number of bugs in the livemap plugin - aside from some codepage issues, it should be bug-free.

Edit: Updated video on the youtubes - regarding what I've done, some experimental in-browser 3D track stuff, web control panels and bonus rambling: http://www.youtube.com/watch?v=vuZly5agOkM

misiek08

9th May 2012, 16:03

Push the changes if you can :)

Kiyoshi

13th May 2012, 12:58

This is very interesting, but a little confusing to an old-fashioned programmer. :)

It looks like at it's core, it basically translates insim packets (from a lfs server, or client?) to JSON versions, and can run local js plugins to 'do stuff' with the insim information locally. But what can it do beyond that? Does the lapper example just serve HTML to any connecting browser, or does it serve JSON insim packets directly to XIaN clients? Or could it do both? Should I just shut up & try to understand the code? :)

I'm only vaguely aware of the capabilities of nodejs, but it seems that this potentially gives it real flexibility. Have you any idea how it performs compared to similar things? That's probably not an issue with server insim information, but could be more important if all the other data is required.

the_angry_angel

13th May 2012, 17:05

You could serve packets to any connecting browser, if a plugin setup a HTTP server - like the livemap plugin does :) Or you could send back packets to LFS, like you would with any other InSim application.

Basically xi4n itself just translates and event-ifies packets from LFS, and a plugin just receives those events from the xi4n core. A plugin doesn't have to talk to a browser, it can talk to anything it needs or wants to :) There's no restriction on what a plugin can do at the moment, so if nodejs itself can do it, so can a plugin - be it talking to a browser, storing chat in a database or talking to another insim program - you name it and it can do it :)

Hope thats a bit clearer? I will be honest the code is a little hacky still and needs a bit of a tidy when I've got stuff working in a reasonable way, so it may not be the easiest to follow right now.

misiek08

13th May 2012, 17:58

How to setup password on InSim connection? I don't see in opts anything called admin or password (clientmanager.js).

the_angry_angel

13th May 2012, 18:58

Apparently I forgot to add password capabilities completely :( I'm about to head away for for work, working very long hours for the next 2 days (behind a restrictive firewall that I don't control - yet), but I'll do my best to get it added into the github repo shortly :) If you need to hack it in for now you can pop it into https://github.com/theangryangel/XI4N/blob/master/lib/client.js#L165

misiek08

13th May 2012, 19:13

Oh, there. Sorry, I was looking in clientmanager.js.

The TV plugins looks nice. I didn't tried the livemap yet, but it can be best thing for 16h race this year. I'm going to host it if organizers will approve it!

EDIT:
TV plugin doesn't know what VIEW_CAM and VIEW_DRIVER is. I don't know why, but it don't read exports.VIEW_ variables.

Kiyoshi

20th May 2012, 12:39

Thanks for the response, it helped a bit. :)

I did manage to get this running at least, though I needed a few more steps than were in your readme - if others are going to try this in my situation, then it may be worth mentioning them:

I am running this on Windows, and have no experience of node.js or nmp.
I had to run 'nmp link' in the xi4n folder.
I also had to run 'npm install <module>' as I had several error messages reporting missing modules. I imagine this was due to a fresh install of node.js, though I'm not at all sure. :)

I haven't looked at the code yet. I don't know a great deal about JS either, but it looks interesting. :D

the_angry_angel

20th May 2012, 14:01

The TV plugins looks nice. I didn't tried the livemap yet, but it can be best thing for 16h race this year. I'm going to host it if organizers will approve it!There are 2 versions at the moment and they both work in very different ways.

The first version, which I videoed is very, very dumb. The second version I've been working with on and off is in the tv.js-priority branch on github and works by scoring - https://github.com/theangryangel/XI4N/blob/tv.js-priority/plugins/tv2.js

If you're interested the current revision will/is using something like this to score:
https://docs.google.com/spreadsheet/ccc?key=0AtZGi7WCMINHdHBfd2JNWFN2clZXRUtvUXRFX05Ba Xc

A lot of stuff isn't in there yet, like distances.

TV plugin doesn't know what VIEW_CAM and VIEW_DRIVER is. I don't know why, but it don't read exports.VIEW_ variables.You using this.insim.VIEW_CAM? Because of how the whole nodejs require stuff works, it's all namespaced and not really like a header definition like you'd get in C/PHP/whatever.

I had to run 'nmp link' in the xi4n folder.Interesting, what version of node are you on? You can it by running node --version.

I also had to run 'npm install <module>' as I had several error messages reporting missing modules. I imagine this was due to a fresh install of node.js, though I'm not at all sure. :)More likely me missing some dependencies in the package.json :) Don't suppose you remember what you needed to install?

I haven't looked at the code yet. I don't know a great deal about JS either, but it looks interesting. :DPatches, etc. welcome :) There are a couple of places where it really needs a tidy - but they all focus around https://github.com/theangryangel/XI4N/issues/9 and https://github.com/theangryangel/XI4N/issues/8 so I've not done the tidying yet :) Specifically the bits that are really nasty is the config handling and plugin loading :tilt: Patches welcome if you wanna come and play though :thumb:

As an FYI to anyone following, I've just rejigged how the plugins can sit in the plugins directory, so if you pull a new revision and it's all busted, that'll be why :x I've tried to make it so that the new style doesn't break the old style though

Kiyoshi

20th May 2012, 15:31

I had a closer look at the dependancy issues I had. It does seem to be a Windows nodejs/npm issue, similar to this one here:
http://stackoverflow.com/questions/5919629/express-module-not-found-when-installed-with-npm

'npm link' doesn't work on Windows, apparently, and I don't think the npm install can install any dependancies into nodejs, which might just be the problem.
When trying to run XI4N, the missing modules nodejs reported were:
express; kdtree.js; socket.io; underscore

I originally got it working by 'npm installing' these in my xi4n folder, but that wasn't the right way to do it. I have now managed to install them to nodejs, though it was fiddly & I had to move their actual folders from node_modules into the nodejs root to get it to actually work.

So, don't let this distract you - I don't think it's any problem with the XI4N package, but a nodejs on Windows issue. nodejs version is 0.6.17 incidentally.

misiek08

20th May 2012, 16:12

I had to type npm install <packet> for 3 packets and it works good, without any problems. I'm on W7 64bit.

the_angry_angel

20th May 2012, 18:55

So, don't let this distract you - I don't think it's any problem with the XI4N package, but a nodejs on Windows issue. nodejs version is 0.6.17 incidentally.Unfortunately this is all my fault :x Doh - sorry about that :Looking_a

All those packages are depended on by plugins, rather than the xi4n core - but at the moment they don't load from their own directories - it's something I'm going to be putting in a fix for soon :)

misiek08

21st May 2012, 16:46

So tickets from me:
- add admin password to config
- add possibility to run from LFS Master (not directly from server)

Other things are made very good and if someone knows basic of programming - they can do everything.

My next question is: will this work with nginx (is live stream of packets, so I'm asking), how many resources does this need to run on Linux (Windows eats too much and I wonna prepare PRISM and node.js hosting for applications like this one).

the_angry_angel

22nd May 2012, 09:05

I've added issues (https://github.com/theangryangel/XI4N/issues?sort=created&state=open) to github for your 2 things, that way I don't forget like I did at the weekend :x I got distracted by shiny things :o

When you say it's using too much memory/cpu/disk, how much and specifically what? If you're running it on windows a perfmon log (with % Processor time, Working and Private Set of the nodejs instance), + xi4n log as a pair would be awesome :) If you're running Linux then can you run top in batch mode (top -b) and also output xi4n to a log file and send those over please? If I can figure out what part of it is eating memory or CPU time, that can hopefully be fixed. I suspect it's probably to do with how packets are thrown about, at a guess.

Only reason I'm asking is because I'm seeing xi4n top out at about 20MB before the GC kicks in... :x

As for using it with nginx, or any other web server, the easiest way is to reverse proxy from your frontend server to nodejs. As it is out of the box, it may need a little content re-writing by the HTTP server. For nginx you want to be looking at the proxy_* directives, and subs_filter_* for the rewriting.

misiek08

22nd May 2012, 15:17

I mean, Windows is taking too much resources :)

I know how to make reverse proxy, but I just don't know how will it behave while getting data live.

misiek08

22nd May 2012, 18:33

I heard about PHP path problems, but here too? The track is wrong oriented. Look here: http://misiek08.mydevil.net:8082/

Dygear

23rd May 2012, 06:19

I heard about PHP path problems, but here too? The track is wrong oriented. Look here: http://misiek08.mydevil.net:8082/

It's not a PHP problem, it's a DirectX idiosyncrasy.

the_angry_angel

23rd May 2012, 08:08

I know how to make reverse proxy, but I just don't know how will it behave while getting data live.ningx added HTTP 1.1 support for proxy_* directives in 1.1, so as long as you're on that or newer it should just work out of the box. If it's older than 1.1 I believe you need to look at just using a raw tcp proxy, which can be done with one of the nginx modules - I've not tried socket.io reverse proxying with older versions tbh.

I heard about PHP path problems, but here too? The track is wrong oriented. Look here: http://misiek08.mydevil.net:8082/Meh, I supposed the cheats way would be to use css to rotate the SVG element :shrug:

misiek08

23rd May 2012, 11:34

Maybe it's possible to reverse values for each node and it will work?

Maybe there was mistake when you saved pth files to .json versions. Can you provide me files to parse .pth to .json? Was it clean PTH File Parser from Dygear and json_encode?

the_angry_angel

23rd May 2012, 17:48

Source is in the repo, but yes it is Dygear's PTH parser.
https://github.com/theangryangel/XI4N/blob/master/build/gen-json-pths.php

misiek08

23rd May 2012, 18:45

@Dygear - is it possible you to make version that exports good coordinates? It doesn't look like basic mirroring points...

Dygear

24th May 2012, 06:23

Source is in the repo, but yes it is Dygear's PTH parser.
https://github.com/theangryangel/XI4N/blob/master/build/gen-json-pths.php

I've never seen the glob (https://github.com/theangryangel/XI4N/blob/master/build/gen-json-pths.php#L7) function before, cool trick, I'll have to remember that from now on.
I should also mention, that the version you're using of the pth parser was last updated by Victor, most of that code is his.

@Dygear - is it possible you to make version that exports good coordinates? It doesn't look like basic mirroring points...

It's just a flipped Y axis. (http://www.lfsforum.net/showthread.php?p=1705502#post1705502)

So an update to TAA's foreach within the pth2json function (https://github.com/theangryangel/XI4N/blob/master/build/gen-json-pths.php#L30) would look like this:
foreach ($pth->Node as $n)
{
if ($i == $pth->FinishLine)
$data['startfinish'] = array('x' => $n->Center->X, 'y' => -$n->Center->Y, 'z' => $n->Center->Z);

if ($i % $resolution)
$data['nodes'][] = array('x' => $n->Center->X, 'y' => -$n->Center->Y, 'z' => $n->Center->Z);
$i++;
}

misiek08

24th May 2012, 13:04

It's just a flipped Y axis. (http://www.lfsforum.net/showthread.php?p=1705502#post1705502)

Thanks for the link. It will help not only me and not only here :)

misiek08

24th May 2012, 21:04

Edit: It was my experiments. It's working.
http://misiek08.mydevil.net:8082/

Edit2: Path is good, but cars are going on old path (like with wrong y's).

misiek08

24th May 2012, 21:21

livemap2.js, line 32:
'y': -p.y, :)

the_angry_angel

25th May 2012, 15:39

The only issue I have with changing the converted PTH files is that then you realistically need to alter the data in the MCI other packets that contain positions as well. That then leads to confusion when or if you ever port to something else. Or have to interoperate with another addon - you don't want users to be surprised when they get different data from xi4n in constrast to another library.

I'd personally much rather put this fix directly into the livemap plugin - which I'll try and make time for before the end of the month.

Edit: As 2 people have now asked, please feel free to fork on github.com and submit pull requests. If you do submit a pull request feel free to add yourself to the package.json as a contributor (see http://package.json.nodejitsu.com/ if you're the first or not familiar with package.json format). I can't guarantee when I'll be able to do much work on xi4n for a few weeks, but I'll try and merge pull requests as often as possible.