0.6A1 InSim.txt and LYT.txt

[Scawen]

Hello Programmers.

Sorry, I just realised I forgot to put the new InSim.txt in the patch. Here it is attached, along with the LYT file specification.

[EQ Worry]

InSim libraries will need some serious updates with version 5. :thumb: Thanks for the admin commands stuff, I already feel safer. :) Hopefully I could test it toning, to see e.g. what "set if user is an admin" actually means, if this byte just uses 0 and 1...

[Scawen]

Yes, that byte should just be 0 or 1.
Because some "admin" commands are available to non-admins if guest selection is enabled, like /track=bl3

[Dygear]

Fifth line reads // InSim for Live for Speed : 0.5Z should be // InSim for Live for Speed : 0.6A(1).

Sixteenth line reads // INSIM VERSION NUMBER (updated for version 0.5Z30) should that not be // INSIM VERSION NUMBER (updated for version 0.6A1) also?

Thank you very much for the IS_ACR packet. That's a great help!

[EQ Worry]

Scawen, I have one incompatibility issue, sorry to bother you again. Almost all the new packet types (layout/hit/contact) are optional, requested on InSim connect. The only exception is the ISP_ACR admin command packet, which is always sent. Unfortunately the library I use is detecting it as bad packet type, assuming communication error, leading to disconnect. Maybe a short-sighted assumption, but it worked very well for 2.5 years. :)

Of course I updated the library and I will think how to make it better future-proof, but all the current Airio versions will cease to function properly after server update to 0.6, which will happen soon, I guess. So, please, is there any chance to make the ISP_ACR packet also optional, that must be required during InSim connect? It would help help a lot now. Thanks for reading. :schwitz:

PS: I guess the demo/tweaked/private bits of server state did not make it to your intended features list? :tilt:

[Dygear]

PRISM did not have this problem, I don't think anyway, but then the packet handling code was made by Victor. :nana:

[EQ Worry]

Well, yeah, I'm trying to explain this weird approach in a bit more detail here (http://www.lfsforum.net/showthread.php?p=1603138#post1603138).

[Scawen]

No... InSim is specifically designed so that you can continue to read without knowing all packet types. I will not make all new packets optional.

Any programmers out there, please read this post I made on the other thread. Very important in case your code doesn't handle the rare but very real situation of split packets.

http://www.lfsforum.net/showthread.php?p=1603154#post1603154

[EQ Worry]

Alright.

[EQ Worry]

PS: I guess the demo/tweaked/private bits of server state did not make it to your intended features list? :tilt:

I wonder if this could be one of the last-minute small updates for the official patch. :schwitz: I promise, this is the last time I mention it, also I have no more ideas/requests (well, for this patch at least). :shy:

Now I remember a long-existing thing, the "unknown car id" displayed sometimes. I wonder if that could be related to the fact that often cars are send to /spec right after pitting (Shift+P) by InSim apps, e.g. to prevent repeated midrace rejoins from pits. Maybe a small lag would cause some LFS pitting routine to be delayed for so long that when it runs the car is not in the pits anymore, but spectating? Just an idea though.

Again, thanks for all the updates! :thumbsup:

[GeForz]

I just invented a new engine by spawning poles at the car's position and let the collision system propulse the car to up to 47kph!

:D

[T3charmy]

only thing i wish, was on OBH and HLV that it is actual speed not closing speed...

[EQ Worry]

only thing i wish, was on OBH and HLV that it is actual speed not closing speed...

If I see correctly, HLV has no speed, only CON and OBH have it. And, maybe I am too tired, but what use would be the actual speed? The closing speed describes impact severity, which seems to be the perfect measure of contacts with other cars and objects.

[T3charmy]

If I see correctly, HLV has no speed, only CON and OBH have it. And, maybe I am too tired, but what use would be the actual speed? The closing speed describes impact severity, which seems to be the perfect measure of contacts with other cars and objects.

yea, IDK why i said HLV, i mean CON and OBH, by actual speed, I meant the speed that the user was going, when he, hit the object :P

[Scawen]

Look more closely at the packets - there is a value "Speed" in the CarContact and CarContOBJ structures.

[DarkKostas]

As the previous topic about insim changes is locked ill post it here.
I would like a small change to this

struct IS_AXI // AutoX Info
{
byte Size; // 40
byte Type; // ISP_AXI
byte ReqI; // 0 unless this is a reply to an TINY_AXI request
byte Zero;

byte AXStart; // autocross start position
byte NumCP; // number of checkpoints
word NumO; // number of objects

char LName[32]; // the name of the layout last loaded (if loaded locally)
};


Is it possible to make it so when a remote player loads a layout, we can get the name of it? Because i want to store layout name and use it.

[T3charmy]

Look more closely at the packets - there is a value "Speed" in the CarContact and CarContOBJ structures.

Oh wow, I didn't even see those *facepalm*

[Scawen]

PS: I guess the demo/tweaked/private bits of server state did not make it to your intended features list? :tilt:These were on my list for a while, but there were a few ways of doing it and I couldn't decide the best way. Some of these flags are stored in different ways. In the end, I came to a point where I couldn't do any more after this 3 months push. I'm tired. Also, in the few days before an official patch I can't take any chances. So... sorry, the flags and demo status are not there for 0.6B.

Now I remember a long-existing thing, the "unknown car id" displayed sometimes. I wonder if that could be related to the fact that often cars are send to /spec right after pitting (Shift+P) by InSim apps, e.g. to prevent repeated midrace rejoins from pits. Maybe a small lag would cause some LFS pitting routine to be delayed for so long that when it runs the car is not in the pits anymore, but spectating? Just an idea though.I thought that was fixed. Though my memory is a bit vague at this point. I thought it was worked out and sorted out. Have you seen that error coming up in any of the recent test patches?

[Flame CZE]

I thought that was fixed. Though my memory is a bit vague at this point. I thought it was worked out and sorted out. Have you seen that error coming up in any of the recent test patches?
I remember happening that too in the recent test patches, I don't know if it depends on the client or host LFS version though.

[EQ Worry]

These were on my list for a while, but there were a few ways of doing it and I couldn't decide the best way. ... I'm tired. So... sorry, the flags and demo status are not there for 0.6B.

Thanks for getting back to the matter. :) I understand, it was required too late and it's not as easy as it may seem. It would be great though to keep it as a future feature. The tweaked bit could be used to e.g. disable lap time storing (maybe also for LFSW). Demo bit would allow easy detection of demo-compatible host setup, which is now impossible (or at least complicated). One final thing missing is a report of currently allowed cars on host, many people find it strange this important info is not in fact available. Also, reporting things as vote, midrace join, car reset, etc. only on race starts is not ideal, I think, it all should be part of server state info state (with cars, demo, ...), sent on every change. Just my feeling though. :)

I thought that was fixed. Though my memory is a bit vague at this point. I thought it was worked out and sorted out. Have you seen that error coming up in any of the recent test patches?

I'm afraid I did not have as much time to be on patched servers, also rarely some such server got full enough. Last thing I remember was you mentioning that you're not sure why this message appears. Now I believe it happens only after pitting, when the car is quickly sent to spectate by an InSim. Just a feeling though.

Thanks again for all the InSim updates, some look really promising and some make me (and other server admins, I hope) feel safer. But I'm afraid the real hard test will start only when official patch is out and the current applications are extended to apply the new principles. Problems may appear then, when there are thousands of actions taking place. But so far everything looks good to me.

Final note, thanks for mentioning the split packet problem. (A friend of mine also mentioned the opposite may happen, two LFS InSim packets arriving combined.) While my library handled the 2nd case correctly, the 1st was actually not handled at all. Fortunately the code update was not too complicated, if anyone's interested, I could post the few lines of (supposedly correct) split/joined packet handling C# code here for review/discussion. :)

[Mischa NED]

Fortunately the code update was not too complicated, if anyone's interested, I could post the few lines of (supposedly correct) split/joined packet handling C# code here for review/discussion. :)

Go ahead :) And indeed thanks to the devs for all these nice updates :thumb:

[EQ Worry]

OK, just the packet(s) reading code, merging and splitting them as needed:

byte[] buff = new byte[256]; // number of bytes read stored in bts
int bts = tcp.Receive(buff, 1, SocketFlags.None); // InSim packet length stored in buff[0]
if (bts == 0) { return new byte[0]; } // disconnect
while (buff[0] > bts) // read from TCP until packet length
bts += tcp.Receive(buff, bts, buff[0] - bts, SocketFlags.None); // read only one packet
Array.Resize(ref buff, bts);
return buff;

No guarantees though that this is 100% correct, I may have overlooked something.

[EQ Worry]

I've noticed this also earlier, though there are no doubt exceptions: Posting any code makes the thread die! :razz:

[Squelch]

I've noticed this also earlier, though there are no doubt exceptions: Posting any code makes the thread die! :razz:Everyone is off and busy coding. :D

Thanks for the code snippet. It's useful boilerplate to hang on to.

[Mischa NED]

I've noticed this also earlier, though there are no doubt exceptions: Posting any code makes the thread die!

Sorry, forgot about the thread because I've been quite busy this weekend. Lets check it

I think Scawen wrote something about invalid packet sizes before. Shouldn't they be ignored if not dividable by 4? In my code (which is slightly different) I added this check:


byte[] buff = new byte[256]; // number of bytes read stored in bts
int bts = tcp.Receive(buff, 1, SocketFlags.None); // InSim packet length stored in buff[0]
if (bts == 0) { return new byte[0]; } // disconnect
if (buff[0] % 4 != 0) { return new byte[0]; } // disconnect (invalid packet size)
while (buff[0] > bts) // read from TCP until packet length
bts += tcp.Receive(buff, bts, buff[0] - bts, SocketFlags.None); // read only one packet
Array.Resize(ref buff, bts);
return buff;

[Krayy]

Is there a list of freeride track codes, or are they just the normal track IDs with freeride enabled?

If thats the case, I cannot see any InSim packet that identifies if we're in freeride mode which may affect split times by people taking shortcuts etc. Am I missing something quite obvious?

[GeForz]

You mean the open configs?
Those have the normal track code appended by either X or Y.

e.g. for AS1

AS1 - normal track
AS1R - reverse track
AS1X - open config with normal AS1 start position
AS1Y - open config with reverse AS1 start position

[Krayy]

That would be the obvious thing that I've missed then.

Thanks

[EQ Worry]

I think Scawen wrote something about invalid packet sizes before. Shouldn't they be ignored if not dividable by 4? In my code (which is slightly different) I added this check: ...

Well, yes, I do that packet size check later on when the read packet is actually being processed, this is just TCP (whole) packet reading code.

Also, I added code to notify me about split packets and indeed it happens sometimes. It seems the code is now correctly handling both split and merged packets though, so it was a great note/tip Scawen made. :)

[Mischa NED]

Well, yes, I do that packet size check later on when the read packet is actually being processed, this is just TCP (whole) packet reading code.

Also, I added code to notify me about split packets and indeed it happens sometimes. It seems the code is now correctly handling both split and merged packets though, so it was a great note/tip Scawen made. :)

Good to hear :) I check the package size in TCP handling already and when the packet size is wrong I terminate the connection. This never happened though (according to my log). My handling is a little different and I think it always handled split and merge packets right. I read the packets like if it's one big stream. All packets go into a buffer and I read only (or until) the size I expect for the specific packet. I will add some logging to see it actually happens. Do you know if there is a specific situation that this splitting/combining occurs or is it completely random?

[EQ Worry]

Hmm, I'm really not sure when it happens. Airio had troubles with remote connection to servers (running on another PC, data transferred over the Internet), especially when the connection was not perfect. I guess split packets were more frequent then, leading to erroneous disconnects for bad packets. But it may as well be just my imagination. :)

[Mischa NED]

Hmm, I'm really not sure when it happens. Airio had troubles with remote connection to servers (running on another PC, data transferred over the Internet), especially when the connection was not perfect. I guess split packets were more frequent then, leading to erroneous disconnects for bad packets. But it may as well be just my imagination. :)

Hmmm I'll try that... and the erroneous disconnects for bad packets? How should I imagine that? :) Insim got disconnected with some error message or?

[Victor]

on the Relay i normally see split packets whenever there has been some lag. Because then after the lag is gone, I'll receive a big bunch of data all at once. And because that data may be bigger than the network buffer, it's just split at the buffer's boundary.

You can say it like this too : LFS always sends small packets. These are so small that they'll always fit into one network packet. Even if LFS needs to send several packets (like MCI) at once, 4 or 5 of them may still fit into one network packet. So these will not be split, but they will be concatenated -> a stream.
But now imagine there can be situations (lag / packet loss) where the amount of data exceeds the network packet's window size - then TCP will split the data after the network packet is full and continues with the data in the next TCP packet.

---

Ok now that I'm writing this anyway, I'd like to make a recommendation : (hm two it seems)

In your piece of network code you're constantly allocating memory for buff. This means the garbage collector will be doing a lot of work. Better create a byte[] buffer once (when you construct the networking class) and use that for all your network buffer reading needs.
In fact, I think reading byte for byte / packet for packet from the socket isn't good practise either. You're better off reading all the data in the network buffer asap, into a separate buffer. Then call a method that processes that data buffer to find the next valid packet.
That way your socket buffer will be more available, because you empty it completely as soon as you detect that it holds any data. Under stress, this allows the kernel to quickly fill the buffer again, keeping the data flow between two sockets going quicker.

So :

-create one buffer property in the class, where you read network data into. I would make it 4KB in size.
-always read the entire data in socket's buffer into our local buffer.
-after reading all data from socket, process the buffer to find the next packet(s)

[EQ Worry]

Victor, great explanation and awesome suggestions, thank you! :) In short, make one big static buffer, always fill it with everything available and process the data. Nice.

As for the erroneous disconnects, I meant that my earlier code was not processing split packets correctly, so it lead to bad packets being detected, especially when run remotely, which meant disconnects, probably just because of bad code and not bad packets. :schwitz:

[Victor]

ah, yeah.
Well if you write that bigger 'static' buffer, then every time you have read data from the socket, you just check the first byte in the buffer, which must always be the size byte. Is it % 4? Is there enough data in the buffer? Then you have a packet. Take the packet out of the byte buffer and pass it to whatever method will be processing individual insim packets.

[Mischa NED]

Victor, great explanation and awesome suggestions, thank you! :) In short, make one big static buffer, always fill it with everything available and process the data. Nice.

As for the erroneous disconnects, I meant that my earlier code was not processing split packets correctly, so it lead to bad packets being detected, especially when run remotely, which meant disconnects, probably just because of bad code and not bad packets. :schwitz:

Thanks indeed Victor! I already used a "receive buffer", but now I notice I always copy the packet out of the buffer to use in my packet handler. So.. there is the overhead too. I should change that. I think i have the same problem as EQ with the "bad code" ;-) That tcp stuff is one of the first things you make when building a InSim library and hmm... I don't remember how long ago I did it, but when I watch the code I don't understand much of it anymore :D I'll try to optimize it and will let you know :D

[EQ Worry]

LOL, Mischa, you're absolutely right, same here. :tilt:

[Mischa NED]

I analyzed my code tonight and even found a bug in it. I already worked with a read buffer and continued reading from the tcp if packets weren't complete. Combined packets were handled correctly, but if there would be a split packet (I never saw them yet :D) the rest of the packet was written on the wrong position in the buffer. I still have to test it and hope it's fine now ;) I'd like to share it with you guys. I removed the error handling to make the code more readable:

// buffer
const int _BUFF_SIZE = 8096;
byte[] _anbBuff = new byte[_BUFF_SIZE];
int _nBuffUsed = 0, int _nBytesRead = 0, int _nOffset = 0; // start reading from the first position in buffer

public void Listen (IAsyncResult iaResult) {
// get tcp data
_nBytesRead = _nsNetworkStream.EndRead (iaResult);

if (_nBytesRead > 0) {
_nBuffUsed += _nBytesRead; // _nBuffUsed will contain total number of bytes in buffer

// as long as the buffer contains (more) data and the size of the data is at least equal to the size of the processed packet
while (_nBuffUsed > _nOffset && _nBuffUsed >= _nOffset + _anbBuff [_nOffset]) {

// if packetsize % 4 (valid packet size)
if (_anbBuff [_nOffset] % 4 == 0) {
// copy and handle the full packet
byte[] anbPack = new byte [_anbBuff [_nOffset]];
Array.Copy(_anbBuff, _nOffset, anbPack, 0, anbPack.Length);
_handlePacket (anbPack);
}
else {
// invalid packet found, throw exception and terminate connection
}

_nOffset += _anbBuff [_nOffset]; // put offset on next packet
}

// if all packets in buffer could be processed, reset buffer pointers
if (_nBuffUsed == _nOffset) {
_nBuffUsed = _nOffset = 0;
}

// wait for more tcp data
_nsNetworkStream.BeginRead (_anbBuff, _nBuffUsed, _BUFF_SIZE - _nBuffUsed, new AsyncCallback (Listen), _tcpClient);
}
else {
// no data received, throw exception and terminate connection
}
}

[Mischa NED]

I've noticed this also earlier, though there are no doubt exceptions: Posting any code makes the thread die! :razz:

You were right EQ :-)

[EQ Worry]

Well, yes. :) Just one thing I noticed during experiments: Split packets are rare but possible, merged (joined) packets quite very common. But even the simple code in one of my previous posts works OK. I believe Victor is right that creating new byte arrays constantly is not optimal performance-wise, but on the other hand it is simple and Airio never uses more than 1-3% of CPU power for normal processing, so... Uhm... :)

[Mischa NED]

Well, yes. :) Just one thing I noticed during experiments: Split packets are rare but possible, merged (joined) packets quite very common. But even the simple code in one of my previous posts works OK. I believe Victor is right that creating new byte arrays constantly is not optimal performance-wise, but on the other hand it is simple and Airio never uses more than 1-3% of CPU power for normal processing, so... Uhm... :)

I still haven't seen split packets, but will test it with dedi on another server, maybe I see them then. The only way I got "split" packets was when I made my receive buffer to small so not all (combined) packets fit into it, so the packet was split in my own buffer. The handling is the same though, so I'm sure it will handle split packets correctly. Have a nice day!

[DarkTimes]

You can look at the packet receive code that InSim.NET uses, which is in the TcpSocket (http://insimdotnet.codeplex.com/SourceControl/changeset/view/25a0ee480303#InSimDotNet%2fTcpSocket.cs) class. Undoubtably not perfect, but it seems to work pretty well. Of course it deals with multiple packets and split packets etc.. If you have any improvement suggestions I'm happy to hear them.

This is the simplest receive loop I can think of. Obviously not the most efficient code, but it makes it easy to see all the parts working.

// Store buffer in generic list.
List<byte> buffer = new List<byte>();

// Temp array to store received data.
byte[] temp = new byte[2048]; // Receive up to 2kb of data at a time.

while (true) {
// Receive data from socket.
int received = socket.Receive(temp);

// Nothing received means connection dropped.
if (received == 0){
throw new Exception("lost connection");
}

// Append received data to buffer.
buffer.AddRange(temp.Take(received));

// Loop through each completed packet in the buffer. The first byte in
// the buffer is the size of the first packet.
while (buffer.Any() && buffer.Count >= buffer.First()) {
int size = buffer.First();

// Check packet is multiple of 4.
if (size % 4 > 0){
throw new Exception("corrupt packet");
}

// Read first packet from buffer and remove it.
byte[] packet = buffer.Take(size).ToArray();
buffer.RemoveRange(0, size);

// Do something with the packet.
// HandlePacket(packet);
}
}

Edit: It should be noted as well, as many InSim devs seem to forget this, that when sending packets you also need to pay attention to how many bytes are sent. The send function returns the number of bytes transferred, and if this is less than the size of the packet you need to resend any data that was omitted. This is simple however.

byte[] buffer = GetPacketToSend();

// Keep looping while there is unsent data.
int sent = 0;
do {
sent += socket.Send(buffer, sent, buffer.Length - sent);
} while (sent < buffer.Length);

[cargame.nl]

and Airio never uses more than 1-3% of CPU power for normal processing, so... Uhm... :)
Ever seen the power which is being used when 400 Mb internal memory is being sorted by a !top or !pi command without indexes? :schwitz:

:razz:

[t1ger]

First of all, a big thank you for implementing the ISP_HLV packet - I have used it to good effect on our qualifying server so we no longer need to build layouts to stop corner cutting. Instead, all laps just have to be HLVC compliant and it is so easy because people can use hotlapping to perfect their lap before joining our quali server.

Anyway, I also just used it to watch for corner cutting during races (by analysing the replays with ISP_HLV turned on) and I found a strange thing. I am tracking all cars with the MCI packets, and I wanted to know which node the HLV violation was fired so I could trend it to see if they cut the same corner every lap.

The corner in question is node 445 on KY3 (which is the corner coming off the oval back on to the road track where there is a nice chicane (perfect for cutting)). On lap 5 and 6 of the replay for a particular car, the HLV packet fires before the car has actually reached the off track bit. When I get the HLV packet, the car is at node 443, which is actually the middle of the track on the line they have taken. It does then cut the corner at 445, where I would expect the HLV packet to be fired.

Is this something to do with lag? Is the HLV coming from the client and being saved in the replay (maybe?) instead of being generated again by the replay when it is running? If it is, I doubt there is an easy fix, but just wanted to understand the reason if nothing else.

[Dygear]

As far as I know, the replay data is engine events from the servers point of view after dealing with lag. InSim packets, are then made based off those events when you run your replay by the engine on the fly. So that might be why there is some inconsistency.

[Scawen]

Dygear is correct. The HLV violation was originally detected on the guest's computer in the correct place. In the MPR your computer is processing that information packet before the car actually gets there. It seems that the car position packets run a little later than the game packets when you watch an MPR. You can see the same effect in an MPR at the start of a race - the lights go green and the cars seem to just sit there for half a second or so before moving away.

[t1ger]

Thank you both for the clarification, and it confirms my suspicions that it is just the way it is and not a fault in my code (which I always like to prove :) ).

Thanks
Tim

[paparazzi]

Hey, there. Why not damage packages? I see this fabulous idea from bluejudas in old thread of this and it would be nice to see into insim too at least more useful utilities can be made with this on.

http://www.lfsforum.net/showthread.php?p=1223721

hope your attention,
greats papa

[DarkKostas]

Recently i started playing with InSim again and i remembered 2 things i wanted. The 1st one is this one which was posted long time ago.
As the previous topic about insim changes is locked ill post it here.
I would like a small change to this

struct IS_AXI // AutoX Info
{
byte Size; // 40
byte Type; // ISP_AXI
byte ReqI; // 0 unless this is a reply to an TINY_AXI request
byte Zero;

byte AXStart; // autocross start position
byte NumCP; // number of checkpoints
word NumO; // number of objects

char LName[32]; // the name of the layout last loaded (if loaded locally)
};


Is it possible to make it so when a remote player loads a layout, we can get the name of it? Because i want to store layout name and use it.

The 2nd is to make PLC packet work as /cars command. What i mean is that when someone for example is driving with XFG(and its the only available car on server), when you type /cars UF1, he will get spectated AND when he tries to join in the race, he goes to the car selection menu to change car.

These are not happening with PLC packet. If you send PLC packet and remove all cars available he 1)wont get spectated, 2)when he goes to pit, he is still able to join the race with same car(he should go to the car selection menu).

A good and simple use of this "fix" use to make a !spec_all command which will send a PLC packet with value 0 and one more with value 0xffffffff.

Even if it there wont be any new LFS version soon, at least please implement it for the future version.

kthxbye
Dark_Kostas