Hello, i put a message from Scawen here to begin discuss about insim button issue.

Quote from Scawen :

Hi. All I can say is I've seen this thread. I can't explain it...

It's not clear from this description if it's a new problem or not, Mischa NED says it's the same with old versions. And nothing has changed since the buttons were first implemented, so I would be surprised if it is a new problem.

If there was an very clear way to reproduce it, every time then I could look into the problem at some point. Though I am very busy right now on a lot of things so... no rush with it.

Maybe you just shouldn't send too many buttons to too many users at once. Are you making sure that every text button packet is of the minimum size needed to send the text in that button?

Maybe some bandwidth calculations would help. Exactly what bandwidth of button packets is being sent to how many users all at once, to cause the bug? Maybe the upload bandwidth is being exceeded.

Could be a good idea to discuss it with the other programmers in the programmer section?

Hello,

I try to explain more clearly the insim button problem.

When button are implemented in LFS, i start to use it but only for 7 or 8 button at time.
Recently, i upgrade LFSLapper and add ability to display the top classification with button and simple scripting language who can display button. To have beautifull presentation i use 5 button for one line of this top list and 19 lines. I display 95 buttons in one time.

I have tried in many configuration.
When lapper is

Connected on dedi LFS on windows.

if i connect to this server with LFS and i'am alone, the command to display top list display 46 buttons, stop for 1 second, then display rest off button.

if i connect to this server with LFS and i'am not alone, when i want to display top list. +/- 46 buttons are displayed, then the other button are displayed every 1 second. After +/- 10 second, i am disconnected to server.

Connected on not dedi LFS under windows and i start a new game

if i request top in the LFS host game, all buttons are displayed

if a player connect to my server and request top list. 46 buttons are displayed, then the other button are displayed every 1 second. After +/- 10 second, he is disconnected to server.

Connected on dedi LFS but under linux OS with wine

All work fine, button are displayed quickly for every player.

I don't know if this issue come with patch Z, because i never use button before last test patch and when i tried this i do it on dedi located on linux server.
In fact i have already see this issue on server on patch Y, but at this time i have put this issue on provider limitation, because on my server on Linux this issue don't appear ( Dedicated server with 100Mb of bandwith and debian linux ).
i put trace on my Lapper app, and button are sended in one shot, no TCP limitation.
I try also to connect to insim in UDP, same thing , i try to change TCP buffer on windows XP and some register value, no change. I have a good bandwith on my windows server. I'am disapointed.

It's easy to reproduce it, run a dedi LFS on windows server, connect Lapper and connect to this server with LFS and type !top. If you are not alone, you are disconnected after +/- 10 seconds.

Scawen, when you have time, i can config a server to make test.

For now last version of Lapper is in test and no much guys use it because professionnal hoster have not updated Lapper with this last version. Actually there are +/- 130 lapper who run with stable version and +/- 20 with new version ( this numbers are only server voluntary registered, i have no idea of the real total number of server running Lapper http://www.frh-team.net/reglapper/ ). But i can't release it with this issue or i do remove top list displayed with button.

Thank's for your answer .

Gai-Luron

Question, can you save the dedicated host replay file and try to recreate it with that?
Could you provide the reply file of your own clients disconnection.
The last thing I would ask, would be a packed sniffed stream coming from the Ethernet port to see all of the raw packets going across the network from and two the port of LFS.

What I am thinking:

• Engine Bug - Should be recreatable in the mpr file as well, or should get some kind of error message in play back (Best we can hope for really).
• Network Issue - Should cure it's self when using the loopback device. (By that I mean connecting to the client and host on the same computer ... 127.0.0.1)
• Network Code Error - on the Windows API level. (Unlikly, but could happen.)

Let's throw some darts at the board and see what we hit!

Quote from Dygear :

Question, can you save the dedicated host replay file and try to recreate it with that?
Could you provide the reply file of your own clients disconnection.

Yes i can send you the replay, but button will be no displayed because insim application send button, you only can view the disconnection.

Quote :

The last thing I would ask, would be a packed sniffed stream coming from the Ethernet port to see all of the raw packets going across the network from and two the port of LFS.

I have neve user sniffer, and i don't have time for now, to learn how to work sniffer and analyse packet. But if you want to test with me, why not?

Quote :

• Engine Bug - Should be recreatable in the mpr file as well, or should get some kind of error message in play back (Best we can hope for really)
  

Without insim app connected, you only see the disconnection

Quote :

• Network Issue - Should cure it's self when using the loopback device. (By that I mean connecting to the client and host on the same computer ... 127.0.0.1)
  

Maybe, but this will occur also when i'am connecting to other computer in the world using lapper on windows system.

Quote :

• Network Code Error - on the Windows API level. (Unlikly, but could happen.)
  

I don't know how to search in this way

Gai-Luron

EDIT: here two replay:
http://ks37764.kimsufi.com/replay/insim/replay_button1.mpr
http://ks37764.kimsufi.com/replay/insim/replay_button2.mpr

EDIT 2 :
I have a private message from a player who tell me that issue is also present on CTRA ( not lapper insim app!! )

Quote from Dustin Dawes :

Hi Gai-Luron:

I'm PMing you...

This issue also happens with the CTRA servers, when the results screen gets shown up (with points), many people lag out of the server because they get flooded with button packets. To counter this, we put in a way to disable, or have them limited by time (rather than instantly, deliver them every 10 seconds, as to avoid spamming clients with packets).

Hope this maybe clears it up, that it is partially a code error, by spamming too many buttons. However it is odd (although expected ) that it works while using WINE in Linux.

Dustin (dawesdust_12)

The only difference is that, i am not spamming all player, but this happen also when i show only one toplist for one player

My results, using dedicated server, amount of buttons received per second successfully (any more makes the server drop my connection):

alone on localhost (windows xp), 7000 buttons per second.

alone on very fast remote server (windows server 2003), 300 buttons per second.

Hello,

Here videos on what happen on Windows and linux server.

http://ks37764.kimsufi.com/replay/insim/videotestbutton.zip

Gai-Luron

PS: My server is on Windows XP machine, not server 2003. Lapper send all button to LFS dedi without a TCP error. Maybe i do an error and CTRA server too, but where, i don't know . We can do a test with Lapper ( lapper on my computer) on your windows 2003 server and on your windows XP server.

Dose the Server's MPR file not save InSim packets also? I think it would seeing as this is the part of LFS that really has the handle the InSim packets. In that case, there is always the packet sniffer way:

__________________

We should really start with the most reliable connection information first then work our way down. Seeing how InSim is a packet based API, for me the most reliable would be:

Test Setup:

The LFS Server, InSim Application (That causes the disconnection of the LFS Client), and the LFS Client will be on the same computer, connected though the LoopBack device. A Packet Sniffer (WireShark - Free, Open Source, Works on All Major Platforms.) will then capture all data going into and out of the LoopBack device pertaining to the LFS and InSim packets. An MPR file will be saved on both the LFS Server and LFS Client.

If you then hand all of this data over to the devs I really don't see why they would not be able to figure it out. It provides the most comprehensive set of data with out having the 'compromise the integrity' of handing out the binary or source code of your InSim application it's self (Not that I'm suggesting that the devs would some how mishandle the information you gave them, but in some cases the less eyes the better, I'm just doing it this way to avoid any possible legal hassles). Due to the nature of the InSim 'API', they don't need your InSim application running as long as they have the packets it generates and what time they are sent. It's the most correct why I can think of to get the whole picture to the devs about what's going on here with out giving them the environment directly, although that would be a much better way of handling this . . .

__________________

I'd like to directly point out that I've tiled this post "The most obtuse way to handle a bug report." While that is true it should also provide a much better picture as to what is REALLY going on.

Hello,

LFSLapper is freeware on GNU license and released with source . It's not top secret App., i think the best way to keep alive an App, it's release source. Most app are dead when insim 4 was released, like GhostCar, LFSStat, PitSpotter. LFSLapper is not Dead because initial dev ( Monkster ) have released source and i think is a good thing. LFSLapper for me it's for fun, not to make money or glory

Gai-Luron

I have to say, I had this very same issue with InSim app's I made. I really couldn't figure it out, then I made my very own InSim code, with dynamic sized IS_BTN packets. That seemed to solve all my issues.

Quote from filur :

My results, using dedicated server, amount of buttons received per second successfully (any more makes the server drop my connection):

alone on localhost (windows xp), 7000 buttons per second.

alone on very fast remote server (windows server 2003), 300 buttons per second.

A little question, how can you send 300 or 7000 buttons, LFS limit this to 255 simultaneously for one player?

Quote from mcgas001 :

I have to say, I had this very same issue with InSim app's I made. I really couldn't figure it out, then I made my very own InSim code, with dynamic sized IS_BTN packets. That seemed to solve all my issues.

I use dynamic length button size. You can see here, button size ( multiple of 4 ), clickID and the corresponding text for top list ( 96 buttons )

Gai-Luron

Quote from Gai-Luron :

how can you send 300 or 7000 buttons

I reuse the id's, overwriting older ones.

I encountered this problem with one my apps: "NextRace".
NextRace can display almost 200 buttons at once like this: http://img88.imageshack.us/my. ... racker310120081806ta5.jpg
However, I experienced the problem myself only once...and only a few users said they had trouble using it....most people could see it just fine.
NextRace was always running on a Windows machine.

I've tried to make a test a few min ago: I've modified one my apps so that it sends thousands of Insim buttons. Using it on "single player", it causes no problem. However, if I start a dedi in local and connect to it, my LFS gets disconnected after the flood of insim buttons.
My LFS just says "Lost connection to host", however, my app registered this message too:
21:59:43,125 [127.0.0.1:17464] received this msg (TCP ERROR : WOULDBLOCK) by ConnId: 0

Looking up in the forum, I found this post:
http://www.lfsforum.net/showth ... ght=WOULDBLOCK#post813975

I think Victor's answer solves the mistery: sending too many buttons at once can saturate the network buffer, thus when the LFS host tries to send some more information it fails to do so and declares the connection dead.
I suppose the network buffer size depends on the machine config, so that would explain why your Linux machine has no problem, while your windows server fails.

Victor also provides the solution:

Quote from Victor :

It's always a good idea to prevent instant-spamming of messages and to create a message sending mechanism for your insim application that waits a little bit in between sending messages, so that it will never flood a host or client with messages.

I've tried to put a simple pause of 1 millisecond between one button and another and it's working: my LFS has been receiving thousands of buttons in the last 3 minutes and it's currently still connected and receiving more buttons...

Starblue

Hello,

I try this tommorrow. I hope this will be solve my problem.

but what i don't understand is if my insim app send request to host to query display button on LFS Client. I can put delay between buttons but for only beetween my insim app and host. I have no control under flood from host to client.

Thank's for your answer

Gai-Luron

PS: if this solution work, i try to create thread to enqueue data and send it with interval beetween request.


EDIT: Work fine with 1ms delay between 2 buttons on windows system. Strange solution. If Scawen can do anything in LFS to avoid to use this solution

Quote from Gai-Luron :

what i don't understand is if my insim app send request to host to query display button on LFS Client. I can put delay between buttons but for only beetween my insim app and host. I have no control under flood from host to client.

I suppose the host sends data to the client as fast he can...so if you flood the host, the host will flood the client. If you wait 1 ms, you don't flood the host and the host doesn't flood the client
That's the control you have on the "host to client flood"

Starblue

Quote from Starblue :

I suppose the host sends data to the client as fast he can...so if you flood the host, the host will flood the client. If you wait 1 ms, you don't flood the host and the host doesn't flood the client
That's the control you have on the "host to client flood"

Starblue

Ok.. that sounds lol.. I was doing the opposite to try to solve this problem.

What did I do? I saw there was a "flood" problem.. what is the problem with flooding? Well LOTTTTSS of small packets sent behind eachother very quickly. Ye.. problem.. your solution, a delay prevents this (great news btw )

I tried to solve this flood before, sending all packages at once (just concat all messages and send over TCP at once)... this way I prevent TCP flooding between my app and LFS host.

To my susprise there were still problems.. so I thought it was a problem between host and client. Your solution, sending packages with a small delay, causes the host to send fewer packages at once...

But then... shouldn't the host send them "queued" to the clients? So shouldn't this problem be solved in LFS dedi?

I want InSim to be as fast as possible.. especially.. cause it's localhost you're talking to...

A 1ms delay for 100 buttons means 0.1 sec / connection. Getting 10 requests for 100 buttons makes total delay 1 whole second..

And.. I mentioned before.. that my experience was.. that the delay occures especially when people with higher ping are connected. People with good ping, no problem for anyone... Only 1 person connected with bad ping, button problems for all... is this possible?

Another thing.. while reseaching this problem I found out Microsoft did some security updates regarding TCP flooding.. I don't remember exactly the topics.. but it had to do with message windows and number of connections... I never experienced this button problem on my very very old laptop without servicepacks... but it's a pity.. that laptop died...

I've seen this problem too in my sumo app. The chart consists of 70-80-90-something buttons, and I used to get disconnected when sending the command to display it. The chart started to show, and after displaying half of the buttons it stopped. Then the buttons was displayed one by one with a long delay between before I ususally got disconnected... I never heard anyone else had the problem, it only seemed to happen to me

But after I switched server from Windows Server 2003 to Linux Debian the problem seems to be gone (both were dedi servers running on rented VPS'). Never got the chance to investigate the problem. I don't have any delay what so ever in my app, just pushing button packets out on the wire.

Dunno if this info was usefull, just wanted to let you know

This is not really a bug with LFS. If you flooded information to a web browser, would it not crash in the same way as LFS? Therefore this is basically just sending a DOS attack using insim. If you make a 1ms delay between buttons, it will only increase the delay by 90ms if you are rendering 90 buttons. That's not going to be very noticeable by the user. I don't see how LFS can be changed to prevent insim flooding.

So the issue is reusing Button ID's? Is that on a per client basis?

So, we would have to make an array of arrays for each client giving you available Button ID's?

$clBid[$UID] = array( ... Available Button IDs ... );

Quote from Dygear :

So the issue is reusing Button ID's?

No, the issue is asking LFS to send too much data too fast to remote clients.

Quote from filur :

No, the issue is asking LFS to send too much data too fast to remote clients.

But what is different on button packets compared to all those other packets? Check how many packets are sent in total... the CompCar packets for example?! They keep coming in, no problem... but "some" buttons lag? What's the diff?

My idea of preventing lag / flooding is to sent multiple packets at once. I tried this as solution months ago already.. Combine all 100 buttons for example in 1 packet and send to host... this works of course... and there is no flooding..

But to bad... I think LFS will send them as seperate packets anyway... so flooding still occurs.

Maybe Scawen can tell what the difference is between sending buttons (from host to client, not insim) compared to other packets. If sending 100 packets in just a very very short period... how can this make buttons come in 1 / second after some buttons... why doesn't this happen to position packets? Or is the delay there long enough already?

And one thing I'm still wondering... If it is no problem on Linux... what is the difference between Linux and Windows then? Can it have to do with TCP settings? Window sizes? Fragmentation? I have no idea yet, but.. shouldn't we search it there somewhere? I think it's only worth the time to find out if we're sure that it's not a nasty .NET thing... is everybody with the lag problems using .NET? Or are there other Windows based insim apps with same problems without using .NET?

Hello,

You are Right, the only person who can answer at all of our question is Scawen, because he can put debug info in his app.
For me button are sended to insim dedi with no issue on TCP and UDP.

Edit : a new info.
If every 30 buttons, i wait 10ms, button are displayed with no problem.
If i remove waiting, button after +/- 45 start to be displayed 1 per second. If i stop insim App. LFS client continue displaying button for a while. I presume it's LFS Dedi who flood LFS Client. But why when i put this waiting value each 30 buttons, this work fine

Scawen if you have time

Gai-Luron

Quote from Gai-Luron :

...why when i put this waiting value each 30 buttons, this work fine...

And why *exactly* 1 second delay between those buttons something just "sleeps" 1000ms to prevent flooding... what/where?

ctrl+f: sleep (1000)

We seem to be running on the assumption that the Button Packets are flooding the client, by sending to many to fast. So we still want to keep the quantity we are sending thus we send them slower to make the connection more maintainable.

Just an example to illustrate:

Still would love to know why Linux is not effected. Did Linus fix something that Gates missed?

Quote from Mischa NED :

And why *exactly* 1 second delay between those buttons something just "sleeps" 1000ms to prevent flooding... what/where?

ctrl+f: sleep (1000)

No, you have to write into your inSim app a mechanism to buffer the buttons you want to send and send them at another time in the future when the appropriate time delay has passed.

We have this issue and it has been compounded by what seems to be the new Z patch. I have to talk to our host about increasing our network buffer or see if its because of Dot NET 2. I've switched it over to Dot Net3.5, but I've been away and not yet seen the result of this.

Keep up the good work Gai-Luron with your testing.

Quote from JasonJ :

No, you have to write into your inSim app a mechanism to buffer the buttons you want to send and send them at another time in the future when the appropriate time delay has passed.

I know about the buffered sending... I tried and it works. What i meant... without buffering... it's just weird that every packet has exactly a 1 second delay... so somewhere must be a delay of 1 second... in TCP settings.. .NET... LFS.. or wherever... some program is "sleeping" 1 second because of the flooding..

I tried .NET 3.5 too and for me it made no difference.. I tested on Windows 2003 server, Windows XP SP2 and SP3 and same thing...