Player Rating System (PLR) Beta

[F.Rizzo]

Hi Everyone

I am currently developing a Player Rating System for LFS and was hoping that I could get some feedback from you guys.

You can see details and keep up to date on my site (http://pos.lfs.marcomark.net)

The PLR allows every licensed user to rate another user (only once, but the rating can be updated by re-rating that player).

The system keeps track of all ratings for each player, across all participating servers.

Server admins can then restrict server access based on customisable rules. For example:

Allow newbies (players with less than X ratings)
Players with more than X ratings must have a rating of at least X

You can have a go at the rating system on my test server "Marco Mark's Dev Server". Commands are:

*rate (rate another player)
*rating (view your rating)

The add-on will of course be made available to server admins once a stable version has been finalised.

Any suggestions/comments are very much appreciated.
Cheers,
Marco

[micha1980de]

how do you plan to distinguish from "ebay"'s failed user rating system?
as far as i understood, there's nothing to back a bad rating towards anyone... (a replay or a second whitness etc.)
if i got your point completely wrong, then sorry in advance...

my 2 cents

[Tweaker]

Sounds cool, but I'd rather not be 'tagged' with a rating from other people, as well as have 'requirements' for ratings on servers. Because the fact of the matter is, this is all rated based on opinionated users, and face it, not everyone out there are experts in judging people's actions in racing. There are always many different perspectives when it comes to rating a driver who is (or isn't) really a "5-star driver", or what ever you may call it.

If you are looking to rate someone simply with a 'good' and 'bad' rating system, it may be misused in ways that could be impartial I think. There are many drivers out there who classify good racing in many different ways. For example, uber clean racing where no drivers EVER touch each other and race like old posh gentleman every corner... they'd call that a good scrap. However, others find a little door-to-door banging a good hard fight, as long as it is safe and unintentional.... while the latter group would say that it is dirty driving I bet.

Every driver also makes mistakes, and likely do not do things intentionally for the betterment of their racing. That said, a rating system could not be fair in this manner too, because people may not be fair in deciding who was at fault.

You can only really base a ratings system on obvious and blatant poor racing behavior that is not tolerated on virtually any server we know of in LFS (eg. Wrecking & rough driving, cheating, bad language, etc), . THAT is one thing that LFS players do look out for in servers, because they know licensed servers should have cleaner racing than anywhere else. Although, like I said, there are players out there with pitchforks that really would just love to give someone a negative rating based on what they believe, or simply prior race encounters, or just have a grudge with them.

Also, for ratings to be truthful and fair, you'd need to do voting and reviews from every driver that raced he or she. Much like a Wreckers Barricade sort of deal, with proper replay review of an incident. Else the ratings will just be like a lightning round, and not thought through carefully.

Sure... it's rather complex the way I am making it up to be... but I can only imagine users bumping up someone's rating for good measure, when in fact they are not a good racer at all.... and vice versa for a user's negative score. The last thing I want is to feel unwelcome in a server because of a rating that is most likely unreasonable... and be hounded by so-called-elitists who 'have' a higher rating than I do.

I understand a reason for having a rating system, such that you want people to gain a better score based on improving their appropriate behavior when they are on a server... it just seems nonsensical to me, especially in the online gaming world where arguments flare up, and ratings are not always accurate.

[F.Rizzo]

That could indeed be an issue. There are a couple of ideas I was toying with, one of which would be that a player is only allowed to rate another player if they both participated in the previous race, and raced at least 75% of the distance (the rating player at least).

That way people can't just join a server and randomly rate people.

Also, rating is completely anon. and there is no "I'll leave a bad rating if you do" scenario.

the *rate command was only really ever intended to be used during testing.

[F.Rizzo]

Let me expand on this based on Tweaker's reply;

First of all Thanks for your feedback. As I said in my above post, the idea would be that after a race, you may rate one player from that race. The idea being that if you had a great race against another player you can rate that player with a good rating. If someone really screwed up your race you can rate that player, or if you had a regular race with a coupld of small incidents, you may choose to rate one of them players with an avergae rating. However you can only every rate the 1 player after any race.

I also think there could be certain server rules that filter out spammers. For example a player with a negative rating, who only every gives negative ratings could be filtered out as "inaccurate". There could be any number of rule sets to filter out a lot of the "spam" ratings.

[micha1980de]

you cannot have a clean objective unbiased rating system, if the raters are full of all sorts of emotions during the heated race...
And even after a race, assuming the race distance was long enough, the rating decision would still be emotional influenced...

i'd say it's not possible to implement that, and even if the decision was made by the machine, based on statistic data accumulated during the race, some racers would bring up arguments about unfair "rating" algorithms, like seen on the CTRA servers and their decision finding process concerning their point system...

regards

[F.Rizzo]

you cannot have a clean objective unbiased rating system, if the raters are full of all sorts of emotions during the heated race...
And even after a race, assuming the race distance was long enough, the rating decision would still be emotional influenced...

That is very true. The decision most certainly would be emotionally influenced, which I don't really see as a problem.

The whole idea is not nesseseraly to rate the player as "this driver is super fast" but rather "I really enjoyed racing with this person". Because after all that is what it is all about on public race servers.

Now if 50 out of 50 people give you a rating of -5, chances are that it will be reasonable accurate and the server setting could easily be set to only consider rating for people with at least 50 ratings.

[Woz]

If you are tracking server results then I have the maths for a chess like rating system for you. All players start with a rating of 1500. Every race that gets adjusted by where you place in the race compare to where you were expecteed to place.

You get less or even no rating increase for beating somone far worse than you etc. The further you get away from 1500 (lower is below average, higer is above average) the harder it is to get further away.

PM me if you want all the details. I was thinking of setting up the system but have too much on at the mo to take that on as well.

[birder]

If you are tracking server results then I have the maths for a chess like rating system for you. All players start with a rating of 1500. Every race that gets adjusted by where you place in the race compare to where you were expecteed to place.

You get less or even no rating increase for beating somone far worse than you etc. The further you get away from 1500 (lower is below average, higer is above average) the harder it is to get further away.

PM me if you want all the details. I was thinking of setting up the system but have too much on at the mo to take that on as well.

Hi

At Conedodgers we have a simple licensing system, basically a list, if you are on the list you can race. We make the list from the PB file from LFSLapper

For months i have wanted to do a "skill" level based system that would work over multiple servers. Gui-lurion who writes Lapper has my request but is not doing anything on lapper at the moment

Any chance we could have your chess rating system math in the hope he can use it

or can it be incorperated into a license system

thanks

[F.Rizzo]

Thanks for the comments guys.

A "skill" based rating system was indeed one of my next projects, however this isn't meant to be quite the same as that. This system is meant to rate a user based on their "racing enjoyment", not their skill.

I think a skill based system should be on a server to server basis and not on a global scale (although I guess there could be both).

My scoring calculations were along them lines as well, where you take the average skill of participating racers and points get awarded based on your skill/position compared to the average score. However a pre-tested working scoring mechanism like the chess one sound slike a much better idea!

I would be happy to work with you guys on something like this. PM me if you want to get something on the way :thumb:

[F.Rizzo]

Back on topic I think I will change the name to "Player Opinion System" as I think there is some confusion when using the term "Rating".

[Dajmin]

TBH I think it's a good idea. If you had a good race against someone you vote them up, if they were a moron you vote them down. And since people can only vote once, it means you can't spam someone out unless you can convince a whole load of people to do it as well. It should all balance out in the end.

But then it does sound a lot like the idea I made about a similar 'reputation' system months ago :)

[micha1980de]

and in the end it's like on ebay where the bad guys give themselfes "good" votes...
i really don't want to bash your idea, but i'm having serious trouble about the honesty of the people...
we'll see how it develops and wish you all good things.

regards

[Hollywood]

Besides that, it seems like just another way to fragment a small community... :(

and in the end it's like on ebay where the bad guys give themselfes "good" votes...
i really don't want to bash your idea, but i'm having serious trouble about the honesty of the people...
we'll see how it develops and wish you all good things.

regards

[F.Rizzo]

Dajmin: Apologies if the idea was previously suggested and thank you for your positive comments.

Micha: That is a very valid point, however if "toublemakers" started doing this I think their "Rating/Reputation" would soon go down-hill when they enter serious raceing servers. Because every user can only rate each other one I think this would not be such a big issue.

Hollywood: I can totally see where you are coming from and that is in no way what I am trying to achieve! It is purely another way of offering serious race servers to elimiate trouble makers. It's not like this will be used on every server.

********************************

I wanted to create a system like this to help improve the level of racing on public race servers. I realise that Wreckers Barricade already does this to some extend and this is not to compete with WB or take away their system - it's just another added level of protection. The main problem with Replay based systems is that someone needs to watch replay, and it's that person's sole opinion that decides if the accused should be marked as a trouble maker or not.

[Hyperactive]

One way of preventing people from giving bad ratings about others and the effect of that might be to give bigger multiplier to the votes from those players who have more positive opinions, who have a bigger positive stat. Kinda like someone who has just started LFS and votes for someone to be a bad racer has quite a small effect on that person's stat. At the same time someone who is annoying sob wrecker (like rooble :p) casts a similar vote but because his own stat is -1000000M points his vote has very very small effect on that other one's stat.

Kinda like a vote from me is worth one billion but rooble's vote has no effect at all since he is an annoying sob wrecker.

The biggest and hardest thing would be to get people to rate others... it needs to be very easy and subtle to do but still visible enough so that you remember to do it. More votes means less cheating with friends' rating friends.

One question is that is just one stat enough? You could have few more, like helpfulness or (can't think of anything else)..

(rooble is not really that annoying :p)

[Hollywood]

A good goal, i.e. improving racing. But its something thats got to be carefully done so as not to run-off well meaning, but not completely competent folks, as opposed to the serious malcontents.

Any sort of system akin to this (even WB, as you pointed out the biggest flaw with it) is the need for oversight, so that someone doesn't get hamstringed by some maliscious folks (i.e. a group getting together going around servers and giving black marks to everyone and anyone, etc.)

In a way, the CTRA license system is very similiar, it just does things on "performance" as opposed to "opinion" and only applies to their couple of servers which they provide oversite for.

So yeah, if this is mainly targeted at server admins that have multiple servers that want something more than the WB, this could work because then it'd be up to the server admin to act as oversite to his servers, which he does already, but gives him a way of allowing people to "police" themselves, so to speak.


I wanted to create a system like this to help improve the level of racing on public race servers. I realise that Wreckers Barricade already does this to some extend and this is not to compete with WB or take away their system - it's just another added level of protection. The main problem with Replay based systems is that someone needs to watch replay, and it's that person's sole opinion that decides if the accused should be marked as a trouble maker or not.

[F.Rizzo]

Hyperactive: That is something I also considered and could still be implemented. I could calculate a users reputation by vote+voter's rating rather than just the straight rating. This would also keep it dynamic and update as individual users' rating changes. This might actually be an easier way of dealing with "bad voters" rather than running server rules to filter them out. I will certainly look into this as an option.

I think one vote/rating is enough as any more would imo make it too complicated/confusing for users.

For those that have had a look at the system on my server I am sure you have noticed that I tried to make it subtle so it isn't annoying but obvious enough to get it noticed.

Hollywood: I have now added an option to either use a "Global" rating that is shared between all participating servers, or the server admin can choose to keep ratings for their own server only.

[Dajmin]

I think maybe multiple ratings would help, and might be something to aim for once the system is fully established.

Would only need a couple of things though.
Fairness: How well they obeyed rules and how well they behaved in close-quarters races (ie no bashing, bumping, etc.)
Attitude: How friendly they were if something went wrong. Stuff like not shouting abuse or immediately pitting just because you passed them.

You could put speed in as well, but it'd need to have a much smaller modifier since you can be slow but still fair and it wouldn't be any use to have them marked down for that.

Would that cover everything? Overall rating could be could be called Racecraft, since that's basically what you're looking to promote.

[Woz]

F.Rizzo, sorry for the delay in sending you the details.

I noticed that someone else requested so I will just post some details here when I get the chance. What I currently have is some test maths in an open officer spreadsheet, including macro code so hard to convert to xls.

The basics are that everyone starts with a rating of 1500. This is treated as average. At the end of the race you have your rating adjusted by where you place and where you are expected to place.

You are expected to beat lower rated people and loose to higher. The bigger the gap the more the change in your rating. Also the further your score is away from the average the hard it is to get further away from average.

The basics all come from here though http://en.wikipedia.org/wiki/Elo_rating_system

[Woz]

The forum does not allow odf based files so here is a rundown....

You start every person at 1500. At the end of each race you perform the same maths against every player using the PRE RACE ratings.

GetRatingK() is used to get the scaling factor. The worse you are the more you rating moves by, this makes it easier to climb until you get into the higher ratings.

GetExpectedResult(a, b) works out if you were expected to beat someone and how easy.

Function GetExpectedResult(a, b)
GetNewRating = 1 / (1 + 10 ^ ((b - a) / 400))
End Function

Function GetRatingK(a)
If a < 1800 Then
GetRattingK = 40
ElseIf a < 2200 Then
GetRattingK = 20
Else
GetRattingK = 10
End If
End Function

So the maths that are required for each person in the race are (Pseudo code)

if (p1 ahead of p2 in race)
aheadRating = p1
behindRating = p2
else
aheadRating = p2
behindRating = p1
end
if (p1Rating > p2Rating)
expectedResult = 1
else
expectedResult = 0
end
p1RatingChange = GetRatingK(p1Rating) *
(GetNewRating(aheadRating, behindRating) - expectedResult)
newP1Rating = sum(p1RatingChanges)

so you perform that for each person in the race against everyone else in the race to get all the delta values. For added spice I was planning on making a DNF a loss against everyone else so you would lose a lot of rating for a DNF

[F.Rizzo]

Cheers Woz - I will have a play with this asap. :thumb:

I want to finalise the POS before getting started on this but it is pretty much approaching public beta so not far to go. :)

[Hyperactive]

Basically the ELO rating wouldn't work in a racing sim as such. Unlike in fps games in LFS people don't have equal starting grounds to the race. Grid position is one huge factor. In racing it takes very small error to ruin a race while in fps games you need to basically shoot quite a few team member to make it really to have an effect. Another thing is that there are always people who will disconnect from a race if they see that finishing would hurt their rating. There are also people who will do anything to win.

It will also need to look at the car speed differences, it is not as straight forward as it sounds first. If A drives FZ50 and B drives LX6 on SO4R the LX6 the car will have huge effect on the points. If the points are given without looking at the cars people will choose the winning car even more than they do now. Also the experience on that chosen car is a big factor. More experience = faster, just like with the combos, ASnat nut is always fast in fzr on asnat but useless on ky3r in lx4.

And to be honest, a rating system is just a ladder in the end if you make it completely based on speed and finishing position :)

I admit that the automated system would defenately solve the most biggest issue with the system - to get people to vote :)

[Dajmin]

That's why I've always thought a rating/reputation system would work best. If nobody votes, you remain neutral. If you're a clean racer you have pretty much nothing to worry about, but if you repeatedly do moronic things then you risk eventually not being able to join any servers running the application.

Marking people down just because they aren't very quick is crap, because then they'll never learn to drive faster. And marking people up just because they're fast is crap, because they might have got there by knocking everyone else off the track. Without an application to detect all of this (and my suggestions for wreck detection have been knocked down in the past) the most reliable way is to use people's opinions.
And it can't be all that bad since Microsoft use a basic version on Xbox Live :p

[Hollywood]

And you also run the risk of people voting down people just because.

That's why I've always thought a rating/reputation system would work best. If nobody votes, you remain neutral. If you're a clean racer you have pretty much nothing to worry about, but if you repeatedly do moronic things then you risk eventually not being able to join any servers running the application.

[Woz]

Basically the ELO rating wouldn't work in a racing sim as such. Unlike in fps games in LFS people don't have equal starting grounds to the race. Grid position is one huge factor. In racing it takes very small error to ruin a race while in fps games you need to basically shoot quite a few team member to make it really to have an effect. Another thing is that there are always people who will disconnect from a race if they see that finishing would hurt their rating. There are also people who will do anything to win.:)

That one should be easy to take care of if you scale the expected result (The 1 or 0 value) based on grid position. So divide grid start position by grid size and scale the expected result by that. This then takes into account stat position.

It will also need to look at the car speed differences, it is not as straight forward as it sounds first. If A drives FZ50 and B drives LX6 on SO4R the LX6 the car will have huge effect on the points. If the points are given without looking at the cars people will choose the winning car even more than they do now. Also the experience on that chosen car is a big factor. More experience = faster, just like with the combos, ASnat nut is always fast in fzr on asnat but useless on ky3r in lx4.

The car one is easy again. It would make sense that you lock servers down to a set car class TBO, GTR etc if you run a system like this. If not just apply rating comps when more than 3 people in same class and just rate them on their own in the same way you can have a race with multiple class cars.

As for experience, that is the whole point of the ELO/rating system in the first place. If little Timmy is good in the LX6 and not in other cars they will not get a good rating over time unless they ONLY race that car. The better you are in a higher range of cars and more races you finish the higher you will rise. The rating is just rating you against overs, nothing more. Only those that are consistent rise to the top and those that have a high DNF ratio will sink like stones.

It could also expend so you hold a rating for each car per person and theirr overall ating is an average of that. This way you have to be good in EVERY car to have the best rating.

The ratings build over time. Its not just a single race in the same way a race is not about T1. Its a long haul to make you WANT to finish every race.

And to be honest, a rating system is just a ladder in the end if you make it completely based on speed and finishing position :)

I admit that the automated system would defenately solve the most biggest issue with the system - to get people to vote :)

Yep, ladder like but so is the ELO system. But its more subtle than that and most of all its automatic.

I like CTRA but if everyone raced enough they would all have highest rating. Here you get a bell cure that shows your true position compaed to others :)

[birder]

Hollywood: I can totally see where you are coming from and that is in no way what I am trying to achieve! It is purely another way of offering serious race servers to elimiate trouble makers. It's not like this will be used on every server.

********************************

I wanted to create a system like this to help improve the level of racing on public race servers. I realise that Wreckers Barricade already does this to some extend and this is not to compete with WB or take away their system - it's just another added level of protection. The main problem with Replay based systems is that someone needs to watch replay, and it's that person's sole opinion that decides if the accused should be marked as a trouble maker or not.

I see your vote system as being a great idea but as with ebay feedback you get people giving you negative feedback when you have done nothing wrong and a bunch of mates could gang up on someone, giving them negative points just for the fun of it

One way you could cure that is only allow trusted "listed" players to vote. In other words you have a file "birder,F.risso,someone_else" with all the people you trust in it and only they can vote.

or

Only allow players with points higher than X to vote

On ConeDodgers CD1 (Public) the racing is manic due to all the newbies, yet cross over to CD2 (Licensed) and the racing is great so such a system does work.

F.risso: I have PMed you about our system, if you can help (or anyone else - please)

[S14 DRIFT]

Personally I'd rather not be tagged, as some people have said.

Theres always a few people who get angry at you for no reason, or, for example, a small collision in a full field which means a small contact, but results in that person losing xx places, or having to retire means that you'll get a poor rating.

People often mis-understand the racing rules, and will just see it as purposeful wrecking.

People could get their freinds/other accounts to give you multiple poor ratings, affecting your ability to race on servers with your own freinds.

If its going to be done like this, I'd rather it be done by distance. Most wreckertype people have 0>1500 miles. From above around 5000miles is when people have learnt the cars, some of the tracks, got a general gist of the idea of racing rules, what to do and so on.

Something like that CTRA report system, where reports are stored (with replays) then INDEPENDANTLY viewed by a neutral party, upon which his/her rating of said player is then accepted or canceled.


s14.

[F.Rizzo]

Thanks for all the feedback.

Firstly, a single rating isn't going to mark you as a wrecker.

With regards to people spamming other driver's ratings, this is unlikely to happen as you can only rate each driver once, and you can only rate 1 player after every race. The idea being you rate the player that you felt had the most influence on your race. Good or bad.

I have released a very early Alpha version of the add-on. Feel free to give it a try and let me know what you think. Any feedback is appreciated. You can download it at http://pos.lfs.marcomark.net