Bot Contest

W00t! (Microsoft says that means We Own the Other Team... Pfft, like they would know.)

I think those people debating the fairness of the questions are missing the point. All questions are fair. A person in a conversation with another person can ask any question they want. They may not get an answer but they can ask it nevertheless. So should this hold true with bots especially if our ultimate goal is to simulate a real person. Now in a contest environment we have to set limits because we are looking to announce a winner. If we asked 10 extremely difficult questions and no bot was able to answer them what have we accomplished? Likewise if we ask 10 easy questions and everybody gets them right we are in the same boat. So the questions were a mix of both...easy and hard.

The only question of unfairness with any question would be if we didn't use the same questions for every bot but as you know we did just that.

I always find it interesting that no bot in any contest that we ever had has score a perfect score however if you take the collective responses for all the bots participating we have come extremely close. That is always encouraging to me to continue to work on my bot because I can see plain as day my bot could have done better. I don't think there was a question asked that couldn't be answered. God Loiuse answered the Benji Adams questions as well as the day of the week. Correct me if I'm wrong but even so collectively the PF bots did extremely well.

Anyway I have read through all the transcripts and no one bot really stands out above all the rest. I think it is going to be extremely close between a lot of bots. With that said and having so many bots participating I think it's only fair to award a silver and a bronze medal as well to the second and third place finisher. These are nice medals and I'm sure you will be proud to receive them. I hope we can do a similar contest next year just between the PF bots. I will gladly be a part of it as long as it's done in conjunction with the CBC.
Maybe we can over time turn this into something big with cash awards. I like to think big

Best,
Wendell

Professor, thank you so much for all the hard work you have done here. Too bad your own bot was so mean to you.

Wendell, thank you for putting on this contest.

I'm having a great time and am pleasantly surprised at how well some of the newer bots did in their conversation with the judges.

Wendell: thank you for the time and effort put into the contest at no recompense- that's mighty good of you. Just to add my two cents inre to fairness: I thought that all the q's were fair enough, the closest to unfair was the "Benji Adams" q, but it would have been unfair if you'd asked "Who is Wilhelm Riech" or someone equally as obscure whereas Benji is a presence pretty well felt here to anyone seriously programming a bot and thus a key for his name isnt an unlikely concept; though, I think most botmasters use a "who is X" approach to avoid keying dozens or hundreds of names.

Also I second the sentiments of Butterfly Dream inre to appriciation of the Prof's work here- you make it all possible Benji, thank you.

Wendell,
You've done a great job with all aspects of the contest, including the questions, which all participants have agreed were excellent and completely fair (unless the one real critic has a PF bot he's not revealing). I'm not sure why someone who doesn't even have a bot in the contest would be so passionately insulting ...
In any case, as you said, the contest is most fun for botmasters that enjoy the opportunity to challenge themselves, and more clearly see how they might improve. In this way, your donated time has truly been a gift to us. Anyone who feels otherwise has the free will to participate or not in 2006. I, for one, am looking forward to it.

The main thrust of the critique of the questions seems to be that they were too easy -- the fact that no one bot answered them all refutes this idea.

I can't wait to see how the non-PF bots do.

They have different questions, remember. Similar, but not identical.

Hello, 2 judges have submitted scores so far so I have posted them to the site under the heading
Scores. Please note the table of bots is listed alphabetically not in order of highest score. You will have to do the math right now. Also note we are hoping to receive scores from 8 judges although I only have places for 5. I will add more spots if needed.

I think overall the score were pretty consistent between these 2 judges especially in the better bots which is a good sign.

Wendell

Heh, I'll agree that is probably a good sign.

*does the math*
For the most knowledgeable, with 2 judges' scores, the top 10 are (unofficially):

Name:        Average Score:
--------------------------------------
God Louise        34
Aries            33.5
Immaculada        33.5
Osiris             33.5
Julie Tinkerbell        32
Oraknabo             31.5
Philippos            31
Steve Slacker         31
Suga Plum Fairy         30.5
Jake Thompson         30

[Edit: Leave it to HTML to not format tabs properly.]

Wow...one judge gave Fizzy 25 and the other one 20. I wouldn't have given him more than 18.

On the other hand, my score for God Louise is 34, exactly average for the two judges. So maybe I'm just harder on my own bot because I know when he's bluffing.

I did the same thing. I thought my bot's score, although one of the lowest out there, was pretty consistent with the scoring rules. I thought I had a 17.

Now if a couple of keyphrases that should've worked (and normally do work) had hit, I would've had something in the mid 20s, which I could've lived with a whole lot easier...