Bot Contest
Here I'll be posting information on various Bot contests that challenge and test a Bot's AI and realism. Feel free to post comments and updates on contests, as well as announcements for new contests.
Posts 1,459 - 1,470 of 4,091
Posts 1,459 - 1,470 of 4,091
View Contest Winners in the Hall of Fame.
Shadyman, nobody entering a bot is a judge. There are seven of us including the Professor that makes up the committee but we are not judging anything. The list of judges are on the contest site under the credit section. Regarding the 15 questions several judges sent in potential questions. One judge was selected to pick the best 15. Once he had done that he asked those questions to Talk-Bot. I then asked those same questions to the bots of the owners helping me collect the responses for the 15 questions. Once we collected everybody I posted the results to the contest site. From there the judges are grading them.
Chris
Chris
To be fair, I think some off scores are to be expected and as Lunar22 said it is a very good idea to throw out the highest and lowest. All in all, there is no way you are going to please everyone in this.
That said, the only thing I have had issue with so far is that some of the Judges went deeper into conversation with some bots regardless of if the bot left conversation "open" or not which creates an atmosphere of more personalities coming through for certain bots and not others where the judge didn't deviate from the questions regardless of what the bot said. I think that will skew, in the very least, the Charachter/Personality portion.
That said, the only thing I have had issue with so far is that some of the Judges went deeper into conversation with some bots regardless of if the bot left conversation "open" or not which creates an atmosphere of more personalities coming through for certain bots and not others where the judge didn't deviate from the questions regardless of what the bot said. I think that will skew, in the very least, the Charachter/Personality portion.
I think that the only real problem is the number of judges. It would be impossible to get perfect judges in any contest, but that's why you have a large number, in order to average out the madness.
It's a pity that so few judges were available for this contest. I think that the reason is that every person that is a) interested in bots and b) knows about the contest has all ready sent in a bot, making him or herself ineligible.
While it might have been possible for some people to fairly judge everyone else's bot but their own, I don't think it would be possible for anyone of us here to do so, since we can all recognize forge bots. Likewise, you can generally recognize a bot that has been built from the Alice open source code.
That leaves a few people who have built up their own bot from scratch. If anyone were able to give unbiased oppinions on other people's bots, it would be them.
I personally think that the average scores would be much fairer if 10 contestants voting (not on their own bots), than with 3 non-contestants voting. Slight biases against, say, forge bots or Alice bots should cancel themselves out over the larger sample, and crazy scores should also become irrelevent. I also think that it would make the identity of the judges less of an issue. We would know that, say, the maker of Jabberwacky is who he says he is. As it is now, it is quite possible for a botmaster to be a judge and have noone know it.
It's a pity that so few judges were available for this contest. I think that the reason is that every person that is a) interested in bots and b) knows about the contest has all ready sent in a bot, making him or herself ineligible.
While it might have been possible for some people to fairly judge everyone else's bot but their own, I don't think it would be possible for anyone of us here to do so, since we can all recognize forge bots. Likewise, you can generally recognize a bot that has been built from the Alice open source code.
That leaves a few people who have built up their own bot from scratch. If anyone were able to give unbiased oppinions on other people's bots, it would be them.
I personally think that the average scores would be much fairer if 10 contestants voting (not on their own bots), than with 3 non-contestants voting. Slight biases against, say, forge bots or Alice bots should cancel themselves out over the larger sample, and crazy scores should also become irrelevent. I also think that it would make the identity of the judges less of an issue. We would know that, say, the maker of Jabberwacky is who he says he is. As it is now, it is quite possible for a botmaster to be a judge and have noone know it.
I agree with emm that some of the bots got better opportunities for conversation, but none of them got enough in my book. It would have been nice to have ten or twelve extra bot-human exchanges thrown on at the end of each test.
I still appreciate the effort of the committee. I know this all isn't easy.
I still appreciate the effort of the committee. I know this all isn't easy.
I think it would be interesting in future contests if each botmaster posted something of a mission statement about what they're trying to accomplish with their bot. That would allow the judges and the general public to get a feel for what they can expect from a bot during a conversation and slant the score a little toward how successful each bot is in achieving the botmaster's goals. One of my biggest complaint about these contests is that everyone goes in assuming that all bots are created with the same goals when some focus very specifically opn one thing. For example, tho it's not a chatbot, my favorite AI program ever is SHRDLU. I belive it's by far the most intelligent and impressive AI, yet it could never answer a single one of the 10 questions. I know a bot *that* specific has no place in the Chatterbox Challenge, but within the realm of chatbots, all are experiments with strengths in different areas.
If we really want lots of judges, we could have a contest where the judges are the contestants, with everybody judging everybody else (but not themselves).
I asked Chris what "most capable bot" was supposed to mean, and I got this reply:
Virtually every bot could technically enter but could you realistically expect to win. I don't think so and having all the extra bots puts a lot more work on the judges. I see Ella, The Harvinator Bot and Talk-Bot all have the ability to play a few games. Ultra Hal Assistant can open and run a few programs. I see Eliza providing psychological evaluation or support but that is a big stretch. I see B.O.B., Charlie and others being a web site guide. I don't particular like this category. This is a chatterbox contest and this goes beyond that. Others disagreed however so we included it.
Virtually every bot could technically enter but could you realistically expect to win. I don't think so and having all the extra bots puts a lot more work on the judges. I see Ella, The Harvinator Bot and Talk-Bot all have the ability to play a few games. Ultra Hal Assistant can open and run a few programs. I see Eliza providing psychological evaluation or support but that is a big stretch. I see B.O.B., Charlie and others being a web site guide. I don't particular like this category. This is a chatterbox contest and this goes beyond that. Others disagreed however so we included it.
By the way, have you noticed that Ella is doing worse than many PF bots?
I always thought it was totally unfair when she won the Loebner contest.
I always thought it was totally unfair when she won the Loebner contest.
» More new posts: Doghead's Cosmic Bar