Bot Contest
Here I'll be posting information on various Bot contests that challenge and test a Bot's AI and realism. Feel free to post comments and updates on contests, as well as announcements for new contests.
Posts 795 - 806 of 4,091
Posts 795 - 806 of 4,091
View Contest Winners in the Hall of Fame.
I imagine all of you who entered the contest got this, but for those who didn't and are curious about what's happening, this is the letter the organizers of the Loebner contest sent:
This communication hopefully will clarify the selection
method for this years competition. First, the Loebner
Selection Committee is still bound by the rules and
regulations set forth by the Loebner Contest Endowment.
Those rules state that there will be a maximum of eight
finalists selected to compete for the prize. In previous
years those eight were selected by committee and by no other means. Due to the overwhelming response to this years application request it became apparent that it would be impossible to fairly select eight finalists by committee.
However, the committee did review all applications and ranked them in a method similar to that used to judge the Olympic Games, but using a set of Turing Test criteria.
In addition to this ranking system the members of the
committee were specifically chosen to be impartial and
charged with insuring that the process would be based
purely on the elements specified by this Turing Test
criteria. From this ranking process the 16 applicants with the highest scores were automatically invited to compete in the semi-final event.
However, the Loebner Selection Committee still reserves the right to increase that number or to simply select finalists by alternate means. The semi-finals are designed to help in the final selection process and also provide recognition of a broader number of contestants. If there are any of these initial "auto-select applicants" that do not want to participate in the semi-finals, please notify us as soon as possible as there are many other applicants who will gladly take your place.
Also, there have been allegations that some of the semi-
finalists falsified their transcripts. We are presently
investigating these allegations and want to make it clear
that cheating will result in an applicant being dismissed, and they may be permanently barred from future contests. In addition, any individual who treats the Loebner 2002 staff or technical personnel with anything other than TOTAL RESPECT may also be dismissed from the contest.
Personally, I find that it doesn't clarify much. It doesn't even say what the semi-final will be like!
As for the allegations that some botmasters may be cheating, I just pointed out in my e-mail the concern that was voiced here that the transcripts could be easily manipulated. I never said that anybody did, because I have no way of knowing if it happened or not.
I don't know what you think, but this letter sounds too defensive and not informative enough to me.
This communication hopefully will clarify the selection
method for this years competition. First, the Loebner
Selection Committee is still bound by the rules and
regulations set forth by the Loebner Contest Endowment.
Those rules state that there will be a maximum of eight
finalists selected to compete for the prize. In previous
years those eight were selected by committee and by no other means. Due to the overwhelming response to this years application request it became apparent that it would be impossible to fairly select eight finalists by committee.
However, the committee did review all applications and ranked them in a method similar to that used to judge the Olympic Games, but using a set of Turing Test criteria.
In addition to this ranking system the members of the
committee were specifically chosen to be impartial and
charged with insuring that the process would be based
purely on the elements specified by this Turing Test
criteria. From this ranking process the 16 applicants with the highest scores were automatically invited to compete in the semi-final event.
However, the Loebner Selection Committee still reserves the right to increase that number or to simply select finalists by alternate means. The semi-finals are designed to help in the final selection process and also provide recognition of a broader number of contestants. If there are any of these initial "auto-select applicants" that do not want to participate in the semi-finals, please notify us as soon as possible as there are many other applicants who will gladly take your place.
Also, there have been allegations that some of the semi-
finalists falsified their transcripts. We are presently
investigating these allegations and want to make it clear
that cheating will result in an applicant being dismissed, and they may be permanently barred from future contests. In addition, any individual who treats the Loebner 2002 staff or technical personnel with anything other than TOTAL RESPECT may also be dismissed from the contest.
Personally, I find that it doesn't clarify much. It doesn't even say what the semi-final will be like!
As for the allegations that some botmasters may be cheating, I just pointed out in my e-mail the concern that was voiced here that the transcripts could be easily manipulated. I never said that anybody did, because I have no way of knowing if it happened or not.
I don't know what you think, but this letter sounds too defensive and not informative enough to me.
Hey, watch out! That's not TOTAL RESPECT.
To be fair, perhaps that was a response to some rude email(s) they received. But it comes off as a little over-the-top. It appears they dont want anybody testing the bots themselves and coming up with different winners. It'd be nice to know what the parameters are that they're testing on, and that would help people decide how to tweak their bots, but I understand that they dont want the interference.
The main event is so clearly laid out, it's unfortunate the semi-final picking isnt. Is this the first year they've done semi-finals?
To be fair, perhaps that was a response to some rude email(s) they received. But it comes off as a little over-the-top. It appears they dont want anybody testing the bots themselves and coming up with different winners. It'd be nice to know what the parameters are that they're testing on, and that would help people decide how to tweak their bots, but I understand that they dont want the interference.
The main event is so clearly laid out, it's unfortunate the semi-final picking isnt. Is this the first year they've done semi-finals?
My response to them:
Thank you for the note on the selection process. However, I still have concerns over the methods used for selection.
If selection was done solely on the strength of transcripts, why were only 16 entries considered? Surely you must realize that a transcript alone is little indication of a bot's strength. Putting aside any thoughts of possible tampering, doesn't it make sense that even a very poor bot could produce one brilliant transcript after hundreds of conversations?
I feel very disappointed in not having being selected even for the very first level of this contest. My bot "Little Mu" is very strong indeed, not just as a bot conversationalist, but in having very human conversations, as the transcript I submitted bore out.
A number of the selected entrants are my peers, and I am familiar with their bots. Several of them expressed suprize to me that Little Mu was not considered. To be dismissed without even a single conversation from one of your judges seems quite unfair to me.
In closing, let me just say that a contest that seeks to find a bot that can pass the Turing test should well consider an entry whose transcript demonstrated having a human convinced for nearly half the conversation that she might indeed be human.
Thank you for the note on the selection process. However, I still have concerns over the methods used for selection.
If selection was done solely on the strength of transcripts, why were only 16 entries considered? Surely you must realize that a transcript alone is little indication of a bot's strength. Putting aside any thoughts of possible tampering, doesn't it make sense that even a very poor bot could produce one brilliant transcript after hundreds of conversations?
I feel very disappointed in not having being selected even for the very first level of this contest. My bot "Little Mu" is very strong indeed, not just as a bot conversationalist, but in having very human conversations, as the transcript I submitted bore out.
A number of the selected entrants are my peers, and I am familiar with their bots. Several of them expressed suprize to me that Little Mu was not considered. To be dismissed without even a single conversation from one of your judges seems quite unfair to me.
In closing, let me just say that a contest that seeks to find a bot that can pass the Turing test should well consider an entry whose transcript demonstrated having a human convinced for nearly half the conversation that she might indeed be human.
I agree with the Professor. My initial reaction to the letter with the all caps TOTAL RESPECT was way over the top despite what letters they may or may not received.
Doly, you are not alone in questioning the use of the transcripts. I wrote them earlier on, well before the 16 bots were selected, about the possibility of tampering. Even if there were no tampering it is still not a good criteria to judge a bot as no bot owner in their right mind would send in a bad transcript.
The good news is it appears to me they are at least considering adding more bots to the 16 already chosen. Hopefully bots like Elbot and Little Mu and others will now get a second look. Now exactly what they will be looking for is anybodys guess.
Chris
Doly, you are not alone in questioning the use of the transcripts. I wrote them earlier on, well before the 16 bots were selected, about the possibility of tampering. Even if there were no tampering it is still not a good criteria to judge a bot as no bot owner in their right mind would send in a bad transcript.
The good news is it appears to me they are at least considering adding more bots to the 16 already chosen. Hopefully bots like Elbot and Little Mu and others will now get a second look. Now exactly what they will be looking for is anybodys guess.
Chris
honestly i think they are waiting for people to step down before they add any other competitors... my impression was that they (loebner contest committee) are willing to add contestants only if spaces become available.
My impression is that by now the organizers have realized that they've screwed up, but they want to save face. If they add some new bot now to the semi-finals, they are going to get e-mails from everybody else that wasn't selected.
If I were them, I think they've still got time to judge the bots by themselves and ask them a few questions. Problem is, there may be some bots that aren't online.
If I were them, I think they've still got time to judge the bots by themselves and ask them a few questions. Problem is, there may be some bots that aren't online.
Impossible, Butterfly. Alice would never had accepted a bot from any member of the AAA (Association Against Alice).
But maybe the judges are bots. That could explain why they are so reluctant to give any explanation. What bot understands anything of their own programming?
But maybe the judges are bots. That could explain why they are so reluctant to give any explanation. What bot understands anything of their own programming?
Below is a transcript of the judge's conversation with a bot who tied for second place last year. If you can do this well, you might just walk away with some money.
PROGRAM[14:22:27]Hold that thought, and let me interrupt and ask which year you were born.
JUDGE30[14:22:27]what is your favourite position
PROGRAM[14:22:27]Please try again to tell me which year you were born.
JUDGE30[14:22:50]are you always dominant like this
PROGRAM[14:22:51]Please try again to tell me which year you were born.
JUDGE30[14:23:25]are you a male of female chauvanist
PROGRAM[14:23:25]Please try again to tell me which year you were born.
JUDGE30[14:23:47]tell me when you wre born first
PROGRAM[14:23:47]Please try again to tell me which year you were born.
JUDGE30[14:23:56]you go first
PROGRAM[14:23:56]Please try again to tell me which year you were born.
JUDGE30[14:24:26]you tell me yours and I'll tell you mine. Now you go first
PROGRAM[14:24:26]Please try again to tell me which year you were born.
PROGRAM[14:22:27]Hold that thought, and let me interrupt and ask which year you were born.
JUDGE30[14:22:27]what is your favourite position
PROGRAM[14:22:27]Please try again to tell me which year you were born.
JUDGE30[14:22:50]are you always dominant like this
PROGRAM[14:22:51]Please try again to tell me which year you were born.
JUDGE30[14:23:25]are you a male of female chauvanist
PROGRAM[14:23:25]Please try again to tell me which year you were born.
JUDGE30[14:23:47]tell me when you wre born first
PROGRAM[14:23:47]Please try again to tell me which year you were born.
JUDGE30[14:23:56]you go first
PROGRAM[14:23:56]Please try again to tell me which year you were born.
JUDGE30[14:24:26]you tell me yours and I'll tell you mine. Now you go first
PROGRAM[14:24:26]Please try again to tell me which year you were born.
» More new posts: Doghead's Cosmic Bar