Problem 2390. Fair Scoring for Cody

Created by Nicholas Howe

Solve Later

{"relationships":[{"relationshipType":"http://schemas.mathworks.com/matlab/code/2013/relationships/document","targetMode":"","relationshipId":"rId1","target":"/matlab/document.xml"},{"relationshipType":"http://schemas.mathworks.com/matlab/code/2013/relationships/output","targetMode":"","relationshipId":"rId2","target":"/matlab/output.xml"}],"parts":[{"partUri":"/matlab/document.xml","relationship":[],"contentType":"application/vnd.mathworks.matlab.code.document+xml","content":"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<w:document xmlns:w=\"http://schemas.openxmlformats.org/wordprocessingml/2006/main\"><w:body><w:p><w:pPr><w:pStyle w:val=\"text\"/></w:pPr><w:r><w:t>The length-based scoring system for Cody has a few well-known loopholes that allow clever programmers to hide complex code inside certain functions so that it doesn't register in the score. This problem represents an attempt to generate a fairer Cody scoring function. It doesn't try to replace the length-based criterion with something more elegant; merely to eliminate places where longer code can be hidden and remain undetected by the scoring function.</w:t></w:r></w:p><w:p><w:pPr><w:pStyle w:val=\"text\"/></w:pPr><w:r><w:t>I have included a few examples that I consider abusive to start with. These are just my own opinions. Please feel free to add your suggestions for others in the comments section for this problem, and I will add good ideas to the test suite when I have the time. Thanks for playing!</w:t></w:r></w:p><w:p><w:pPr><w:pStyle w:val=\"ListParagraph\"/><w:numPr><w:numId w:val=\"1\"/></w:numPr></w:pPr><w:r><w:t>regexp: count number of nodes inside (?@cmd) and add to total</w:t></w:r></w:p><w:p><w:pPr><w:pStyle w:val=\"ListParagraph\"/><w:numPr><w:numId w:val=\"1\"/></w:numPr></w:pPr><w:r><w:t>regexprep: count number of nodes inside {$cmd} and add to total</w:t></w:r></w:p><w:p><w:pPr><w:pStyle w:val=\"text\"/></w:pPr><w:r><w:t>Using this scoring function, it might even be possible to allow use of eval.</w:t></w:r></w:p><w:p><w:pPr><w:pStyle w:val=\"ListParagraph\"/><w:numPr><w:numId w:val=\"1\"/></w:numPr></w:pPr><w:r><w:t>eval: count number of nodes inside argument to be evaluated and add to total</w:t></w:r></w:p><w:p><w:pPr><w:pStyle w:val=\"text\"/></w:pPr><w:r><w:t>Other possibilities -- should str2num count the numbers inside its argument? Suggestions welcome! And if you support the idea of changing the scoring function along these lines, please 'like' this problem. Thanks!</w:t></w:r></w:p><w:p><w:pPr><w:pStyle w:val=\"text\"/></w:pPr><w:r><w:t>See also</w:t></w:r><w:r><w:t> </w:t></w:r><w:hyperlink w:docLocation=\"http://www.mathworks.com/matlabcentral/cody/problems/2033-poll-would-you-like-the-regexp-cmd-functionality-to-be-banned-in-cody\"><w:r><w:t>http://www.mathworks.com/matlabcentral/cody/problems/2033-poll-would-you-like-the-regexp-cmd-functionality-to-be-banned-in-cody</w:t></w:r></w:hyperlink></w:p></w:body></w:document>"},{"partUri":"/matlab/output.xml","contentType":"text/xml","content":"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\" ?><embeddedOutputs><metaData><evaluationState>manual</evaluationState><layoutState>code</layoutState><outputStatus>ready</outputStatus></metaData><outputArray type=\"array\"/><regionArray type=\"array\"/></embeddedOutputs>"}]}

The length-based scoring system for Cody has a few well-known loopholes that allow clever programmers to hide complex code inside certain functions so that it doesn't register in the score. This problem represents an attempt to generate a fairer Cody scoring function. It doesn't try to replace the length-based criterion with something more elegant; merely to eliminate places where longer code can be hidden and remain undetected by the scoring function.I have included a few examples that I consider abusive to start with. These are just my own opinions. Please feel free to add your suggestions for others in the comments section for this problem, and I will add good ideas to the test suite when I have the time. Thanks for playing!<ul><li>regexp: count number of nodes inside (?@cmd) and add to total</li><li>regexprep: count number of nodes inside {$cmd} and add to total</li></ul>Using this scoring function, it might even be possible to allow use of eval.<ul><li>eval: count number of nodes inside argument to be evaluated and add to total</li></ul>Other possibilities -- should str2num count the numbers inside its argument? Suggestions welcome! And if you support the idea of changing the scoring function along these lines, please 'like' this problem. Thanks!See also <a href = "http://www.mathworks.com/matlabcentral/cody/problems/2033-poll-would-you-like-the-regexp-cmd-functionality-to-be-banned-in-cody">http://www.mathworks.com/matlabcentral/cody/problems/2033-poll-would-you-like-the-regexp-cmd-functionality-to-be-banned-in-cody</a>

The length-based scoring system for Cody has a few well-known loopholes that allow clever programmers to hide complex code inside certain functions so that it doesn't register in the score.  This problem represents an attempt to generate a fairer Cody scoring function.  It doesn't try to replace the length-based criterion with something more elegant; merely to eliminate places where longer code can be hidden and remain undetected by the scoring function.

I have included a few examples that I consider abusive to start with.  These are just my own opinions.  Please feel free to add your suggestions for others in the comments section for this problem, and I will add good ideas to the test suite when I have the time.  Thanks for playing!

* regexp:  count number of nodes inside (?@cmd) and add to total
* regexprep:  count number of nodes inside {$cmd} and add to total

Using this scoring function, it might even be possible to allow use of eval.

* eval:  count number of nodes inside argument to be evaluated and add to total

Other possibilities -- should str2num count the numbers inside its argument?  Suggestions welcome!  And if you support the idea of changing the scoring function along these lines, please 'like' this problem.  Thanks!

See also http://www.mathworks.com/matlabcentral/cody/problems/2033-poll-would-you-like-the-regexp-cmd-functionality-to-be-banned-in-cody

Solve

Solution Stats

12 Solutions
4 Solvers

Last Solution submitted on Nov 01, 2020

Last 200 Solutions

Problem Comments

9 Comments

9 Comments

Show 6 older comments

J.R.! Menzinger on 25 Jun 2014

I think this is a very interesting proposal.
So this kind of tricks still allowed in Cody, but they will be counted in a fair way.
I used this a couple of times, and i think it is useful to learn this kind of coding methods, but it is not the real meaning of Cody.

The most trick I used is the "str2num" one. To count all the numbers inside is a little bit hard, but I will accept this method also. It's fair! ;-)

Alfonso Nieto-Castanon on 26 Jun 2014

I believe you need to redefine the 'newline' variable in all of your testsuite cases?

Alfonso Nieto-Castanon on 26 Jun 2014

this is a very hard problem to solve in general. For example consider the following cases (the score in the first three cases should be penalized with additional 4 points, while in the latter three cases it should not): 1) regexp('',strcat('(?','@a=1)')); 2) str='(?@a=1)';regexp('',str); 3) regexp(' ',char(')@Ab>2*'-1)); 4) regexp('',strcat('(?','a=1)')); 5) str='(?@a=1)';regexp(str,''); 6) regexp(' ',char(')@Ab>2*'+1));

Nicholas Howe on 26 Jun 2014

Alfonso, thank you for your excellent thinking on this subject. You're quite right that an approach based only on parsing the program text would be hard to perfect. One could at least make people work harder to get around the scoring system! But what do you think of a different approach: modifying the definitions of regexp and its ilk on the Cody machines so that they dynamically alter the score when called with eval-like arguments?

Alfonso Nieto-Castanon on 26 Jun 2014

Hi Nicholas, that is a very good idea, the main problem would be to solve the correspondence between the different times regexp is called and the different times it might appear in the code, but perhaps using dbstack we could resolve this? (of course we could just penalize for each time it is called even if that might produce very high penalties for loops, etc.) thoughts?

Nicholas Howe on 26 Jun 2014

The same regexp expression could potentially be called more than once with different dynamic code each time. It might be best to count the score of each unique code string from each separate invocation point -- that seems most in keeping with the spirit of the current rules. You're right that it would be quite complex to keep track.

Thanks for pointing out the problem with newline, by the way. I think it is fixed now.

Rafael S.T. Vieira on 1 Nov 2020

The easiest way would be to add a fixed size length penalty whenever using the regexp function, for instance, a 1000 points. When a problem requires the regexp function, every player would have the same penalty so it would be meaningless. And when the problem doesn't require it, the Cody player would be disencouraged to use it.

PS: Instead of banning functions, as Cody currently does, a fixed point penalty for certain functions would disencourage their misuse while allowing players to fully explore MATLAB features.

Rafael S.T. Vieira on 1 Nov 2020

And as a bonus, hacks would all have a high score, making them all the worst possible solutions. Never the leading ones.

goc3 on 3 Nov 2020

I like your ideas in this regard, Rafael.

Solution Comments

Solution 462917

1 Comment

1 Comment

Nicholas Howe on 27 Jun 2014

I believe this solution misses the point! ;-)

Problem Recent Solvers4

Problem Tags

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!