JavaScript in HTML code: Random numbers are not evenly distributed

Question

I've got the following problem:
I'm uploading a survey on amazon mturk using Python and the survey is done via HTML and javascript. I show one of three different versions of the survey to participants, which I select by generating a random number via javascript. I store the number in local storage to prevent refreshing the website from resetting it. The problem I find is that more people seem to get versions 1 than version 3. But I cannot recreate the problem for myself when running the code in Tryit Editor online.

Could you please help me understand (and fix) why this happens? The following is the (trimmed) HTML code that I upload. I replaced text and removed fluff.

<HTMLQuestion xmlns="http://mechanicalturk.amazonaws.com/AWSMechanicalTurkDataSchemas/2011-11-11/HTMLQuestion.xsd">
<HTMLContent><![CDATA[
<!-- YOUR HTML BEGINS -->
<!DOCTYPE html>
<html>
<head>
<meta http-equiv='Content-Type' content='text/html; charset=UTF-8'/>
<script type='text/javascript' src='https://s3.amazonaws.com/mturk-public/externalHIT_v1.js'></script>
<script>
function test(){
    document.getElementById('txt-field').value = "1";
}
</script>
</head>
<body>
<form name='mturk_form' method='post' id='mturk_form' action='https://www.mturk.com/mturk/externalSubmit'><input type='hidden' value='' name='assignmentId' id='assignmentId'/>
  <span>
<INPUT TYPE="text" NAME="link_click" id='txt-field' value="0" style="display: none">    
<div><h3><a href="www.google.com" target="_blank" id='report420' onclick="test()" >link</a></h3>
Instructions</div>
<div><table border="1" style="height: 258px;" width="196"><tbody>Table</tbody></table></div>
</span>
<!--I think the relevant part starts here-->
<script> 

document.write("Miscellaneous question");

var i = localStorage.getItem('i') || Math.floor(3*Math.random());
localStorage.setItem('i',i);

if (i==0){
document.write("Version 1");
}
if (i==1){
document.write("Version 2");
}
if (i==2){
document.write("Version 3");
}
document.write("Miscellaneous question");

</script>
<p><input type='submit' id='submitButton' value='Submit' /></p></form>
<script language='Javascript'>turkSetAssignmentID();</script>
</body></html>
<!-- YOUR HTML ENDS -->
]]>
</HTMLContent>
<FrameHeight>600</FrameHeight>
</HTMLQuestion>

Do you have stats for the number of clients getting each version ? — Seblor
– Seblor, Commented May 27, 2018 at 14:02
I have them for slightly different versions, but accumulated they are: version 1: n=156, version 2: n=138, version 3: n=106. With a Pearson's Chi-Square test (df=2), it's significant at 99% (Chi-square=9.62). — Mirko
– Mirko, Commented May 27, 2018 at 14:16
That's a really hard assumption, in small amounts randomness may seem deviant, if you throw a coin ten times and get 8 heads and only two tails, analyzing this sampling alone, you will have the illusion of a unnatural randomness. On how many surveys are you basing your assumption? — desoares
– desoares, Commented May 27, 2018 at 14:21
Across different versions with the same layout of the randomization, I have n=400 now. A common statistical test suggests it's biased (Chi-square=9.62, p<0.01). — Mirko
– Mirko, Commented May 27, 2018 at 14:24
I only thing I use in the code itself is that you are using a pretty common name for your local storage variable, i. It is pretty easy to think of a scenario where someone else has already used this variable, so you might be trying to save a variable that already has a value, and there is no error checking for if i already exists as not a number or is greater than 3. — ecg8
– ecg8, Commented May 27, 2018 at 14:41

GramThanos · Accepted Answer · 2018-05-27 15:38:29Z

1

The random function Math.floor(3*Math.random()) has uniform distribution, but I don't think that 400 samples are enough so that you can see it in action (as @desoares mentioned).

Testing code:

var count = [0, 0, 0];
var n = 1000000;
document.write('Testing for ' + n + ' samples : ');
for (var i = 0; i < n; i++) {
    count[Math.floor(3*Math.random())]++;
}

document.write(JSON.stringify(count));

var count = [0, 0, 0];
var n = 400;
document.write('Testing for ' + n + ' samples : ');
for (var i = 0; i < n; i++) {
    count[Math.floor(3*Math.random())]++;
}

document.write(JSON.stringify(count));

Also, if you want to be sure that people from the same computer are not forced to take the same version, you should clear the saved variable localStorage.removeItem('i'); on submit. You may also add an expiration mechanic.

answered May 27, 2018 at 15:38

GramThanos

3,6321 gold badge24 silver badges34 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Mirko Over a year ago

I think that it depends on the result. Suppose you found there to be no occurrence of the third version - that would be quite indicative of a biased random generator or a mistake.

GramThanos Over a year ago

Javascript runs on the client on a browser, the browser should follow the specification of the javascript (you can't control that), based on that and the info you gave us, the code should produce a uniform distribution. But there is always a small possibility that we live on the parallel universe that your code will never select the 3rd version.

Collectives™ on Stack Overflow

JavaScript in HTML code: Random numbers are not evenly distributed

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related