Retrieving data-* value
You refer to the attribute as table[data-table-values], which is incorrect on two accounts:
table[] is not necessary. I'm not sure why you were using that. This isn't a selector, this is a property lookup.
data-table-values should be table-values. Your HTML attribute is called "data-table-values", but the .data() lookup does not need that for any of the HTML data-* attributes.
The end result of both of these is to have .data('table-values') as evidenced in the example at the bottom.
If you wanted to look up the actual attribute value, then you would need to include the data- prefix, because you're searching for that specific HTML attribute; e.g., $test.attr('data-table-values'), but it's against common practice to do this for data-* attributes.
One thing you should note about the .data() method is that, it operates in memory. It doesn't update your HTML. If you use any sort of inspector you would see that setting $test.data('foo','bar'), it does not bind a foo attribute to the element, nor modifies the DOM, it manages the value internally to jQuery.
Fixing JSON
To use the value as JSON, you need to store it as wellformed JSON, right now you're using single quotes as the encapsulating JSON string delimiter '{'':[]}' and also to delimit the object keys. You must use double quotes for valid JSON.
In the example below, I trim off your leading and ending single quotes (') and then replace all the single quotes around the keys with double quotes ". I broke this into several lines for clarity, but could be condensed into several lines.
Note: if the JSON is valid and properly stored in the data-* attribute, then the client wouldn't need to do so much work scrubbing it and the code would be much more simpler
var $test = $('#test');
var str = $test.data('table-values'); // <=== was:"table[data-table-values]"
console.log('data-* value:', str);
// format to a valid JSON string
str = str.replace(/^'+/, ''); // remove leading single quote
str = str.replace(/'+$/, ''); // remove ending single quote
str = str.replace(/'/g, '"'); // use double quotes
console.log('Replaced single quotes:', str);
// make JS object
var obj = JSON.parse(str);
console.log('obj:', obj);
console.log('obj.header[1]:', obj.header[1]);
// make JSON string
console.log('JSON:', JSON.stringify(obj));
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
<table id="test" data-table-values="'{'header': ['value1', 'value2']}'"></table>
Better HTML
Here is an example per @guest271314's suggestion, which demonstrates having well-formed JSON in your HTML.
Notice there are far fewer lines (only one required to get the JSON object). Provided is the jQuery and the ES6 equivalent:
// jQuery
var obj = $('#test').data('table-values');
console.log('obj:', obj);
console.log('obj.header[1]:', obj.header[1]);
console.log('JSON.stringify(obj):', JSON.stringify(obj));
// ES6
var obj2 = JSON.parse(document.querySelector('#test').dataset['tableValues']);
console.log('obj2:', obj2);
console.log('obj2.header[1]:', obj2.header[1]);
console.log('JSON.stringify(obj2):', JSON.stringify(obj2));
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
<table id="test" data-table-values='{"header": ["value1", "value2"]}'></table>
table[data-table-values]through the function. So I didn't know if it would bring along double quotes or not...and why it didn't work when I did it. BTW I say rails but these are javascript files within rails Im talking about.