66

I have a rails application running on production mode, but all of the sudden this error came up today when a user tried to save a record.

Mysql2::Error: Incorrect string value

More details (from production log):

Parameters: {"utf8"=>"â<9c><93>" ... 

Mysql2::Error: Incorrect string value: '\xC5\x99\xC3\xA1k 

Mysql2::Error: Incorrect string value: '\xC5\x99\xC3\xA1k 

Now I saw some solutions that required dropping the databases and recreating it, but I cannot do that.

Now mysql shows this:

mysql> show variables like 'char%';
+--------------------------+----------------------------+
| Variable_name            | Value                      |
+--------------------------+----------------------------+
| character_set_client     | utf8                       |
| character_set_connection | utf8                       |
| character_set_database   | latin1                     |
| character_set_filesystem | binary                     |
| character_set_results    | utf8                       |
| character_set_server     | latin1                     |
| character_set_system     | utf8                       |
| character_sets_dir       | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.04 sec)

What is wrong and how can I change it so I do not have any problems with any characters?

Also: Is this problem solvable with javascript? Convert it before sending it ?

Thanks

2

7 Answers 7

88

the problem is caused by charset of your mysql server side. You can config manually like:

ALTER TABLE your_database_name.your_table CONVERT TO CHARACTER SET utf8

or drop the table and recreate it like:

rake db:drop
rake db:create
rake db:migrate

references:

https://stackoverflow.com/a/18498210/2034097

https://stackoverflow.com/a/16934647/2034097

UPDATE

the first command only affect specified table, if you want to change all the tables in a database, you can do like

ALTER DATABASE databasename CHARACTER SET utf8 COLLATE utf8_general_ci;

reference:

https://stackoverflow.com/a/6115705/2034097

Sign up to request clarification or add additional context in comments.

7 Comments

I run this ALTER TABLE your_database_name.your_table CONVERT TO CHARACTER SET utf8 against all my tables.
My records are still there, I can use them and create new ones. Great. Now I will have to run this again if I want to create other tables right?
congrats, to answer your question, I updated my answer, please take a look.
Running the ALTER DATABASE command did not change the character set of my problem table. Running ALTER TABLE did work. I was able to check the character set name using the following command. SELECT CCSA.character_set_name FROM information_schema.TABLES T, information_schema.COLLATION_CHARACTER_SET_APPLICABILITY CCSA WHERE CCSA.collation_name = T.table_collation AND T.table_schema = "your_databasename" AND T.table_name = "your_tablename"; reference: stackoverflow.com/questions/1049728/…
I was already in utf8, but looks like I needed utf8mb4 stackoverflow.com/questions/13653712/…
|
57

I managed to store emojis (which take up 4 bytes) by following this blog post:

Rails 4, MySQL, and Emoji (Mysql2::Error: Incorrect string value error.)

You might think that you’re safe inserting most utf8 data in to mysql when you’ve specified that the charset is utf-8. Sadly, however, you’d be wrong. The problem is that the utf8 character set takes up 3 bytes when stored in a VARCHAR column. Emoji characters, on the other hand, take up 4 bytes.

The solution is in 2 parts:

Change the encoding of your table and fields:

ALTER TABLE `[table]` 
  CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_bin,
MODIFY [column] VARCHAR(250)
    CHARACTER SET utf8mb4 COLLATE utf8mb4_bin

Tell the mysql2 adapter about it:

development:
  adapter: mysql2
  database: db
  username: 
  password:
  encoding: utf8mb4
  collation: utf8mb4_unicode_ci

Hope this helps someone!

Then I had to restart my app and it worked. Please note that some emojis will work without this fix, while some won't:

  • ➡️ Did work
  • 🔵 Did not work until I applied the fix described above.

5 Comments

to access mysql via terminal: sudo mysql -u root
Why did you set the collation to utf8mb4_unicode_ci in your database.yml rather than utf8mb4_bin?
This finally works for me with Chinese character set, thanks!
Works with Rails 6. Not sure why this is not the default.
@CannonMoyer i think it's because ActiveRecord doesn't recognize utf8mb4_unicode_bin: ActiveRecord::StatementInvalid (Mysql2::Error: Unknown collation: 'utf8mb4_unicode_bin': SET NAMES utf8mb4 COLLATE utf8mb4_unicode_bin. I'm going with utf8mb4_unicode_ci in the db and in database.yml, looks like it's working for the emojis i've tested so far.
29

You can use a migration like this to convert your tables to utf8:

class ConvertTablesToUtf8 < ActiveRecord::Migration
  def change_encoding(encoding,collation)
    connection = ActiveRecord::Base.connection
    tables = connection.tables
    dbname =connection.current_database
    execute <<-SQL
      ALTER DATABASE #{dbname} CHARACTER SET #{encoding} COLLATE #{collation};
    SQL
    tables.each do |tablename|
      execute <<-SQL
        ALTER TABLE #{dbname}.#{tablename} CONVERT TO CHARACTER SET #{encoding} COLLATE #{collation};
      SQL
    end
  end

  def change
    reversible do |dir|
      dir.up do
        change_encoding('utf8','utf8_general_ci')
      end
      dir.down do
        change_encoding('latin1','latin1_swedish_ci')
      end
    end
  end
end

Comments

4

If you want to the store emoji, you need to do the following:

  1. Create a migration (thanks @mfazekas)

    class ConvertTablesToUtf8 < ActiveRecord::Migration def change_encoding(encoding,collation) connection = ActiveRecord::Base.connection tables = connection.tables dbname =connection.current_database execute <<-SQL ALTER DATABASE #{dbname} CHARACTER SET #{encoding} COLLATE #{collation}; SQL tables.each do |tablename| execute <<-SQL ALTER TABLE #{dbname}.#{tablename} CONVERT TO CHARACTER SET #{encoding} COLLATE #{collation}; SQL end end

    def change reversible do |dir| dir.up do change_encoding('utf8mb4','utf8mb4_bin') end dir.down do change_encoding('latin1','latin1_swedish_ci') end end end end

  2. Change rails charset to utf8mb4 (thanks @selvamani-p)

    production: encoding: utf8mb4

References:

https://stackoverflow.com/a/39465494/1058096

https://stackoverflow.com/a/26273185/1058096

3 Comments

Thanks for this. I think you could just change the character set and collation for specific columns which require emoji support - if for example changing all tables to utf8mb4 means existing index keys are too long. The migration could execute SQL like: execute <<-SQL ALTER TABLE #{table} MODIFY #{column} text CHARACTER SET #{character_set} COLLATE #{collation}; SQL
Great, but if you forget to change collation you will get errors. It happens if you use a tool to manage the database. Everything depends of the case, if you need performance you should pay attention on it and your solution is better than mine.
This answer looks more like a comment on @mfazekas' answer to me. See "Why and how are some answers deleted?"
1

Need to change CHARACTER SET and COLLATE for already created database:

ALTER DATABASE databasename CHARACTER SET utf8 COLLATE utf8_unicode_ci;

Or it was necessary to create a database with pre-set parameters:

CREATE DATABASE databasename CHARACTER SET utf8 COLLATE utf8_general_ci;

Comments

0

It seems like an encoding problem while getting data from database. Try adding the below to your database.yml file

   encoding: utf8 

Hope this solves your issue

1 Comment

MySQL's utf8 character set only allows for a subset of unicode. Additionally, changing the encoding in the database.yml file does not convert the database tables themselves to the specified encoding.
0

Also, if you don't want to do changes in your database structure, you could opt by serializing the field in question.

class MyModel < ActiveRecord::Base
  serialize :content

  attr_accessible :content, :title
end

5 Comments

Works in a pinch but definitely not the long-term solution you're looking for. Multiple drawbacks to serializing content. Much better to get your database properly configured at some point.
@JoshuaPinter Could you expand upon why using serialization isn't a good long-term solution? Currently trying to decide how to handle this situation.
@BrandonSturgeon There's lots but ones that I've run up against: as the value of the field grows, it can become unweildly and very inefficient. And it might hit a character limit in the field database. We were storing an array of id values that we were serializing instead of having a proper join table and it means you can't utilize database queries properly when retrieving those ids. Lots of stuff that is obvious and lots that might not be. Just use it sparingly and be aware of the implications.
@JoshuaPinter Thank you for the response! I decided to collate all tables that needed collation thru a migration. This seems to function just as well. Another consideration, I believe, is if you need to access the database outside of a rails project, you'd have to implement a deserialization method separately.
@BrandonSturgeon Good point! Again, great in a pinch but generally not a long-term, robust solution.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.