0

I'm developing a web application using Flask and a Postgresql database (using SQLAlchemy). It works just fine. Except for a little problem: all text data comes in some encoding that it's not utf-8 (at least I think it's not).

I looked around to see any possible solutions and this is what I've tried:

  • Include this line at every Python file in the project:
# -*- coding: utf-8 -*-
  • Set the engine like this:
def __init__(self):
        self.engine = sql.create_engine(os.environ['DATABASE_URL'], encoding='utf8')
        self.conn = self.engine.connect()
        self.metadata = sql.MetaData()

And a few variations of these previous items like client_encoding='utf8 instead of encoding='utf8. I also tried these solutions:

For example, this is one of my select functions (but the problem happens in every select / insert / update functions):

def get_produtos(self, id):
        produtos = sql.Table('produtos', self.metadata, autoload=True,
                             autoload_with=self.engine).columns
        parceiros = sql.Table('parceiros', self.metadata, autoload=True,
                              autoload_with=self.engine).columns
        q = sql.select([produtos.id, produtos.nome, parceiros.nome_parceiro,
                        produtos.valor, produtos.qtd_desconto,
                        parceiros.id_parceiro])
        q = q.where(produtos.id == id)
        return self.conn.execute(q).fetchall()

The program does not raises any exception but the data just comes with the wrong encoding.

This is how the data is in my database: https://i.sstatic.net/wvRTN.png

This is how the data shows up in my app: https://i.sstatic.net/HCj0C.png

5
  • Can you edit your question to add the output of this command from the psql console: \l <name of your database (or psql --list | grep '<name of your database>'. This will show us the defaults for your database. Commented Sep 25, 2019 at 6:40
  • The application is running on Heroku so I can only acces the database itself. Running the command you said logged in my database (not sure if it's the right thing to do), I got this: i.imgur.com/S1ZQxOT.png Commented Sep 25, 2019 at 16:00
  • 1
    A flagrant mojibake case. You are right suspecting that all text data comes in some encoding that it's not utf-8. 'Se‡Æo Eletr“nicos' is definitely a result from this: 'Seção Eletrônicos'.encode('cp850').decode('cp1252') Commented Jan 21, 2021 at 16:09
  • @JosefZ this problem was actually solved a while ago, but thanks for answering since it might help other people Commented Jan 21, 2021 at 18:26
  • 1
    @MarcosSombra Well. Then, you should answer the question yoursef (and finally accept the answer (See this page for an explanation of why this is important.) Commented Jan 21, 2021 at 20:16

1 Answer 1

1

As JosefZ said, it's a Mojibake case. This problem in particular occured because some entries were inserted via shell and there was no encoding and led to this inconsistency.

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.