Handling null bytes in Postgres

rubyonrails-core
,
1

Postgres doesn’t support null bytes in string columns. However, null bytes do show up in requests (for whatever reason) and at the moment result in a 500 error, as in the following example:

class BooksController < ActionController::Base
  def show
    @books = Book.find_by(name: book_params[:name])
    render plain: @books.count
  end

  private

  def book_params
    params.require(:search).permit(:name)
  end
end

class BooksControllerTest < Minitest::Test
  include Rack::Test::Methods

  def test_index
    get "/book", search: { name: "\u0000" }
    assert last_response.ok? # ArgumentError: string contains null byte
  end
end

Possible solutions

In order to solve the problem in our applications, we discussed multiple strategies on how to handle null bytes in our application. In particular, we tried to figure out where the best place is to handle them:

CleanShot 2025-11-05 at 14.32.13@2x
CleanShot 2025-11-05 at 14.32.13@2x2302×670 51.4 KB

  • Rack Middleware: could come with quite an overhead and Model.find_by(attribute: "test\00") still raises
  • case-by-case: normalizes could be used to sanitise certain attributes. class User < ApplicationRecord normalizes :email, with: ->(str) { str&.delete("\u0000") }) end. However, we would have to remember to apply this change everywhere and a query like User.exists?(["email = ?" , "test\00"]) will still raise

For our solution, we’ve defined a set of requirements:

RSpec.describe 'Postgres string null byte support' do
  let(:search_term) { "test\00" }

  context 'when quering directly on the database' do
    let(:query) do
      ActiveRecord::Base
        .connection
        .execute("SELECT * FROM materials where materialcode = '#{search_term}'")
    end
    it 'raises an error as postgres does NOT support null bytes' do
      expect { query }.to raise_error(ArgumentError).with_message('string contains null byte')
    end
  end

  context 'when using ActiveRecord' do
    context 'with attribute-aware methods' do
      let(:query) { Material.where(name: search_term) }
      it 'does NOT raise an error' do
        expect { query }.not_to raise_error
        expect(query.first).to be_nil
      end
    end
    
    context 'without attribute-aware methods' do
      let(:query) { Manufacturer.where('manufacturers.name ILIKE ? '
                                         , search_term) }
      it 'does NOT raise an error' do
        expect { query }.not_to raise_error
        expect(query.first).to be_nil
      end
    end
  end
end

In the end we decided to implement a new OID::PostgresString class:

module ActiveRecord
  module ConnectionAdapters
    module PostgreSQL
      module OID
        class PostgresString < Type::String
          def serialize(value)
            super(PostgreSQL::StringUtils.strip_null_bytes(value))
          end

          private

          def cast_value(value)
            super(PostgreSQL::StringUtils.strip_null_bytes(value))
          end
        end
      end
    end
  end
end

which then gets registered in our PostgreSQLAdapter patch:

# config/initializers/postgres_string_null_byte_support.rb
module ActiveRecord
  module ConnectionAdapters
    class PostgreSQLAdapter < AbstractAdapter
      class << self
        alias original_initialize_type_map initialize_type_map

        def initialize_type_map(m = type_map)
          original_initialize_type_map(m)
          register_class_with_limit m,'varchar', OID::PostgresString
        end
      end

      ActiveRecord::Type.register(:string, OID::PostgresString, adapter: :postgresql)
    end
  end
end

This then allowed us to strip null bytes in all finders and assignments.

Where do you think makes most sense to implement a fix for this? Maybe in the PG adapter as a configurable option (similiar as the ActiveRecord::ConnectionAdapters::PostgreSQLAdapter.decode_dates flag)? Or rather in ruby-pg?

References

Their were also various issues about this already:

I also made a talk about this problem with more detailed snippets: https://github.com/renuo/postgres-null-bytes-talk/blob/main/presentation.pdf

1 Like