Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow SQL warnings to be reported. #46690

Merged
merged 1 commit into from Jan 18, 2023

Conversation

adrianna-chang-shopify
Copy link
Contributor

@adrianna-chang-shopify adrianna-chang-shopify commented Dec 9, 2022

Motivation / Background

At Shopify, we use https://github.com/Shopify/activerecord-pedantmysql2-adapter to raise SQL warnings as errors so they can be identified and fixed. We believe this provides value upstream, so we'd like to propose an API in Rails that provides warning reporting.

Activerecord::PedantMysql2 provides a more comprehensive API, including the ability to capture / silence warnings within certain scopes. We've reduced the API to what we believe is most critical: allowing users to raise, log, or ignore warnings. Users can also specify a custom proc to be called when a warning is encountered, which provides some of the flexibility that the pedant gem was offering.

Detail

We've introduced two configurations to Active Record:

  • config.active_record.db_warnings_action can be used to configure the action to take when a query produces a warning. The warning can be ignored, logged, raised, or trigger custom behaviour provided via a proc.
  • config.active_record.db_warnings_ignore allows applications to set an allowlist of SQL warnings that should always be ignored, regardless of the configured action.

For now, this is only scoped to MySQL adapters, internally relying on MySQL's SHOW WARNINGS statement, and the PostgreSQL adapter, relying on #set_notice_processor.

Checklist

Before submitting the PR make sure the following are checked:

  • This Pull Request is related to one change. Changes that are unrelated should be opened in separate PRs.
  • Commit message has a detailed description of what changed and why. If this PR fixes a related issue include it in the commit message. Ex: [Fix #issue-number]
  • Tests are added or updated if you fix a bug or add a feature.
  • CHANGELOG files are updated for the changed libraries if there is a behavior change or additional feature. Minor bug fixes and documentation changes should not be included.

@adrianna-chang-shopify adrianna-chang-shopify force-pushed the ac-db-warning-reporting branch 2 times, most recently from 7051422 to 66d3b34 Compare Dec 9, 2022
Copy link
Contributor

@casperisfine casperisfine left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few comments but generally LGTM.

Not too sure about the naming of the two configuration options, and whether we should scope them under MySQL or not. Do other DBs even have warnings? @matthewd since you're the PG expert in my eyes 😄

@eileencodes eileencodes requested a review from matthewd Dec 12, 2022
@eileencodes eileencodes self-assigned this Dec 12, 2022
@matthewd
Copy link
Member

The Postgres equivalent is https://github.com/ged/ruby-pg/blob/1dd1039983b0f5c15e3cb23e3bf6475dc0b9ee6a/ext/pg_connection.c#L2786

It looks like SQLite has a similar mechanism in https://www.sqlite.org/errlog.html, though that's perhaps too limited for us by being process-global (and, AFAICS, not currently exposed by the gem?)

@adrianna-chang-shopify
Copy link
Contributor Author

👋 Hi folks, thanks for the feedback! We can definitely expand this to include PG support -- I couldn't for the life of me find an equivalent warnings feature in PG when we spun this up, so thanks for linking those docs @matthewd !

What may make sense would be to pass the pool config or something to the callback to it can behave differently if needed or even just include the DB name.

This makes sense to me -- maybe let's start with passing the DB name to the proc for now, unless we see some other use case where having the full pool config would be necessary?

@casperisfine
Copy link
Contributor

casperisfine commented Dec 15, 2022

maybe let's start with passing the DB name to the proc for now

You mean connection name? Because the DB name would change between environments.

I think passing a rich object is better than just a name. That is assuming the object is public.

@adrianna-chang-shopify
Copy link
Contributor Author

You mean connection name?

Yes, sorry I meant the db config names 🤦‍♀️

I think passing a reach object is better than just a name

Yeah, I guess it makes more sense to pass an object here rather than a magic string with the name 😄 PoolConfig is nodoc'ed though, does that make it private? Should we maybe pass a ActiveRecord::DatabaseConfigurations::HashConfig instead?

@casperisfine
Copy link
Contributor

👍 to pass ActiveRecord::DatabaseConfigurations::DatabaseConfig.

@adrianna-chang-shopify adrianna-chang-shopify force-pushed the ac-db-warning-reporting branch 2 times, most recently from 0071e93 to 07f799f Compare Dec 16, 2022
@adrianna-chang-shopify
Copy link
Contributor Author

Still need to implement Postgres support, feel free to hold off on reviewing until that's up. I'll re-request reviews when it's ready 😄

activerecord/CHANGELOG.md Outdated Show resolved Hide resolved
guides/source/configuring.md Outdated Show resolved Hide resolved
@adrianna-chang-shopify adrianna-chang-shopify force-pushed the ac-db-warning-reporting branch 2 times, most recently from b65192b to 81f501a Compare Dec 16, 2022
@eileencodes
Copy link
Member

I talked about this with Adrianna today and if there are no strong objections I think we should implement this for application level only in this PR and in a followup PR figure out how we want to do it per-connection (either waiting until someone needs / requests it or ensuring it gets in the next release).

@byroot
Copy link
Member

byroot commented Jan 6, 2023

figure out how we want to do it per-connection

Wasn't the idea that you could filter that from the callback using the connection that is passed as argument?

Even in multi-db scenarios, 99% of the time you won't have different handling for different DBs. It's nice that you can do it, but I see no need to streamline it more than it already is.

@adrianna-chang-shopify
Copy link
Contributor Author

adrianna-chang-shopify commented Jan 6, 2023

Wasn't the idea that you could filter that from the callback using the connection that is passed as argument?

@eileencodes and I discussed reverting back to the original API and foregoing passing a second argument to the callback (either a connection / db config) until we were more convinced of a per-connection use case. In other words, db_warnings_action can be a proc that takes a single SqlWarning object, and db_warnings_ignore remains a list of Strings / regexps to match against. We felt that the API being discussed didn't seem like quite the right fit (ignoring warnings across an entire db via emit_db_warnings vs a proc for db_warnings_action that receives two arguments vs a proc for db_warnings_ignore that should only receive the SqlWarning itself so as not to be redundant... It felt like there was a lot of overlap within the API).

If we're concerned about the use case Matthew presented (which seems the most viable) of ignoring warnings on secondary databases, we could introduce an ignore_db_warnings config at the database level that "wins out" over the global config. But even that could be delayed until we're more convinced that it's necessary?

@byroot
Copy link
Member

byroot commented Jan 6, 2023

Fine by me.

@adrianna-chang-shopify adrianna-chang-shopify force-pushed the ac-db-warning-reporting branch 2 times, most recently from 2657b63 to d6b9431 Compare Jan 6, 2023
@adrianna-chang-shopify adrianna-chang-shopify force-pushed the ac-db-warning-reporting branch 2 times, most recently from a7438e7 to 3e45a37 Compare Jan 9, 2023
Copy link
Contributor

@casperisfine casperisfine left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some very minor stuff.

@adrianna-chang-shopify adrianna-chang-shopify force-pushed the ac-db-warning-reporting branch 2 times, most recently from 60f618e to 5ad6136 Compare Jan 10, 2023
@adrianna-chang-shopify
Copy link
Contributor Author

Appreciate the feedback, folks ❤️ I've gone back to individual commits to make this a bit easier to review, I'l squash again when this is ready.

@adrianna-chang-shopify adrianna-chang-shopify force-pushed the ac-db-warning-reporting branch 2 times, most recently from 8436d48 to e49fd9c Compare Jan 11, 2023
->(warning) do
warning_message = "[#{warning.class}] #{warning.message}"
warning_message += " (#{warning.code})" if warning.code
ActiveRecord::Base.logger.warn(warning_message)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will load ActiveRecord::Base when this config is set, and we don't want that. But it inside a on_load block.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so. We're in a callback here, it won't be invoked until a query is performed and it emits warnings.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, right.

->(warning) do
warning_message = "[#{warning.class}] #{warning.message}"
warning_message += " (#{warning.code})" if warning.code
ActiveRecord::Base.logger.warn(warning_message)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, right.

Copy link
Contributor

@casperisfine casperisfine left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@byroot
Copy link
Member

byroot commented Jan 16, 2023

@matthewd @jhawthorn if you don't have any other concerns I'd like to merge this.

@adrianna-chang-shopify adrianna-chang-shopify force-pushed the ac-db-warning-reporting branch 2 times, most recently from cbb5fe7 to dbc5432 Compare Jan 16, 2023
@nvasilevski
Copy link
Contributor

Was wondering if we should consider adding an API on top of this one to override the behavior per block. Similar to PedantMysql2.capture_warnings. Doesn't have to be a part of this PR though.

I can imagine someone wanting to perform a query despite of warnings but without adding the warning to the ignore list so they could rescue it like:

   def update_record(params)
       ActiveRecord::Base.connection.execute("query...")
   rescue SQLWarning => e
       # no-op
       # I know my manual query issues a warning but I'm okay with that
       # however I don't want to add the warning to the ignore list as this is the only place which I expect to raise a warning. Every other part of the app should still raise
   end

But this example won't work for a read query as we don't get to use the result if we raise.
And with a wrapper-like API we will be able to do:

   def update_record(params)
     warnings = ActiveRecord.capture_warnings do
        # we know this query issues warnings but this is a specific use-case so we want to ignore it but only here
        result = ActiveRecord::Base.connection.execute("query...")
     end
   end

@adrianna-chang-shopify
Copy link
Contributor Author

Was wondering if we should consider adding an API on top of this one to override the behavior per block

I think this is a valid use case for sure 👍 This PR was meant to introduce warning reporting as a simple AR config to start, paving the way for more complex use cases later on. I think introducing #capture_warnings makes sense, but there are implications for thread safety there, so I definitely think we'd want to tackle this in a separate PR.

Copy link
Member

@jhawthorn jhawthorn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One non-blocking question, otherwise looks good to me!

activerecord/lib/active_record/errors.rb Outdated Show resolved Hide resolved
`ActiveRecord.db_warnings_action` can be used to configure the
action to take when a query produces a warning. The warning can be
logged, raised, or trigger custom behaviour provided via a proc.

`ActiveRecord.db_warnings_ignore` allows applications to set an
allowlist of SQL warnings that should always be ignored, regardless
of the configured action.

Co-authored-by: Paarth Madan <paarth.madan@shopify.com>
Copy link
Member

@jhawthorn jhawthorn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Thank you

@byroot byroot merged commit fc04ad8 into rails:main Jan 18, 2023
3 of 4 checks passed
@adrianna-chang-shopify adrianna-chang-shopify deleted the ac-db-warning-reporting branch Jan 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants