Skip to content

Configurable idle timeout - #1098

Merged
MaxGabriel merged 5 commits into
masterfrom
configurableIdleTimeout
Nov 2, 2020
Merged

MaxGabriel merged 5 commits into
masterfrom
configurableIdleTimeout

Conversation

@MaxGabriel

@MaxGabriel MaxGabriel commented Jul 13, 2020 •

Copy link
Copy Markdown
Member

This PR is a draft shows what it might look like to have a configurable idle timeout and stripe size for Persistent's SQL backends. This would resolve the issues brought up in #775.

This PR is just a draft. It's untested and needs a few small additions, but most things are in place. (Edit: added a test)

Before submitting your PR, check that you've:

After submitting your PR:

  • Update the Changelog.md file with a link to your PR
  • Bumped the version number if there isn't an (unreleased) on the Changelog
  • Check that CI passes (or if it fails, for reasons unrelated to your change, like CI timeouts)


, pgPoolStripes :: Int
-- ^ How many stripes to divide the pool into. See "Data.Pool" for details.
, pgPoolIdleTimeout :: Integer

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally this would be NominalDiffTime. However, NominalDiffTime lacks a Read instance, and PostgresConf derives Read. I requested a Read instance here: haskell/time#130

Comment on lines +185 to +186
=> (PG.Connection -> IO (Maybe Double)) -- ^ Action to perform to get the server version.
-> (PG.Connection -> IO ()) -- ^ Action to perform after connection is created.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two functions are kind of annoying, naming wise. Something like createPostgresqlPoolModifiedWithVersionWithConf would be gross (createPostgresqlPoolModifiedWithVersion is pretty bad on its own tbh). I'm also somewhat worried that another callback will eventually be added here (Perhaps a callback for when a connection is destroyed, or some other Postgres specific thing?)

Possibly a separate data type is called for to hold these functions (PostgresConfCallbacks isn't quite right, but something like that), with a function to create a default data structure?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hooks is another term that gets used for this concept a bit -

data PostgresConfHooks = PostgresConfHooks
  { getServerVersion :: IO (Maybe Double)
  , afterCreate :: IO ()
  }

createPostgresqlPoolWithHooks :: (PG.Connection -> PostgresConfHooks) -> ...

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, I was investigating this today and I was thinking this could be an opportunity to clean up the server version API. I have these complaints:

  1. Double is a weird way to represent a version. It doesn't work for 3 digit versions (x.y.z) (though Postgres moved to just 2 digit versions recently which helps). I think something like NonEmpty Integer makes more sense, e.g. [10, 2]

  2. The Maybe behavior feels weird to me. If you return Nothing, Persistent assumes the lowest possible version, not trying to do things like upsert. This isn't documented afaict, and would subtly degrade behavior if someone happened to return Nothing.

As long as we're adding a new API, I would propose something like:

getServerVersion :: IO (NonEmpty Integer)

In the case of Redshift users who can't get the server version (assuming that's still a thing), they can return a hard-coded value that matches whatever Postgres capabilities Redshift has, e.g. pure (10 :| [2]).

If there was something better suited than NonEmpty Integer I'd be fine with that too.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implementation would be something like:

getServerVersion2 :: PG.Connection -> IO (NonEmpty Word)
getServerVersion2 conn = do
  [PG.Only version] <- PG.query_ conn "show server_version";
  case AT.parseOnly parseVersion (T.pack version) of
    Left err -> throwIO $ PostgresServerVersionError $ "Failed to parse Postgres version. Got: " <> version <> ". Error: " <> err
    Right versionComponents -> case NEL.nonEmpty versionComponents of
      Nothing -> throwIO $ PostgresServerVersionError $ "Empty Postgres version string. Got: " <> version
      Just neVersion -> pure neVersion

  where
    -- Partially copied from the `versions` package
    parseVersion = AT.decimal `AT.sepBy` AT.char '.' <* AT.endOfInput

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds great to me

{ pgConnStr :: ConnectionString
-- ^ The connection string.

, pgPoolStripes :: Int

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding new fields could be considered a breaking change for persistent-postgresql

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If they're exported, yeah, it's technically a breaking change.

, connectionPoolConfigIdleTimeout :: NominalDiffTime
, connectionPoolConfigSize :: Int
}
deriving (Show)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think having a separate data type for the connection pool configuration is nicer than 3 separate function parameters. Thoughts?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In an ideal world for me, there is a field of PostgresConf named poolConfig :: ConnectionPoolConfig. Unfortunately, PostgresConf already has an existing field for the pool size, so I broke out stripes + idle timeout to separate fields there.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But if adding the two new fields is a breaking change, possibly we could go all-out and replace the fields with a ConnectionPoolConfig field?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there are breaking changes, and then there is Breaking Changes, and, weeeeeell, I'd rather the easiest breaking change as possible. Making these types more abstract is a goal of #920 so we can leave a TODO here to clean it up with persistent-3.0 possibly.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok cool, that's helpful guidance 👍

Comment on lines +63 to +65
{ connectionPoolConfigStripes :: Int
, connectionPoolConfigIdleTimeout :: NominalDiffTime
, connectionPoolConfigSize :: Int

@MaxGabriel MaxGabriel Jul 13, 2020 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've tried to keep the order of stripes + idle timeout + pool size consistent, and in the same order as they are in resource-pool. I think it's somewhat unfortunate that it's easy to mix up stripes and pool size though.

Comment thread persistent-postgresql/Database/Persist/Postgresql.hs Outdated
@MaxGabriel

Copy link
Copy Markdown
Member Author

Ok, I think this is pretty ready for review. It's just missing changelog + cabal bumps at this point. But I've left a few TODOs where I was uncertain about things

@MaxGabriel
MaxGabriel marked this pull request as ready for review July 19, 2020 23:48
@MaxGabriel

Copy link
Copy Markdown
Member Author

Ok, added to the Changelog and @since 2.11.0.0

Comment thread persistent/Database/Persist/Sql/Types.hs Outdated
@parsonsmatt parsonsmatt added this to the 2.11 milestone Sep 18, 2020
@parsonsmatt

Copy link
Copy Markdown
Collaborator

Everything looks great to me! I'm happy to merge now or we can incorporate some of the changes I mentioned. Either way is good for me.

@MaxGabriel

Copy link
Copy Markdown
Member Author

I'll take a look at your suggestions 👍

@MaxGabriel
MaxGabriel force-pushed the configurableIdleTimeout branch from 3e83476 to 74b1711 Compare October 25, 2020 16:24
@MaxGabriel

Copy link
Copy Markdown
Member Author

Ok, I pushed a change which moves to using NonEmpty Word as the new way of representing Postgres versions, while being backwards compatible with Maybe Double. See #1098 (comment).

Is this an approach we want to go with? If so, I can just do final touchups on this PR and we should be good to go

@parsonsmatt parsonsmatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great 😄 A few suggestions but I'm excited to get this in.

Comment thread persistent-postgresql/Database/Persist/Postgresql.hs Outdated
Comment thread persistent-postgresql/Database/Persist/Postgresql.hs Outdated
Comment thread persistent-postgresql/Database/Persist/Postgresql.hs Outdated
Comment thread persistent-postgresql/Database/Persist/Postgresql.hs
@MaxGabriel
MaxGabriel force-pushed the configurableIdleTimeout branch from dbf9926 to b6dd509 Compare November 2, 2020 01:40
@MaxGabriel

MaxGabriel commented Nov 2, 2020 •

Copy link
Copy Markdown
Member Author

Talked with Matt a little on Slack: he's good with no specific attempt make to make data types backwards compatible with future updates. In Persistent 3.0, we'll move to some model like that.

Going to merge this PR once I get tests passing

Comment thread persistent-postgresql/test/PgInit.hs Outdated
@MaxGabriel

Copy link
Copy Markdown
Member Author

Looks like the postgres server version on CI is formatted differently; working on a patch to make the parser more flexible

@MaxGabriel
MaxGabriel merged commit 75b5ac7 into master Nov 2, 2020
@parsonsmatt
parsonsmatt deleted the configurableIdleTimeout branch April 1, 2021 16:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants