Upgrade to pgx v5

[jschaf]

#74

[kirk-anchor]

Broken text replace

[DavidArchibald]

Hey! I was interested in getting this at least sort of working and put together a set of minimal fixes here: mypricehealth@ec3a10c

The main things fixed are that the complete removal of {{- range .Declarers}}{{- "\n\n" -}}{{ .Declare $.PkgPath }}{{ end -}} ended up removing all the struct declarations, that the pgtype.XArray types are now pgtype.FlatArray[X] or maybe pgtype.Array[X] types, and that EmitScanColumn didn't scan into sub fields.

This was enough to at least get the generation of some of the examples folder building. I might continue working on some of this but I'm wondering how much interest there is in collaboration here. There's definitely some there's a non insignificant remaining work to get it beaten into shape. I couldn't quite get it to working enough that internal/pg/query.sql.go works for example.

[kirk-anchor]

@DavidArchibald I am interested in upgrading to pgx-v5. Do your changes remove support for composite types or arrays of composite types? joe/pgxv5...mypricehealth:pggen:joe/pgxv5

[DavidArchibald]

@kirk-anchor on the tip of my branch I was able to get all the code I personally needed generating which included generating structs for all the queries I needed, arrays included. I don't know if I tested composite types explicitly so I guess I'd recommend trying it out and seeing if it works for your queries.

If it's not working I'd appreciate a simple reproduction and then I'd probably be able to fix it. Though just to be explicit, I whipped this up in an afternoon and I'd actually be surprised if my set of fixes didn't in of themselves include multiple problems. What I can say is that I audited all the code I generated and it all worked for the queries I was writing but that's it. I'm aware of a handful of issues but they were in types I didn't personally use and I couldn't figure out a straightforward migration from v4 -> v5.

[kirk-anchor]

@DavidArchibald When I use a composite type, it tries to reference the now removed q.types field.

func (q *DBQuerier) InsertMyTable(ctx context.Context, myRows []MyTable) (pgconn.CommandTag, error) {
	ctx = context.WithValue(ctx, QueryName{}, "InsertMyTable")
	cmdTag, err := q.conn.Exec(ctx, insertMyTableSQL, q.types.newMyTableArrayInit(myRows))
	if err != nil {
		return pgconn.CommandTag{}, fmt.Errorf("exec query InsertMyTable: %w", err)
	}
	return cmdTag, err
}

I have queries like this.

-- name: InsertMyTable :exec
INSERT INTO my_table (my_col1, my_col2, my_col3)
SELECT
    my_col1,
    my_col2,
    my_col3
FROM
    unnest(pggen.arg('my_rows')::my_table[]);

-- name: GetMyTable :many
SELECT
    my_table
FROM
    my_table;

This way I have a common MyTable struct type to use on all queries.

	InsertMyTable(ctx context.Context, myRows []MyTable) (pgconn.CommandTag, error)
	GetMyTable(ctx context.Context) ([]MyTable, error)

[kirk-anchor]

Looks like on pgx-v5, I can get the composite type with dt, err := conn.LoadType(ctx, "my_table")
https://github.com/jackc/pgx/blob/60a01d044a5b3f65b9eea866954fdeea1e7d3f00/pgtype/composite_test.go#L231

@jschaf Do you plan to continue development? I may see if adding composite types to the fork will get my queries working. Arrays use Go generics now so arrays of composites should work too. Do you plan to upstream the changes from the fork or take another approach?

[kirk-anchor]

I was able to get the composite type example queries I provided working. Also fixed some bugs. mypricehealth/pggen@joe/pgxv5...kirk-anchor:pggen:pgxv5

[bweston92]

I can't see the failed test logs, but is this something if someone was to pickup you'd be happy to review?

[jschaf]

I can't see the failed test logs, but is this something if someone was to pickup you'd be happy to review?

Yes, work has me swamped, but I'd be happy to review a PR.

[bweston92]

@kirk-anchor I see that you've maintained that fork and bumped pgx to v5. Do you feel like this is something that can be worked into a PR? Happy to help.

[kirk-anchor]

I've been continuing to update it here https://github.com/kirk-anchor/pggen

I recently upgraded Go module versions. I already added support for composite types. The code needs some cleanup.

I am also considering adding some other features, maybe a pggen.any_arg() that makes the generated var any type so pgx can handle JSON marshaling instead of having to use json.Marshal() in Go first to pass in a JSON value. I would also like to be able to use custom types for a composite type, --go-type mostly works but omits the Go package prefix from the type in the generated code when using a custom type on a composite type.

[kirk-anchor]

Another issue I have to solve is the order of RegisterType() for composite types. It currently does them in alphabetical order but if you have a composite type that contains another composite type, you must register the composite type it contains before registering that composite type.

[bweston92]

@kirk-anchor is that something you can put in a PR to here?

[kirk-anchor]

Potentially, if there's interest. It uses pgx.CollectRows(rows, pgx.RowToStructByName[T]) instead of creating and caching a ScanPlan. @jschaf would you accept this approach?

[jschaf]

Yes, I’m okay with that approach. The public interface won’t change, right?

[bweston92]

@kirk-anchor if there is anything you need help with getting a PR in I would be willing to help as much as I can

[DavidArchibald]

@kirk-anchor I have a few changes on top of your's at https://github.com/mypricehealth/pggen/ again. It includes a lot of small things, some of which I argue should be removed completely for this repo.

The biggest one though is multi-statement support which you may be interested in.

[kirk-anchor]

I will take a look at it later this week, I have not had time. @DavidArchibald I did see you made some changes on top of mine, I will go through them. At a glance, I'm not sure I understand the multi-statement support. It doesn't seem to be batch queries like pggen used to have. If it's multiple hardcoded statements in the .sql file that are executed together, I just use a CTE for that.

[DavidArchibald]

@kirk-anchor I'm admittedly unfamiliar with the batch queries pggen used to have. It is indeed multiple hardcoded statements that are executed together. It's helpful for cases where CTEs wouldn't work or would otherwise be clunky.

[jschaf]

I'm admittedly unfamiliar with the batch queries pggen used to have.

The pggen batch queries were pgx batch queries, which were just concatenated SQL queries, e.g., SELECT 1; SELECT 2; SELECT 3. The benefit is that if you didn't have data dependencies between queries, you could send them all in one round trip.

pgx5 supports Postgres query pipelining, allowing precise control over when statements execute.

See jackc/pgx#1774 for more details.