Repository navigation
fix: Keep entity rows apart in the SQL point-in-time joins - #7124
Conversation
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #7124 +/- ##
=======================================
Coverage 52.45% 52.45%
=======================================
Files 459 459
Lines 58172 58172
Branches 8577 8577
=======================================
Hits 30515 30515
Misses 25573 25573
Partials 2084 2084
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
|
The only red job is |
The point-in-time join templates of the Redshift, Snowflake, BigQuery, Postgres, Athena, Spark, Trino, ClickHouse and Couchbase offline stores identify each entity row by concatenating the string values of its join keys and event timestamp. Different rows could produce the same id, e.g. driver_id=1, customer_id=12 and driver_id=11, customer_id=2 at the same timestamp. Such rows got the other row's feature value, or, when both feature rows had the same event timestamp, each came back twice, once with each value. Length-prefix each join key value in the id, as the ibis join does since feast-dev#6884, so the id is only shared by rows with equal values. Signed-off-by: sohailshk <122166523+sohailshk@users.noreply.github.com>
…Trino template Signed-off-by: sohailshk <122166523+sohailshk@users.noreply.github.com>
6cc9ef4 to
8e53dfe
Compare
What this PR does / why we need it:
The point-in-time join templates of the Redshift, Snowflake, BigQuery, Postgres, Athena, Spark, Trino, ClickHouse and Couchbase offline stores build
<feature_view>__entity_row_unique_idby concatenating the join key values and the entity timestamp with no separator, and join the feature rows back to the entity rows on that id. Two entity rows can get the same id:driver_id=1, customer_id=12anddriver_id=11, customer_id=2at the same timestamp both become112<timestamp>.One of these rows then gets the other's feature value, or, if both feature rows have the same event timestamp, each entity row comes back twice, once with each value.
get_historical_featureswith the Postgres offline store on a local PostgreSQL 18, feature view keyed ondriver_idandcustomer_id:Each join key value is now length-prefixed in the id (
<length>:<value>), as #6884 did for the ibis point-in-time join, so the id is only shared by rows with equal key values.Adds
test_sql_entity_row_unique_id.py. It runs the rendered Redshift, Athena, Postgres, Trino and ClickHouse queries on DuckDB with two such entity rows and checks that each keeps its own feature value (4 rows instead of 2 on master). The BigQuery, Snowflake, Spark and Couchbase templates use SQL that DuckDB can't run, so for those the test only checks that each join key is length-prefixed in the id.Which issue(s) this PR fixes:
N/A (no existing issue; #6884 fixed the same problem in the ibis join)
Checks
git commit -s)Testing Strategy
Misc
pytest tests/unit/infra/offline_stores: 309 passed, 20 skipped; the 2test_ibis.py::test_point_in_time_join*failures (sqlglot TypeError) also occur on unmodified master. The Snowflake, BigQuery, Spark, Couchbase, Redshift, Athena, Trino and ClickHouse queries were not run against those engines.