Skip to content

fix: Use valid Postgres type names for Arrow types in entity dataframe uploads - #7142

Merged
ntkathole merged 1 commit into
feast-dev:masterfrom
sohailshk:fix/arrow-to-pg-type-invalid-types
Nov 13, 2026
Merged

ntkathole merged 1 commit into
feast-dev:masterfrom
sohailshk:fix/arrow-to-pg-type-invalid-types

Conversation

@sohailshk

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

arrow_to_pg_type builds the CREATE TABLE for a pandas entity dataframe in the Postgres offline store. Some of its names are not Postgres types, and some types are not mapped:

Arrow type master this PR
int8 tinyint (error: type does not exist) smallint
binary binary (error: type does not exist) bytea (also large_binary)
null (all-None column) null (syntax error) text
date32, date64 ValueError: Unsupported type date
decimal128(p, s) ValueError: Unsupported type numeric

Each case was checked with df_to_postgres_table on a local PostgreSQL 18.4: 5 of 5 fail on master, 5 of 5 create and fill the table with this change (date64 checked only in the unit test). Decimal maps to plain numeric because this function only gets the type string, and an unconstrained Postgres numeric stores any decimal value exactly.

Adds a parametrized test in test_type_map.py (7 cases, all fail on master).

Which issue(s) this PR fixes:

N/A (no existing issue)

Checks

  • I've made sure the tests are passing.
  • My commits are signed off (git commit -s)
  • My PR title follows conventional commits format

Testing Strategy

  • Unit tests
  • Integration tests
  • Manual tests
  • Testing is not required for this change

Misc

pytest tests/unit/test_type_map.py: 369 passed.

@sohailshk
sohailshk requested a review from a team as a code owner November 12, 2026 14:58
@codecov-commenter

codecov-commenter commented Nov 12, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 52.40%. Comparing base (5eefef9) to head (a220e8a).
⚠️ Report is 4 commits behind head on master.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##           master    #7142   +/-   ##
=======================================
  Coverage   52.39%   52.40%           
=======================================
  Files         459      459           
  Lines       58156    58160    +4     
  Branches     8572     8574    +2     
=======================================
+ Hits        30470    30476    +6     
+ Misses      25606    25604    -2     
  Partials     2080     2080           
Flag Coverage Δ
go-feature-server 30.96% <ø> (ø)
python-unit 53.90% <100.00%> (+<0.01%) ⬆️
Files with missing lines Coverage Δ
sdk/python/feast/type_map.py 65.78% <100.00%> (+0.40%) ⬆️

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5eefef9...a220e8a. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@sohailshk

Copy link
Copy Markdown
Contributor Author

The only red job is unit-test-python (3.11, macos-15): 3987 passed, 54 skipped and 1 error in the setup of tests/unit/test_feature_server.py::test_push_and_get (AssertionError: stdout: b'', the feature server subprocess printed nothing). That is the same macOS 3.11 setup error as on #7124, and this PR only changes arrow_to_pg_type and its tests. Could someone re-run that job?

…e uploads

arrow_to_pg_type, which builds the CREATE TABLE for a pandas entity dataframe,
returned type names Postgres does not have: int8 -> tinyint, binary -> binary,
null -> null. It also raised for date32 and decimal columns. Map them to
smallint, bytea, text, date and numeric.

Signed-off-by: sohailshk <122166523+sohailshk@users.noreply.github.com>
@ntkathole
ntkathole force-pushed the fix/arrow-to-pg-type-invalid-types branch from b809993 to a220e8a Compare November 13, 2026 13:45
@ntkathole
ntkathole merged commit 171d601 into feast-dev:master Nov 13, 2026
17 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants